prole/docs/TODO.md

Master TODO index

Single source of truth for unfinished work in this repo. Captures items that surfaced across the autobuild pipeline phases, the 2026-04-28 → 2026-04-29 cluster recovery / Maven build readiness work, the 2026-04-29 CNPG operator + backup stabilization work, and reality-vs-intent drift in the docs.

When you pick something up, move it from the ranked queue into the Now section. When it lands, archive it into the Done section at the bottom (or delete if it's not worth memorializing).

When you discover a new item, add it to the ranked queue. Default rank: bottom. Move it up later if it gets pulled in.

The Kanban "Now" section at top is the only place this doc imposes structure. Everything below "Now" is a numbered list — top of the list is next-most-important.

---

Now (Kanban)

In progress

• TODO-1 — cfg save path refuses non-string widget values — assigned to Junie. Followup from commit dba8a2d's findings. ConfigMixin._save_knoe_cfg in knoe/ui/screens/cfg.py calls var.get() on Tk widget vars and serializes the result; if a var is a MagicMock (interactive run in a non-Tk context, headless env, partial mock setup, etc.), the save path writes <MagicMock name='Canvas().tk.call().strip()' id='…'> strings into conf/<mode>.cfg. The next installer pass then os.makedirs() on those values, producing absurdly-named directories on disk. Saw it firsthand on 2026-05-02; cleaned up in dba8a2d. This brief is the durable fix: TDD-style failing test first, then a _str_value() validator in the save path that raises TypeError on non-str widget reads. Brief: docs/plans/junie/todo-1-cfg-save-path-bug.md.

• k3d-mirror-of-GKE Phase 3 — knoe-auth as a pod inside k3d — assigned to Junie. Pre-merge smoke loop: build the knoe-auth:latest image (using the make docker-build-auth target shipped in commit 903f84f), import it into the k3d cluster (k3d image import), apply a k3d-flavoured Deployment manifest (sibling of deploy/gcp/gke/knoe-auth-deployment.yaml, but with imagePullPolicy: Never and realm KNOE.LOCAL), and run the keytab-bootstrap initContainer + KDC sidecar pattern in-cluster. New make k3d-knoe-{deploy,redeploy,undeploy} targets. Phase 2 OIDC signing key flows from etc/secrets/knoe-auth-oidc-key.b64 into a K8s Secret (knoe-auth-oidc-signing-key in knoe-system) so the in-cluster pod gets it the same way GKE does. Brief: docs/plans/junie/k3d-knoe-auth-pod-deploy.md. Architectural plan: docs/plans/k3d-gke-mirror.md §6 Phase 3.

Paused

• pg_oauth in install/deploy.sh + k3d build — previously assigned to Junie; paused. The Phase 1 work (per-engineer roles, external LB at pg.0.knoe.dev, cert SANs, pg_hba tightening, docs/db-access.md) is committed. This phase replaces the SCRAM password mechanism with PG18 native OAUTHBEARER, wires it into install.sh / supabase/deploy.sh so a fresh deploy lands with the OIDC mechanism active by default, and ports the Service + cluster cert pattern to k3d. Connection target stays at pg.0.knoe.dev; only the credential issuance changes (libpq Device Flow → knoe-auth Bearer token). OIDC issuer now deployed to GKE (2026-05-02) — pg_oauth can resume.

Up next

(empty — pull from the ranked queue below)

Blocked

(none currently)

---

Next (ranked queue — top of list = next-most-important)

Note: numbering retained from the original queue (no #1) so the Junie briefs in plans/junie/ keep their <NN>-… filename match. #1 (installer env-contamination guard) shipped 2026-05-01; see the Done section.

8. Port oauth2-proxy manifest to k3d / k3s / min modes — deploy/gcp/gke/oauth2-proxy-deployment.yaml is GKE-shaped (uses cloud.google.com/v1 BackendConfig + GCE health-check semantics). For non-GCE deploy modes the same Deployment runs unchanged but the ingress glue differs (Traefik / nginx-ingress / no LB at all). Add a deploy/k3s/oauth2-proxy-deployment.yaml with the Traefik-flavored ingress annotations once we deploy that mode.

9. Wire GoTrue (supabase-auth) Google provider for Supabase API clients — different surface than the Studio gate (which oauth2-proxy now handles). Once we have client apps that authenticate end-users via Supabase Auth, configure GoTrue's Google provider with its own OAuth client. See docs/oidc-setup.md for the secret indirection pattern.

10. Audit-log story for the multi-tenant Studio phase — Studio doesn't track per-user actions in self-hosted; oauth2-proxy forwards the Authorization: Bearer <id_token> header to Kong (--pass-authorization-header=true / --set-authorization-header=true) but nothing reads it. When the multi-tenant control plane lands (per-user / per-project Studios), wire a Kong logging plugin that captures the email claim from the JWT and emits to a structured log sink.

11. Retire api.0.knoe.dev Ingress rule + managed cert — db.0.knoe.dev now serves both Studio (Google-gated) and the SDK API surface (Kong key-auth) via oauth2-proxy --skip-auth-route. api.0.knoe.dev remains as an alias for backward compatibility, but new docs/onboarding should point to db.0.knoe.dev. When external clients have all migrated: drop the api.0.knoe.dev rule from the supabase-kong Ingress and the supabase-api-managed-cert ManagedCertificate. Until then, the alias is harmless (same backend, same key-auth).

12. Fork the Supabase Studio image to wire in-app help / support / feedback buttons to mailto:support@knoe.dev (or https://db.0.knoe.dev/support) — upstream Studio (supabase/studio:2026.02.16-sha-26c615c) hardcodes those URLs to Supabase-cloud endpoints (supabase.com/dashboard/api/feedback etc.) which are unreachable from self-hosted, so the in-app "Report a problem" / "Send feedback" / "Get help" flows error out. No env-var hook exists in upstream — verified by Explore search. Fork the image, patch the relevant TSX (apps/studio/components/layouts/AppLayout/AppLayout.tsx, support-dialog component, settings/help links — handful of files), publish to our registry, bump image.studio.repository in supabase/helm/knoe-supabase/values.yaml. Same fork is the natural place to fix the OpenAI key panel, telemetry endpoints, and any other in-Studio assumptions about Supabase cloud as we encounter them. Stop-gap until then: https://db.0.knoe.dev/support 302s to mailto:support@knoe.dev (Kong route support); just need to tell users to bookmark or remember it.

13. Ekosystem UUID system — CNPG wiring and follow-on work — Schema files written (knoe-db/schema/ekosystem.sql, knoe-db/schema/ekosystem_objects.sql); init_prole_app.sql wired for direct psql runs. Remaining: (a) ConfigMap + postInitApplicationSQLRefs for all three CNPG manifests (k3s prod, GKE, OpenTofu); (b) register prole as tenant 1; (c) Python counterpart utility; (d) align knoe.user with ekosystem UUIDs; (e) LDAP/Samba AD group sync from cross_grants. Brief: docs/plans/junie/ekosystem-uuid-cnpg-wire.md.

14. Migrate cnpg-prometheus datasource UID to the stable cnpg-prometheus name — currently the live datasource has the auto-generated UID P5531627C358300FE from the original kps install. We pinned uid: cnpg-prometheus in monitoring/kps-values-gke.yaml so any FRESH kps install lands on the stable name, but Grafana refuses to change the UID of an already-provisioned datasource (read-only via API; rollout-restart doesn't migrate it). The dashboard transform monitoring/cnpg-dashboard-transforms.yaml ds_prometheus_default_to_cnpg carries the auto-uid as a workaround. Migration path on the next clean kps re-install (or after a maintenance window where we can wipe the Grafana sqlite DB to drop datasources): swap value: P5531627C358300FE → value: cnpg-prometheus in the transform and re-run the sync tool. No client-visible change either way.

---

Reality-vs-intent gaps

Items where docs describe an intended state that doesn't match live state. Each links to where reality is documented. Closing the gap means either updating the doc to match reality permanently, or doing the work to make reality match.

#	Reality (current)	Intended state	Documented at	Tracked work item

---

Later

11. Round 1.5: OpenBao transit-key encryption for per-user secrets at rest — authority/src/main/java/dev/knoe/auth/enroll/UserProvisioningService.java:84 has // TODO Round 1.5: encrypt secret with OpenBao transit key before storing. Currently TOTP secrets and similar long-lived per-user state go to knoe.* tables with PG-level encryption-at-rest only. Round 1.5 wraps writes/reads with an OpenBao transit-key envelope. See docs/plans/knoe-auth-round-1.md §8 "Out of scope for Round 1".

12. knoe-db/docker-entrypoint.sh:3 swap to set -Eeuo pipefail — current set -e only. Marked as TODO in the file: "swap to -Eeuo pipefail above (after handling all potentially-unset variables)". Needs an audit pass over the script for unset-var safety first.

13. Customer-deploy-resync execution decision — currently no separate ~/dev/prole working tree under development; the plan in docs/plans/customer-deploy-resync.md is dormant. Decision: do we ever activate the per-customer branching workflow (e.g. for a future second customer), or formalize "single-tenant for now" and remove the plan?

---

Deferred / Out of scope

• Studio Database Advisor — Performance: unindexed FKs in upstream-managed schemas — 6 FKs flagged on a fresh project: auth.mfa_challenges.factor_id, auth.saml_relay_states.flow_state_id, auth.oauth_authorizations.{client_id, user_id}, storage.s3_multipart_uploads_parts.{upload_id, bucket_id}. Adding indexes to upstream-managed tables risks conflict on the next supabase-auth / supabase-storage migration. Wait for upstream fixes; revisit if these turn out to be load-bearing for our usage.

• com.knoey.auth open-source split — saved for the major collaboration (per user memory: "we are saving knoey.com/auth for the major collaboration"). Will require a separate repo extraction and licensing pass.

• "Test ssh to Gitea" task — orphaned during a previous Cowork+Code session when the Dispatch connection was lost. May or may not still be relevant.

• Volume snapshot backups (CNPG VolumeSnapshot method) — alternative to barman tarball. Considered during the 2026-04-29 outage debugging when barman-cloud-backup looked broken; now that GCS+barman is verified working, no immediate need. Worth revisiting if barman shows further fragility.

• File the upstream plugin-barman-cloud issue about the misleading ReferenceError: weakly-referenced object no longer exists traceback that fires on persistent 5xx responses. Issue #707 is closed but didn't capture this pattern; ours is a clearer reproduction. Defer — the upstream is responsive but the workaround (sufficient bucket capacity + correct IAM) is good enough.

---

Done

(items get archived here when they land, with a date and commit reference; or just delete if not worth memorializing)

• 2026-05-02 — Phase 2 OIDC provider — GKE deploy (brief: docs/plans/junie/phase2-oidc-gke-deploy.md). KNOE_AUTH_OIDC_ENABLED=true and KNOE_AUTH_OIDC_ISSUER=https://api.knoe.dev/auth set in deploy/gcp/gke/knoe-auth-deployment.yaml. Kong /auth route added to supabase/helm/knoe-supabase/templates/kong/config.yaml (proxies to knoe-auth.knoe-system.svc.cluster.local:8080; GoTrue /auth/v1/* routes unaffected). studioIngress and knoeAuth default blocks added to values.yaml (fixes helm lint). Signing key wired via knoe-auth-oidc K8s Secret sourced from 1Password via init_knoe_auth.sh initialize. Unblocks pg_oauth resume.

• 2026-05-02 — Image rename knoe-authority → knoe-auth (queue #3 / drift R6). authority/Dockerfile.app added (Spring Boot multi-stage build). make docker-build-auth / make docker-push-auth targets added. Three image refs in deploy/gcp/gke/knoe-auth-deployment.yaml and two in deploy/opentofu/k3s/manifests/knoe/knoe-auth-deployment.yaml updated from knoe-authority:latest → knoe-auth:latest. docs/knoe-system.md §3 caveat removed. Brief: docs/plans/junie/03-image-rename-knoe-authority-to-knoe-auth.md.

• 2026-05-02 — Image rename knoe-authority → knoe-auth (queue #3 / drift R6). authority/Dockerfile.app added (Spring Boot multi-stage build). make docker-build-auth / make docker-push-auth targets added. Three image refs in deploy/gcp/gke/knoe-auth-deployment.yaml and two in deploy/opentofu/k3s/manifests/knoe/knoe-auth-deployment.yaml updated from knoe-authority:latest → knoe-auth:latest. docs/knoe-system.md §3 caveat removed. Brief: docs/plans/junie/03-image-rename-knoe-authority-to-knoe-auth.md.

• 2026-05-02 — k3d-mirror-of-GKE Phase 1: laptop dev loop for knoe-auth (k3d brief). Smallest k3d-resident stack (CNPG single-replica + standalone KDC, realm KNOE.LOCAL) that lets a host-side knoe-auth iterate against real Postgres + Kerberos. Deliverables: k8s/knoe/knoe-kdc-{configmap,deployment,service,pvc,init-job}.yaml, etc/init_knoe_auth.sh --mode k3d flag (extends existing script; GKE behavior unchanged), make k3d-knoe-{up,pf,down} targets, etc/krb5.local.conf (TCP-forced, udp_preference_limit=1), docs/local-dev-knoe-auth.md, scripts/k3d-knoe-{up,pf,down,smoke}.sh. Design choice: extended init_knoe_auth.sh with --mode k3d rather than a sibling script — minimizes drift, same schema/status subcommands work in both modes. Brief: docs/plans/junie/k3d-knoe-auth-dev-loop.md.

• 2026-05-02 — k3d dev-user seed: chrisfu + knoe_developer (commit pending). etc/init_knoe_auth.sh --mode k3d (called by cmd_initialize_k3d) now runs seed_dev_users_k3d() after the schema bootstrap. Creates the knoe_developer group role with R/W on knoe+public (mirrors the GKE production layout from docs/db-access.md) and a chrisfu LOGIN role with password chrisfu-dev granted into the group. Idempotent: rebuild loop (make k3d-knoe-down && make k3d-knoe-up) deterministically resets chrisfu's password back to chrisfu-dev so the engineer always knows what to use post-rebuild. From the host with port-forward up: PGPASSWORD=chrisfu-dev psql "postgresql://chrisfu@localhost:5432/knoe-db?sslmode=require" connects. docs/local-dev-knoe-auth.md "Verify psql connectivity" + "Reset / rebuild loop" sections updated. docs/knoe-system.md §"One-time cluster setup" mentions the chrisfu seed. Closes the local-side of "I want auth as chrisfu@knoey.com to my local cnpg" — Kerberos principal + knoe.user/knoe.identity rows are deferred (separate work if needed).

• 2026-05-02 — Phase 2 OIDC k3d dev sandbox (commit pending). Companion to the k3d Phase 1 brief — completes the laptop dev loop for knoe-auth Phase 2. New authority/src/main/resources/application-k3d.yml Spring profile activates the OIDC surface with local issuer (http://localhost:8080); new etc/gen_oidc_signing_key.sh idempotently generates an RS256 PKCS#8 keypair at etc/secrets/knoe-auth-oidc-key.{pem,b64} (gitignored); scripts/k3d-knoe-up.sh runs the generator after schema seed; scripts/k3d-knoe-pf.sh prints the env-var export the engineer needs (export KNOE_AUTH_OIDC_SIGNING_KEY=$(cat etc/secrets/knoe-auth-oidc-key.b64)) plus the OIDC endpoint surface to curl-test. docs/local-dev-knoe-auth.md updated to use -Dspring-boot.run.profiles=k3d, the env var is exported, and the verify section now includes a /jwks.json curl. docs/knoe-system.md Phase 2 status flipped from "Planned" to "Shipped (k3d sandbox)" / "Pending (GKE deploy)". GKE-side Phase 2 deploy is now the open work item in TODO §"In progress" (gated on queue #3 image rename for the rebuild).

• 2026-05-02 — k3d Phase 1 dev loop for knoe-auth (commit 5d36008). Junie's Phase 1 brief landed — laptop dev loop with CNPG (single-replica) + KDC (realm KNOE.LOCAL) reachable from host via make k3d-knoe-{up,pf,down}. New: 5 KDC manifests under k8s/knoe/knoe-kdc-*.yaml, 4 helper scripts under scripts/k3d-knoe-*.sh, etc/krb5.local.conf (with udp_preference_limit=1 to dodge kubectl port-forward UDP flakiness on macOS), docs/local-dev-knoe-auth.md engineer doc, and docs/knoe-system.md unified reference. etc/init_knoe_auth.sh extended with --mode k3d flag (GKE behavior unchanged). Brief: docs/plans/junie/k3d-knoe-auth-dev-loop.md. Architectural plan: docs/plans/k3d-gke-mirror.md.

• 2026-05-02 — Garage decommissioned on app cluster knoe-dev-0 (queue #4 closure). Reframed mid-flight: the original "verify with garage repair --yes blocks" premise didn't match reality. Live garage was unconfigured (garage status → NO ROLE ASSIGNED, garage layout show → version 0, 21d of "Ring not yet ready" warnings); cross-cluster sweep confirmed nothing referenced its S3 endpoint anymore (CNPG backups moved to GCS in 2026-04-29 migration). Removed the entire workload from knoe-system namespace mirroring the 2026-04-29 DB-cluster removal: sts/garage, svc/garage, cm/garage-config, secret/garage-secrets, pvc/data-garage-0 (29 Gi, was on garage-hdd storage class), and the garage-hdd StorageClass itself. PV had reclaim=Retain so the underlying GCE pd-standard disk (pvc-656e1936-… in us-west3-c) persisted past the kubectl-side cleanup; reclaimed via gcloud compute disks delete. Note: app cluster only had a ClusterIP service — no garage-s3-ilb LoadBalancer (DB cluster had that). Source manifests remain at k8s/knoe/garage-*.yaml, deploy/opentofu/k3s/manifests/knoe/garage-*.yaml, and etc/init_garage_store.sh for the k3d/k3s/min deploy modes (and any future GKE deploy that wants to wire garage back). Whether the GKE deploy mode should stop applying garage altogether is a separate decision.

• 2026-05-02 — Phase B orphan cleanup (queue #5). Five resources deleted across both clusters. App cluster (knoe-dev-0): gitlab/gitlab-migrations-58a3e27-97-23-n72j8 (Failed/Evicted, 11d stale; the parallel …-nxzj5 Completed pod kept for log preservation), and three Released PVs from the 2026-04-28 monitoring stack rebuild — pvc-42e8545b-… (10Gi, claim=monitoring/storage-kps-grafana-0), pvc-86422179-… (30Gi, claim=monitoring/prometheus-…-prometheus-0), pvc-e2c29f7a-… (5Gi, claim=monitoring/alertmanager-…-alertmanager-0); all on standard-hdd, all 4d old, all matching the expected claimRef pattern, no surprises. DB cluster (knoe-dev-cnpg-0): kube-system/prole-supabase-dirprep Job (FailureTarget, 22d stale, prole-era). Post-state: kubectl get pv shows 0 Released; gitlab namespace has only the Completed migration pod; DB-cluster kube-system has no prole-* artifacts. No source-side changes (kubectl-only ops).

• 2026-05-02 — Reframe queue #4 (Garage block-resync repair → garage decommission decision). Tried to run the planned garage repair --yes blocks but found the cluster in an unconfigured state: garage status shows the single node with NO ROLE ASSIGNED, layout version 0 (was at v10 after 2026-04-29 storage resize — likely lost during a subsequent reconcile or PVC operation). Pod is up 21d but the "Ring not yet ready" warnings indicate it's been non-functional since whenever the layout was lost. Sweep across all namespaces' configmaps/secrets shows nothing outside garage's own state references the S3 endpoint or 10.180.15.239:3900. Conclusion: garage on app-cluster is a dead pod consuming 80GiB and serving no workload. Original block-repair premise is moot. Reframed the queue item as a decommission-or-restore decision; details in the entry above. Action deferred pending direction.

• 2026-05-02 — Apply Junie's PodMonitor migration + DASHBOARD removal to live (queue #13 + #15 follow-up; commit 03bb731 for chart fixes). Source landed in c3fae73; this is the corresponding live-cluster apply on knoe-dev-cnpg-0 + knoe-dev-0. Sequence: applied updated knoe-db.yaml → CNPG GC'd its auto-managed PodMonitor (deleted on poll attempt 1, ~2s) → applied manual knoe-db-podmonitor.yaml (now ownerRef=<none>, survives future cluster spec changes) → verified Prometheus still scraping all 3 postgres exporters (health=up on 10.24.{0,1,2}.x:9187) and cnpg_pg_stat_archiver_archived_count{cluster="knoe-db"} returns data confirming cluster-label relabel still works. Then helm upgrade supabase ./supabase/helm/knoe-supabase --reset-then-reuse-values (revision 1 → 3 after a partial-failure retry); configmap field-manager dual-ownership conflict resolved by deleting + recreating fresh under helm. Two chart cleanups required as follow-ups (committed as 03bb731): deleted templates/test/kong.yaml (helm-test job for the now-dead dashboard basic-auth gate) and added secret.oauth2proxy defaults in values.yaml so the vestigial templates/studio/oauth2proxy-*.yaml chart resources (Junie Phase-2 OIDC artifacts; the deployed oauth2-proxy lives outside this chart at deploy/gcp/gke/oauth2-proxy-deployment.yaml and goes in FRONT of supabase-kong, not behind it) can evaluate guards cleanly. Side-finding closed live: all 3 knoe-dev-0 nodes had the legacy prole.org/node-role=general label only; Junie's chart defaults to nodeAffinity: knoe.dev/node-role=general, which made the new kong pod stuck in Pending. Labelled all 3 nodes with knoe.dev/node-role=general (additive — kept the prole.org label too for any other manifests still referencing it). DB cluster still on legacy-label-only; reapply when relevant. External smoke tests post-upgrade: /onboard.html→200, /rest/v1/ no-auth→401, /→302, /healthz→302, svc.knoe.dev/grafana/login→200.

• 2026-05-02 — Wire cnpg-backup-sa into CNPG cluster spec + init script (queue #7 / drift R8). deploy/gcp/gke/knoe-db.yaml gains spec.serviceAccountName: cnpg-backup-sa (requires CNPG v1.29+). etc/init_cnpg_gke.sh: operator install URL now uses CNPG_OPERATOR_VERSION variable (default 1.29.0); new §11 patches both knoe-db and knoe-db-barman-cloud RoleBindings to add cnpg-backup-sa as a subject if not already present, matching the 2026-04-29 live stabilization. bash -n clean. Brief: docs/plans/junie/07-init-cnpg-gke-sa-wiring.md.

• 2026-05-02 — Migrate off CNPG-deprecated enablePodMonitor + podMonitorRelabelings (queue #13). Both deprecated fields removed from deploy/gcp/gke/knoe-db.yaml spec.monitoring. New sibling manifest deploy/gcp/gke/knoe-db-podmonitor.yaml carries the PodMonitor with the cluster relabeling rule (sourced from cnpg.io/cluster pod label; required for all 85 CNPG Grafana dashboard panels). Apply alongside knoe-db.yaml. Brief: docs/plans/junie/13-podmonitor-manual-management.md.

• 2026-05-02 — Remove dead DASHBOARD consumer + basicauth_credentials from Kong chart (queue #15). supabase/helm/knoe-supabase: removed DASHBOARD_USERNAME/DASHBOARD_PASSWORD envsubst lines from wrapper.sh, DASHBOARD consumer + basicauth_credentials block from config.yaml, and both env-var secret refs from kong/deployment.yaml. values.yaml secret.dashboard section renamed to secret.openai (only apiKey field; username/password dropped). templates/secrets/dashboard.yaml + _helpers.tpl updated to openai/supabase.secret.openai. studio/deployment.yaml updated to read from secret.openai.apiKey. ci/example.yaml updated. helm template confirms knoe-supabase-openai secret referenced, no DASHBOARD output. Brief: docs/plans/junie/15-remove-dead-dashboard-consumer.md.

• 2026-05-02 — k3s manifest rename: prole-* → knoe-* (queue #2 / drift R5) in commit fb7e8b7. Five git mv in deploy/opentofu/k3s/manifests/knoe/: prole-{configmap,deployment,service,kdc-configmap,kdc-secrets.example}.yaml → knoe-*. Content sweep of knoe-kdc-configmap.yaml: replaced stale PROLE.LOCAL/PROLE.ORG realm refs and PROLE_KDC_* env vars with KNOE.DEV/KNOE_KDC_* to match the canonical GKE configmap. Kustomize renders cleanly (kubectl kustomize → 1193 lines, exit 0). Brief: docs/plans/junie/02-k3s-prole-rename.md.

• 2026-05-02 — Fix scripts/patch_garage_cross_cluster.sh (queue #6 / drift R9) in commits 34a25dd + e9df3c4. Three defects from the 2026-04-29 one-shot run: (A) DB_CLUSTER default knoe-cnpg-0 → knoe-dev-cnpg-0 (stale rebrand artifact; header comment + overrides block also updated); (B) Phase 1 deletion loop now includes service/garage-s3-ilb (the LoadBalancer that was left orphaned after the original run); (C) deploy/gcp/gke/knoe-db-backup-gcs.yaml split — legacy ScheduledBackup with deprecated method: barmanObjectStore removed, file is now SA-only. bash -n clean; CONFIRM=false dry-run prints correct cluster names. Brief: docs/plans/junie/06-patch-garage-script-fixes.md.

• 2026-05-01 — Installer env-contamination guard (queue #1 / drift R4) (commit pending). Filed in response to the 2026-04-28 14:00 UTC backup outage (an install.sh --mode k3d run with the shell pointed at GKE replaced the GCS-backed ObjectStore with a Garage-backed one). New shared bash helper etc/preflight_kubecontext.sh with two functions: verify_kubecontext_matches_config <cfg> (strict gate; reads [Global] APP_CLUSTER_KUBECONTEXT from the config and exits 1 if kubectl config current-context differs), and print_kubecontext_notice (informational). deploy.sh sources the helper and calls the strict gate before invoking the Python deploy pipeline. install.sh calls the informational notice (gated on not---min) so the user sees the inherited context up-front before the TUI launches; the mode-aware strict gate for the interactive path is a follow-up for the welcome screen once it records mode. Bypass with KNOE_SKIP_KUBECONTEXT_GUARD=true for deliberate cross-cluster maintenance. Verified end-to-end: deploy.sh refuses on mismatch with a clear remediation message; bypass override works; install.sh's --min path skips the notice; install.sh's no-flag and other-flag paths print the notice. CLAUDE.md drift table row R4 removed; §"Env-contamination warning" rewritten to describe the live guard. Closes the 2026-04-28 14:00 UTC outage feedback loop.

• 2026-05-01 — Cross-cluster CNPG metrics via DB-cluster Prometheus (commit pending). Solves the cloudnative-pg Grafana dashboard's "No data" problem from queue item #12. Picked path (c) over the originally-recommended (b) for DB-cluster autonomy + pattern symmetry — each cluster owns its observability footprint, and a future second DB cluster reuses the recipe one-for-one. Installed kube-prometheus-stack v84.3.0 on knoe-dev-cnpg-0 (monitoring namespace), Grafana + Alertmanager disabled (single canonical Grafana stays on app cluster). All PVCs explicitly pinned to standard-hdd (per the user's standing rule on SSD-quota preservation; verified prometheus-...-0 PVC bound 20Gi standard-hdd). New ILB prometheus-cnpg-ilb in DB-cluster monitoring ns at 10.180.15.216:9090, reachable from app-cluster pods over the shared VPC. New cnpg-prometheus datasource added to app-cluster Grafana via monitoring/kps-values-gke.yaml grafana.additionalDataSources. Side-fix: removed an orphan knoe-grafana-datasource ConfigMap that etc/init_monitoring.sh:apply_grafana_datasource() had been creating with a duplicate uid: prometheus — caused Grafana's provisioning reload to fail with HTTP 500 once any second datasource arrived. Function neutered to a no-op with a clear comment; chart values are now the single source of truth for datasources. Verified end-to-end: Prometheus on DB cluster discovers all 3 CNPG postgres pods (knoe-db-2/3/4 at 10.24.x.x:9187, all health=up); Grafana datasource health endpoint returns Successfully queried the Prometheus API; cnpg_pg_stat_archiver_archived_count query through Grafana proxy returns live data (704 archived WAL files at time of test). To complete: switch each cnpg-grafana dashboard's DS_PROMETHEUS template variable to cnpg-prometheus (one-click). New files: monitoring/kps-cnpg-values.yaml, deploy/gcp/gke/cnpg-prometheus-ilb.yaml. Modified: monitoring/kps-values-gke.yaml (datasource added), etc/init_monitoring.sh (function no-op'd). Out of scope (Phase 2): federate alerts back to app-cluster Alertmanager; reserve a static internal IP so the ILB doesn't drift on re-create.

• 2026-04-30 — Bump postgres pod memory limit 512Mi → 2Gi (commit pending). Long-standing throughput cliff: barman-cloud-backup is single-threaded gzip + GCS upload buffering, ceiling-bound by the pod's memory limit. The 9 GB DB took 30–90 min to back up at 512Mi; 2Gi cuts that to 5–10 min. requests left at 128Mi (no scheduling change; this is purely a ceiling bump). Patched live with kubectl patch cluster knoe-db --type=merge -p '{"spec":{"resources":{"limits":{"memory":"2Gi"}}}}'; CNPG operator performed a supervised rolling restart (replicas first, primary last). Repo source updated at deploy/gcp/gke/knoe-db.yaml. Pre-flight verified no external psql sessions (chrisfu/ron not connected); 39 internal sessions FATAL'd through the rolling restart and reconnected via their pools — supabase + gitlab + streaming_replica all stable post-rollout. Closes drift R3. k3s + k8s mode resource limits remain at 512Mi (not exposed in those manifests today); cross-mode parity folded into queue item #8 ("Port oauth2-proxy manifest to k3d/k3s/min modes").

• 2026-04-30 — Align knoe/core/actions.py cluster-name + machine-type defaults with live state (commit pending). Two stale constants since the rebrand: DEFAULT_DB_CLUSTER_NAME = "knoe-cnpg-0" (live: knoe-dev-cnpg-0) and DEFAULT_APP_CLUSTER_MACHINE_TYPE = "e2-small" (live: e2-standard-2). Updated both at knoe/core/actions.py lines 121 + 123. Verified: python3 -c "from knoe.core.actions import ..." reads the new values; pytest collection error count went from 8 → 7 post-edit (one of the pre-existing collection errors apparently went away when the import resolved differently — no regression). Closes drift R1 + R2 in CLAUDE.md (footnotes + drift table rows + the # STALE inline comments in the §"Cluster code constants" code block all dropped). Implicitly closes the formerly-#8 queue item ("Update CLAUDE.md cluster table once defaults match") in this same commit. Queue renumbered: #2 (this item) and #8 dropped, gaps closed.

• 2026-04-30 — Drop redundant GCP IAM workloadIdentityUser binding (commit pending). v1.28-era leftover: [knoe-db-0/knoe-db] was bound alongside the active [knoe-db-0/cnpg-backup-sa] on the cnpg-backup@… GCP SA. Pods have used cnpg-backup-sa since the 2026-04-29 stabilization (commit e92cd8b), so the older binding was dead config — kept around because removing it required a reasoned check that no in-flight job still authenticated under the old identity. Verified live (gcloud iam service-accounts get-iam-policy), dropped via gcloud iam service-accounts remove-iam-policy-binding, re-verified the SA's policy now has only [knoe-db-0/cnpg-backup-sa]. CNPG primary still healthy (SELECT 1 returned cleanly post-removal). Closes drift R7. Removed the matching CLAUDE.md Reality TODO + drift table row.

• 2026-04-30 — Grafana at svc.knoe.dev/grafana with Google OAuth (commit pending). Diagnosed the svc.knoe.dev "name resolution failed" error: knoe-svc-kong-config had its grafana service pointing at prometheus-grafana.monitoring.svc.cluster.local:80 — the OLD upstream name from before the kube-prometheus-stack chart renamed to kps-grafana. Kong was DNS-failing every request to that vanished service and surfacing the Kong-style "name resolution failed" error to clients. Fixed the upstream + scoped the route to /grafana (so other surfaces can land at svc.knoe.dev/* later). Layered Grafana's native Google OAuth via auth.google config in monitoring/kps-values-gke.yaml (new — captures previously-uncommitted live Helm values + adds the auth additions). Role mapping via JMESPath: contains(['chrisfu@knoey.com', 'ron@knoey.com'], email) && 'Admin' || 'Editor' — anyone @knoey.com gets Editor, chrisfu+ron get Admin. New manifests: deploy/gcp/gke/grafana-google-oidc-secret.example.yaml, etc/init_grafana_oauth.sh, monitoring/kps-values-gke.yaml. Repo Kong source updated at deploy/opentofu/k3s/manifests/knoe/kong-configmap.yaml (.org→.dev rebrand, healthz route restored, /grafana scoping). Verified externally: /grafana/login → 200; /grafana/login/google → 302 accounts.google.com with the right client_id; subpath assets load; Studio + SDK paths on db.0.knoe.dev unchanged. Browser walkthrough (chrisfu sign-in → Admin role; ron same; outside-domain rejected by allowed_domains: knoey.com) is the manual follow-up. Stop-gap adminPassword: admin in values stays (local break-glass) — rotate this when convenient as a follow-up.

• 2026-04-30 — Reusable engineer onboarding via one-time link + QR (commit pending). Replaces the assumed-but-nonexistent shared 1Password vault with a self-contained delivery flow: ./etc/onboard_engineer.sh <user> <email> provisions the postgres role + 24h temp password and outputs (1) a single-shot URL, (2) a QR-code rendering for screenshare, (3) the plaintext fallback. Engineer opens URL or scans QR → static page at db.0.knoe.dev/onboard.html (vanilla HTML+JS, no backend) reads the URL fragment client-side and shows password + psql connection string + rotation snippet with [Copy] buttons; saves to engineer's personal 1Password. Engineer rotates immediately via \password. New manifests: deploy/gcp/gke/knoe-onboard.yaml (nginx:alpine + ConfigMap + Service), Kong route /onboard.html → knoe-onboard:80 in supabase/helm/knoe-supabase/templates/kong/config.yaml, oauth2-proxy --skip-auth-route=^/onboard\.html$ in deploy/gcp/gke/oauth2-proxy-deployment.yaml. Pivotal pg_hba change: rule changed from hostssl all chrisfu,ron all to hostssl all +knoe_developer all so future onboards don't require pg_hba edits — the script's GRANT knoe_developer TO <user> is enough. CA cert committed at etc/knoe-db-ca.crt (public root, not a secret) so engineers reference sslrootcert=etc/knoe-db-ca.crt after cloning. Engineer-facing docs: docs/onboarding.md. docs/db-access.md updated to remove the broken 1Password reference. Verified end-to-end: ./etc/onboard_engineer.sh test-onboard test-onboard@knoey.com → URL + QR rendered → psql ...user=test-onboard... connects with the temp password and reports pg_has_role(current_user,'knoe_developer','MEMBER')=t → --revoke drops the role cleanly.

• 2026-04-30 — Per-engineer psql access via pg.0.knoe.dev (Phase 1) (commit pending). External GKE TCP LoadBalancer at 34.106.156.196 (forwarding rule a61028024f2314e06902bafc1b4c3a08); user adds Namecheap A record pg.0.knoe.dev → 34.106.156.196 (manual; Namecheap UI). CNPG serverAltDNSNames extended with pg.0.knoe.dev so the auto-issued server cert validates externally. New knoe_developer group role with R/W on knoe+public and R/O on auth+storage+extensions. Per-engineer roles chrisfu and ron with strong random passwords (192-bit base64) VALID UNTIL '2026-05-30', members of knoe_developer. pg_hba rewritten so external requires TLS+SCRAM and only allows chrisfu, ron (other roles postgres, supabase_admin, authenticator stay cluster-internal). externalTrafficPolicy: Local on the LB to preserve client source IP (otherwise the SNAT'd node IP made externals look like internal cluster traffic and bypassed the hostnossl reject rule). Documentation in docs/db-access.md. Verified: chrisfu+ron both connect via TLS, plaintext rejected with the proper pg_hba error showing the real client IP, postgres role connection from outside fails auth, internal supabase services keep their existing TLS connections (gitlab on TLS 1.3, supabase_admin/authenticator on plaintext over RFC1918 — both still allowed by the new rules). Phase 2 (libpq OAUTHBEARER + validator library, bundled into install/deploy.sh + k3d) handed off to Junie — see "In progress" at top of file.

• 2026-04-30 — pg_stat_statements moved out of public schema across deploy modes (commit pending). Studio's Database Advisor was flagging public.pg_stat_statements as a Security warning the moment any user opened the dashboard. Root cause: relocatable extensions without an explicit SCHEMA clause on CREATE EXTENSION land in the first writable schema in the connecting role's search_path, which is public for the postgres role. Live: ALTER EXTENSION pg_stat_statements SET SCHEMA extensions;. Bootstrap (3 files, all 4 deploy modes): deploy/gcp/gke/knoe-db.yaml (gke), deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml (k3s), k8s/knoe/knoe-db.yaml (k3d / min) — each postInitTemplateSQL block now creates the extensions schema and CREATE EXTENSION ... SCHEMA extensions;. Plus supabase/deploy.sh gets GRANT USAGE ON SCHEMA extensions TO anon, authenticated, service_role; so PostgREST clients (and the advisor itself) can resolve the schema. Other security checks verified clean (no tables in public without RLS, no SECURITY DEFINER without search_path, no auth.users-exposed views, no duplicate indexes, no extensions left in public). Performance side has 6 unindexed FKs in upstream auth.* and storage.* tables — captured as deferred (touching upstream-managed schemas is the wrong move; they'll get fixed in supabase-auth / supabase-storage releases).

• 2026-04-30 — Studio post-launch fixes (storage permissions + /support stop-gap) in commit 79a3f76:

  • Storage list-buckets unblocked. supabase_storage_admin role was missing membership in service_role / authenticated / anon, so SET LOCAL role = 'service_role' (run by supabase-storage on every request) failed with PG 42501 at guc.c:6936 call_string_check_hook, surfaced as a misleading "new row violates row-level security policy" error. One GRANT applied live (GRANT anon, authenticated, service_role TO supabase_storage_admin;) and added durably to supabase/deploy.sh right after the existing GRANT service_role TO authenticator block.
  • oauth2-proxy auth-header passthrough fixed. The --pass-authorization-header=true and --set-authorization-header=true flags were stomping the Authorization: Bearer <apikey> header that Studio / supabase-py send on /storage/v1/... calls (oauth2-proxy was rewriting it with the Google OIDC id_token, which storage couldn't validate against JWT_SECRET). Both flags removed from deploy/gcp/gke/oauth2-proxy-deployment.yaml. When per-user identity passthrough lands post-knoe-auth Round 1, use --pass-user-headers / --set-xauthrequest (X-Forwarded-* headers) instead — those don't collide with Supabase's auth.
  • /support route stop-gap. https://db.0.knoe.dev/support 302s to mailto:support@knoe.dev so users have a working escape hatch while the in-Studio support buttons stay broken (separate Studio-fork follow-up #15). Kong service support added to templates/kong/config.yaml using request-termination (302) + response-transformer (Location header). KONG_PLUGINS allowlist extended with response-transformer in values.yaml. oauth2-proxy gets --skip-auth-route=^/support$ so an unauthenticated user looking for help isn't bounced into a Google sign-in detour.

• 2026-04-30 — Unified db.0.knoe.dev for Studio + SDK in commit f805404: added two --skip-auth-route args to oauth2-proxy so SDK requests on /auth/v1, /rest/v1, /realtime/v1, /storage/v1, /functions/v1, /graphql/v1, /pg/* bypass the Google sign-in flow and pass through to supabase-kong, where Kong's existing key-auth plugin enforces anon/service-role keys. Studio UI at / remains Google-gated. One URL now covers humans-in-Studio AND robots-via-supabase-py — same security envelope as api.0.knoe.dev had on those paths. api.0.knoe.dev kept as a backward-compat alias (see follow-up #14). External verification: curl -H "apikey: <KEY>" https://db.0.knoe.dev/rest/v1/ → 200 OpenAPI; curl https://db.0.knoe.dev/rest/v1/ (no key) → 401 from Kong; curl https://db.0.knoe.dev/ → 302 to Google (Studio gate intact); curl https://db.0.knoe.dev/api/profile → 302 to Google (Studio internals stay gated).

• 2026-04-30 — Google sign-in to db.0.knoe.dev for chrisfu + ron (shared Studio) in commit 25f1b2e: oauth2-proxy v7.15.2 deployed in front of supabase-kong as the Studio access gate (Google Workspace OIDC, --email-domain=knoey.com); supabase-kong Ingress patched so db.0.knoe.dev → oauth2-proxy:80 while api.0.knoe.dev → supabase-kong:8000 is untouched (key-auth still gates the API surface); Kong dashboard route's basic-auth plugin removed (oauth2-proxy is the gate now — double-auth was friction without security gain). External verification: db.0.knoe.dev/ → 302 accounts.google.com/o/oauth2/auth?client_id=200699021716-…&redirect_uri=https://db.0.knoe.dev/oauth2/callback&scope=openid+email+profile; api.0.knoe.dev/rest/v1/ → 401 from Kong key-auth (untouched); /healthz → 200. Three new files + chart edit: deploy/gcp/gke/oauth2-proxy-deployment.yaml, deploy/gcp/gke/oauth2-proxy-google-oidc-secret.example.yaml, etc/init_oauth2_proxy.sh, and the basic-auth plugin removed from supabase/helm/knoe-supabase/templates/kong/config.yaml. Future swap to knoe-auth Round 1: change one arg in oauth2-proxy from --provider=google to --provider=oidc --oidc-issuer-url=https://api.knoe.dev/auth and reapply.

• 2026-04-29 — CNPG operator + backup stabilization session (4 queue items in one wave; live-cluster ops, no commit yet — doc-update commit pending):

  • #4 done: plugin-barman-cloud operator restarted (kubectl rollout restart deployment/barman-cloud -n cnpg-system); stale Garage retention-worker endpoint cache cleared
  • #11 done: cnpg-controller-manager restarted; clean plugin re-discovery confirmed in logs
  • CNPG operator upgraded v1.28.1 → v1.29.0 to expose spec.serviceAccountName field (added in v1.29.0 per upstream release notes); operator rollout clean, cluster healthy throughout
  • #2 done: cluster.spec.serviceAccountName=cnpg-backup-sa applied; supervised rolling restart cycled all 3 pods (db-3 first failed with RBAC error — patched RoleBindings knoe-db and knoe-db-barman-cloud to add cnpg-backup-sa as a subject; db-3 then recovered, db-4 and db-2 rolled cleanly); 23:03 UTC scheduled backup completed with new SA, verifying end-to-end
  • #3 done: Garage workload removed from knoe-dev-cnpg-0 (statefulset, both services including garage-s3-ilb, configmap, PVC data-garage-0, StorageClass garage-hdd); ObjectStore untouched, backups still progressing post-removal

• 2026-04-29 — Doc true-up + master TODO index in commit f5b24c7. Reality-vs-intent gaps captured across CLAUDE.md / AGENTS.md / pipeline-phases / plans / canonical ObjectStore manifest; new docs/TODO.md Kanban-style index.

• 2026-04-29 — GCS migration completed. ObjectStore gs://knoe-0-backups/ with googleCredentials.gkeEnvironment: true; WI auth via knoe-db SA. 2 verified backups in catalog. ScheduledBackup unsuspended. (live-cluster ops; no commit)

• 2026-04-29 — Garage cluster recovered. PVC 29 → 80 GiB, layout 10 → 60 GB, bucket cleared (8382 objects + 12 orphan multiparts). (live-cluster ops; no commit)

• 2026-04-28/29 — knoe-db-1 fix. 14d crashloop resolved via PVC delete + CNPG re-bootstrap; cluster failover-cycled all 3 pods; 3/3 healthy. (live-cluster ops; no commit)

• 2026-04-28 — Phase A repo cleanup + Phase C Maven build readiness in commit b355855. tmp/ archived (87 files); org.knoe → dev.knoe rename; authority → auth artifactId; sources moved to dev/knoe/auth/; make build-auth green. 4 k3s prole-auth-*.yaml manifests renamed to knoe-auth-*.yaml.

• 2026-04-28 — Pipeline Phase 0 (test pipeline foundation: pyproject.toml, IntelliJ run configs, coverage fix, welcome mode selector). See docs/pipeline-phases.md §Phase 0.