prole/docs/TODO.md

Master TODO index

Single source of truth for unfinished work in this repo. Captures items that surfaced across the autobuild pipeline phases, the 2026-04-28 → 2026-04-29 cluster recovery / Maven build readiness work, the 2026-04-29 CNPG operator + backup stabilization work, and reality-vs-intent drift in the docs.

When you pick something up, move it from the ranked queue into the Now section. When it lands, archive it into the Done section at the bottom (or delete if it's not worth memorializing).

When you discover a new item, add it to the ranked queue. Default rank: bottom. Move it up later if it gets pulled in.

The Kanban "Now" section at top is the only place this doc imposes structure. Everything below "Now" is a numbered list — top of the list is next-most-important.

---

Now (Kanban)

In progress

• Phase 2: pg_oauth in install/deploy.sh + k3d build — assigned to Junie. The Phase 1 work (per-engineer roles, external LB at pg.0.knoe.dev, cert SANs, pg_hba tightening, docs/db-access.md) is committed. Phase 2 replaces the SCRAM password mechanism with PG18 native OAUTHBEARER, wires it into install.sh / supabase/deploy.sh so a fresh deploy lands with the OIDC mechanism active by default, and ports the Service + cluster cert pattern to k3d. Connection target stays at pg.0.knoe.dev; only the credential issuance changes (libpq Device Flow → Google Bearer token). Plan reference: ~/.claude/plans/we-re-continuing-work-on-happy-toast.md.

Up next

(empty — pull from the ranked queue below)

Blocked

(none currently)

---

Next (ranked queue — top of list = next-most-important)

1. Installer env-contamination guard — install.sh / deploy.sh should refuse to run if ${KUBECONTEXT} from conf/${mode}.cfg doesn't match kubectl config current-context (or doesn't match what was passed via --context). Caused the 2026-04-28 14:00 UTC backup outage: an install.sh --mode k3d run with a GKE shell context overwrote the GCS-backed ObjectStore with a Garage-backed one, then Garage filled up. See CLAUDE.md §"Env-contamination warning".

2. k3s manifest cleanup: prole-* filename rename — deploy/opentofu/k3s/manifests/knoe/prole-{configmap,deployment,service,kdc-configmap,kdc-secrets.example}.yaml still carry the old prefix. The kustomization expects knoe-* names — kustomize is currently broken until rename. Same git mv pattern we used for prole-auth-* → knoe-auth-* in commit b355855. Brief: docs/plans/junie/02-k3s-prole-rename.md. Assigned to Junie.

3. Image rename knoe-authority → knoe-auth — three deployment manifests still reference knoe-authority:latest:

   • deploy/gcp/gke/knoe-auth-deployment.yaml (3 lines)
   • deploy/opentofu/k3s/manifests/knoe/knoe-auth-deployment.yaml (2 lines) The Maven artifact is now knoe-auth.jar (post-rename in commit b355855). Image rebuild + registry push needed; both manifests then bump.

4. Garage block-resync corruption — after the 2026-04-29 bulk delete (8382 objects + 12 multipart aborts), garage-0 (on the app cluster knoe-dev-0 — not the DB cluster, that one was removed) background workers logged Could not fetch needed block X, no node returned valid data for orphaned block refs. Eventually self-heals via garage_block::resync worker, but worth verifying with garage repair --yes blocks once the dust settles.

5. Phase B orphan cleanup — leftover from the cluster recovery work that didn't get cleaned because we focused on CNPG/GCS:

   • App cluster (knoe-dev-0): delete gitlab/gitlab-migrations-58a3e27-97-23-n72j8 Error pod (7d+ stale); reap 3 Released PVs in monitoring/ (4h orphans from monitoring stack recreation)
   • DB cluster (knoe-dev-cnpg-0): delete kube-system/prole-supabase-dirprep Job (18d stale, prole-era)

6. Fix scripts/patch_garage_cross_cluster.sh — three bugs surfaced when we used it on 2026-04-29:

   • Default DB_CLUSTER=knoe-cnpg-0 (line 39) is the stale name; should be knoe-dev-cnpg-0 (matches conf/gke.cfg)
   • Phase 1 only deletes service/garage (ClusterIP); misses service/garage-s3-ilb (the LoadBalancer at the IP the ObjectStore actually points at)
   • Phase 2 applies deploy/gcp/gke/knoe-db-backup-gcs.yaml which contains a SECOND ScheduledBackup using the legacy method: barmanObjectStore — would double-up our existing plugin-method ScheduledBackup, and that legacy method is being removed in CNPG v1.30. The script should split SA-only apply from ScheduledBackup apply.
   • Phase 3 unconditionally re-applies the canonical ObjectStore manifest. Now safe (manifest matches live), but worth a guard. Brief: docs/plans/junie/06-patch-garage-script-fixes.md. Assigned to Junie.

7. Update etc/init_cnpg_gke.sh to set cluster.spec.serviceAccountName: cnpg-backup-sa and create the RoleBinding additions — current script creates cnpg-backup-sa but doesn't wire it as the cluster pod SA. After 2026-04-29 stabilization the live cluster does this; the init script should reflect it so a fresh deploy lands in the same end state. Brief: docs/plans/junie/07-init-cnpg-gke-sa-wiring.md. Assigned to Junie.

8. Port oauth2-proxy manifest to k3d / k3s / min modes — deploy/gcp/gke/oauth2-proxy-deployment.yaml is GKE-shaped (uses cloud.google.com/v1 BackendConfig + GCE health-check semantics). For non-GCE deploy modes the same Deployment runs unchanged but the ingress glue differs (Traefik / nginx-ingress / no LB at all). Add a deploy/k3s/oauth2-proxy-deployment.yaml with the Traefik-flavored ingress annotations once we deploy that mode.

9. Wire GoTrue (supabase-auth) Google provider for Supabase API clients — different surface than the Studio gate (which oauth2-proxy now handles). Once we have client apps that authenticate end-users via Supabase Auth, configure GoTrue's Google provider with its own OAuth client. See docs/oidc-setup.md for the secret indirection pattern.

10. Audit-log story for the multi-tenant Studio phase — Studio doesn't track per-user actions in self-hosted; oauth2-proxy forwards the Authorization: Bearer <id_token> header to Kong (--pass-authorization-header=true / --set-authorization-header=true) but nothing reads it. When the multi-tenant control plane lands (per-user / per-project Studios), wire a Kong logging plugin that captures the email claim from the JWT and emits to a structured log sink.

11. Retire api.0.knoe.dev Ingress rule + managed cert — db.0.knoe.dev now serves both Studio (Google-gated) and the SDK API surface (Kong key-auth) via oauth2-proxy --skip-auth-route. api.0.knoe.dev remains as an alias for backward compatibility, but new docs/onboarding should point to db.0.knoe.dev. When external clients have all migrated: drop the api.0.knoe.dev rule from the supabase-kong Ingress and the supabase-api-managed-cert ManagedCertificate. Until then, the alias is harmless (same backend, same key-auth).

12. Fork the Supabase Studio image to wire in-app help / support / feedback buttons to mailto:support@knoe.dev (or https://db.0.knoe.dev/support) — upstream Studio (supabase/studio:2026.02.16-sha-26c615c) hardcodes those URLs to Supabase-cloud endpoints (supabase.com/dashboard/api/feedback etc.) which are unreachable from self-hosted, so the in-app "Report a problem" / "Send feedback" / "Get help" flows error out. No env-var hook exists in upstream — verified by Explore search. Fork the image, patch the relevant TSX (apps/studio/components/layouts/AppLayout/AppLayout.tsx, support-dialog component, settings/help links — handful of files), publish to our registry, bump image.studio.repository in supabase/helm/knoe-supabase/values.yaml. Same fork is the natural place to fix the OpenAI key panel, telemetry endpoints, and any other in-Studio assumptions about Supabase cloud as we encounter them. Stop-gap until then: https://db.0.knoe.dev/support 302s to mailto:support@knoe.dev (Kong route support); just need to tell users to bookmark or remember it.

13. Migrate off CNPG-deprecated monitoring.enablePodMonitor + podMonitorRelabelings fields — both fields in deploy/gcp/gke/knoe-db.yaml spec.monitoring are flagged deprecated by the CNPG operator and "will be removed in a future release". Migration path per the operator's warning: manually manage the PodMonitor resource (i.e. write deploy/gcp/gke/knoe-db-podmonitor.yaml ourselves and apply alongside the Cluster). New file would carry the relabelings (cluster label injection, currently sourced from pod label cnpg.io/cluster) and any future scrape customization. Cluster spec keeps only the bare CNPG operator config. No urgency — works fine on the current operator (CNPG v1.29) — but file before the next CNPG upgrade. Reference: kubectl deprecation warning emitted on every patch of these fields. Brief: docs/plans/junie/13-podmonitor-manual-management.md. Assigned to Junie.

14. Migrate cnpg-prometheus datasource UID to the stable cnpg-prometheus name — currently the live datasource has the auto-generated UID P5531627C358300FE from the original kps install. We pinned uid: cnpg-prometheus in monitoring/kps-values-gke.yaml so any FRESH kps install lands on the stable name, but Grafana refuses to change the UID of an already-provisioned datasource (read-only via API; rollout-restart doesn't migrate it). The dashboard transform monitoring/cnpg-dashboard-transforms.yaml ds_prometheus_default_to_cnpg carries the auto-uid as a workaround. Migration path on the next clean kps re-install (or after a maintenance window where we can wipe the Grafana sqlite DB to drop datasources): swap value: P5531627C358300FE → value: cnpg-prometheus in the transform and re-run the sync tool. No client-visible change either way.

15. Remove DASHBOARD consumer + basicauth_credentials from supabase/helm/knoe-supabase/templates/kong/config.yaml — now-dead config: the basic-auth plugin block was removed in commit (this oauth2-proxy work), so the consumer + credentials it gated have no enforcer. Lines 30-32 ({{- if .Values.secret.dashboard }} - username: DASHBOARD {{- end }}) and lines 44-49 (the basicauth_credentials: block) can be deleted; then drop secret.dashboard from values.yaml and the DASHBOARD_USERNAME / DASHBOARD_PASSWORD envsubst lines from wrapper.sh. Cleanup-only; current state is harmless. Brief: docs/plans/junie/15-remove-dead-dashboard-consumer.md. Assigned to Junie. Note: brief flags the OpenAI key trap — secret.dashboard.openAiApiKey is the only LIVE consumer of that section, so don't blanket-delete.

---

Reality-vs-intent gaps

Items where docs describe an intended state that doesn't match live state. Each links to where reality is documented. Closing the gap means either updating the doc to match reality permanently, or doing the work to make reality match.

#	Reality (current)	Intended state	Documented at	Tracked work item
R4	Installer can clobber GKE state when run in k3d mode with a GKE shell context (caused 2026-04-28 14:00 UTC outage)	Installer refuses to run on context mismatch	CLAUDE.md §"install.sh / deploy.sh pre-flight" warning	Queue item #1
R5	k3s manifests prole-{configmap,deployment,service,kdc-*}.yaml still carry old prefix	All renamed to knoe-* (kustomization already references knoe-*)	Phase C plan in ~/.claude/plans/we-re-continuing-work-on-happy-toast.md follow-ups	Queue item #2
R6	Image refs knoe-authority:latest in deploy manifests	knoe-auth:latest (matches Maven artifact knoe-auth.jar post-rename)	Phase C plan follow-ups	Queue item #3
R8	etc/init_cnpg_gke.sh doesn't set cluster.spec.serviceAccountName or extend the role bindings to cnpg-backup-sa	Script lands a fresh deploy in the same end state as live (with serviceAccountName: cnpg-backup-sa and role-binding subjects added)	Live cluster spec + RoleBindings	Queue item #7
R9	scripts/patch_garage_cross_cluster.sh has stale DB_CLUSTER default, misses garage-s3-ilb service, and bundles legacy ScheduledBackup apply	Three issues fixed; phases split so removal is independent of SA/ObjectStore re-apply	Live script	Queue item #6

---

Later

11. Round 1.5: OpenBao transit-key encryption for per-user secrets at rest — authority/src/main/java/dev/knoe/auth/enroll/UserProvisioningService.java:84 has // TODO Round 1.5: encrypt secret with OpenBao transit key before storing. Currently TOTP secrets and similar long-lived per-user state go to knoe.* tables with PG-level encryption-at-rest only. Round 1.5 wraps writes/reads with an OpenBao transit-key envelope. See docs/plans/knoe-auth-round-1.md §8 "Out of scope for Round 1".

12. knoe-db/docker-entrypoint.sh:3 swap to set -Eeuo pipefail — current set -e only. Marked as TODO in the file: "swap to -Eeuo pipefail above (after handling all potentially-unset variables)". Needs an audit pass over the script for unset-var safety first.

13. Customer-deploy-resync execution decision — currently no separate ~/dev/prole working tree under development; the plan in docs/plans/customer-deploy-resync.md is dormant. Decision: do we ever activate the per-customer branching workflow (e.g. for a future second customer), or formalize "single-tenant for now" and remove the plan?

---

Deferred / Out of scope

• Studio Database Advisor — Performance: unindexed FKs in upstream-managed schemas — 6 FKs flagged on a fresh project: auth.mfa_challenges.factor_id, auth.saml_relay_states.flow_state_id, auth.oauth_authorizations.{client_id, user_id}, storage.s3_multipart_uploads_parts.{upload_id, bucket_id}. Adding indexes to upstream-managed tables risks conflict on the next supabase-auth / supabase-storage migration. Wait for upstream fixes; revisit if these turn out to be load-bearing for our usage.

• com.knoey.auth open-source split — saved for the major collaboration (per user memory: "we are saving knoey.com/auth for the major collaboration"). Will require a separate repo extraction and licensing pass.

• "Test ssh to Gitea" task — orphaned during a previous Cowork+Code session when the Dispatch connection was lost. May or may not still be relevant.

• Volume snapshot backups (CNPG VolumeSnapshot method) — alternative to barman tarball. Considered during the 2026-04-29 outage debugging when barman-cloud-backup looked broken; now that GCS+barman is verified working, no immediate need. Worth revisiting if barman shows further fragility.

• File the upstream plugin-barman-cloud issue about the misleading ReferenceError: weakly-referenced object no longer exists traceback that fires on persistent 5xx responses. Issue #707 is closed but didn't capture this pattern; ours is a clearer reproduction. Defer — the upstream is responsive but the workaround (sufficient bucket capacity + correct IAM) is good enough.

---

Done

(items get archived here when they land, with a date and commit reference; or just delete if not worth memorializing)

• 2026-05-01 — Cross-cluster CNPG metrics via DB-cluster Prometheus (commit pending). Solves the cloudnative-pg Grafana dashboard's "No data" problem from queue item #12. Picked path (c) over the originally-recommended (b) for DB-cluster autonomy + pattern symmetry — each cluster owns its observability footprint, and a future second DB cluster reuses the recipe one-for-one. Installed kube-prometheus-stack v84.3.0 on knoe-dev-cnpg-0 (monitoring namespace), Grafana + Alertmanager disabled (single canonical Grafana stays on app cluster). All PVCs explicitly pinned to standard-hdd (per the user's standing rule on SSD-quota preservation; verified prometheus-...-0 PVC bound 20Gi standard-hdd). New ILB prometheus-cnpg-ilb in DB-cluster monitoring ns at 10.180.15.216:9090, reachable from app-cluster pods over the shared VPC. New cnpg-prometheus datasource added to app-cluster Grafana via monitoring/kps-values-gke.yaml grafana.additionalDataSources. Side-fix: removed an orphan knoe-grafana-datasource ConfigMap that etc/init_monitoring.sh:apply_grafana_datasource() had been creating with a duplicate uid: prometheus — caused Grafana's provisioning reload to fail with HTTP 500 once any second datasource arrived. Function neutered to a no-op with a clear comment; chart values are now the single source of truth for datasources. Verified end-to-end: Prometheus on DB cluster discovers all 3 CNPG postgres pods (knoe-db-2/3/4 at 10.24.x.x:9187, all health=up); Grafana datasource health endpoint returns Successfully queried the Prometheus API; cnpg_pg_stat_archiver_archived_count query through Grafana proxy returns live data (704 archived WAL files at time of test). To complete: switch each cnpg-grafana dashboard's DS_PROMETHEUS template variable to cnpg-prometheus (one-click). New files: monitoring/kps-cnpg-values.yaml, deploy/gcp/gke/cnpg-prometheus-ilb.yaml. Modified: monitoring/kps-values-gke.yaml (datasource added), etc/init_monitoring.sh (function no-op'd). Out of scope (Phase 2): federate alerts back to app-cluster Alertmanager; reserve a static internal IP so the ILB doesn't drift on re-create.

• 2026-04-30 — Bump postgres pod memory limit 512Mi → 2Gi (commit pending). Long-standing throughput cliff: barman-cloud-backup is single-threaded gzip + GCS upload buffering, ceiling-bound by the pod's memory limit. The 9 GB DB took 30–90 min to back up at 512Mi; 2Gi cuts that to 5–10 min. requests left at 128Mi (no scheduling change; this is purely a ceiling bump). Patched live with kubectl patch cluster knoe-db --type=merge -p '{"spec":{"resources":{"limits":{"memory":"2Gi"}}}}'; CNPG operator performed a supervised rolling restart (replicas first, primary last). Repo source updated at deploy/gcp/gke/knoe-db.yaml. Pre-flight verified no external psql sessions (chrisfu/ron not connected); 39 internal sessions FATAL'd through the rolling restart and reconnected via their pools — supabase + gitlab + streaming_replica all stable post-rollout. Closes drift R3. k3s + k8s mode resource limits remain at 512Mi (not exposed in those manifests today); cross-mode parity folded into queue item #8 ("Port oauth2-proxy manifest to k3d/k3s/min modes").

• 2026-04-30 — Align knoe/core/actions.py cluster-name + machine-type defaults with live state (commit pending). Two stale constants since the rebrand: DEFAULT_DB_CLUSTER_NAME = "knoe-cnpg-0" (live: knoe-dev-cnpg-0) and DEFAULT_APP_CLUSTER_MACHINE_TYPE = "e2-small" (live: e2-standard-2). Updated both at knoe/core/actions.py lines 121 + 123. Verified: python3 -c "from knoe.core.actions import ..." reads the new values; pytest collection error count went from 8 → 7 post-edit (one of the pre-existing collection errors apparently went away when the import resolved differently — no regression). Closes drift R1 + R2 in CLAUDE.md (footnotes + drift table rows + the # STALE inline comments in the §"Cluster code constants" code block all dropped). Implicitly closes the formerly-#8 queue item ("Update CLAUDE.md cluster table once defaults match") in this same commit. Queue renumbered: #2 (this item) and #8 dropped, gaps closed.

• 2026-04-30 — Drop redundant GCP IAM workloadIdentityUser binding (commit pending). v1.28-era leftover: [knoe-db-0/knoe-db] was bound alongside the active [knoe-db-0/cnpg-backup-sa] on the cnpg-backup@… GCP SA. Pods have used cnpg-backup-sa since the 2026-04-29 stabilization (commit e92cd8b), so the older binding was dead config — kept around because removing it required a reasoned check that no in-flight job still authenticated under the old identity. Verified live (gcloud iam service-accounts get-iam-policy), dropped via gcloud iam service-accounts remove-iam-policy-binding, re-verified the SA's policy now has only [knoe-db-0/cnpg-backup-sa]. CNPG primary still healthy (SELECT 1 returned cleanly post-removal). Closes drift R7. Removed the matching CLAUDE.md Reality TODO + drift table row.

• 2026-04-30 — Grafana at svc.knoe.dev/grafana with Google OAuth (commit pending). Diagnosed the svc.knoe.dev "name resolution failed" error: knoe-svc-kong-config had its grafana service pointing at prometheus-grafana.monitoring.svc.cluster.local:80 — the OLD upstream name from before the kube-prometheus-stack chart renamed to kps-grafana. Kong was DNS-failing every request to that vanished service and surfacing the Kong-style "name resolution failed" error to clients. Fixed the upstream + scoped the route to /grafana (so other surfaces can land at svc.knoe.dev/* later). Layered Grafana's native Google OAuth via auth.google config in monitoring/kps-values-gke.yaml (new — captures previously-uncommitted live Helm values + adds the auth additions). Role mapping via JMESPath: contains(['chrisfu@knoey.com', 'ron@knoey.com'], email) && 'Admin' || 'Editor' — anyone @knoey.com gets Editor, chrisfu+ron get Admin. New manifests: deploy/gcp/gke/grafana-google-oidc-secret.example.yaml, etc/init_grafana_oauth.sh, monitoring/kps-values-gke.yaml. Repo Kong source updated at deploy/opentofu/k3s/manifests/knoe/kong-configmap.yaml (.org→.dev rebrand, healthz route restored, /grafana scoping). Verified externally: /grafana/login → 200; /grafana/login/google → 302 accounts.google.com with the right client_id; subpath assets load; Studio + SDK paths on db.0.knoe.dev unchanged. Browser walkthrough (chrisfu sign-in → Admin role; ron same; outside-domain rejected by allowed_domains: knoey.com) is the manual follow-up. Stop-gap adminPassword: admin in values stays (local break-glass) — rotate this when convenient as a follow-up.

• 2026-04-30 — Reusable engineer onboarding via one-time link + QR (commit pending). Replaces the assumed-but-nonexistent shared 1Password vault with a self-contained delivery flow: ./etc/onboard_engineer.sh <user> <email> provisions the postgres role + 24h temp password and outputs (1) a single-shot URL, (2) a QR-code rendering for screenshare, (3) the plaintext fallback. Engineer opens URL or scans QR → static page at db.0.knoe.dev/onboard.html (vanilla HTML+JS, no backend) reads the URL fragment client-side and shows password + psql connection string + rotation snippet with [Copy] buttons; saves to engineer's personal 1Password. Engineer rotates immediately via \password. New manifests: deploy/gcp/gke/knoe-onboard.yaml (nginx:alpine + ConfigMap + Service), Kong route /onboard.html → knoe-onboard:80 in supabase/helm/knoe-supabase/templates/kong/config.yaml, oauth2-proxy --skip-auth-route=^/onboard\.html$ in deploy/gcp/gke/oauth2-proxy-deployment.yaml. Pivotal pg_hba change: rule changed from hostssl all chrisfu,ron all to hostssl all +knoe_developer all so future onboards don't require pg_hba edits — the script's GRANT knoe_developer TO <user> is enough. CA cert committed at etc/knoe-db-ca.crt (public root, not a secret) so engineers reference sslrootcert=etc/knoe-db-ca.crt after cloning. Engineer-facing docs: docs/onboarding.md. docs/db-access.md updated to remove the broken 1Password reference. Verified end-to-end: ./etc/onboard_engineer.sh test-onboard test-onboard@knoey.com → URL + QR rendered → psql ...user=test-onboard... connects with the temp password and reports pg_has_role(current_user,'knoe_developer','MEMBER')=t → --revoke drops the role cleanly.

• 2026-04-30 — Per-engineer psql access via pg.0.knoe.dev (Phase 1) (commit pending). External GKE TCP LoadBalancer at 34.106.156.196 (forwarding rule a61028024f2314e06902bafc1b4c3a08); user adds Namecheap A record pg.0.knoe.dev → 34.106.156.196 (manual; Namecheap UI). CNPG serverAltDNSNames extended with pg.0.knoe.dev so the auto-issued server cert validates externally. New knoe_developer group role with R/W on knoe+public and R/O on auth+storage+extensions. Per-engineer roles chrisfu and ron with strong random passwords (192-bit base64) VALID UNTIL '2026-05-30', members of knoe_developer. pg_hba rewritten so external requires TLS+SCRAM and only allows chrisfu, ron (other roles postgres, supabase_admin, authenticator stay cluster-internal). externalTrafficPolicy: Local on the LB to preserve client source IP (otherwise the SNAT'd node IP made externals look like internal cluster traffic and bypassed the hostnossl reject rule). Documentation in docs/db-access.md. Verified: chrisfu+ron both connect via TLS, plaintext rejected with the proper pg_hba error showing the real client IP, postgres role connection from outside fails auth, internal supabase services keep their existing TLS connections (gitlab on TLS 1.3, supabase_admin/authenticator on plaintext over RFC1918 — both still allowed by the new rules). Phase 2 (libpq OAUTHBEARER + validator library, bundled into install/deploy.sh + k3d) handed off to Junie — see "In progress" at top of file.

• 2026-04-30 — pg_stat_statements moved out of public schema across deploy modes (commit pending). Studio's Database Advisor was flagging public.pg_stat_statements as a Security warning the moment any user opened the dashboard. Root cause: relocatable extensions without an explicit SCHEMA clause on CREATE EXTENSION land in the first writable schema in the connecting role's search_path, which is public for the postgres role. Live: ALTER EXTENSION pg_stat_statements SET SCHEMA extensions;. Bootstrap (3 files, all 4 deploy modes): deploy/gcp/gke/knoe-db.yaml (gke), deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml (k3s), k8s/knoe/knoe-db.yaml (k3d / min) — each postInitTemplateSQL block now creates the extensions schema and CREATE EXTENSION ... SCHEMA extensions;. Plus supabase/deploy.sh gets GRANT USAGE ON SCHEMA extensions TO anon, authenticated, service_role; so PostgREST clients (and the advisor itself) can resolve the schema. Other security checks verified clean (no tables in public without RLS, no SECURITY DEFINER without search_path, no auth.users-exposed views, no duplicate indexes, no extensions left in public). Performance side has 6 unindexed FKs in upstream auth.* and storage.* tables — captured as deferred (touching upstream-managed schemas is the wrong move; they'll get fixed in supabase-auth / supabase-storage releases).

• 2026-04-30 — Studio post-launch fixes (storage permissions + /support stop-gap) in commit 79a3f76:

  • Storage list-buckets unblocked. supabase_storage_admin role was missing membership in service_role / authenticated / anon, so SET LOCAL role = 'service_role' (run by supabase-storage on every request) failed with PG 42501 at guc.c:6936 call_string_check_hook, surfaced as a misleading "new row violates row-level security policy" error. One GRANT applied live (GRANT anon, authenticated, service_role TO supabase_storage_admin;) and added durably to supabase/deploy.sh right after the existing GRANT service_role TO authenticator block.
  • oauth2-proxy auth-header passthrough fixed. The --pass-authorization-header=true and --set-authorization-header=true flags were stomping the Authorization: Bearer <apikey> header that Studio / supabase-py send on /storage/v1/... calls (oauth2-proxy was rewriting it with the Google OIDC id_token, which storage couldn't validate against JWT_SECRET). Both flags removed from deploy/gcp/gke/oauth2-proxy-deployment.yaml. When per-user identity passthrough lands post-knoe-auth Round 1, use --pass-user-headers / --set-xauthrequest (X-Forwarded-* headers) instead — those don't collide with Supabase's auth.
  • /support route stop-gap. https://db.0.knoe.dev/support 302s to mailto:support@knoe.dev so users have a working escape hatch while the in-Studio support buttons stay broken (separate Studio-fork follow-up #15). Kong service support added to templates/kong/config.yaml using request-termination (302) + response-transformer (Location header). KONG_PLUGINS allowlist extended with response-transformer in values.yaml. oauth2-proxy gets --skip-auth-route=^/support$ so an unauthenticated user looking for help isn't bounced into a Google sign-in detour.

• 2026-04-30 — Unified db.0.knoe.dev for Studio + SDK in commit f805404: added two --skip-auth-route args to oauth2-proxy so SDK requests on /auth/v1, /rest/v1, /realtime/v1, /storage/v1, /functions/v1, /graphql/v1, /pg/* bypass the Google sign-in flow and pass through to supabase-kong, where Kong's existing key-auth plugin enforces anon/service-role keys. Studio UI at / remains Google-gated. One URL now covers humans-in-Studio AND robots-via-supabase-py — same security envelope as api.0.knoe.dev had on those paths. api.0.knoe.dev kept as a backward-compat alias (see follow-up #14). External verification: curl -H "apikey: <KEY>" https://db.0.knoe.dev/rest/v1/ → 200 OpenAPI; curl https://db.0.knoe.dev/rest/v1/ (no key) → 401 from Kong; curl https://db.0.knoe.dev/ → 302 to Google (Studio gate intact); curl https://db.0.knoe.dev/api/profile → 302 to Google (Studio internals stay gated).

• 2026-04-30 — Google sign-in to db.0.knoe.dev for chrisfu + ron (shared Studio) in commit 25f1b2e: oauth2-proxy v7.15.2 deployed in front of supabase-kong as the Studio access gate (Google Workspace OIDC, --email-domain=knoey.com); supabase-kong Ingress patched so db.0.knoe.dev → oauth2-proxy:80 while api.0.knoe.dev → supabase-kong:8000 is untouched (key-auth still gates the API surface); Kong dashboard route's basic-auth plugin removed (oauth2-proxy is the gate now — double-auth was friction without security gain). External verification: db.0.knoe.dev/ → 302 accounts.google.com/o/oauth2/auth?client_id=200699021716-…&redirect_uri=https://db.0.knoe.dev/oauth2/callback&scope=openid+email+profile; api.0.knoe.dev/rest/v1/ → 401 from Kong key-auth (untouched); /healthz → 200. Three new files + chart edit: deploy/gcp/gke/oauth2-proxy-deployment.yaml, deploy/gcp/gke/oauth2-proxy-google-oidc-secret.example.yaml, etc/init_oauth2_proxy.sh, and the basic-auth plugin removed from supabase/helm/knoe-supabase/templates/kong/config.yaml. Future swap to knoe-auth Round 1: change one arg in oauth2-proxy from --provider=google to --provider=oidc --oidc-issuer-url=https://api.knoe.dev/auth and reapply.

• 2026-04-29 — CNPG operator + backup stabilization session (4 queue items in one wave; live-cluster ops, no commit yet — doc-update commit pending):

  • #4 done: plugin-barman-cloud operator restarted (kubectl rollout restart deployment/barman-cloud -n cnpg-system); stale Garage retention-worker endpoint cache cleared
  • #11 done: cnpg-controller-manager restarted; clean plugin re-discovery confirmed in logs
  • CNPG operator upgraded v1.28.1 → v1.29.0 to expose spec.serviceAccountName field (added in v1.29.0 per upstream release notes); operator rollout clean, cluster healthy throughout
  • #2 done: cluster.spec.serviceAccountName=cnpg-backup-sa applied; supervised rolling restart cycled all 3 pods (db-3 first failed with RBAC error — patched RoleBindings knoe-db and knoe-db-barman-cloud to add cnpg-backup-sa as a subject; db-3 then recovered, db-4 and db-2 rolled cleanly); 23:03 UTC scheduled backup completed with new SA, verifying end-to-end
  • #3 done: Garage workload removed from knoe-dev-cnpg-0 (statefulset, both services including garage-s3-ilb, configmap, PVC data-garage-0, StorageClass garage-hdd); ObjectStore untouched, backups still progressing post-removal

• 2026-04-29 — Doc true-up + master TODO index in commit f5b24c7. Reality-vs-intent gaps captured across CLAUDE.md / AGENTS.md / pipeline-phases / plans / canonical ObjectStore manifest; new docs/TODO.md Kanban-style index.

• 2026-04-29 — GCS migration completed. ObjectStore gs://knoe-0-backups/ with googleCredentials.gkeEnvironment: true; WI auth via knoe-db SA. 2 verified backups in catalog. ScheduledBackup unsuspended. (live-cluster ops; no commit)

• 2026-04-29 — Garage cluster recovered. PVC 29 → 80 GiB, layout 10 → 60 GB, bucket cleared (8382 objects + 12 orphan multiparts). (live-cluster ops; no commit)

• 2026-04-28/29 — knoe-db-1 fix. 14d crashloop resolved via PVC delete + CNPG re-bootstrap; cluster failover-cycled all 3 pods; 3/3 healthy. (live-cluster ops; no commit)

• 2026-04-28 — Phase A repo cleanup + Phase C Maven build readiness in commit b355855. tmp/ archived (87 files); org.knoe → dev.knoe rename; authority → auth artifactId; sources moved to dev/knoe/auth/; make build-auth green. 4 k3s prole-auth-*.yaml manifests renamed to knoe-auth-*.yaml.

• 2026-04-28 — Pipeline Phase 0 (test pipeline foundation: pyproject.toml, IntelliJ run configs, coverage fix, welcome mode selector). See docs/pipeline-phases.md §Phase 0.