diff --git a/docs/TODO.md b/docs/TODO.md new file mode 100644 index 0000000..bb92adc --- /dev/null +++ b/docs/TODO.md @@ -0,0 +1,145 @@ +# Master TODO index + +Single source of truth for unfinished work in this repo. Captures items that surfaced across the autobuild pipeline phases, the 2026-04-28 → 2026-04-29 cluster recovery / Maven build readiness work, the 2026-04-29 CNPG operator + backup stabilization work, and reality-vs-intent drift in the docs. + +When you pick something up, move it from the ranked queue into the **Now** section. When it lands, archive it into the `Done` section at the bottom (or delete if it's not worth memorializing). + +When you discover a new item, add it to the ranked queue. Default rank: bottom. Move it up later if it gets pulled in. + +The Kanban "Now" section at top is the only place this doc imposes structure. Everything below "Now" is a numbered list — top of the list is next-most-important. + +--- + +## Now (Kanban) + +### In progress +- **Phase 2: pg_oauth in install/deploy.sh + k3d build** — assigned to Junie. The Phase 1 work (per-engineer roles, external LB at `pg.0.knoe.dev`, cert SANs, pg_hba tightening, [`docs/db-access.md`](db-access.md)) is committed. Phase 2 replaces the SCRAM password mechanism with PG18 native OAUTHBEARER, wires it into `install.sh` / `supabase/deploy.sh` so a fresh deploy lands with the OIDC mechanism active by default, and ports the Service + cluster cert pattern to k3d. Connection target stays at `pg.0.knoe.dev`; only the credential issuance changes (libpq Device Flow → Google Bearer token). Plan reference: `~/.claude/plans/we-re-continuing-work-on-happy-toast.md`. + +### Up next +*(empty — pull from the ranked queue below)* + +### Blocked +*(none currently)* + +--- + +## Next (ranked queue — top of list = next-most-important) + +1. **Installer env-contamination guard** — `install.sh` / `deploy.sh` should refuse to run if `${KUBECONTEXT}` from `conf/${mode}.cfg` doesn't match `kubectl config current-context` (or doesn't match what was passed via `--context`). Caused the 2026-04-28 14:00 UTC backup outage: an `install.sh --mode k3d` run with a GKE shell context overwrote the GCS-backed ObjectStore with a Garage-backed one, then Garage filled up. See [`CLAUDE.md`](../CLAUDE.md) §"Env-contamination warning". + +2. **k3s manifest cleanup: `prole-*` filename rename** — `deploy/opentofu/k3s/manifests/knoe/prole-{configmap,deployment,service,kdc-configmap,kdc-secrets.example}.yaml` still carry the old prefix. The kustomization expects `knoe-*` names — kustomize is currently broken until rename. Same `git mv` pattern we used for `prole-auth-*` → `knoe-auth-*` in commit [b355855](#). **Brief: [`docs/plans/junie/02-k3s-prole-rename.md`](plans/junie/02-k3s-prole-rename.md). Assigned to Junie.** + +3. **Image rename `knoe-authority` → `knoe-auth`** — three deployment manifests still reference `knoe-authority:latest`: + - `deploy/gcp/gke/knoe-auth-deployment.yaml` (3 lines) + - `deploy/opentofu/k3s/manifests/knoe/knoe-auth-deployment.yaml` (2 lines) + The Maven artifact is now `knoe-auth.jar` (post-rename in commit [b355855](#)). Image rebuild + registry push needed; both manifests then bump. + +4. **Garage block-resync corruption** — after the 2026-04-29 bulk delete (8382 objects + 12 multipart aborts), garage-0 (on the **app cluster** `knoe-dev-0` — not the DB cluster, that one was removed) background workers logged `Could not fetch needed block X, no node returned valid data` for orphaned block refs. Eventually self-heals via `garage_block::resync` worker, but worth verifying with `garage repair --yes blocks` once the dust settles. + +5. **Phase B orphan cleanup** — leftover from the cluster recovery work that didn't get cleaned because we focused on CNPG/GCS: + - **App cluster (`knoe-dev-0`):** delete `gitlab/gitlab-migrations-58a3e27-97-23-n72j8` Error pod (7d+ stale); reap 3 Released PVs in `monitoring/` (4h orphans from monitoring stack recreation) + - **DB cluster (`knoe-dev-cnpg-0`):** delete `kube-system/prole-supabase-dirprep` Job (18d stale, prole-era) + +6. **Fix `scripts/patch_garage_cross_cluster.sh`** — three bugs surfaced when we used it on 2026-04-29: + - Default `DB_CLUSTER=knoe-cnpg-0` (line 39) is the stale name; should be `knoe-dev-cnpg-0` (matches `conf/gke.cfg`) + - Phase 1 only deletes `service/garage` (ClusterIP); misses `service/garage-s3-ilb` (the LoadBalancer at the IP the ObjectStore actually points at) + - Phase 2 applies `deploy/gcp/gke/knoe-db-backup-gcs.yaml` which contains a SECOND ScheduledBackup using the legacy `method: barmanObjectStore` — would double-up our existing plugin-method ScheduledBackup, and that legacy method is being removed in CNPG v1.30. The script should split SA-only apply from ScheduledBackup apply. + - Phase 3 unconditionally re-applies the canonical ObjectStore manifest. Now safe (manifest matches live), but worth a guard. + **Brief: [`docs/plans/junie/06-patch-garage-script-fixes.md`](plans/junie/06-patch-garage-script-fixes.md). Assigned to Junie.** + +7. **Update `etc/init_cnpg_gke.sh` to set `cluster.spec.serviceAccountName: cnpg-backup-sa` and create the RoleBinding additions** — current script creates `cnpg-backup-sa` but doesn't wire it as the cluster pod SA. After 2026-04-29 stabilization the live cluster does this; the init script should reflect it so a fresh deploy lands in the same end state. **Brief: [`docs/plans/junie/07-init-cnpg-gke-sa-wiring.md`](plans/junie/07-init-cnpg-gke-sa-wiring.md). Assigned to Junie.** + +8. **Port oauth2-proxy manifest to k3d / k3s / min modes** — `deploy/gcp/gke/oauth2-proxy-deployment.yaml` is GKE-shaped (uses `cloud.google.com/v1` BackendConfig + GCE health-check semantics). For non-GCE deploy modes the same Deployment runs unchanged but the ingress glue differs (Traefik / nginx-ingress / no LB at all). Add a `deploy/k3s/oauth2-proxy-deployment.yaml` with the Traefik-flavored ingress annotations once we deploy that mode. + +9. **Wire GoTrue (`supabase-auth`) Google provider for Supabase API clients** — different surface than the Studio gate (which oauth2-proxy now handles). Once we have client apps that authenticate end-users via Supabase Auth, configure GoTrue's Google provider with its own OAuth client. See [`docs/oidc-setup.md`](oidc-setup.md) for the secret indirection pattern. + +10. **Audit-log story for the multi-tenant Studio phase** — Studio doesn't track per-user actions in self-hosted; oauth2-proxy forwards the `Authorization: Bearer ` header to Kong (`--pass-authorization-header=true` / `--set-authorization-header=true`) but nothing reads it. When the multi-tenant control plane lands (per-user / per-project Studios), wire a Kong logging plugin that captures the email claim from the JWT and emits to a structured log sink. + +11. **Retire `api.0.knoe.dev` Ingress rule + managed cert** — `db.0.knoe.dev` now serves both Studio (Google-gated) and the SDK API surface (Kong key-auth) via oauth2-proxy `--skip-auth-route`. `api.0.knoe.dev` remains as an alias for backward compatibility, but new docs/onboarding should point to `db.0.knoe.dev`. When external clients have all migrated: drop the `api.0.knoe.dev` rule from the supabase-kong Ingress and the `supabase-api-managed-cert` ManagedCertificate. Until then, the alias is harmless (same backend, same key-auth). + +12. **Fork the Supabase Studio image to wire in-app help / support / feedback buttons to `mailto:support@knoe.dev` (or `https://db.0.knoe.dev/support`)** — upstream Studio (`supabase/studio:2026.02.16-sha-26c615c`) hardcodes those URLs to Supabase-cloud endpoints (`supabase.com/dashboard/api/feedback` etc.) which are unreachable from self-hosted, so the in-app "Report a problem" / "Send feedback" / "Get help" flows error out. No env-var hook exists in upstream — verified by Explore search. Fork the image, patch the relevant TSX (`apps/studio/components/layouts/AppLayout/AppLayout.tsx`, support-dialog component, settings/help links — handful of files), publish to our registry, bump `image.studio.repository` in [`supabase/helm/knoe-supabase/values.yaml`](../supabase/helm/knoe-supabase/values.yaml). Same fork is the natural place to fix the OpenAI key panel, telemetry endpoints, and any other in-Studio assumptions about Supabase cloud as we encounter them. Stop-gap until then: `https://db.0.knoe.dev/support` 302s to `mailto:support@knoe.dev` (Kong route `support`); just need to tell users to bookmark or remember it. + +13. **Migrate off CNPG-deprecated `monitoring.enablePodMonitor` + `podMonitorRelabelings` fields** — both fields in [`deploy/gcp/gke/knoe-db.yaml`](../deploy/gcp/gke/knoe-db.yaml) `spec.monitoring` are flagged deprecated by the CNPG operator and "will be removed in a future release". Migration path per the operator's warning: manually manage the PodMonitor resource (i.e. write `deploy/gcp/gke/knoe-db-podmonitor.yaml` ourselves and apply alongside the Cluster). New file would carry the `relabelings` (cluster label injection, currently sourced from pod label `cnpg.io/cluster`) and any future scrape customization. Cluster spec keeps only the bare CNPG operator config. No urgency — works fine on the current operator (CNPG v1.29) — but file before the next CNPG upgrade. Reference: kubectl deprecation warning emitted on every patch of these fields. **Brief: [`docs/plans/junie/13-podmonitor-manual-management.md`](plans/junie/13-podmonitor-manual-management.md). Assigned to Junie.** + +14. **Migrate cnpg-prometheus datasource UID to the stable `cnpg-prometheus` name** — currently the live datasource has the auto-generated UID `P5531627C358300FE` from the original kps install. We pinned `uid: cnpg-prometheus` in [`monitoring/kps-values-gke.yaml`](../monitoring/kps-values-gke.yaml) so any FRESH kps install lands on the stable name, but Grafana refuses to change the UID of an already-provisioned datasource (read-only via API; rollout-restart doesn't migrate it). The dashboard transform [`monitoring/cnpg-dashboard-transforms.yaml`](../monitoring/cnpg-dashboard-transforms.yaml) `ds_prometheus_default_to_cnpg` carries the auto-uid as a workaround. Migration path on the next clean kps re-install (or after a maintenance window where we can wipe the Grafana sqlite DB to drop datasources): swap `value: P5531627C358300FE` → `value: cnpg-prometheus` in the transform and re-run the sync tool. No client-visible change either way. + +15. **Remove DASHBOARD consumer + `basicauth_credentials` from `supabase/helm/knoe-supabase/templates/kong/config.yaml`** — now-dead config: the basic-auth plugin block was removed in commit (this oauth2-proxy work), so the consumer + credentials it gated have no enforcer. Lines 30-32 (`{{- if .Values.secret.dashboard }} - username: DASHBOARD {{- end }}`) and lines 44-49 (the `basicauth_credentials:` block) can be deleted; then drop `secret.dashboard` from `values.yaml` and the `DASHBOARD_USERNAME` / `DASHBOARD_PASSWORD` envsubst lines from `wrapper.sh`. Cleanup-only; current state is harmless. **Brief: [`docs/plans/junie/15-remove-dead-dashboard-consumer.md`](plans/junie/15-remove-dead-dashboard-consumer.md). Assigned to Junie. Note: brief flags the OpenAI key trap — `secret.dashboard.openAiApiKey` is the only LIVE consumer of that section, so don't blanket-delete.** + +--- + +## Reality-vs-intent gaps + +Items where docs describe an intended state that doesn't match live state. Each links to where reality is documented. Closing the gap means either updating the doc to match reality permanently, or doing the work to make reality match. + +| # | Reality (current) | Intended state | Documented at | Tracked work item | +|---|---|---|---|---| +| R4 | Installer can clobber GKE state when run in `k3d` mode with a GKE shell context (caused 2026-04-28 14:00 UTC outage) | Installer refuses to run on context mismatch | [`CLAUDE.md`](../CLAUDE.md) §"install.sh / deploy.sh pre-flight" warning | Queue item #1 | +| R5 | k3s manifests `prole-{configmap,deployment,service,kdc-*}.yaml` still carry old prefix | All renamed to `knoe-*` (kustomization already references `knoe-*`) | Phase C plan in `~/.claude/plans/we-re-continuing-work-on-happy-toast.md` follow-ups | Queue item #2 | +| R6 | Image refs `knoe-authority:latest` in deploy manifests | `knoe-auth:latest` (matches Maven artifact `knoe-auth.jar` post-rename) | Phase C plan follow-ups | Queue item #3 | +| R8 | `etc/init_cnpg_gke.sh` doesn't set `cluster.spec.serviceAccountName` or extend the role bindings to `cnpg-backup-sa` | Script lands a fresh deploy in the same end state as live (with `serviceAccountName: cnpg-backup-sa` and role-binding subjects added) | Live cluster spec + RoleBindings | Queue item #7 | +| R9 | `scripts/patch_garage_cross_cluster.sh` has stale `DB_CLUSTER` default, misses `garage-s3-ilb` service, and bundles legacy ScheduledBackup apply | Three issues fixed; phases split so removal is independent of SA/ObjectStore re-apply | Live script | Queue item #6 | + +--- + +## Later + +11. **Round 1.5: OpenBao transit-key encryption for per-user secrets at rest** — `authority/src/main/java/dev/knoe/auth/enroll/UserProvisioningService.java:84` has `// TODO Round 1.5: encrypt secret with OpenBao transit key before storing.` Currently TOTP secrets and similar long-lived per-user state go to `knoe.*` tables with PG-level encryption-at-rest only. Round 1.5 wraps writes/reads with an OpenBao transit-key envelope. See [`docs/plans/knoe-auth-round-1.md`](plans/knoe-auth-round-1.md) §8 "Out of scope for Round 1". + +12. **`knoe-db/docker-entrypoint.sh:3` swap to `set -Eeuo pipefail`** — current `set -e` only. Marked as TODO in the file: "swap to -Eeuo pipefail above (after handling all potentially-unset variables)". Needs an audit pass over the script for unset-var safety first. + +13. **Customer-deploy-resync execution decision** — currently no separate `~/dev/prole` working tree under development; the plan in [`docs/plans/customer-deploy-resync.md`](plans/customer-deploy-resync.md) is dormant. Decision: do we ever activate the per-customer branching workflow (e.g. for a future second customer), or formalize "single-tenant for now" and remove the plan? + +--- + +## Deferred / Out of scope + +- **Studio Database Advisor — Performance: unindexed FKs in upstream-managed schemas** — 6 FKs flagged on a fresh project: `auth.mfa_challenges.factor_id`, `auth.saml_relay_states.flow_state_id`, `auth.oauth_authorizations.{client_id, user_id}`, `storage.s3_multipart_uploads_parts.{upload_id, bucket_id}`. Adding indexes to upstream-managed tables risks conflict on the next supabase-auth / supabase-storage migration. Wait for upstream fixes; revisit if these turn out to be load-bearing for our usage. + +- **`com.knoey.auth` open-source split** — saved for the major collaboration (per user memory: "we are saving knoey.com/auth for the major collaboration"). Will require a separate repo extraction and licensing pass. +- **"Test ssh to Gitea" task** — orphaned during a previous Cowork+Code session when the Dispatch connection was lost. May or may not still be relevant. +- **Volume snapshot backups (CNPG `VolumeSnapshot` method)** — alternative to barman tarball. Considered during the 2026-04-29 outage debugging when barman-cloud-backup looked broken; now that GCS+barman is verified working, no immediate need. Worth revisiting if barman shows further fragility. +- **File the upstream `plugin-barman-cloud` issue** about the misleading `ReferenceError: weakly-referenced object no longer exists` traceback that fires on persistent 5xx responses. Issue [#707](https://github.com/cloudnative-pg/plugin-barman-cloud/issues/707) is closed but didn't capture this pattern; ours is a clearer reproduction. Defer — the upstream is responsive but the workaround (sufficient bucket capacity + correct IAM) is good enough. + +--- + +## Done + +*(items get archived here when they land, with a date and commit reference; or just delete if not worth memorializing)* + +- 2026-05-01 — **Cross-cluster CNPG metrics via DB-cluster Prometheus** (commit pending). Solves the `cloudnative-pg` Grafana dashboard's "No data" problem from queue item #12. Picked path (c) over the originally-recommended (b) for DB-cluster autonomy + pattern symmetry — each cluster owns its observability footprint, and a future second DB cluster reuses the recipe one-for-one. Installed kube-prometheus-stack v84.3.0 on `knoe-dev-cnpg-0` (`monitoring` namespace), Grafana + Alertmanager disabled (single canonical Grafana stays on app cluster). All PVCs explicitly pinned to `standard-hdd` (per the user's standing rule on SSD-quota preservation; verified `prometheus-...-0` PVC bound 20Gi `standard-hdd`). New ILB `prometheus-cnpg-ilb` in DB-cluster `monitoring` ns at `10.180.15.216:9090`, reachable from app-cluster pods over the shared VPC. New `cnpg-prometheus` datasource added to app-cluster Grafana via `monitoring/kps-values-gke.yaml` `grafana.additionalDataSources`. **Side-fix**: removed an orphan `knoe-grafana-datasource` ConfigMap that `etc/init_monitoring.sh:apply_grafana_datasource()` had been creating with a duplicate `uid: prometheus` — caused Grafana's provisioning reload to fail with HTTP 500 once any second datasource arrived. Function neutered to a no-op with a clear comment; chart values are now the single source of truth for datasources. Verified end-to-end: Prometheus on DB cluster discovers all 3 CNPG postgres pods (knoe-db-2/3/4 at 10.24.x.x:9187, all `health=up`); Grafana datasource health endpoint returns `Successfully queried the Prometheus API`; `cnpg_pg_stat_archiver_archived_count` query through Grafana proxy returns live data (704 archived WAL files at time of test). To complete: switch each cnpg-grafana dashboard's `DS_PROMETHEUS` template variable to `cnpg-prometheus` (one-click). New files: [`monitoring/kps-cnpg-values.yaml`](../monitoring/kps-cnpg-values.yaml), [`deploy/gcp/gke/cnpg-prometheus-ilb.yaml`](../deploy/gcp/gke/cnpg-prometheus-ilb.yaml). Modified: [`monitoring/kps-values-gke.yaml`](../monitoring/kps-values-gke.yaml) (datasource added), [`etc/init_monitoring.sh`](../etc/init_monitoring.sh) (function no-op'd). Out of scope (Phase 2): federate alerts back to app-cluster Alertmanager; reserve a static internal IP so the ILB doesn't drift on re-create. + +- 2026-04-30 — **Bump postgres pod memory limit 512Mi → 2Gi** (commit pending). Long-standing throughput cliff: `barman-cloud-backup` is single-threaded gzip + GCS upload buffering, ceiling-bound by the pod's memory limit. The 9 GB DB took 30–90 min to back up at 512Mi; 2Gi cuts that to 5–10 min. requests left at 128Mi (no scheduling change; this is purely a ceiling bump). Patched live with `kubectl patch cluster knoe-db --type=merge -p '{"spec":{"resources":{"limits":{"memory":"2Gi"}}}}'`; CNPG operator performed a supervised rolling restart (replicas first, primary last). Repo source updated at [`deploy/gcp/gke/knoe-db.yaml`](../deploy/gcp/gke/knoe-db.yaml). Pre-flight verified no external psql sessions (chrisfu/ron not connected); 39 internal sessions FATAL'd through the rolling restart and reconnected via their pools — supabase + gitlab + streaming_replica all stable post-rollout. Closes drift R3. k3s + k8s mode resource limits remain at 512Mi (not exposed in those manifests today); cross-mode parity folded into queue item #8 ("Port oauth2-proxy manifest to k3d/k3s/min modes"). + +- 2026-04-30 — **Align `knoe/core/actions.py` cluster-name + machine-type defaults with live state** (commit pending). Two stale constants since the rebrand: `DEFAULT_DB_CLUSTER_NAME = "knoe-cnpg-0"` (live: `knoe-dev-cnpg-0`) and `DEFAULT_APP_CLUSTER_MACHINE_TYPE = "e2-small"` (live: `e2-standard-2`). Updated both at [`knoe/core/actions.py`](../knoe/core/actions.py) lines 121 + 123. Verified: `python3 -c "from knoe.core.actions import ..."` reads the new values; pytest collection error count went from 8 → 7 post-edit (one of the pre-existing collection errors apparently went away when the import resolved differently — no regression). Closes drift R1 + R2 in CLAUDE.md (footnotes + drift table rows + the `# STALE` inline comments in the §"Cluster code constants" code block all dropped). Implicitly closes the formerly-#8 queue item ("Update CLAUDE.md cluster table once defaults match") in this same commit. Queue renumbered: #2 (this item) and #8 dropped, gaps closed. + +- 2026-04-30 — **Drop redundant GCP IAM workloadIdentityUser binding** (commit pending). v1.28-era leftover: `[knoe-db-0/knoe-db]` was bound alongside the active `[knoe-db-0/cnpg-backup-sa]` on the `cnpg-backup@…` GCP SA. Pods have used `cnpg-backup-sa` since the 2026-04-29 stabilization (commit [e92cd8b](#)), so the older binding was dead config — kept around because removing it required a reasoned check that no in-flight job still authenticated under the old identity. Verified live (`gcloud iam service-accounts get-iam-policy`), dropped via `gcloud iam service-accounts remove-iam-policy-binding`, re-verified the SA's policy now has only `[knoe-db-0/cnpg-backup-sa]`. CNPG primary still healthy (`SELECT 1` returned cleanly post-removal). Closes drift R7. Removed the matching CLAUDE.md Reality TODO + drift table row. + +- 2026-04-30 — **Grafana at `svc.knoe.dev/grafana` with Google OAuth** (commit pending). Diagnosed the `svc.knoe.dev` "name resolution failed" error: `knoe-svc-kong-config` had its grafana service pointing at `prometheus-grafana.monitoring.svc.cluster.local:80` — the OLD upstream name from before the kube-prometheus-stack chart renamed to `kps-grafana`. Kong was DNS-failing every request to that vanished service and surfacing the Kong-style "name resolution failed" error to clients. Fixed the upstream + scoped the route to `/grafana` (so other surfaces can land at `svc.knoe.dev/*` later). Layered Grafana's native Google OAuth via auth.google config in [`monitoring/kps-values-gke.yaml`](../monitoring/kps-values-gke.yaml) (new — captures previously-uncommitted live Helm values + adds the auth additions). Role mapping via JMESPath: `contains(['chrisfu@knoey.com', 'ron@knoey.com'], email) && 'Admin' || 'Editor'` — anyone @knoey.com gets Editor, chrisfu+ron get Admin. New manifests: [`deploy/gcp/gke/grafana-google-oidc-secret.example.yaml`](../deploy/gcp/gke/grafana-google-oidc-secret.example.yaml), [`etc/init_grafana_oauth.sh`](../etc/init_grafana_oauth.sh), [`monitoring/kps-values-gke.yaml`](../monitoring/kps-values-gke.yaml). Repo Kong source updated at [`deploy/opentofu/k3s/manifests/knoe/kong-configmap.yaml`](../deploy/opentofu/k3s/manifests/knoe/kong-configmap.yaml) (`.org→.dev` rebrand, healthz route restored, `/grafana` scoping). Verified externally: `/grafana/login` → 200; `/grafana/login/google` → 302 `accounts.google.com` with the right client_id; subpath assets load; Studio + SDK paths on db.0.knoe.dev unchanged. Browser walkthrough (chrisfu sign-in → Admin role; ron same; outside-domain rejected by `allowed_domains: knoey.com`) is the manual follow-up. Stop-gap `adminPassword: admin` in values stays (local break-glass) — rotate this when convenient as a follow-up. + +- 2026-04-30 — **Reusable engineer onboarding via one-time link + QR** (commit pending). Replaces the assumed-but-nonexistent shared 1Password vault with a self-contained delivery flow: `./etc/onboard_engineer.sh ` provisions the postgres role + 24h temp password and outputs (1) a single-shot URL, (2) a QR-code rendering for screenshare, (3) the plaintext fallback. Engineer opens URL or scans QR → static page at `db.0.knoe.dev/onboard.html` (vanilla HTML+JS, no backend) reads the URL fragment client-side and shows password + psql connection string + rotation snippet with [Copy] buttons; saves to engineer's **personal** 1Password. Engineer rotates immediately via `\password`. New manifests: [`deploy/gcp/gke/knoe-onboard.yaml`](../deploy/gcp/gke/knoe-onboard.yaml) (nginx:alpine + ConfigMap + Service), Kong route `/onboard.html` → knoe-onboard:80 in [`supabase/helm/knoe-supabase/templates/kong/config.yaml`](../supabase/helm/knoe-supabase/templates/kong/config.yaml), oauth2-proxy `--skip-auth-route=^/onboard\.html$` in [`deploy/gcp/gke/oauth2-proxy-deployment.yaml`](../deploy/gcp/gke/oauth2-proxy-deployment.yaml). Pivotal pg_hba change: rule changed from `hostssl all chrisfu,ron all` to `hostssl all +knoe_developer all` so future onboards don't require pg_hba edits — the script's `GRANT knoe_developer TO ` is enough. CA cert committed at [`etc/knoe-db-ca.crt`](../etc/knoe-db-ca.crt) (public root, not a secret) so engineers reference `sslrootcert=etc/knoe-db-ca.crt` after cloning. Engineer-facing docs: [`docs/onboarding.md`](onboarding.md). [`docs/db-access.md`](db-access.md) updated to remove the broken 1Password reference. Verified end-to-end: `./etc/onboard_engineer.sh test-onboard test-onboard@knoey.com` → URL + QR rendered → `psql ...user=test-onboard...` connects with the temp password and reports `pg_has_role(current_user,'knoe_developer','MEMBER')=t` → `--revoke` drops the role cleanly. + +- 2026-04-30 — **Per-engineer psql access via `pg.0.knoe.dev` (Phase 1)** (commit pending). External GKE TCP LoadBalancer at `34.106.156.196` (forwarding rule `a61028024f2314e06902bafc1b4c3a08`); user adds Namecheap A record `pg.0.knoe.dev → 34.106.156.196` (manual; Namecheap UI). CNPG `serverAltDNSNames` extended with `pg.0.knoe.dev` so the auto-issued server cert validates externally. New `knoe_developer` group role with R/W on `knoe`+`public` and R/O on `auth`+`storage`+`extensions`. Per-engineer roles `chrisfu` and `ron` with strong random passwords (192-bit base64) `VALID UNTIL '2026-05-30'`, members of `knoe_developer`. pg_hba rewritten so external requires TLS+SCRAM and only allows `chrisfu, ron` (other roles `postgres`, `supabase_admin`, `authenticator` stay cluster-internal). `externalTrafficPolicy: Local` on the LB to preserve client source IP (otherwise the SNAT'd node IP made externals look like internal cluster traffic and bypassed the `hostnossl reject` rule). Documentation in [`docs/db-access.md`](db-access.md). Verified: chrisfu+ron both connect via TLS, plaintext rejected with the proper pg_hba error showing the real client IP, `postgres` role connection from outside fails auth, internal supabase services keep their existing TLS connections (gitlab on TLS 1.3, supabase_admin/authenticator on plaintext over RFC1918 — both still allowed by the new rules). Phase 2 (libpq OAUTHBEARER + validator library, bundled into install/deploy.sh + k3d) handed off to Junie — see "In progress" at top of file. + +- 2026-04-30 — **`pg_stat_statements` moved out of `public` schema across deploy modes** (commit pending). Studio's Database Advisor was flagging `public.pg_stat_statements` as a Security warning the moment any user opened the dashboard. Root cause: relocatable extensions without an explicit `SCHEMA` clause on `CREATE EXTENSION` land in the first writable schema in the connecting role's search_path, which is `public` for the `postgres` role. Live: `ALTER EXTENSION pg_stat_statements SET SCHEMA extensions;`. Bootstrap (3 files, all 4 deploy modes): [`deploy/gcp/gke/knoe-db.yaml`](../deploy/gcp/gke/knoe-db.yaml) (gke), [`deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml`](../deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml) (k3s), [`k8s/knoe/knoe-db.yaml`](../k8s/knoe/knoe-db.yaml) (k3d / min) — each `postInitTemplateSQL` block now creates the `extensions` schema and `CREATE EXTENSION ... SCHEMA extensions;`. Plus [`supabase/deploy.sh`](../supabase/deploy.sh) gets `GRANT USAGE ON SCHEMA extensions TO anon, authenticated, service_role;` so PostgREST clients (and the advisor itself) can resolve the schema. Other security checks verified clean (no tables in public without RLS, no SECURITY DEFINER without search_path, no auth.users-exposed views, no duplicate indexes, no extensions left in public). Performance side has 6 unindexed FKs in upstream `auth.*` and `storage.*` tables — captured as deferred (touching upstream-managed schemas is the wrong move; they'll get fixed in supabase-auth / supabase-storage releases). + +- 2026-04-30 — **Studio post-launch fixes (storage permissions + /support stop-gap)** in commit [79a3f76](#): + - **Storage list-buckets unblocked.** `supabase_storage_admin` role was missing membership in `service_role` / `authenticated` / `anon`, so `SET LOCAL role = 'service_role'` (run by supabase-storage on every request) failed with PG `42501` at `guc.c:6936 call_string_check_hook`, surfaced as a misleading "new row violates row-level security policy" error. One GRANT applied live (`GRANT anon, authenticated, service_role TO supabase_storage_admin;`) and added durably to [`supabase/deploy.sh`](../supabase/deploy.sh) right after the existing `GRANT service_role TO authenticator` block. + - **oauth2-proxy auth-header passthrough fixed.** The `--pass-authorization-header=true` and `--set-authorization-header=true` flags were stomping the `Authorization: Bearer ` header that Studio / supabase-py send on `/storage/v1/...` calls (oauth2-proxy was rewriting it with the Google OIDC id_token, which storage couldn't validate against `JWT_SECRET`). Both flags removed from [`deploy/gcp/gke/oauth2-proxy-deployment.yaml`](../deploy/gcp/gke/oauth2-proxy-deployment.yaml). When per-user identity passthrough lands post-knoe-auth Round 1, use `--pass-user-headers` / `--set-xauthrequest` (X-Forwarded-* headers) instead — those don't collide with Supabase's auth. + - **`/support` route stop-gap.** `https://db.0.knoe.dev/support` 302s to `mailto:support@knoe.dev` so users have a working escape hatch while the in-Studio support buttons stay broken (separate Studio-fork follow-up #15). Kong service `support` added to [`templates/kong/config.yaml`](../supabase/helm/knoe-supabase/templates/kong/config.yaml) using `request-termination` (302) + `response-transformer` (Location header). `KONG_PLUGINS` allowlist extended with `response-transformer` in [`values.yaml`](../supabase/helm/knoe-supabase/values.yaml). oauth2-proxy gets `--skip-auth-route=^/support$` so an unauthenticated user looking for help isn't bounced into a Google sign-in detour. + +- 2026-04-30 — **Unified `db.0.knoe.dev` for Studio + SDK** in commit [f805404](#): added two `--skip-auth-route` args to oauth2-proxy so SDK requests on `/auth/v1`, `/rest/v1`, `/realtime/v1`, `/storage/v1`, `/functions/v1`, `/graphql/v1`, `/pg/*` bypass the Google sign-in flow and pass through to supabase-kong, where Kong's existing `key-auth` plugin enforces anon/service-role keys. Studio UI at `/` remains Google-gated. One URL now covers humans-in-Studio AND robots-via-supabase-py — same security envelope as `api.0.knoe.dev` had on those paths. `api.0.knoe.dev` kept as a backward-compat alias (see follow-up #14). External verification: `curl -H "apikey: " https://db.0.knoe.dev/rest/v1/` → 200 OpenAPI; `curl https://db.0.knoe.dev/rest/v1/` (no key) → 401 from Kong; `curl https://db.0.knoe.dev/` → 302 to Google (Studio gate intact); `curl https://db.0.knoe.dev/api/profile` → 302 to Google (Studio internals stay gated). + +- 2026-04-30 — **Google sign-in to db.0.knoe.dev for chrisfu + ron (shared Studio)** in commit [25f1b2e](#): oauth2-proxy v7.15.2 deployed in front of supabase-kong as the Studio access gate (Google Workspace OIDC, `--email-domain=knoey.com`); supabase-kong Ingress patched so `db.0.knoe.dev` → `oauth2-proxy:80` while `api.0.knoe.dev` → `supabase-kong:8000` is untouched (key-auth still gates the API surface); Kong `dashboard` route's `basic-auth` plugin removed (oauth2-proxy is the gate now — double-auth was friction without security gain). External verification: db.0.knoe.dev/ → 302 `accounts.google.com/o/oauth2/auth?client_id=200699021716-…&redirect_uri=https://db.0.knoe.dev/oauth2/callback&scope=openid+email+profile`; api.0.knoe.dev/rest/v1/ → 401 from Kong key-auth (untouched); /healthz → 200. Three new files + chart edit: [`deploy/gcp/gke/oauth2-proxy-deployment.yaml`](../deploy/gcp/gke/oauth2-proxy-deployment.yaml), [`deploy/gcp/gke/oauth2-proxy-google-oidc-secret.example.yaml`](../deploy/gcp/gke/oauth2-proxy-google-oidc-secret.example.yaml), [`etc/init_oauth2_proxy.sh`](../etc/init_oauth2_proxy.sh), and the `basic-auth` plugin removed from [`supabase/helm/knoe-supabase/templates/kong/config.yaml`](../supabase/helm/knoe-supabase/templates/kong/config.yaml). Future swap to knoe-auth Round 1: change one arg in oauth2-proxy from `--provider=google` to `--provider=oidc --oidc-issuer-url=https://api.knoe.dev/auth` and reapply. + +- 2026-04-29 — **CNPG operator + backup stabilization session** (4 queue items in one wave; live-cluster ops, no commit yet — doc-update commit pending): + - **#4 done:** plugin-barman-cloud operator restarted (`kubectl rollout restart deployment/barman-cloud -n cnpg-system`); stale Garage retention-worker endpoint cache cleared + - **#11 done:** cnpg-controller-manager restarted; clean plugin re-discovery confirmed in logs + - **CNPG operator upgraded v1.28.1 → v1.29.0** to expose `spec.serviceAccountName` field (added in v1.29.0 per upstream release notes); operator rollout clean, cluster healthy throughout + - **#2 done:** `cluster.spec.serviceAccountName=cnpg-backup-sa` applied; supervised rolling restart cycled all 3 pods (db-3 first failed with RBAC error — patched RoleBindings `knoe-db` and `knoe-db-barman-cloud` to add `cnpg-backup-sa` as a subject; db-3 then recovered, db-4 and db-2 rolled cleanly); 23:03 UTC scheduled backup completed with new SA, verifying end-to-end + - **#3 done:** Garage workload removed from `knoe-dev-cnpg-0` (statefulset, both services including `garage-s3-ilb`, configmap, PVC `data-garage-0`, StorageClass `garage-hdd`); ObjectStore untouched, backups still progressing post-removal +- 2026-04-29 — **Doc true-up + master TODO index** in commit [f5b24c7](#). Reality-vs-intent gaps captured across CLAUDE.md / AGENTS.md / pipeline-phases / plans / canonical ObjectStore manifest; new `docs/TODO.md` Kanban-style index. +- 2026-04-29 — **GCS migration completed.** ObjectStore `gs://knoe-0-backups/` with `googleCredentials.gkeEnvironment: true`; WI auth via `knoe-db` SA. 2 verified backups in catalog. ScheduledBackup unsuspended. (live-cluster ops; no commit) +- 2026-04-29 — **Garage cluster recovered.** PVC 29 → 80 GiB, layout 10 → 60 GB, bucket cleared (8382 objects + 12 orphan multiparts). (live-cluster ops; no commit) +- 2026-04-28/29 — **knoe-db-1 fix.** 14d crashloop resolved via PVC delete + CNPG re-bootstrap; cluster failover-cycled all 3 pods; 3/3 healthy. (live-cluster ops; no commit) +- 2026-04-28 — **Phase A repo cleanup + Phase C Maven build readiness** in commit [b355855](#). tmp/ archived (87 files); `org.knoe` → `dev.knoe` rename; `authority` → `auth` artifactId; sources moved to `dev/knoe/auth/`; `make build-auth` green. 4 k3s `prole-auth-*.yaml` manifests renamed to `knoe-auth-*.yaml`. +- 2026-04-28 — **Pipeline Phase 0** (test pipeline foundation: pyproject.toml, IntelliJ run configs, coverage fix, welcome mode selector). See [`docs/pipeline-phases.md`](pipeline-phases.md) §Phase 0. diff --git a/docs/plans/README.md b/docs/plans/README.md index 2e64976..5a43e19 100644 --- a/docs/plans/README.md +++ b/docs/plans/README.md @@ -19,6 +19,9 @@ If a sentence in a plan assumes knowledge you don't have, that's a bug in the pl | --- | --- | --- | | [`knoe-auth-round-1.md`](knoe-auth-round-1.md) | Identity backbone for the platform: MIT Kerberos KDC + invite-anchored web enrollment + TOTP 2FA. Round 1 of N. **Shipped.** | Before touching `authority/`, `etc/init_kdc.sh`, `etc/init_knoe_users.sh`, anything in `deploy/gcp/gke/knoe-auth-*` or `knoe-kdc-*`, or the `knoe.*` database schema. | | [`deployment-modes.md`](deployment-modes.md) | Four-mode installer (`min` / `k3d` / `k3s` / `gke`) with a welcome-screen mode selector and a `min`-mode fast-path through the wizard. **Shipped in Phase 0.** | Before touching `knoe/ui/screens/welcome.py`, `knoe/ui/screens/navigation.py`, or adding any new wizard screen. | +| [`customer-deploy-resync.md`](customer-deploy-resync.md) | Original plan to converge `~/dev/prole` onto `knoe-db/main` as a customer-deploy branch. **Dormant** — prole rebrand merged into `main`; no separate prole working tree currently under development. | Only if you're considering activating a per-customer branching workflow. | +| [`junie/`](junie/README.md) (subdirectory) | Self-contained, single-task **work briefs** for Junie to consume from inside the IDE. Tactical, single-MR scope, paired against numbered items in `../TODO.md`. | When you want to hand a discrete task to Junie or audit what's been queued. | +| [`../TODO.md`](../TODO.md) (sibling) | Master TODO index — single source of truth for unfinished work, including reality-vs-intent gaps flagged in `CLAUDE.md`/`AGENTS.md`. | Before picking up any task, to see what's already on the queue. | Each plan follows the same shape: **Context → How it's wired (file references) → Architecture → Schema / API → Step-by-step → Verification → Glossary.** diff --git a/docs/plans/junie/02-k3s-prole-rename.md b/docs/plans/junie/02-k3s-prole-rename.md new file mode 100644 index 0000000..b719512 --- /dev/null +++ b/docs/plans/junie/02-k3s-prole-rename.md @@ -0,0 +1,128 @@ +# Junie brief — Queue #2: k3s manifest cleanup, `prole-*` filename rename + +> **Self-contained brief.** No prior session context required. Mechanical +> rename + a few content sweeps. Should land as a single commit. + +--- + +## 1. Why + +`deploy/opentofu/k3s/manifests/knoe/kustomization.yaml` resolves resources by +filename. After the prole→knoe rebrand, the kustomization was updated to +reference `knoe-{configmap,deployment,service,kdc-configmap}.yaml` — but the +files on disk still carry the `prole-` prefix. **Kustomize is broken on the +k3s deploy mode right now**: `kubectl apply -k deploy/opentofu/k3s/manifests/knoe` +errors with "no such file" on the missing `knoe-*.yaml` resources. + +This is the same `git mv` pattern that landed the auth-side files in commit +[`b355855`](#) (the `prole-auth-*.yaml` → `knoe-auth-*.yaml` rename). Mirror +that approach. + +## 2. The mapping + +Files currently in `deploy/opentofu/k3s/manifests/knoe/`: + +| Current filename | Rename to | Resource inside (verify) | +|---|---|---| +| `prole-configmap.yaml` | `knoe-configmap.yaml` | ConfigMap `knoe-nginx-config` | +| `prole-deployment.yaml` | `knoe-deployment.yaml` | Deployment `knoe` | +| `prole-service.yaml` | `knoe-service.yaml` | Service `knoe` | +| `prole-kdc-configmap.yaml` | `knoe-kdc-configmap.yaml` | ConfigMap `knoe-kdc-config` | +| `prole-kdc-secrets.example.yaml` | `knoe-kdc-secrets.example.yaml` | Secret template (not in kustomization — example only) | + +Use `git mv` for all five so history is preserved. + +## 3. Adjacent content sweeps to do in the same commit + +Don't ship just the rename. While each file is open, scan for stale +`prole.local` / `PROLE.LOCAL` / `prole-*` references in the **content**: + +- **`prole-kdc-configmap.yaml` (post-rename: `knoe-kdc-configmap.yaml`)** — + the embedded `krb5.conf` currently has `default_realm = PROLE.LOCAL`. The + Kerberos realm in `authority/` and the rest of the platform is + `KNOE.LOCAL` (or the production realm; verify by grepping + `default_realm` across the repo and matching what the running KDC pod + uses). Update accordingly. If you find a value other than `KNOE.LOCAL` is + in production use, ASK before changing — don't guess. + +- **Any `host: prole.local` / `prole.org` ingress hostnames** in the renamed + files, or in `ingress.yaml` (which is already the right name). Cross-check + against the GKE manifests in `deploy/gcp/gke/` for the canonical + hostnames. If a k3s-only hostname is intentional (different deploy mode), + leave it but add a comment noting why. + +- **`metadata.labels.app: knoe`** is already correct in the deployment + + service. Verify no `app: prole` label survives. + +Anything that isn't a clear "this is stale, no question" — leave a TODO +comment with the specific question and link to this brief, rather than +making the call yourself. + +## 4. Verification + +1. **Kustomize dry-run resolves all resources cleanly:** + ```bash + kubectl kustomize deploy/opentofu/k3s/manifests/knoe/ > /tmp/k3s-rendered.yaml + echo "exit: $?" + wc -l /tmp/k3s-rendered.yaml + ``` + No "file not found" errors. Output rendered manifest has all expected + resources (knoe Deployment, knoe Service, knoe-nginx-config CM, knoe-kdc-config CM, etc.). + +2. **No stragglers:** + ```bash + grep -rni 'prole' deploy/opentofu/k3s/manifests/knoe/ + ``` + Expected output: nothing, or only intentional historical comments + (annotated with the reason). + +3. **`git log --follow` works for one of the renamed files:** + ```bash + git log --follow --oneline deploy/opentofu/k3s/manifests/knoe/knoe-configmap.yaml | head -5 + ``` + Should show pre-rename history (proof `git mv` preserved it). + +4. **Smoke test in k3d if available** (optional but nice): `make k3d-up` + followed by `kubectl apply -k deploy/opentofu/k3s/manifests/knoe/` — pods + should start. If k3d isn't available locally, skip; CI will catch it. + +## 5. Commit message + +``` +chore(k3s): rename prole-*.yaml manifests to knoe-* and align contents + +Mirrors commit b355855 (auth-side rename). The kustomization at +deploy/opentofu/k3s/manifests/knoe/kustomization.yaml has been +referencing knoe-* names since the rebrand; kustomize was broken +on the k3s deploy mode until this lands. + +Five git mv: + prole-configmap.yaml -> knoe-configmap.yaml + prole-deployment.yaml -> knoe-deployment.yaml + prole-service.yaml -> knoe-service.yaml + prole-kdc-configmap.yaml -> knoe-kdc-configmap.yaml + prole-kdc-secrets.example.yaml-> knoe-kdc-secrets.example.yaml + +Plus content sweeps for stale prole.local / PROLE.LOCAL realm +references in the embedded krb5.conf. + +Closes queue item #2 in docs/TODO.md (drift R5). +``` + +## 6. Out of scope + +- Image refs `knoe-authority` → `knoe-auth` in deployment manifests — that's + queue item **#3**, separate brief, separate commit. (Touches both + `deploy/gcp/gke/knoe-auth-deployment.yaml` and the renamed k3s + `knoe-auth-deployment.yaml`.) +- KDC realm value changes if `default_realm` is currently other than + `PROLE.LOCAL` in some files — ask before rewriting. + +## 7. Definition of done + +- [ ] Five `git mv` operations recorded in the commit (preserves history). +- [ ] `kubectl kustomize` produces a clean rendered manifest with no errors. +- [ ] `grep -rni 'prole' deploy/opentofu/k3s/manifests/knoe/` returns clean. +- [ ] Single commit; message starts with `chore(k3s):`. +- [ ] `docs/TODO.md` queue item #2 moved to **Done** section with date and + commit ref (or, if you prefer, leave that to whoever merges). diff --git a/docs/plans/junie/06-patch-garage-script-fixes.md b/docs/plans/junie/06-patch-garage-script-fixes.md new file mode 100644 index 0000000..fbe8f16 --- /dev/null +++ b/docs/plans/junie/06-patch-garage-script-fixes.md @@ -0,0 +1,191 @@ +# Junie brief — Queue #6: fix `scripts/patch_garage_cross_cluster.sh` + +> **Self-contained brief.** Three concrete defects in the script + a related +> manifest split. Land as one or two commits (your call — see §5). + +--- + +## 1. Why + +`scripts/patch_garage_cross_cluster.sh` was written one-shot during the +2026-04-29 cluster recovery to migrate CNPG backups off Garage and onto +GCS+Workload-Identity. It worked at the time, but reading it back surfaced +three defects that would bite the next person who runs it. We want it to +be re-runnable cleanly (e.g. for a future second customer or DR rebuild) +without surprises. + +Live-cluster context (do not change): GCS migration is complete, ObjectStore +is active, ScheduledBackup is plugin-method, three CNPG pods on +`knoe-dev-cnpg-0` are healthy. **You don't need to run the script to fix +it.** This is a code-only commit. + +## 2. The three defects + +### Defect A — stale `DB_CLUSTER` default + +`scripts/patch_garage_cross_cluster.sh:39` + +```bash +DB_CLUSTER="${DB_CLUSTER:-knoe-cnpg-0}" +``` + +Should be `knoe-dev-cnpg-0` (matches `conf/gke.cfg` line `init_cluster.db_cluster_kubecontext` +and `CLAUDE.md` cluster table). The current default resolves to the wrong +context and `kubectl` operations would silently target the wrong cluster +(or fail with "context not found", depending on the environment). + +The script's `APP_CLUSTER="${APP_CLUSTER:-knoe-dev-0}"` on line 40 is +already correct. + +Header comment on lines 16-18 also references the wrong name in the +`./etc/init_cnpg_gke.sh ... --cluster knoe-cnpg-0` example — fix that too. + +### Defect B — Phase 1 misses `service/garage-s3-ilb` + +`scripts/patch_garage_cross_cluster.sh:86-95` — Phase 1's deletion loop: + +```bash +for resource in "statefulset/garage" "service/garage" "configmap/garage-config"; do + ... +done +``` + +The loop deletes `service/garage` (the ClusterIP) but not +`service/garage-s3-ilb` (the LoadBalancer). The ObjectStore's +`endpointURL` historically pointed at the ILB, so leaving the LB orphaned +costs money and confuses cluster state. The 2026-04-29 cleanup did this +manually after the script ran. + +**Fix**: add `"service/garage-s3-ilb"` to the loop's resource list. Order +doesn't matter (all `--ignore-not-found=true`). + +### Defect C — Phase 2 applies a manifest containing both SA AND a legacy ScheduledBackup + +`scripts/patch_garage_cross_cluster.sh:114-122` — Phase 2 applies +`deploy/gcp/gke/knoe-db-backup-gcs.yaml`. That manifest currently contains: + +1. **ServiceAccount `cnpg-backup-sa`** — wanted; this is the WI-bound SA. +2. **ScheduledBackup `knoe-db-daily`** with `method: barmanObjectStore` — + **unwanted**. Two problems: + - Live cluster already has a plugin-method ScheduledBackup. Applying + this would create a duplicate that runs alongside the active one. + - `method: barmanObjectStore` (the legacy in-tree method) is being + removed in CNPG v1.30 per upstream release notes. New work should not + pin to it. + +**Fix** (one of two paths — pick the cleanest): + +**Path 1 (recommended): split the manifest.** Move the ScheduledBackup +out of `deploy/gcp/gke/knoe-db-backup-gcs.yaml` into its own file (or +delete it outright if there's no consumer — verify by `grep -r "knoe-db-daily"` +across the repo first; if nothing references it, just remove it). Keep +`knoe-db-backup-gcs.yaml` as the canonical SA-only manifest. The script's +Phase 2 stays unchanged (still applies the SA file). + +**Path 2 (minimal): adjust the script only.** Use a kubectl filter so +Phase 2 only applies the ServiceAccount portion of the file: + +```bash +GCP_PROJECT_ID="${GCP_PROJECT}" envsubst '${GCP_PROJECT_ID}' < "${GCS_MANIFEST}" \ + | yq 'select(.kind == "ServiceAccount")' \ + | kubectl --context="${DB_CTX}" apply -f - +``` + +(`yq` is already used elsewhere in `etc/`. If it's not in the script's +preflight `command -v` check, add it.) + +**Path 1 is recommended** because it makes the manifest file's contents +match its filename's intent (`-gcs.yaml` should be GCS plumbing, not +backup scheduling), and removes the deprecated-method footgun for any +future runner. + +If you take Path 1, also verify the live ScheduledBackup in +`knoe-db-0` namespace still exists and uses the plugin method — but **do +not run kubectl against the live cluster from the script change.** Just +inspect via `kubectl --context=$DB get scheduledbackup -n knoe-db-0 -o yaml` +once if you want to confirm; don't modify it. + +### Defect D (bonus, mentioned in TODO) — Phase 3 unconditional re-apply + +Phase 3 (lines 127-134) unconditionally re-applies +`k8s/knoe/knoe-db-barman-objectstore-gcs.yaml`. Manifest currently matches +live state, so this is harmless. The TODO suggests "worth a guard" — add +a `kubectl get objectstore knoe-db-barman-objectstore -n $DB_NS` check +that diffs against the manifest and skips re-apply if matching. **Skip +this one if it adds significant scope** — it's a nice-to-have, not a +defect. + +## 3. What to verify + +```bash +# 1) Defaults resolve correctly when no env vars set +bash -n scripts/patch_garage_cross_cluster.sh # syntax check +( unset DB_CLUSTER APP_CLUSTER GCP_PROJECT GCP_REGION SERVICE_NS DB_NS + source <(sed -n '36,50p' scripts/patch_garage_cross_cluster.sh) + echo "DB_CLUSTER=$DB_CLUSTER, DB_CTX=$DB_CTX" ) +# Expected: DB_CLUSTER=knoe-dev-cnpg-0, DB_CTX=gke_..._knoe-dev-cnpg-0 + +# 2) Dry-run the script in CONFIRM=false mode (existing safety: prints the +# plan and exits 1) — verify the printed plan reads sensibly +./scripts/patch_garage_cross_cluster.sh +# Expected: prints "1. Remove Garage from knoe-dev-cnpg-0/knoe-system" etc. + +# 3) Phase 1 deletion list now includes garage-s3-ilb +grep -A8 'for resource in' scripts/patch_garage_cross_cluster.sh +# Expected: list contains "service/garage-s3-ilb" + +# 4) (If Path 1 chosen) the SA-only manifest has only ServiceAccount kind +grep '^kind:' deploy/gcp/gke/knoe-db-backup-gcs.yaml +# Expected: only "kind: ServiceAccount" +``` + +## 4. Out of scope + +- Don't run the live script. This is a code-only fix. The script is for + someone migrating a fresh cluster; we're not migrating. +- Don't change any RBAC or IAM. The Workload Identity binding is correct + in the live cluster; this brief touches the script and (optionally) the + manifest split, nothing more. +- The CNPG SA wiring inside `etc/init_cnpg_gke.sh` (`cluster.spec.serviceAccountName` + + extended RoleBindings) is queue item **#7** with its own brief. Don't + conflate. + +## 5. Commit shape + +If you take Path 1 (manifest split): two commits, atomic each: +1. `chore(deploy): split knoe-db-backup-gcs manifest — SA only, drop legacy ScheduledBackup` +2. `fix(scripts): patch_garage_cross_cluster defaults + missing garage-s3-ilb` + +If you take Path 2 (script-only): one commit. + +``` +fix(scripts): patch_garage_cross_cluster — three defects from 2026-04-29 review + +Defects A, B, C from docs/plans/junie/06-patch-garage-script-fixes.md: + + A. DB_CLUSTER default was knoe-cnpg-0 (stale rebrand artifact); + should be knoe-dev-cnpg-0 to match conf/gke.cfg. + + B. Phase 1 deletion loop missed service/garage-s3-ilb (the + LoadBalancer the ObjectStore endpoint historically pointed at). + Now included. + + C. Phase 2 was applying knoe-db-backup-gcs.yaml whole, including a + legacy ScheduledBackup using method:barmanObjectStore (being + removed in CNPG v1.30). Manifest split: SA stays in + knoe-db-backup-gcs.yaml; ScheduledBackup removed. + +Header comment + usage example also updated for the rename. +Closes queue item #6 in docs/TODO.md (drift R9). +``` + +## 6. Definition of done + +- [ ] `DB_CLUSTER` default + header comment updated. +- [ ] Phase 1 loop includes `service/garage-s3-ilb`. +- [ ] Phase 2 doesn't apply a legacy ScheduledBackup (Path 1 or 2). +- [ ] `bash -n` clean; CONFIRM=false dry-run prints sensible plan. +- [ ] If Path 1: `kind: ServiceAccount` is the only kind in + `deploy/gcp/gke/knoe-db-backup-gcs.yaml`. +- [ ] `docs/TODO.md` queue item #6 + drift R9 row archived to **Done** with + date and commit ref. diff --git a/docs/plans/junie/07-init-cnpg-gke-sa-wiring.md b/docs/plans/junie/07-init-cnpg-gke-sa-wiring.md new file mode 100644 index 0000000..fce8c20 --- /dev/null +++ b/docs/plans/junie/07-init-cnpg-gke-sa-wiring.md @@ -0,0 +1,215 @@ +# Junie brief — Queue #7: wire `cnpg-backup-sa` into the CNPG cluster spec via `etc/init_cnpg_gke.sh` + +> **Self-contained brief.** Drift R8: live cluster state has the SA wired +> through, the init script doesn't. Make the script land a fresh deploy in +> the same end state. Live cluster is healthy — do **not** run anything +> against it. + +--- + +## 1. Why + +`etc/init_cnpg_gke.sh` runs from a clean state (e.g. for a future second +DB cluster, or DR rebuild) and lays down everything: GCS buckets, GCP +service account, Workload Identity binding, K8s service account, namespace, +CNPG operator, secrets, the CNPG `Cluster` resource. The 2026-04-29 +stabilization session added two pieces that the script doesn't yet +reproduce: + +1. **`cluster.spec.serviceAccountName: cnpg-backup-sa`** on the live + cluster. This field was added in CNPG **v1.29.0**. The init script + currently pins CNPG **v1.24.0** (line 402: + `cnpg-1.24.0.yaml`), which doesn't support the field — the cluster + would either ignore it or fail validation. Bump the operator version + AND add the field to the rendered manifest. + +2. **RoleBindings extended** with `cnpg-backup-sa` as a subject. Live + `knoe-db` and `knoe-db-barman-cloud` RoleBindings (auto-created by + CNPG at cluster creation) each have **two** subjects: + `knoe-db` (the default SA CNPG creates) and `cnpg-backup-sa`. With v1.29+ + and `spec.serviceAccountName` set from the start, CNPG should + auto-create RoleBindings naming the explicit SA — but verify, because + the live state was patched after the fact. + +The result of this brief: a fresh `init_cnpg_gke.sh` run produces a +cluster whose RBAC and SA wiring matches `kubectl get cluster knoe-db -o yaml` ++ `kubectl get rolebinding -n knoe-db-0` from the live system, with no +post-install patches required. + +## 2. Live state to match + +```bash +DB=gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0 +kubectl --context=$DB -n knoe-db-0 get cluster knoe-db -o jsonpath='{.spec.serviceAccountName}' +# -> cnpg-backup-sa + +kubectl --context=$DB -n knoe-db-0 get rolebinding knoe-db -o jsonpath='{.subjects}' | jq +# -> [{kind:ServiceAccount, name:knoe-db, namespace:knoe-db-0}, +# {kind:ServiceAccount, name:cnpg-backup-sa, namespace:knoe-db-0}] + +kubectl --context=$DB -n knoe-db-0 get rolebinding knoe-db-barman-cloud -o jsonpath='{.subjects}' | jq +# -> same two subjects +``` + +## 3. What changes + +### 3.1 — `etc/init_cnpg_gke.sh` + +Around line 402 (CNPG operator install): + +```bash +# Current (stale) +kubectl apply --server-side -f \ + https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/releases/cnpg-1.24.0.yaml +``` + +Bump to **v1.29.0** (or a newer stable; verify against +[CNPG releases](https://github.com/cloudnative-pg/cloudnative-pg/releases) +that the version supports `Cluster.spec.serviceAccountName`). Add a code +comment explaining the version constraint: + +```bash +# v1.29.0+ is required: spec.serviceAccountName lets the cluster pods run +# as cnpg-backup-sa (annotated with iam.gke.io/gcp-service-account for WI), +# which is how the GCS-backed barman ObjectStore authenticates without a +# static key. Older operators (we used 1.24.0 historically) would silently +# drop the field. +CNPG_OPERATOR_VERSION="1.29.0" +kubectl apply --server-side -f \ + "https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/releases/cnpg-${CNPG_OPERATOR_VERSION}.yaml" +``` + +(Promote the version to a top-of-file variable so future bumps are one +line.) + +### 3.2 — `deploy/gcp/gke/knoe-db.yaml` (CNPG `Cluster` manifest) + +Add `serviceAccountName: cnpg-backup-sa` under `spec:`. Use the same +indentation as the surrounding fields. Place near the top of `spec` (right +after `description` / `imageName` / `instances`, before `bootstrap` — +follow the live cluster's field order if you want to be tidy: +`kubectl get cluster knoe-db -o yaml`). + +Add an inline comment: + +```yaml + # Run cluster pods as cnpg-backup-sa (annotated for Workload Identity to + # the cnpg-backup@... GCP SA). This is how barman-cloud authenticates + # to gs://knoe-0-backups/ without a static key. Requires CNPG v1.29+. + # The SA is provisioned by etc/init_cnpg_gke.sh § "Apply ServiceAccount + # + annotate with WI". + serviceAccountName: cnpg-backup-sa +``` + +### 3.3 — RoleBinding subjects (decide path) + +Two paths, depending on what CNPG v1.29 actually does at cluster +creation time. Test both before deciding. + +**Path A — operator does the right thing automatically.** With +`spec.serviceAccountName: cnpg-backup-sa` set from the start, CNPG v1.29+ +auto-creates `knoe-db` and `knoe-db-barman-cloud` RoleBindings with +`cnpg-backup-sa` as the (sole) subject. If true: nothing more to do in the +script — the live two-subject state is just a historical artifact of +having patched after-the-fact, and a fresh install would have a single +subject. + +**Path B — operator still creates with a default SA name.** If CNPG +auto-creates the RoleBindings with subject `knoe-db` (the default SA name +matching the cluster name) regardless of the explicit +`serviceAccountName`, then the script needs to patch them. Add a section +after cluster apply: + +```bash +# ── 11. Ensure RoleBindings reference cnpg-backup-sa ───────────────────────── +log "Ensuring CNPG-managed RoleBindings include cnpg-backup-sa ..." +for rb in "${CNPG_CLUSTER_NAME}" "${CNPG_CLUSTER_NAME}-barman-cloud"; do + # Wait briefly for CNPG to create them on first install + for _ in 1 2 3 4 5; do + kubectl -n "$CNPG_NAMESPACE" get rolebinding "$rb" >/dev/null 2>&1 && break + sleep 2 + done + if kubectl -n "$CNPG_NAMESPACE" get rolebinding "$rb" >/dev/null 2>&1; then + if ! kubectl -n "$CNPG_NAMESPACE" get rolebinding "$rb" -o jsonpath='{.subjects[*].name}' \ + | grep -qw cnpg-backup-sa; then + kubectl -n "$CNPG_NAMESPACE" patch rolebinding "$rb" --type='json' -p="$(cat <<'EOF' +[{"op":"add","path":"/subjects/-","value":{"kind":"ServiceAccount","name":"cnpg-backup-sa","namespace":"NAMESPACE_PLACEHOLDER"}}] +EOF +)" --patch-file=/dev/stdin 2>/dev/null || true + fi + fi +done +``` + +(Sketch — finalize the syntax. The `NAMESPACE_PLACEHOLDER` would be +substituted; `--patch-file` doesn't accept `-` in all kubectl versions, so +you may need a temp file.) + +**How to decide A vs B**: spin up a throwaway k3d cluster (or a CNPG-only +test namespace on `knoe-dev-cnpg-0` if you can without disturbing +`knoe-db-0`), apply a minimal `Cluster` resource with +`serviceAccountName: cnpg-backup-sa`, observe the RoleBinding subjects +that CNPG auto-creates. If the SA you specified is the only subject: +Path A. Otherwise: Path B. + +If you can't easily test, lean toward Path B (the patch is idempotent and +costs nothing if the operator already did the right thing). + +## 4. Verification + +Without running against the live cluster: + +1. **Manifest renders cleanly:** + ```bash + ARTIFACT_REGISTRY=us-west3-docker.pkg.dev/plenary-truck-485623-p7/knoe-system \ + KNOE_DB_IMAGE_TAG=18-055 \ + envsubst '${ARTIFACT_REGISTRY} ${KNOE_DB_IMAGE_TAG}' < deploy/gcp/gke/knoe-db.yaml \ + | yq '.spec.serviceAccountName' + # Expected: cnpg-backup-sa + ``` + +2. **Operator version is 1.29+:** + ```bash + grep -E 'cnpg-[0-9]' etc/init_cnpg_gke.sh + # Expected: cnpg-1.29.0.yaml (or newer) + ``` + +3. **`bash -n etc/init_cnpg_gke.sh`** clean. + +4. **(Optional) Dry-run apply against k3d** if you have one running. The + cluster won't reach Healthy in k3d (no Workload Identity), but + `kubectl get cluster -o yaml` should show `spec.serviceAccountName: + cnpg-backup-sa` set. + +## 5. Out of scope + +- Don't touch the live cluster. This is purely script + manifest work. +- Don't change the `cnpg-backup-sa` annotation pattern in + `deploy/gcp/gke/knoe-db-backup-gcs.yaml` — that's how the SA gets the + WI annotation, separate concern. +- The `patch_garage_cross_cluster.sh` defects are queue item **#6** with + its own brief. If your changes here surface a discrepancy with that + brief, flag it but don't try to fix both in one commit. +- The PodMonitor manual-management migration (queue #13) doesn't interact + with this brief. + +## 6. Commit shape + +Likely two commits: +1. `chore(deploy): add serviceAccountName: cnpg-backup-sa to knoe-db cluster spec` +2. `fix(init): bump CNPG operator to v1.29 + ensure RoleBindings include cnpg-backup-sa` + +Or one combined commit titled +`fix(init): wire cnpg-backup-sa as cluster SA + bump CNPG to v1.29 (drift R8)`. + +## 7. Definition of done + +- [ ] `etc/init_cnpg_gke.sh` references CNPG operator v1.29.0+ with a + comment explaining why (the SA field requirement). +- [ ] `deploy/gcp/gke/knoe-db.yaml` carries + `spec.serviceAccountName: cnpg-backup-sa` with explanatory comment. +- [ ] RoleBinding handling: either documented as auto-correct (Path A, + with a one-liner test result included in the commit message), OR + scripted patch (Path B). +- [ ] `bash -n` passes; manifest renders. +- [ ] `docs/TODO.md` queue item #7 + drift R8 row archived to **Done**. diff --git a/docs/plans/junie/13-podmonitor-manual-management.md b/docs/plans/junie/13-podmonitor-manual-management.md new file mode 100644 index 0000000..ee5b07b --- /dev/null +++ b/docs/plans/junie/13-podmonitor-manual-management.md @@ -0,0 +1,277 @@ +# Junie brief — Queue #13: migrate off CNPG-deprecated `enablePodMonitor` + `podMonitorRelabelings` + +> **Self-contained brief.** Manage the PodMonitor as a sibling resource +> instead of through CNPG operator fields. Closes a deprecation warning +> we get on every `kubectl patch` of the cluster spec. + +--- + +## 1. Why + +CNPG operator (running v1.29.0 on `knoe-dev-cnpg-0`) emits these warnings +on every patch of `Cluster.spec.monitoring`: + +``` +Warning: spec.monitoring.enablePodMonitor is deprecated and will be removed + in a future release. Please migrate to manually managing your PodMonitor + resources. Set this field to false and create a PodMonitor resource for + your cluster as described in the documentation. +Warning: spec.monitoring.podMonitorRelabelings is deprecated and will be + removed in a future release. Please migrate to manually managing your + PodMonitor resources with custom relabeling configurations. +``` + +Migration is straightforward: write the PodMonitor as a sibling YAML doc, +apply it alongside the Cluster, set `enablePodMonitor: false` (or remove +the whole `monitoring` block once that's also been deprecated). No +functional change — just lifts the relabel config out of the CNPG-managed +field and into a standard prometheus-operator resource. + +The cnpg-grafana dashboard relies on the `cluster` label; **breaking the +relabel breaks the dashboard**, so verify carefully. + +## 2. Live state to match + +```bash +DB=gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0 +kubectl --context=$DB -n knoe-db-0 get podmonitor knoe-db -o yaml +``` + +The CNPG operator currently auto-creates this PodMonitor. Relevant fields: + +```yaml +spec: + namespaceSelector: {} + podMetricsEndpoints: + - port: metrics + relabelings: + - action: replace + sourceLabels: [__meta_kubernetes_pod_label_cnpg_io_cluster] + targetLabel: cluster + selector: + matchLabels: + cnpg.io/cluster: knoe-db + cnpg.io/podRole: instance +``` + +You'll write this manifest into the repo, lose the CNPG-managed metadata +(labels/annotations/ownerReferences) since you own it now, and apply it +yourself. + +## 3. What changes + +### 3.1 — New file: `deploy/gcp/gke/knoe-db-podmonitor.yaml` + +```yaml +--- +# Manually-managed PodMonitor for the CNPG `knoe-db` cluster. +# +# Migrated 2026-05-XX from the (now-deprecated) CNPG fields +# `Cluster.spec.monitoring.enablePodMonitor` + +# `Cluster.spec.monitoring.podMonitorRelabelings`. Same scrape spec, same +# relabel config — now living as a first-class resource we control. +# +# The selector matches every CNPG instance pod via the operator-applied +# labels `cnpg.io/cluster=knoe-db` + `cnpg.io/podRole=instance`. The +# `metrics` port is the postgres-exporter (sidecar) on the CNPG pod. +# +# The relabel injects a `cluster` label on every scraped sample, sourced +# from the `cnpg.io/cluster` pod label. The cnpg-grafana dashboards filter +# every panel by `cluster=$cluster`; without this relabel only the +# operator-collector metrics carry the cluster label and 83 of 85 panels +# render empty. +# +# Apply alongside `knoe-db.yaml` from etc/init_cnpg_gke.sh. +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: knoe-db + namespace: knoe-db-0 + labels: + # Match what kube-prometheus-stack discovers via Helm-managed Prometheus + # serviceMonitorSelector / podMonitorSelector. kps's defaults match + # release=kps; verify in the live cluster: + # kubectl -n monitoring get prometheus -o yaml | grep -A2 podMonitorSelector + # If the selector requires a specific label, set it here. + release: kps +spec: + namespaceSelector: {} + selector: + matchLabels: + cnpg.io/cluster: knoe-db + cnpg.io/podRole: instance + podMetricsEndpoints: + - port: metrics + relabelings: + - action: replace + sourceLabels: + - __meta_kubernetes_pod_label_cnpg_io_cluster + targetLabel: cluster +``` + +**Verify the `release: kps` label trick.** Inspect the live Prometheus's +`podMonitorSelector` and ensure your manifest matches it. The current +auto-created PodMonitor doesn't carry `release: kps` (it uses +`app.kubernetes.io/managed-by: cloudnative-pg`), but it still gets +discovered — meaning the kps Prometheus must be configured with +`podMonitorSelector: {}` (any) or `podMonitorSelectorNilUsesHelmValues: +false` (which is what `monitoring/kps-cnpg-values.yaml` sets). Confirm +this and document in the manifest comment whether the `release: kps` +label is actually required. + +### 3.2 — `deploy/gcp/gke/knoe-db.yaml` + +Replace the entire `monitoring:` block: + +```yaml + monitoring: + # NOTE: enablePodMonitor + podMonitorRelabelings are CNPG-deprecated and + # will be removed in a future release. Migration path: manually manage + # the PodMonitor resource. Tracked in docs/TODO.md as a follow-up. + enablePodMonitor: true + podMonitorRelabelings: + - sourceLabels: ["__meta_kubernetes_pod_label_cnpg_io_cluster"] + targetLabel: cluster + action: replace +``` + +with: + +```yaml + monitoring: + # PodMonitor is managed manually as a sibling resource: + # deploy/gcp/gke/knoe-db-podmonitor.yaml. The CNPG-managed alternative + # (enablePodMonitor + podMonitorRelabelings) is deprecated. + enablePodMonitor: false +``` + +### 3.3 — `etc/init_cnpg_gke.sh` + +Find the section that applies `knoe-db.yaml` (search for +`render_cnpg_manifest "$storage_class" "$tmp_manifest"`). After the +cluster apply succeeds (and after the cluster reaches a working storage +class), apply the PodMonitor: + +```bash +# ── 11. Apply the manually-managed PodMonitor ──────────────────────────────── +log "Applying PodMonitor for $CNPG_CLUSTER_NAME ..." +kubectl apply -f "$GKE_MANIFEST_DIR/knoe-db-podmonitor.yaml" +``` + +Place this AFTER the cluster apply completes (since the PodMonitor doesn't +depend on the cluster being healthy — it just needs the namespace) but +BEFORE any verification waits. + +Also: when CNPG-managed PodMonitor was set, the operator auto-deleted it +when `enablePodMonitor` flipped to false. Verify what happens during the +transition on a live cluster: when you change `enablePodMonitor: true → +false` AND apply your manual PodMonitor with the same name in the same +namespace, does CNPG try to delete your manual one? Based on +`ownerReferences` on the auto-created PodMonitor, the answer should be +"the auto-created one gets deleted by garbage collection when the +controller flag flips, and your manually-applied one (no ownerReference) +survives". Test this on a non-prod cluster before claiming victory in the +commit message. + +### 3.4 — Cross-mode parity (k3s + k8s/min) + +Both `deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml` and +`k8s/knoe/knoe-db.yaml` have `monitoring.enablePodMonitor: false` already +— they don't run the dashboard, so no PodMonitor at all today. **Leave +those modes alone for now.** They'll need their own PodMonitor manifest if +we ever wire monitoring there, but that's separate work. Add a comment to +each pointing at the GKE manifest as the reference if/when needed. + +## 4. Verification + +End-to-end — must pass before merging: + +1. **Apply the new manifest doesn't 404:** + ```bash + kubectl --dry-run=server -f deploy/gcp/gke/knoe-db-podmonitor.yaml apply + ``` + +2. **Cluster spec change doesn't break operator:** + ```bash + yq '.spec.monitoring' deploy/gcp/gke/knoe-db.yaml + # Expected: only enablePodMonitor: false (no podMonitorRelabelings) + ``` + +3. **Live transition test (the scary one).** On the live cluster, do the + transition AND verify dashboard still works: + ```bash + DB=gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0 + APP=gke_plenary-truck-485623-p7_us-west3_knoe-dev-0 + + # 1. Apply the new manual PodMonitor + kubectl --context=$DB apply -f deploy/gcp/gke/knoe-db-podmonitor.yaml + + # 2. Verify it's there + kubectl --context=$DB -n knoe-db-0 get podmonitor + + # 3. Apply the cluster spec change (flips enablePodMonitor false) + kubectl --context=$DB apply -f deploy/gcp/gke/knoe-db.yaml + # (or however the script renders + applies it) + + # 4. Confirm only ONE PodMonitor remains (the manual one) — the + # CNPG-owned one should garbage-collect once the field flips + kubectl --context=$DB -n knoe-db-0 get podmonitor + + # 5. Wait 30s and confirm DB-cluster Prometheus still has scrape targets: + kubectl --context=$APP -n monitoring exec sts/kps-grafana -c grafana -- \ + wget -qO- "http://admin:$(kubectl --context=$APP -n monitoring get secret kps-grafana -o jsonpath='{.data.admin-password}' | base64 -d)@localhost:3000/api/datasources/proxy/uid/P5531627C358300FE/api/v1/targets?state=active" \ + | python3 -c 'import sys,json; t=json.load(sys.stdin)["data"]["activeTargets"]; print(f"{len(t)} active targets, postgres exporters: {sum(1 for x in t if \"9187\" in x[\"scrapeUrl\"])}")' + # Expected: ≥3 postgres exporters active (our 3 CNPG pods) + + # 6. Refresh the cnpg-grafana dashboard, confirm panels still populate + ``` + +4. **Rollback path documented.** If step 3.5 shows dropped targets, the + rollback is to flip `enablePodMonitor: true` and delete the manual + PodMonitor. Note this in the commit message. + +## 5. Out of scope + +- Any work on `monitoring/kps-cnpg-values.yaml` itself. + `podMonitorSelectorNilUsesHelmValues: false` stays as is. +- Cross-mode PodMonitor (k3s/k8s/min) — defer. +- The `cluster=knoe-db` label vs. dynamic templating — currently hardcoded + to `knoe-db`. If a second CNPG cluster ever lands, the manual PodMonitor + needs to be parameterized. Leave a TODO comment but don't solve. +- Don't touch the cnpg-grafana dashboard transforms + (`monitoring/cnpg-dashboard-transforms.yaml`) — those are unrelated. + +## 6. Commit shape + +``` +feat(monitoring): manage knoe-db PodMonitor manually (drop CNPG-deprecated fields) + +CNPG operator emits deprecation warnings on every cluster.spec.monitoring +patch: + - enablePodMonitor will be removed in a future release + - podMonitorRelabelings will be removed in a future release + +Migrate to a manually-managed PodMonitor: + + deploy/gcp/gke/knoe-db-podmonitor.yaml (NEW — same spec as auto-created) + ~ deploy/gcp/gke/knoe-db.yaml (enablePodMonitor: true → false, + podMonitorRelabelings dropped) + ~ etc/init_cnpg_gke.sh (apply PodMonitor after cluster) + +Verified live: cnpg-grafana dashboard panels render unchanged after the +flip; 3 postgres-exporter targets active in cnpg-prometheus. + +Closes queue item #13 in docs/TODO.md. +``` + +## 7. Definition of done + +- [ ] `deploy/gcp/gke/knoe-db-podmonitor.yaml` created, content verified + against live PodMonitor spec. +- [ ] `deploy/gcp/gke/knoe-db.yaml` no longer contains + `podMonitorRelabelings`; `enablePodMonitor: false` with comment. +- [ ] `etc/init_cnpg_gke.sh` applies the new manifest after cluster apply. +- [ ] Live cnpg-grafana dashboard verified after the transition: panels + populate, 3 postgres-exporter targets in Prometheus. +- [ ] No new deprecation warnings on `kubectl apply` of the cluster manifest. +- [ ] `docs/TODO.md` queue item #13 archived to **Done** with verification + notes. diff --git a/docs/plans/junie/15-remove-dead-dashboard-consumer.md b/docs/plans/junie/15-remove-dead-dashboard-consumer.md new file mode 100644 index 0000000..980e254 --- /dev/null +++ b/docs/plans/junie/15-remove-dead-dashboard-consumer.md @@ -0,0 +1,249 @@ +# Junie brief — Queue #15: remove dead `DASHBOARD` consumer + `basicauth_credentials` from supabase Kong chart + +> **Self-contained brief.** Pure cleanup. The basic-auth plugin block was +> removed in commit [`25f1b2e`](#) (oauth2-proxy → Kong fronting); the +> consumer + credentials it gated have no enforcer. **Watch the OpenAI key +> trap** — it lives in the same `secret.dashboard` block in +> `values.yaml`, so don't blanket-delete the section. + +--- + +## 1. Why + +Before commit [`25f1b2e`](#) (2026-04-30, oauth2-proxy fronting), Studio +was gated by Kong's `basic-auth` plugin against credentials in +`secret.dashboard.{username,password}`. That gate was replaced by +oauth2-proxy + Google OAuth, so the `basic-auth` plugin reference was +removed from the Kong route definition. + +Three artifacts of the old gate remain: + +1. **Kong consumer named `DASHBOARD`** in + `supabase/helm/knoe-supabase/templates/kong/config.yaml` — has no + plugin enforcing it. Dead. +2. **`basicauth_credentials` block** in the same file — provides + credentials for a consumer no plugin is checking. Dead. +3. **`DASHBOARD_USERNAME` / `DASHBOARD_PASSWORD` env vars** flowing through + the Kong deployment + the wrapper-script's envsubst. Dead. + +The `secret.dashboard.{username,password}` values in `values.yaml` +themselves are unused after this cleanup, but the **`secret.dashboard.openAiApiKey` +field is NOT dead** — it's consumed by +`templates/studio/deployment.yaml:92` to populate `OPENAI_API_KEY` in the +Studio pod env. **Don't delete the whole `secret.dashboard` block.** +Restructure carefully so the OpenAI key still has a home. + +## 2. The four artifacts to remove + +### Artifact 1 — `supabase/helm/knoe-supabase/templates/kong/config.yaml` + +**Lines 19-20** (the wrapper-script's sed replacements): +```yaml + -e "s|\${DASHBOARD_USERNAME}|${DASHBOARD_USERNAME}|" \ + -e "s|\${DASHBOARD_PASSWORD}|${DASHBOARD_PASSWORD}|" \ +``` +Delete both lines. + +**Lines 30-32** (the consumer): +```yaml + {{- if .Values.secret.dashboard }} + - username: DASHBOARD + {{- end }} +``` +Delete all three lines (the `if`-end pair around the consumer). + +**Lines 44-49** (the credentials block): +```yaml + {{- if .Values.secret.dashboard }} + basicauth_credentials: + - consumer: DASHBOARD + username: ${DASHBOARD_USERNAME} + password: ${DASHBOARD_PASSWORD} + {{- end }} +``` +Delete all six lines. + +### Artifact 2 — `supabase/helm/knoe-supabase/templates/kong/deployment.yaml` + +**Lines 74-92** (the two env-var blocks): both `DASHBOARD_USERNAME` and +`DASHBOARD_PASSWORD` env-from-secret references. They're both wrapped in +`{{- if .Values.secret.dashboard }} ... {{- end }}`. Delete the whole +conditional block (both env vars, the wrapper, the inner conditional for +`secretRef` vs default secret). + +After deletion, verify the surrounding env list still parses (Kong +deployment manifest must still be a valid Pod spec). + +### Artifact 3 — `supabase/helm/knoe-supabase/values.yaml` + +**Lines 104-118** (the `secret.dashboard` section): + +```yaml + ## Studio dashboard credentials + ## + dashboard: + username: "supabase" + password: "this_password_is_insecure_and_should_be_updated" + openAiApiKey: "key_super_secret" + ## Reference to existing secret + # secretRef: "" + ## Map to actual keys inside secretRef if they differ + # secretRefKey: + # username: username + # password: password + # openAiApiKey: openAiApiKey +``` + +**Restructure** — don't delete. Two paths: + +**Path A (recommended): rename the section to `openai`.** Remove +`username`/`password`. Rename `dashboard:` to `openai:`. Update the one +consumer (`templates/studio/deployment.yaml:92`) to read from +`.Values.secret.openai.apiKey` (renamed from `openAiApiKey`). Update the +secretRefKey mapping similarly. Cleanest. + +```yaml + ## OpenAI API key — passed to Studio's AI Assist sidebar. + ## Optional; if absent, the AI Assist panel renders an error toast but + ## the rest of Studio works fine. We default to a placeholder string so + ## helm template doesn't fail; replace via secretRef in production. + openai: + apiKey: "key_super_secret" + # secretRef: "" + # secretRefKey: + # apiKey: apiKey +``` + +**Path B (minimal): drop username/password only.** Keep the section name +`dashboard:` since renaming touches more files. End up with a `dashboard:` +section that holds only `openAiApiKey`. Slightly confusing name vs. +contents, but the diff is smaller. + +Pick Path A unless the touch surface scares you. The `if .Values.secret.dashboard` +conditional in the kong-deployment env block goes away in Path A +naturally; in Path B, replace those `if`s with explicit +`if .Values.secret.dashboard.username` / `.password` checks before +removing the env vars (since the block-level check used to mean "any +field set"; now it must be field-specific). + +### Artifact 4 — `supabase/helm/knoe-supabase/templates/studio/deployment.yaml` + +If you take Path A above, update the env-var binding at line 92: + +```yaml + - name: OPENAI_API_KEY + valueFrom: + secretKeyRef: + {{- if .Values.secret.openai.secretRef }} + name: {{ .Values.secret.openai.secretRef }} + key: {{ .Values.secret.openai.secretRefKey.apiKey | default "apiKey" }} + {{- else }} + name: {{ include "supabase.secret.openai" . }} + key: apiKey + {{- end }} +``` + +(rename `dashboard` → `openai`, `openAiApiKey` → `apiKey`). + +Also rename or re-source the helper template `supabase.secret.dashboard` +(if it exists in `_helpers.tpl`) to `supabase.secret.openai`. If the +helper doesn't exist (the grep earlier returned no match), skip. + +### Artifact 5 — `supabase/helm/knoe-supabase/ci/example.yaml` + +Has a parallel `secret.dashboard.openAiApiKey` for chart CI. Update to +match Path A or B. + +## 3. Don't break + +- **OpenAI API key flow**: pre-and-post-cleanup the Studio pod must + receive `OPENAI_API_KEY` from a Secret, identically. Verify with + `helm template` before vs. after. +- **Existing live deployments using `secretRef`** — if any production + deploy uses `secret.dashboard.secretRef` to point at an existing K8s + Secret with `username`/`password`/`openAiApiKey` keys, that secret + needs renaming too. Grep for `secret.dashboard.secretRef` in any + real values overrides (not just the chart defaults). If found, + ASK before proceeding. + +## 4. Verification + +1. **`helm template` the chart and diff for OPENAI_API_KEY:** + ```bash + helm template supabase/helm/knoe-supabase \ + | grep -A 6 'name: OPENAI_API_KEY' + ``` + Expected: env var still binds to a Secret. The Secret name + key may + differ from before (if you renamed); the **flow** must still resolve. + +2. **`helm template` the chart and confirm DASHBOARD is gone:** + ```bash + helm template supabase/helm/knoe-supabase | grep -i DASHBOARD + ``` + Expected: no output. + +3. **`helm lint`** the chart: + ```bash + helm lint supabase/helm/knoe-supabase + ``` + Expected: clean. + +4. **No stale references in the rest of the repo:** + ```bash + grep -rni DASHBOARD_USERNAME supabase/ k8s/ deploy/ etc/ + grep -rni DASHBOARD_PASSWORD supabase/ k8s/ deploy/ etc/ + ``` + Expected: no output (or only docs explaining the old flow that you + then update / remove). + +5. **OpenAI key still works in Studio**: hard to verify without rolling + out, but a `helm template` showing the env var still bound is + sufficient for chart correctness. Live re-roll of supabase-studio is + out of scope; flag it as a manual follow-up if anything changes about + the secret name. + +## 5. Out of scope + +- **Don't roll out the chart change to live `knoe-dev-0`.** Chart + cleanup is code-only; the current live deployment uses the OLD chart + (with `secret.dashboard`), and migrating production secret structure + is a separate decision. +- The Studio fork (queue item #12) — totally separate. +- Any other Kong route changes — leave the file alone outside the + surgical lines listed. + +## 6. Commit shape + +If Path A: + +``` +chore(supabase): remove dead DASHBOARD consumer + basic-auth credentials + +The basic-auth plugin was retired in commit 25f1b2e (oauth2-proxy +took over Studio gating via Google OAuth). Three artifacts of the +old gate remained as no-op config: + + - Kong consumer named DASHBOARD + - basicauth_credentials block + - DASHBOARD_USERNAME / DASHBOARD_PASSWORD env in kong deployment + +All removed. + +The secret.dashboard.openAiApiKey field — the only LIVE consumer +of secret.dashboard — was migrated to secret.openai.apiKey. Studio's +OPENAI_API_KEY env binding updated; helm template diff confirms +the same Secret-source flow. + +Closes queue item #15 in docs/TODO.md. +``` + +If Path B: simpler subject — `chore(supabase): drop dead DASHBOARD username/password`. + +## 7. Definition of done + +- [ ] Four artifacts removed from `kong/config.yaml`, + `kong/deployment.yaml`, `values.yaml`, `ci/example.yaml`. +- [ ] OpenAI key still resolves through `helm template`. +- [ ] `helm lint` clean. +- [ ] `grep -rni DASHBOARD_USERNAME` returns nothing in repo. +- [ ] `docs/TODO.md` queue item #15 archived to **Done**. diff --git a/docs/plans/junie/README.md b/docs/plans/junie/README.md new file mode 100644 index 0000000..034c109 --- /dev/null +++ b/docs/plans/junie/README.md @@ -0,0 +1,66 @@ +# Junie briefs + +Self-contained, single-task briefs for [Junie](https://www.jetbrains.com/junie/) +to consume from inside the IntelliJ IDE. Each file is a hand-off — Junie +reads the brief cold (no shared chat history), implements the change, and +opens an MR. + +## What lives here vs. `../` + +The `..` parent directory holds **architectural plans** — multi-section +documents covering an initiative's strategy, schema, and component design. +They outlive the implementation and become the architectural reference +once shipped. + +This subdirectory holds **work briefs** — smaller, more tactical, scoped +to a single MR's worth of changes. They reference the master TODO index +(`../../TODO.md`) and contain enough context for Junie to land the change +without escalating questions. + +## Brief shape (convention) + +Each brief follows this skeleton: + +1. **Why** — one or two paragraphs of context, including the trigger event + if any (e.g. "the 4/28 14:00 UTC outage"). +2. **What changes** — concrete file paths, line numbers, before/after where + useful. Don't make Junie re-derive the change. +3. **Verification** — runnable commands or `helm template` diffs that + confirm the change. Each brief ends with a **Definition of done** + checklist. +4. **Out of scope** — explicit fences. The TODO is interconnected; without + this section briefs creep. +5. **Commit shape** — proposed commit message + structure. + +## Status tracking + +Every brief here corresponds to a numbered item in +[`../../TODO.md`](../../TODO.md) (or has an explicit "doesn't yet exist +in TODO" note). When Junie lands a brief: + +1. Move the corresponding item from the TODO ranked queue into the + **Done** section with date + commit ref (or the convention used by + the rest of the file). +2. Don't delete the brief from this directory — it stays as the design + record. + +## Current briefs (as of 2026-05-01) + +| File | Queue # | Subject | +|---|---|---| +| [`02-k3s-prole-rename.md`](02-k3s-prole-rename.md) | #2 | Rename k3s `prole-*.yaml` → `knoe-*.yaml` (kustomize is broken) | +| [`06-patch-garage-script-fixes.md`](06-patch-garage-script-fixes.md) | #6 | Three defects in `scripts/patch_garage_cross_cluster.sh` | +| [`07-init-cnpg-gke-sa-wiring.md`](07-init-cnpg-gke-sa-wiring.md) | #7 | Wire `cnpg-backup-sa` into CNPG cluster spec; bump operator to v1.29 | +| [`13-podmonitor-manual-management.md`](13-podmonitor-manual-management.md) | #13 | Migrate off CNPG-deprecated `enablePodMonitor` + `podMonitorRelabelings` | +| [`15-remove-dead-dashboard-consumer.md`](15-remove-dead-dashboard-consumer.md) | #15 | Remove dead Kong DASHBOARD consumer + `basicauth_credentials` | + +## How a session fires off a batch + +The driving session (Claude Code, Cowork+Code, or a human) writes the +briefs into this directory and points Junie at one or more of them. Junie +reads the brief, implements, runs the verification checklist, opens an +MR. Each brief is independent — Junie can take them in any order, or in +parallel across separate IDE sessions. + +The brief is the contract. If something is unclear, the brief is buggy +and should be edited before Junie continues.