chrisfu/prole
1
0
Fork 0
mirror of https://github.com/dredx/prole.git synced 2026-09-23 11:03:59 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

docs(plans): file Junie briefs for queue items #2, #6, #7, #13, #15

Browse Source 
Five self-contained work briefs in a new docs/plans/junie/ subdirectory,
each tagged against a numbered item in docs/TODO.md so Junie can take
them independently in any order.

  02-k3s-prole-rename.md            -> queue #2  (drift R5)
  06-patch-garage-script-fixes.md   -> queue #6  (drift R9)
  07-init-cnpg-gke-sa-wiring.md     -> queue #7  (drift R8)
  13-podmonitor-manual-management.md-> queue #13
  15-remove-dead-dashboard-consumer.md-> queue #15

Each brief follows the same shape: Why -> What changes (concrete file
paths + line numbers + before/after) -> Verification -> Out of scope ->
Commit shape -> Definition of done. The intent is that Junie reads cold
(no shared chat history) and lands the change without escalating
questions.

Also adds:
  - docs/plans/junie/README.md describing the convention.
  - Row in docs/plans/README.md so newcomers find the subdirectory.
  - Brief reference + "Assigned to Junie" tag on each of the five queue
    items in docs/TODO.md.

Existing in-progress assignment to Junie (Phase 2 pg_oauth) is unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
chrisfu 2026-05-01 15:35:57 -07:00
parent 9daa94b860
commit f83ec8169c
8 changed files with 1274 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

145
docs/TODO.md Normal file
View File

@ -0,0 +1,145 @@
# Master TODO index
Single source of truth for unfinished work in this repo. Captures items that surfaced across the autobuild pipeline phases, the 2026-04-28 → 2026-04-29 cluster recovery / Maven build readiness work, the 2026-04-29 CNPG operator + backup stabilization work, and reality-vs-intent drift in the docs.
When you pick something up, move it from the ranked queue into the **Now** section. When it lands, archive it into the `Done` section at the bottom (or delete if it's not worth memorializing).
When you discover a new item, add it to the ranked queue. Default rank: bottom. Move it up later if it gets pulled in.
The Kanban "Now" section at top is the only place this doc imposes structure. Everything below "Now" is a numbered list — top of the list is next-most-important.
---
## Now (Kanban)
### In progress
- **Phase 2: pg_oauth in install/deploy.sh + k3d build** — assigned to Junie. The Phase 1 work (per-engineer roles, external LB at `pg.0.knoe.dev`, cert SANs, pg_hba tightening, [`docs/db-access.md`](db-access.md)) is committed. Phase 2 replaces the SCRAM password mechanism with PG18 native OAUTHBEARER, wires it into `install.sh` / `supabase/deploy.sh` so a fresh deploy lands with the OIDC mechanism active by default, and ports the Service + cluster cert pattern to k3d. Connection target stays at `pg.0.knoe.dev`; only the credential issuance changes (libpq Device Flow → Google Bearer token). Plan reference: `~/.claude/plans/we-re-continuing-work-on-happy-toast.md`.
### Up next
*(empty — pull from the ranked queue below)*
### Blocked
*(none currently)*
---
## Next (ranked queue — top of list = next-most-important)
1. **Installer env-contamination guard** — `install.sh` / `deploy.sh` should refuse to run if `${KUBECONTEXT}` from `conf/${mode}.cfg` doesn't match `kubectl config current-context` (or doesn't match what was passed via `--context`). Caused the 2026-04-28 14:00 UTC backup outage: an `install.sh --mode k3d` run with a GKE shell context overwrote the GCS-backed ObjectStore with a Garage-backed one, then Garage filled up. See [`CLAUDE.md`](../CLAUDE.md) §"Env-contamination warning".
2. **k3s manifest cleanup: `prole-*` filename rename** — `deploy/opentofu/k3s/manifests/knoe/prole-{configmap,deployment,service,kdc-configmap,kdc-secrets.example}.yaml` still carry the old prefix. The kustomization expects `knoe-*` names — kustomize is currently broken until rename. Same `git mv` pattern we used for `prole-auth-*` → `knoe-auth-*` in commit [b355855](#). **Brief: [`docs/plans/junie/02-k3s-prole-rename.md`](plans/junie/02-k3s-prole-rename.md). Assigned to Junie.**
3. **Image rename `knoe-authority` → `knoe-auth`** — three deployment manifests still reference `knoe-authority:latest`:
- `deploy/gcp/gke/knoe-auth-deployment.yaml` (3 lines)
- `deploy/opentofu/k3s/manifests/knoe/knoe-auth-deployment.yaml` (2 lines)
The Maven artifact is now `knoe-auth.jar` (post-rename in commit [b355855](#)). Image rebuild + registry push needed; both manifests then bump.
4. **Garage block-resync corruption** — after the 2026-04-29 bulk delete (8382 objects + 12 multipart aborts), garage-0 (on the **app cluster** `knoe-dev-0` — not the DB cluster, that one was removed) background workers logged `Could not fetch needed block X, no node returned valid data` for orphaned block refs. Eventually self-heals via `garage_block::resync` worker, but worth verifying with `garage repair --yes blocks` once the dust settles.
5. **Phase B orphan cleanup** — leftover from the cluster recovery work that didn't get cleaned because we focused on CNPG/GCS:
- **App cluster (`knoe-dev-0`):** delete `gitlab/gitlab-migrations-58a3e27-97-23-n72j8` Error pod (7d+ stale); reap 3 Released PVs in `monitoring/` (4h orphans from monitoring stack recreation)
- **DB cluster (`knoe-dev-cnpg-0`):** delete `kube-system/prole-supabase-dirprep` Job (18d stale, prole-era)
6. **Fix `scripts/patch_garage_cross_cluster.sh`** — three bugs surfaced when we used it on 2026-04-29:
- Default `DB_CLUSTER=knoe-cnpg-0` (line 39) is the stale name; should be `knoe-dev-cnpg-0` (matches `conf/gke.cfg`)
- Phase 1 only deletes `service/garage` (ClusterIP); misses `service/garage-s3-ilb` (the LoadBalancer at the IP the ObjectStore actually points at)
- Phase 2 applies `deploy/gcp/gke/knoe-db-backup-gcs.yaml` which contains a SECOND ScheduledBackup using the legacy `method: barmanObjectStore` — would double-up our existing plugin-method ScheduledBackup, and that legacy method is being removed in CNPG v1.30. The script should split SA-only apply from ScheduledBackup apply.
- Phase 3 unconditionally re-applies the canonical ObjectStore manifest. Now safe (manifest matches live), but worth a guard.
**Brief: [`docs/plans/junie/06-patch-garage-script-fixes.md`](plans/junie/06-patch-garage-script-fixes.md). Assigned to Junie.**
7. **Update `etc/init_cnpg_gke.sh` to set `cluster.spec.serviceAccountName: cnpg-backup-sa` and create the RoleBinding additions** — current script creates `cnpg-backup-sa` but doesn't wire it as the cluster pod SA. After 2026-04-29 stabilization the live cluster does this; the init script should reflect it so a fresh deploy lands in the same end state. **Brief: [`docs/plans/junie/07-init-cnpg-gke-sa-wiring.md`](plans/junie/07-init-cnpg-gke-sa-wiring.md). Assigned to Junie.**
8. **Port oauth2-proxy manifest to k3d / k3s / min modes** — `deploy/gcp/gke/oauth2-proxy-deployment.yaml` is GKE-shaped (uses `cloud.google.com/v1` BackendConfig + GCE health-check semantics). For non-GCE deploy modes the same Deployment runs unchanged but the ingress glue differs (Traefik / nginx-ingress / no LB at all). Add a `deploy/k3s/oauth2-proxy-deployment.yaml` with the Traefik-flavored ingress annotations once we deploy that mode.
9. **Wire GoTrue (`supabase-auth`) Google provider for Supabase API clients** — different surface than the Studio gate (which oauth2-proxy now handles). Once we have client apps that authenticate end-users via Supabase Auth, configure GoTrue's Google provider with its own OAuth client. See [`docs/oidc-setup.md`](oidc-setup.md) for the secret indirection pattern.
10. **Audit-log story for the multi-tenant Studio phase** — Studio doesn't track per-user actions in self-hosted; oauth2-proxy forwards the `Authorization: Bearer <id_token>` header to Kong (`--pass-authorization-header=true` / `--set-authorization-header=true`) but nothing reads it. When the multi-tenant control plane lands (per-user / per-project Studios), wire a Kong logging plugin that captures the email claim from the JWT and emits to a structured log sink.
11. **Retire `api.0.knoe.dev` Ingress rule + managed cert** — `db.0.knoe.dev` now serves both Studio (Google-gated) and the SDK API surface (Kong key-auth) via oauth2-proxy `--skip-auth-route`. `api.0.knoe.dev` remains as an alias for backward compatibility, but new docs/onboarding should point to `db.0.knoe.dev`. When external clients have all migrated: drop the `api.0.knoe.dev` rule from the supabase-kong Ingress and the `supabase-api-managed-cert` ManagedCertificate. Until then, the alias is harmless (same backend, same key-auth).
12. **Fork the Supabase Studio image to wire in-app help / support / feedback buttons to `mailto:support@knoe.dev` (or `https://db.0.knoe.dev/support`)** — upstream Studio (`supabase/studio:2026.02.16-sha-26c615c`) hardcodes those URLs to Supabase-cloud endpoints (`supabase.com/dashboard/api/feedback` etc.) which are unreachable from self-hosted, so the in-app "Report a problem" / "Send feedback" / "Get help" flows error out. No env-var hook exists in upstream — verified by Explore search. Fork the image, patch the relevant TSX (`apps/studio/components/layouts/AppLayout/AppLayout.tsx`, support-dialog component, settings/help links — handful of files), publish to our registry, bump `image.studio.repository` in [`supabase/helm/knoe-supabase/values.yaml`](../supabase/helm/knoe-supabase/values.yaml). Same fork is the natural place to fix the OpenAI key panel, telemetry endpoints, and any other in-Studio assumptions about Supabase cloud as we encounter them. Stop-gap until then: `https://db.0.knoe.dev/support` 302s to `mailto:support@knoe.dev` (Kong route `support`); just need to tell users to bookmark or remember it.
13. **Migrate off CNPG-deprecated `monitoring.enablePodMonitor` + `podMonitorRelabelings` fields** — both fields in [`deploy/gcp/gke/knoe-db.yaml`](../deploy/gcp/gke/knoe-db.yaml) `spec.monitoring` are flagged deprecated by the CNPG operator and "will be removed in a future release". Migration path per the operator's warning: manually manage the PodMonitor resource (i.e. write `deploy/gcp/gke/knoe-db-podmonitor.yaml` ourselves and apply alongside the Cluster). New file would carry the `relabelings` (cluster label injection, currently sourced from pod label `cnpg.io/cluster`) and any future scrape customization. Cluster spec keeps only the bare CNPG operator config. No urgency — works fine on the current operator (CNPG v1.29) — but file before the next CNPG upgrade. Reference: kubectl deprecation warning emitted on every patch of these fields. **Brief: [`docs/plans/junie/13-podmonitor-manual-management.md`](plans/junie/13-podmonitor-manual-management.md). Assigned to Junie.**
14. **Migrate cnpg-prometheus datasource UID to the stable `cnpg-prometheus` name** — currently the live datasource has the auto-generated UID `P5531627C358300FE` from the original kps install. We pinned `uid: cnpg-prometheus` in [`monitoring/kps-values-gke.yaml`](../monitoring/kps-values-gke.yaml) so any FRESH kps install lands on the stable name, but Grafana refuses to change the UID of an already-provisioned datasource (read-only via API; rollout-restart doesn't migrate it). The dashboard transform [`monitoring/cnpg-dashboard-transforms.yaml`](../monitoring/cnpg-dashboard-transforms.yaml) `ds_prometheus_default_to_cnpg` carries the auto-uid as a workaround. Migration path on the next clean kps re-install (or after a maintenance window where we can wipe the Grafana sqlite DB to drop datasources): swap `value: P5531627C358300FE` → `value: cnpg-prometheus` in the transform and re-run the sync tool. No client-visible change either way.
15. **Remove DASHBOARD consumer + `basicauth_credentials` from `supabase/helm/knoe-supabase/templates/kong/config.yaml`** — now-dead config: the basic-auth plugin block was removed in commit (this oauth2-proxy work), so the consumer + credentials it gated have no enforcer. Lines 30-32 (`{{- if .Values.secret.dashboard }} - username: DASHBOARD {{- end }}`) and lines 44-49 (the `basicauth_credentials:` block) can be deleted; then drop `secret.dashboard` from `values.yaml` and the `DASHBOARD_USERNAME` / `DASHBOARD_PASSWORD` envsubst lines from `wrapper.sh`. Cleanup-only; current state is harmless. **Brief: [`docs/plans/junie/15-remove-dead-dashboard-consumer.md`](plans/junie/15-remove-dead-dashboard-consumer.md). Assigned to Junie. Note: brief flags the OpenAI key trap — `secret.dashboard.openAiApiKey` is the only LIVE consumer of that section, so don't blanket-delete.**
---
## Reality-vs-intent gaps
Items where docs describe an intended state that doesn't match live state. Each links to where reality is documented. Closing the gap means either updating the doc to match reality permanently, or doing the work to make reality match.
| # | Reality (current) | Intended state | Documented at | Tracked work item |
|---|---|---|---|---|
| R4 | Installer can clobber GKE state when run in `k3d` mode with a GKE shell context (caused 2026-04-28 14:00 UTC outage) | Installer refuses to run on context mismatch | [`CLAUDE.md`](../CLAUDE.md) §"install.sh / deploy.sh pre-flight" warning | Queue item #1 |
| R5 | k3s manifests `prole-{configmap,deployment,service,kdc-*}.yaml` still carry old prefix | All renamed to `knoe-*` (kustomization already references `knoe-*`) | Phase C plan in `~/.claude/plans/we-re-continuing-work-on-happy-toast.md` follow-ups | Queue item #2 |
| R6 | Image refs `knoe-authority:latest` in deploy manifests | `knoe-auth:latest` (matches Maven artifact `knoe-auth.jar` post-rename) | Phase C plan follow-ups | Queue item #3 |
| R8 | `etc/init_cnpg_gke.sh` doesn't set `cluster.spec.serviceAccountName` or extend the role bindings to `cnpg-backup-sa` | Script lands a fresh deploy in the same end state as live (with `serviceAccountName: cnpg-backup-sa` and role-binding subjects added) | Live cluster spec + RoleBindings | Queue item #7 |
| R9 | `scripts/patch_garage_cross_cluster.sh` has stale `DB_CLUSTER` default, misses `garage-s3-ilb` service, and bundles legacy ScheduledBackup apply | Three issues fixed; phases split so removal is independent of SA/ObjectStore re-apply | Live script | Queue item #6 |
---
## Later
11. **Round 1.5: OpenBao transit-key encryption for per-user secrets at rest** — `authority/src/main/java/dev/knoe/auth/enroll/UserProvisioningService.java:84` has `// TODO Round 1.5: encrypt secret with OpenBao transit key before storing.` Currently TOTP secrets and similar long-lived per-user state go to `knoe.*` tables with PG-level encryption-at-rest only. Round 1.5 wraps writes/reads with an OpenBao transit-key envelope. See [`docs/plans/knoe-auth-round-1.md`](plans/knoe-auth-round-1.md) §8 "Out of scope for Round 1".
12. **`knoe-db/docker-entrypoint.sh:3` swap to `set -Eeuo pipefail`** — current `set -e` only. Marked as TODO in the file: "swap to -Eeuo pipefail above (after handling all potentially-unset variables)". Needs an audit pass over the script for unset-var safety first.
13. **Customer-deploy-resync execution decision** — currently no separate `~/dev/prole` working tree under development; the plan in [`docs/plans/customer-deploy-resync.md`](plans/customer-deploy-resync.md) is dormant. Decision: do we ever activate the per-customer branching workflow (e.g. for a future second customer), or formalize "single-tenant for now" and remove the plan?
---
## Deferred / Out of scope
- **Studio Database Advisor — Performance: unindexed FKs in upstream-managed schemas** — 6 FKs flagged on a fresh project: `auth.mfa_challenges.factor_id`, `auth.saml_relay_states.flow_state_id`, `auth.oauth_authorizations.{client_id, user_id}`, `storage.s3_multipart_uploads_parts.{upload_id, bucket_id}`. Adding indexes to upstream-managed tables risks conflict on the next supabase-auth / supabase-storage migration. Wait for upstream fixes; revisit if these turn out to be load-bearing for our usage.
- **`com.knoey.auth` open-source split** — saved for the major collaboration (per user memory: "we are saving knoey.com/auth for the major collaboration"). Will require a separate repo extraction and licensing pass.
- **"Test ssh to Gitea" task** — orphaned during a previous Cowork+Code session when the Dispatch connection was lost. May or may not still be relevant.
- **Volume snapshot backups (CNPG `VolumeSnapshot` method)** — alternative to barman tarball. Considered during the 2026-04-29 outage debugging when barman-cloud-backup looked broken; now that GCS+barman is verified working, no immediate need. Worth revisiting if barman shows further fragility.
- **File the upstream `plugin-barman-cloud` issue** about the misleading `ReferenceError: weakly-referenced object no longer exists` traceback that fires on persistent 5xx responses. Issue [#707](https://github.com/cloudnative-pg/plugin-barman-cloud/issues/707) is closed but didn't capture this pattern; ours is a clearer reproduction. Defer — the upstream is responsive but the workaround (sufficient bucket capacity + correct IAM) is good enough.
---
## Done
*(items get archived here when they land, with a date and commit reference; or just delete if not worth memorializing)*
- 2026-05-01 — **Cross-cluster CNPG metrics via DB-cluster Prometheus** (commit pending). Solves the `cloudnative-pg` Grafana dashboard's "No data" problem from queue item #12. Picked path (c) over the originally-recommended (b) for DB-cluster autonomy + pattern symmetry — each cluster owns its observability footprint, and a future second DB cluster reuses the recipe one-for-one. Installed kube-prometheus-stack v84.3.0 on `knoe-dev-cnpg-0` (`monitoring` namespace), Grafana + Alertmanager disabled (single canonical Grafana stays on app cluster). All PVCs explicitly pinned to `standard-hdd` (per the user's standing rule on SSD-quota preservation; verified `prometheus-...-0` PVC bound 20Gi `standard-hdd`). New ILB `prometheus-cnpg-ilb` in DB-cluster `monitoring` ns at `10.180.15.216:9090`, reachable from app-cluster pods over the shared VPC. New `cnpg-prometheus` datasource added to app-cluster Grafana via `monitoring/kps-values-gke.yaml` `grafana.additionalDataSources`. **Side-fix**: removed an orphan `knoe-grafana-datasource` ConfigMap that `etc/init_monitoring.sh:apply_grafana_datasource()` had been creating with a duplicate `uid: prometheus` — caused Grafana's provisioning reload to fail with HTTP 500 once any second datasource arrived. Function neutered to a no-op with a clear comment; chart values are now the single source of truth for datasources. Verified end-to-end: Prometheus on DB cluster discovers all 3 CNPG postgres pods (knoe-db-2/3/4 at 10.24.x.x:9187, all `health=up`); Grafana datasource health endpoint returns `Successfully queried the Prometheus API`; `cnpg_pg_stat_archiver_archived_count` query through Grafana proxy returns live data (704 archived WAL files at time of test). To complete: switch each cnpg-grafana dashboard's `DS_PROMETHEUS` template variable to `cnpg-prometheus` (one-click). New files: [`monitoring/kps-cnpg-values.yaml`](../monitoring/kps-cnpg-values.yaml), [`deploy/gcp/gke/cnpg-prometheus-ilb.yaml`](../deploy/gcp/gke/cnpg-prometheus-ilb.yaml). Modified: [`monitoring/kps-values-gke.yaml`](../monitoring/kps-values-gke.yaml) (datasource added), [`etc/init_monitoring.sh`](../etc/init_monitoring.sh) (function no-op'd). Out of scope (Phase 2): federate alerts back to app-cluster Alertmanager; reserve a static internal IP so the ILB doesn't drift on re-create.
- 2026-04-30 — **Bump postgres pod memory limit 512Mi → 2Gi** (commit pending). Long-standing throughput cliff: `barman-cloud-backup` is single-threaded gzip + GCS upload buffering, ceiling-bound by the pod's memory limit. The 9 GB DB took 30–90 min to back up at 512Mi; 2Gi cuts that to 5–10 min. requests left at 128Mi (no scheduling change; this is purely a ceiling bump). Patched live with `kubectl patch cluster knoe-db --type=merge -p '{"spec":{"resources":{"limits":{"memory":"2Gi"}}}}'`; CNPG operator performed a supervised rolling restart (replicas first, primary last). Repo source updated at [`deploy/gcp/gke/knoe-db.yaml`](../deploy/gcp/gke/knoe-db.yaml). Pre-flight verified no external psql sessions (chrisfu/ron not connected); 39 internal sessions FATAL'd through the rolling restart and reconnected via their pools — supabase + gitlab + streaming_replica all stable post-rollout. Closes drift R3. k3s + k8s mode resource limits remain at 512Mi (not exposed in those manifests today); cross-mode parity folded into queue item #8 ("Port oauth2-proxy manifest to k3d/k3s/min modes").
- 2026-04-30 — **Align `knoe/core/actions.py` cluster-name + machine-type defaults with live state** (commit pending). Two stale constants since the rebrand: `DEFAULT_DB_CLUSTER_NAME = "knoe-cnpg-0"` (live: `knoe-dev-cnpg-0`) and `DEFAULT_APP_CLUSTER_MACHINE_TYPE = "e2-small"` (live: `e2-standard-2`). Updated both at [`knoe/core/actions.py`](../knoe/core/actions.py) lines 121 + 123. Verified: `python3 -c "from knoe.core.actions import ..."` reads the new values; pytest collection error count went from 8 → 7 post-edit (one of the pre-existing collection errors apparently went away when the import resolved differently — no regression). Closes drift R1 + R2 in CLAUDE.md (footnotes + drift table rows + the `# STALE` inline comments in the §"Cluster code constants" code block all dropped). Implicitly closes the formerly-#8 queue item ("Update CLAUDE.md cluster table once defaults match") in this same commit. Queue renumbered: #2 (this item) and #8 dropped, gaps closed.
- 2026-04-30 — **Drop redundant GCP IAM workloadIdentityUser binding** (commit pending). v1.28-era leftover: `[knoe-db-0/knoe-db]` was bound alongside the active `[knoe-db-0/cnpg-backup-sa]` on the `cnpg-backup@…` GCP SA. Pods have used `cnpg-backup-sa` since the 2026-04-29 stabilization (commit [e92cd8b](#)), so the older binding was dead config — kept around because removing it required a reasoned check that no in-flight job still authenticated under the old identity. Verified live (`gcloud iam service-accounts get-iam-policy`), dropped via `gcloud iam service-accounts remove-iam-policy-binding`, re-verified the SA's policy now has only `[knoe-db-0/cnpg-backup-sa]`. CNPG primary still healthy (`SELECT 1` returned cleanly post-removal). Closes drift R7. Removed the matching CLAUDE.md Reality TODO + drift table row.
- 2026-04-30 — **Grafana at `svc.knoe.dev/grafana` with Google OAuth** (commit pending). Diagnosed the `svc.knoe.dev` "name resolution failed" error: `knoe-svc-kong-config` had its grafana service pointing at `prometheus-grafana.monitoring.svc.cluster.local:80` — the OLD upstream name from before the kube-prometheus-stack chart renamed to `kps-grafana`. Kong was DNS-failing every request to that vanished service and surfacing the Kong-style "name resolution failed" error to clients. Fixed the upstream + scoped the route to `/grafana` (so other surfaces can land at `svc.knoe.dev/*` later). Layered Grafana's native Google OAuth via auth.google config in [`monitoring/kps-values-gke.yaml`](../monitoring/kps-values-gke.yaml) (new — captures previously-uncommitted live Helm values + adds the auth additions). Role mapping via JMESPath: `contains(['chrisfu@knoey.com', 'ron@knoey.com'], email) && 'Admin' || 'Editor'` — anyone @knoey.com gets Editor, chrisfu+ron get Admin. New manifests: [`deploy/gcp/gke/grafana-google-oidc-secret.example.yaml`](../deploy/gcp/gke/grafana-google-oidc-secret.example.yaml), [`etc/init_grafana_oauth.sh`](../etc/init_grafana_oauth.sh), [`monitoring/kps-values-gke.yaml`](../monitoring/kps-values-gke.yaml). Repo Kong source updated at [`deploy/opentofu/k3s/manifests/knoe/kong-configmap.yaml`](../deploy/opentofu/k3s/manifests/knoe/kong-configmap.yaml) (`.org→.dev` rebrand, healthz route restored, `/grafana` scoping). Verified externally: `/grafana/login` → 200; `/grafana/login/google` → 302 `accounts.google.com` with the right client_id; subpath assets load; Studio + SDK paths on db.0.knoe.dev unchanged. Browser walkthrough (chrisfu sign-in → Admin role; ron same; outside-domain rejected by `allowed_domains: knoey.com`) is the manual follow-up. Stop-gap `adminPassword: admin` in values stays (local break-glass) — rotate this when convenient as a follow-up.
- 2026-04-30 — **Reusable engineer onboarding via one-time link + QR** (commit pending). Replaces the assumed-but-nonexistent shared 1Password vault with a self-contained delivery flow: `./etc/onboard_engineer.sh <user> <email>` provisions the postgres role + 24h temp password and outputs (1) a single-shot URL, (2) a QR-code rendering for screenshare, (3) the plaintext fallback. Engineer opens URL or scans QR → static page at `db.0.knoe.dev/onboard.html` (vanilla HTML+JS, no backend) reads the URL fragment client-side and shows password + psql connection string + rotation snippet with [Copy] buttons; saves to engineer's **personal** 1Password. Engineer rotates immediately via `\password`. New manifests: [`deploy/gcp/gke/knoe-onboard.yaml`](../deploy/gcp/gke/knoe-onboard.yaml) (nginx:alpine + ConfigMap + Service), Kong route `/onboard.html` → knoe-onboard:80 in [`supabase/helm/knoe-supabase/templates/kong/config.yaml`](../supabase/helm/knoe-supabase/templates/kong/config.yaml), oauth2-proxy `--skip-auth-route=^/onboard\.html$` in [`deploy/gcp/gke/oauth2-proxy-deployment.yaml`](../deploy/gcp/gke/oauth2-proxy-deployment.yaml). Pivotal pg_hba change: rule changed from `hostssl all chrisfu,ron all` to `hostssl all +knoe_developer all` so future onboards don't require pg_hba edits — the script's `GRANT knoe_developer TO <user>` is enough. CA cert committed at [`etc/knoe-db-ca.crt`](../etc/knoe-db-ca.crt) (public root, not a secret) so engineers reference `sslrootcert=etc/knoe-db-ca.crt` after cloning. Engineer-facing docs: [`docs/onboarding.md`](onboarding.md). [`docs/db-access.md`](db-access.md) updated to remove the broken 1Password reference. Verified end-to-end: `./etc/onboard_engineer.sh test-onboard test-onboard@knoey.com` → URL + QR rendered → `psql ...user=test-onboard...` connects with the temp password and reports `pg_has_role(current_user,'knoe_developer','MEMBER')=t` → `--revoke` drops the role cleanly.
- 2026-04-30 — **Per-engineer psql access via `pg.0.knoe.dev` (Phase 1)** (commit pending). External GKE TCP LoadBalancer at `34.106.156.196` (forwarding rule `a61028024f2314e06902bafc1b4c3a08`); user adds Namecheap A record `pg.0.knoe.dev → 34.106.156.196` (manual; Namecheap UI). CNPG `serverAltDNSNames` extended with `pg.0.knoe.dev` so the auto-issued server cert validates externally. New `knoe_developer` group role with R/W on `knoe`+`public` and R/O on `auth`+`storage`+`extensions`. Per-engineer roles `chrisfu` and `ron` with strong random passwords (192-bit base64) `VALID UNTIL '2026-05-30'`, members of `knoe_developer`. pg_hba rewritten so external requires TLS+SCRAM and only allows `chrisfu, ron` (other roles `postgres`, `supabase_admin`, `authenticator` stay cluster-internal). `externalTrafficPolicy: Local` on the LB to preserve client source IP (otherwise the SNAT'd node IP made externals look like internal cluster traffic and bypassed the `hostnossl reject` rule). Documentation in [`docs/db-access.md`](db-access.md). Verified: chrisfu+ron both connect via TLS, plaintext rejected with the proper pg_hba error showing the real client IP, `postgres` role connection from outside fails auth, internal supabase services keep their existing TLS connections (gitlab on TLS 1.3, supabase_admin/authenticator on plaintext over RFC1918 — both still allowed by the new rules). Phase 2 (libpq OAUTHBEARER + validator library, bundled into install/deploy.sh + k3d) handed off to Junie — see "In progress" at top of file.
- 2026-04-30 — **`pg_stat_statements` moved out of `public` schema across deploy modes** (commit pending). Studio's Database Advisor was flagging `public.pg_stat_statements` as a Security warning the moment any user opened the dashboard. Root cause: relocatable extensions without an explicit `SCHEMA` clause on `CREATE EXTENSION` land in the first writable schema in the connecting role's search_path, which is `public` for the `postgres` role. Live: `ALTER EXTENSION pg_stat_statements SET SCHEMA extensions;`. Bootstrap (3 files, all 4 deploy modes): [`deploy/gcp/gke/knoe-db.yaml`](../deploy/gcp/gke/knoe-db.yaml) (gke), [`deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml`](../deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml) (k3s), [`k8s/knoe/knoe-db.yaml`](../k8s/knoe/knoe-db.yaml) (k3d / min) — each `postInitTemplateSQL` block now creates the `extensions` schema and `CREATE EXTENSION ... SCHEMA extensions;`. Plus [`supabase/deploy.sh`](../supabase/deploy.sh) gets `GRANT USAGE ON SCHEMA extensions TO anon, authenticated, service_role;` so PostgREST clients (and the advisor itself) can resolve the schema. Other security checks verified clean (no tables in public without RLS, no SECURITY DEFINER without search_path, no auth.users-exposed views, no duplicate indexes, no extensions left in public). Performance side has 6 unindexed FKs in upstream `auth.*` and `storage.*` tables — captured as deferred (touching upstream-managed schemas is the wrong move; they'll get fixed in supabase-auth / supabase-storage releases).
- 2026-04-30 — **Studio post-launch fixes (storage permissions + /support stop-gap)** in commit [79a3f76](#):
- **Storage list-buckets unblocked.** `supabase_storage_admin` role was missing membership in `service_role` / `authenticated` / `anon`, so `SET LOCAL role = 'service_role'` (run by supabase-storage on every request) failed with PG `42501` at `guc.c:6936 call_string_check_hook`, surfaced as a misleading "new row violates row-level security policy" error. One GRANT applied live (`GRANT anon, authenticated, service_role TO supabase_storage_admin;`) and added durably to [`supabase/deploy.sh`](../supabase/deploy.sh) right after the existing `GRANT service_role TO authenticator` block.
- **oauth2-proxy auth-header passthrough fixed.** The `--pass-authorization-header=true` and `--set-authorization-header=true` flags were stomping the `Authorization: Bearer <apikey>` header that Studio / supabase-py send on `/storage/v1/...` calls (oauth2-proxy was rewriting it with the Google OIDC id_token, which storage couldn't validate against `JWT_SECRET`). Both flags removed from [`deploy/gcp/gke/oauth2-proxy-deployment.yaml`](../deploy/gcp/gke/oauth2-proxy-deployment.yaml). When per-user identity passthrough lands post-knoe-auth Round 1, use `--pass-user-headers` / `--set-xauthrequest` (X-Forwarded-* headers) instead — those don't collide with Supabase's auth.
- **`/support` route stop-gap.** `https://db.0.knoe.dev/support` 302s to `mailto:support@knoe.dev` so users have a working escape hatch while the in-Studio support buttons stay broken (separate Studio-fork follow-up #15). Kong service `support` added to [`templates/kong/config.yaml`](../supabase/helm/knoe-supabase/templates/kong/config.yaml) using `request-termination` (302) + `response-transformer` (Location header). `KONG_PLUGINS` allowlist extended with `response-transformer` in [`values.yaml`](../supabase/helm/knoe-supabase/values.yaml). oauth2-proxy gets `--skip-auth-route=^/support$` so an unauthenticated user looking for help isn't bounced into a Google sign-in detour.
- 2026-04-30 — **Unified `db.0.knoe.dev` for Studio + SDK** in commit [f805404](#): added two `--skip-auth-route` args to oauth2-proxy so SDK requests on `/auth/v1`, `/rest/v1`, `/realtime/v1`, `/storage/v1`, `/functions/v1`, `/graphql/v1`, `/pg/*` bypass the Google sign-in flow and pass through to supabase-kong, where Kong's existing `key-auth` plugin enforces anon/service-role keys. Studio UI at `/` remains Google-gated. One URL now covers humans-in-Studio AND robots-via-supabase-py — same security envelope as `api.0.knoe.dev` had on those paths. `api.0.knoe.dev` kept as a backward-compat alias (see follow-up #14). External verification: `curl -H "apikey: <KEY>" https://db.0.knoe.dev/rest/v1/` → 200 OpenAPI; `curl https://db.0.knoe.dev/rest/v1/` (no key) → 401 from Kong; `curl https://db.0.knoe.dev/` → 302 to Google (Studio gate intact); `curl https://db.0.knoe.dev/api/profile` → 302 to Google (Studio internals stay gated).
- 2026-04-30 — **Google sign-in to db.0.knoe.dev for chrisfu + ron (shared Studio)** in commit [25f1b2e](#): oauth2-proxy v7.15.2 deployed in front of supabase-kong as the Studio access gate (Google Workspace OIDC, `--email-domain=knoey.com`); supabase-kong Ingress patched so `db.0.knoe.dev` → `oauth2-proxy:80` while `api.0.knoe.dev` → `supabase-kong:8000` is untouched (key-auth still gates the API surface); Kong `dashboard` route's `basic-auth` plugin removed (oauth2-proxy is the gate now — double-auth was friction without security gain). External verification: db.0.knoe.dev/ → 302 `accounts.google.com/o/oauth2/auth?client_id=200699021716-…&redirect_uri=https://db.0.knoe.dev/oauth2/callback&scope=openid+email+profile`; api.0.knoe.dev/rest/v1/ → 401 from Kong key-auth (untouched); /healthz → 200. Three new files + chart edit: [`deploy/gcp/gke/oauth2-proxy-deployment.yaml`](../deploy/gcp/gke/oauth2-proxy-deployment.yaml), [`deploy/gcp/gke/oauth2-proxy-google-oidc-secret.example.yaml`](../deploy/gcp/gke/oauth2-proxy-google-oidc-secret.example.yaml), [`etc/init_oauth2_proxy.sh`](../etc/init_oauth2_proxy.sh), and the `basic-auth` plugin removed from [`supabase/helm/knoe-supabase/templates/kong/config.yaml`](../supabase/helm/knoe-supabase/templates/kong/config.yaml). Future swap to knoe-auth Round 1: change one arg in oauth2-proxy from `--provider=google` to `--provider=oidc --oidc-issuer-url=https://api.knoe.dev/auth` and reapply.
- 2026-04-29 — **CNPG operator + backup stabilization session** (4 queue items in one wave; live-cluster ops, no commit yet — doc-update commit pending):
- **#4 done:** plugin-barman-cloud operator restarted (`kubectl rollout restart deployment/barman-cloud -n cnpg-system`); stale Garage retention-worker endpoint cache cleared
- **#11 done:** cnpg-controller-manager restarted; clean plugin re-discovery confirmed in logs
- **CNPG operator upgraded v1.28.1 → v1.29.0** to expose `spec.serviceAccountName` field (added in v1.29.0 per upstream release notes); operator rollout clean, cluster healthy throughout
- **#2 done:** `cluster.spec.serviceAccountName=cnpg-backup-sa` applied; supervised rolling restart cycled all 3 pods (db-3 first failed with RBAC error — patched RoleBindings `knoe-db` and `knoe-db-barman-cloud` to add `cnpg-backup-sa` as a subject; db-3 then recovered, db-4 and db-2 rolled cleanly); 23:03 UTC scheduled backup completed with new SA, verifying end-to-end
- **#3 done:** Garage workload removed from `knoe-dev-cnpg-0` (statefulset, both services including `garage-s3-ilb`, configmap, PVC `data-garage-0`, StorageClass `garage-hdd`); ObjectStore untouched, backups still progressing post-removal
- 2026-04-29 — **Doc true-up + master TODO index** in commit [f5b24c7](#). Reality-vs-intent gaps captured across CLAUDE.md / AGENTS.md / pipeline-phases / plans / canonical ObjectStore manifest; new `docs/TODO.md` Kanban-style index.
- 2026-04-29 — **GCS migration completed.** ObjectStore `gs://knoe-0-backups/` with `googleCredentials.gkeEnvironment: true`; WI auth via `knoe-db` SA. 2 verified backups in catalog. ScheduledBackup unsuspended. (live-cluster ops; no commit)
- 2026-04-29 — **Garage cluster recovered.** PVC 29 → 80 GiB, layout 10 → 60 GB, bucket cleared (8382 objects + 12 orphan multiparts). (live-cluster ops; no commit)
- 2026-04-28/29 — **knoe-db-1 fix.** 14d crashloop resolved via PVC delete + CNPG re-bootstrap; cluster failover-cycled all 3 pods; 3/3 healthy. (live-cluster ops; no commit)
- 2026-04-28 — **Phase A repo cleanup + Phase C Maven build readiness** in commit [b355855](#). tmp/ archived (87 files); `org.knoe` → `dev.knoe` rename; `authority` → `auth` artifactId; sources moved to `dev/knoe/auth/`; `make build-auth` green. 4 k3s `prole-auth-*.yaml` manifests renamed to `knoe-auth-*.yaml`.
- 2026-04-28 — **Pipeline Phase 0** (test pipeline foundation: pyproject.toml, IntelliJ run configs, coverage fix, welcome mode selector). See [`docs/pipeline-phases.md`](pipeline-phases.md) §Phase 0.

3
docs/plans/README.md
View File

@ -19,6 +19,9 @@ If a sentence in a plan assumes knowledge you don't have, that's a bug in the pl
| --- | --- | --- | | --- | --- | --- |
| [`knoe-auth-round-1.md`](knoe-auth-round-1.md) | Identity backbone for the platform: MIT Kerberos KDC + invite-anchored web enrollment + TOTP 2FA. Round 1 of N. **Shipped.** | Before touching `authority/`, `etc/init_kdc.sh`, `etc/init_knoe_users.sh`, anything in `deploy/gcp/gke/knoe-auth-*` or `knoe-kdc-*`, or the `knoe.*` database schema. | | [`knoe-auth-round-1.md`](knoe-auth-round-1.md) | Identity backbone for the platform: MIT Kerberos KDC + invite-anchored web enrollment + TOTP 2FA. Round 1 of N. **Shipped.** | Before touching `authority/`, `etc/init_kdc.sh`, `etc/init_knoe_users.sh`, anything in `deploy/gcp/gke/knoe-auth-*` or `knoe-kdc-*`, or the `knoe.*` database schema. |
| [`deployment-modes.md`](deployment-modes.md) | Four-mode installer (`min` / `k3d` / `k3s` / `gke`) with a welcome-screen mode selector and a `min`-mode fast-path through the wizard. **Shipped in Phase 0.** | Before touching `knoe/ui/screens/welcome.py`, `knoe/ui/screens/navigation.py`, or adding any new wizard screen. | | [`deployment-modes.md`](deployment-modes.md) | Four-mode installer (`min` / `k3d` / `k3s` / `gke`) with a welcome-screen mode selector and a `min`-mode fast-path through the wizard. **Shipped in Phase 0.** | Before touching `knoe/ui/screens/welcome.py`, `knoe/ui/screens/navigation.py`, or adding any new wizard screen. |
| [`customer-deploy-resync.md`](customer-deploy-resync.md) | Original plan to converge `~/dev/prole` onto `knoe-db/main` as a customer-deploy branch. **Dormant** — prole rebrand merged into `main`; no separate prole working tree currently under development. | Only if you're considering activating a per-customer branching workflow. |
| [`junie/`](junie/README.md) (subdirectory) | Self-contained, single-task **work briefs** for Junie to consume from inside the IDE. Tactical, single-MR scope, paired against numbered items in `../TODO.md`. | When you want to hand a discrete task to Junie or audit what's been queued. |
| [`../TODO.md`](../TODO.md) (sibling) | Master TODO index — single source of truth for unfinished work, including reality-vs-intent gaps flagged in `CLAUDE.md`/`AGENTS.md`. | Before picking up any task, to see what's already on the queue. |
Each plan follows the same shape: **Context → How it's wired (file references) → Architecture → Schema / API → Step-by-step → Verification → Glossary.** Each plan follows the same shape: **Context → How it's wired (file references) → Architecture → Schema / API → Step-by-step → Verification → Glossary.**

128
docs/plans/junie/02-k3s-prole-rename.md Normal file
View File

@ -0,0 +1,128 @@
# Junie brief — Queue #2: k3s manifest cleanup, `prole-*` filename rename
> **Self-contained brief.** No prior session context required. Mechanical
> rename + a few content sweeps. Should land as a single commit.
---
## 1. Why
`deploy/opentofu/k3s/manifests/knoe/kustomization.yaml` resolves resources by
filename. After the prole→knoe rebrand, the kustomization was updated to
reference `knoe-{configmap,deployment,service,kdc-configmap}.yaml` — but the
files on disk still carry the `prole-` prefix. **Kustomize is broken on the
k3s deploy mode right now**: `kubectl apply -k deploy/opentofu/k3s/manifests/knoe`
errors with "no such file" on the missing `knoe-*.yaml` resources.
This is the same `git mv` pattern that landed the auth-side files in commit
[`b355855`](#) (the `prole-auth-*.yaml` → `knoe-auth-*.yaml` rename). Mirror
that approach.
## 2. The mapping
Files currently in `deploy/opentofu/k3s/manifests/knoe/`:
| Current filename | Rename to | Resource inside (verify) |
|---|---|---|
| `prole-configmap.yaml` | `knoe-configmap.yaml` | ConfigMap `knoe-nginx-config` |
| `prole-deployment.yaml` | `knoe-deployment.yaml` | Deployment `knoe` |
| `prole-service.yaml` | `knoe-service.yaml` | Service `knoe` |
| `prole-kdc-configmap.yaml` | `knoe-kdc-configmap.yaml` | ConfigMap `knoe-kdc-config` |
| `prole-kdc-secrets.example.yaml` | `knoe-kdc-secrets.example.yaml` | Secret template (not in kustomization — example only) |
Use `git mv` for all five so history is preserved.
## 3. Adjacent content sweeps to do in the same commit
Don't ship just the rename. While each file is open, scan for stale
`prole.local` / `PROLE.LOCAL` / `prole-*` references in the **content**:
- **`prole-kdc-configmap.yaml` (post-rename: `knoe-kdc-configmap.yaml`)** —
the embedded `krb5.conf` currently has `default_realm = PROLE.LOCAL`. The
Kerberos realm in `authority/` and the rest of the platform is
`KNOE.LOCAL` (or the production realm; verify by grepping
`default_realm` across the repo and matching what the running KDC pod
uses). Update accordingly. If you find a value other than `KNOE.LOCAL` is
in production use, ASK before changing — don't guess.
- **Any `host: prole.local` / `prole.org` ingress hostnames** in the renamed
files, or in `ingress.yaml` (which is already the right name). Cross-check
against the GKE manifests in `deploy/gcp/gke/` for the canonical
hostnames. If a k3s-only hostname is intentional (different deploy mode),
leave it but add a comment noting why.
- **`metadata.labels.app: knoe`** is already correct in the deployment +
service. Verify no `app: prole` label survives.
Anything that isn't a clear "this is stale, no question" — leave a TODO
comment with the specific question and link to this brief, rather than
making the call yourself.
## 4. Verification
1. **Kustomize dry-run resolves all resources cleanly:**
```bash
kubectl kustomize deploy/opentofu/k3s/manifests/knoe/ > /tmp/k3s-rendered.yaml
echo "exit: $?"
wc -l /tmp/k3s-rendered.yaml
```
No "file not found" errors. Output rendered manifest has all expected
resources (knoe Deployment, knoe Service, knoe-nginx-config CM, knoe-kdc-config CM, etc.).
2. **No stragglers:**
```bash
grep -rni 'prole' deploy/opentofu/k3s/manifests/knoe/
```
Expected output: nothing, or only intentional historical comments
(annotated with the reason).
3. **`git log --follow` works for one of the renamed files:**
```bash
git log --follow --oneline deploy/opentofu/k3s/manifests/knoe/knoe-configmap.yaml | head -5
```
Should show pre-rename history (proof `git mv` preserved it).
4. **Smoke test in k3d if available** (optional but nice): `make k3d-up`
followed by `kubectl apply -k deploy/opentofu/k3s/manifests/knoe/` — pods
should start. If k3d isn't available locally, skip; CI will catch it.
## 5. Commit message
```
chore(k3s): rename prole-*.yaml manifests to knoe-* and align contents
Mirrors commit b355855 (auth-side rename). The kustomization at
deploy/opentofu/k3s/manifests/knoe/kustomization.yaml has been
referencing knoe-* names since the rebrand; kustomize was broken
on the k3s deploy mode until this lands.
Five git mv:
prole-configmap.yaml -> knoe-configmap.yaml
prole-deployment.yaml -> knoe-deployment.yaml
prole-service.yaml -> knoe-service.yaml
prole-kdc-configmap.yaml -> knoe-kdc-configmap.yaml
prole-kdc-secrets.example.yaml-> knoe-kdc-secrets.example.yaml
Plus content sweeps for stale prole.local / PROLE.LOCAL realm
references in the embedded krb5.conf.
Closes queue item #2 in docs/TODO.md (drift R5).
```
## 6. Out of scope
- Image refs `knoe-authority` → `knoe-auth` in deployment manifests — that's
queue item **#3**, separate brief, separate commit. (Touches both
`deploy/gcp/gke/knoe-auth-deployment.yaml` and the renamed k3s
`knoe-auth-deployment.yaml`.)
- KDC realm value changes if `default_realm` is currently other than
`PROLE.LOCAL` in some files — ask before rewriting.
## 7. Definition of done
- [ ] Five `git mv` operations recorded in the commit (preserves history).
- [ ] `kubectl kustomize` produces a clean rendered manifest with no errors.
- [ ] `grep -rni 'prole' deploy/opentofu/k3s/manifests/knoe/` returns clean.
- [ ] Single commit; message starts with `chore(k3s):`.
- [ ] `docs/TODO.md` queue item #2 moved to **Done** section with date and
commit ref (or, if you prefer, leave that to whoever merges).

191
docs/plans/junie/06-patch-garage-script-fixes.md Normal file
View File

@ -0,0 +1,191 @@
# Junie brief — Queue #6: fix `scripts/patch_garage_cross_cluster.sh`
> **Self-contained brief.** Three concrete defects in the script + a related
> manifest split. Land as one or two commits (your call — see §5).
---
## 1. Why
`scripts/patch_garage_cross_cluster.sh` was written one-shot during the
2026-04-29 cluster recovery to migrate CNPG backups off Garage and onto
GCS+Workload-Identity. It worked at the time, but reading it back surfaced
three defects that would bite the next person who runs it. We want it to
be re-runnable cleanly (e.g. for a future second customer or DR rebuild)
without surprises.
Live-cluster context (do not change): GCS migration is complete, ObjectStore
is active, ScheduledBackup is plugin-method, three CNPG pods on
`knoe-dev-cnpg-0` are healthy. **You don't need to run the script to fix
it.** This is a code-only commit.
## 2. The three defects
### Defect A — stale `DB_CLUSTER` default
`scripts/patch_garage_cross_cluster.sh:39`
```bash
DB_CLUSTER="${DB_CLUSTER:-knoe-cnpg-0}"
```
Should be `knoe-dev-cnpg-0` (matches `conf/gke.cfg` line `init_cluster.db_cluster_kubecontext`
and `CLAUDE.md` cluster table). The current default resolves to the wrong
context and `kubectl` operations would silently target the wrong cluster
(or fail with "context not found", depending on the environment).
The script's `APP_CLUSTER="${APP_CLUSTER:-knoe-dev-0}"` on line 40 is
already correct.
Header comment on lines 16-18 also references the wrong name in the
`./etc/init_cnpg_gke.sh ... --cluster knoe-cnpg-0` example — fix that too.
### Defect B — Phase 1 misses `service/garage-s3-ilb`
`scripts/patch_garage_cross_cluster.sh:86-95` — Phase 1's deletion loop:
```bash
for resource in "statefulset/garage" "service/garage" "configmap/garage-config"; do
...
done
```
The loop deletes `service/garage` (the ClusterIP) but not
`service/garage-s3-ilb` (the LoadBalancer). The ObjectStore's
`endpointURL` historically pointed at the ILB, so leaving the LB orphaned
costs money and confuses cluster state. The 2026-04-29 cleanup did this
manually after the script ran.
**Fix**: add `"service/garage-s3-ilb"` to the loop's resource list. Order
doesn't matter (all `--ignore-not-found=true`).
### Defect C — Phase 2 applies a manifest containing both SA AND a legacy ScheduledBackup
`scripts/patch_garage_cross_cluster.sh:114-122` — Phase 2 applies
`deploy/gcp/gke/knoe-db-backup-gcs.yaml`. That manifest currently contains:
1. **ServiceAccount `cnpg-backup-sa`** — wanted; this is the WI-bound SA.
2. **ScheduledBackup `knoe-db-daily`** with `method: barmanObjectStore` —
**unwanted**. Two problems:
- Live cluster already has a plugin-method ScheduledBackup. Applying
this would create a duplicate that runs alongside the active one.
- `method: barmanObjectStore` (the legacy in-tree method) is being
removed in CNPG v1.30 per upstream release notes. New work should not
pin to it.
**Fix** (one of two paths — pick the cleanest):
**Path 1 (recommended): split the manifest.** Move the ScheduledBackup
out of `deploy/gcp/gke/knoe-db-backup-gcs.yaml` into its own file (or
delete it outright if there's no consumer — verify by `grep -r "knoe-db-daily"`
across the repo first; if nothing references it, just remove it). Keep
`knoe-db-backup-gcs.yaml` as the canonical SA-only manifest. The script's
Phase 2 stays unchanged (still applies the SA file).
**Path 2 (minimal): adjust the script only.** Use a kubectl filter so
Phase 2 only applies the ServiceAccount portion of the file:
```bash
GCP_PROJECT_ID="${GCP_PROJECT}" envsubst '${GCP_PROJECT_ID}' < "${GCS_MANIFEST}" \
| yq 'select(.kind == "ServiceAccount")' \
| kubectl --context="${DB_CTX}" apply -f -
```
(`yq` is already used elsewhere in `etc/`. If it's not in the script's
preflight `command -v` check, add it.)
**Path 1 is recommended** because it makes the manifest file's contents
match its filename's intent (`-gcs.yaml` should be GCS plumbing, not
backup scheduling), and removes the deprecated-method footgun for any
future runner.
If you take Path 1, also verify the live ScheduledBackup in
`knoe-db-0` namespace still exists and uses the plugin method — but **do
not run kubectl against the live cluster from the script change.** Just
inspect via `kubectl --context=$DB get scheduledbackup -n knoe-db-0 -o yaml`
once if you want to confirm; don't modify it.
### Defect D (bonus, mentioned in TODO) — Phase 3 unconditional re-apply
Phase 3 (lines 127-134) unconditionally re-applies
`k8s/knoe/knoe-db-barman-objectstore-gcs.yaml`. Manifest currently matches
live state, so this is harmless. The TODO suggests "worth a guard" — add
a `kubectl get objectstore knoe-db-barman-objectstore -n $DB_NS` check
that diffs against the manifest and skips re-apply if matching. **Skip
this one if it adds significant scope** — it's a nice-to-have, not a
defect.
## 3. What to verify
```bash
# 1) Defaults resolve correctly when no env vars set
bash -n scripts/patch_garage_cross_cluster.sh # syntax check
( unset DB_CLUSTER APP_CLUSTER GCP_PROJECT GCP_REGION SERVICE_NS DB_NS
source <(sed -n '36,50p' scripts/patch_garage_cross_cluster.sh)
echo "DB_CLUSTER=$DB_CLUSTER, DB_CTX=$DB_CTX" )
# Expected: DB_CLUSTER=knoe-dev-cnpg-0, DB_CTX=gke_..._knoe-dev-cnpg-0
# 2) Dry-run the script in CONFIRM=false mode (existing safety: prints the
# plan and exits 1) — verify the printed plan reads sensibly
./scripts/patch_garage_cross_cluster.sh
# Expected: prints "1. Remove Garage from knoe-dev-cnpg-0/knoe-system" etc.
# 3) Phase 1 deletion list now includes garage-s3-ilb
grep -A8 'for resource in' scripts/patch_garage_cross_cluster.sh
# Expected: list contains "service/garage-s3-ilb"
# 4) (If Path 1 chosen) the SA-only manifest has only ServiceAccount kind
grep '^kind:' deploy/gcp/gke/knoe-db-backup-gcs.yaml
# Expected: only "kind: ServiceAccount"
```
## 4. Out of scope
- Don't run the live script. This is a code-only fix. The script is for
someone migrating a fresh cluster; we're not migrating.
- Don't change any RBAC or IAM. The Workload Identity binding is correct
in the live cluster; this brief touches the script and (optionally) the
manifest split, nothing more.
- The CNPG SA wiring inside `etc/init_cnpg_gke.sh` (`cluster.spec.serviceAccountName`
+ extended RoleBindings) is queue item **#7** with its own brief. Don't
conflate.
## 5. Commit shape
If you take Path 1 (manifest split): two commits, atomic each:
1. `chore(deploy): split knoe-db-backup-gcs manifest — SA only, drop legacy ScheduledBackup`
2. `fix(scripts): patch_garage_cross_cluster defaults + missing garage-s3-ilb`
If you take Path 2 (script-only): one commit.
```
fix(scripts): patch_garage_cross_cluster — three defects from 2026-04-29 review
Defects A, B, C from docs/plans/junie/06-patch-garage-script-fixes.md:
A. DB_CLUSTER default was knoe-cnpg-0 (stale rebrand artifact);
should be knoe-dev-cnpg-0 to match conf/gke.cfg.
B. Phase 1 deletion loop missed service/garage-s3-ilb (the
LoadBalancer the ObjectStore endpoint historically pointed at).
Now included.
C. Phase 2 was applying knoe-db-backup-gcs.yaml whole, including a
legacy ScheduledBackup using method:barmanObjectStore (being
removed in CNPG v1.30). Manifest split: SA stays in
knoe-db-backup-gcs.yaml; ScheduledBackup removed.
Header comment + usage example also updated for the rename.
Closes queue item #6 in docs/TODO.md (drift R9).
```
## 6. Definition of done
- [ ] `DB_CLUSTER` default + header comment updated.
- [ ] Phase 1 loop includes `service/garage-s3-ilb`.
- [ ] Phase 2 doesn't apply a legacy ScheduledBackup (Path 1 or 2).
- [ ] `bash -n` clean; CONFIRM=false dry-run prints sensible plan.
- [ ] If Path 1: `kind: ServiceAccount` is the only kind in
`deploy/gcp/gke/knoe-db-backup-gcs.yaml`.
- [ ] `docs/TODO.md` queue item #6 + drift R9 row archived to **Done** with
date and commit ref.

215
docs/plans/junie/07-init-cnpg-gke-sa-wiring.md Normal file
View File

@ -0,0 +1,215 @@
# Junie brief — Queue #7: wire `cnpg-backup-sa` into the CNPG cluster spec via `etc/init_cnpg_gke.sh`
> **Self-contained brief.** Drift R8: live cluster state has the SA wired
> through, the init script doesn't. Make the script land a fresh deploy in
> the same end state. Live cluster is healthy — do **not** run anything
> against it.
---
## 1. Why
`etc/init_cnpg_gke.sh` runs from a clean state (e.g. for a future second
DB cluster, or DR rebuild) and lays down everything: GCS buckets, GCP
service account, Workload Identity binding, K8s service account, namespace,
CNPG operator, secrets, the CNPG `Cluster` resource. The 2026-04-29
stabilization session added two pieces that the script doesn't yet
reproduce:
1. **`cluster.spec.serviceAccountName: cnpg-backup-sa`** on the live
cluster. This field was added in CNPG **v1.29.0**. The init script
currently pins CNPG **v1.24.0** (line 402:
`cnpg-1.24.0.yaml`), which doesn't support the field — the cluster
would either ignore it or fail validation. Bump the operator version
AND add the field to the rendered manifest.
2. **RoleBindings extended** with `cnpg-backup-sa` as a subject. Live
`knoe-db` and `knoe-db-barman-cloud` RoleBindings (auto-created by
CNPG at cluster creation) each have **two** subjects:
`knoe-db` (the default SA CNPG creates) and `cnpg-backup-sa`. With v1.29+
and `spec.serviceAccountName` set from the start, CNPG should
auto-create RoleBindings naming the explicit SA — but verify, because
the live state was patched after the fact.
The result of this brief: a fresh `init_cnpg_gke.sh` run produces a
cluster whose RBAC and SA wiring matches `kubectl get cluster knoe-db -o yaml`
+ `kubectl get rolebinding -n knoe-db-0` from the live system, with no
post-install patches required.
## 2. Live state to match
```bash
DB=gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0
kubectl --context=$DB -n knoe-db-0 get cluster knoe-db -o jsonpath='{.spec.serviceAccountName}'
# -> cnpg-backup-sa
kubectl --context=$DB -n knoe-db-0 get rolebinding knoe-db -o jsonpath='{.subjects}' | jq
# -> [{kind:ServiceAccount, name:knoe-db, namespace:knoe-db-0},
# {kind:ServiceAccount, name:cnpg-backup-sa, namespace:knoe-db-0}]
kubectl --context=$DB -n knoe-db-0 get rolebinding knoe-db-barman-cloud -o jsonpath='{.subjects}' | jq
# -> same two subjects
```
## 3. What changes
### 3.1 — `etc/init_cnpg_gke.sh`
Around line 402 (CNPG operator install):
```bash
# Current (stale)
kubectl apply --server-side -f \
https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/releases/cnpg-1.24.0.yaml
```
Bump to **v1.29.0** (or a newer stable; verify against
[CNPG releases](https://github.com/cloudnative-pg/cloudnative-pg/releases)
that the version supports `Cluster.spec.serviceAccountName`). Add a code
comment explaining the version constraint:
```bash
# v1.29.0+ is required: spec.serviceAccountName lets the cluster pods run
# as cnpg-backup-sa (annotated with iam.gke.io/gcp-service-account for WI),
# which is how the GCS-backed barman ObjectStore authenticates without a
# static key. Older operators (we used 1.24.0 historically) would silently
# drop the field.
CNPG_OPERATOR_VERSION="1.29.0"
kubectl apply --server-side -f \
"https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/releases/cnpg-${CNPG_OPERATOR_VERSION}.yaml"
```
(Promote the version to a top-of-file variable so future bumps are one
line.)
### 3.2 — `deploy/gcp/gke/knoe-db.yaml` (CNPG `Cluster` manifest)
Add `serviceAccountName: cnpg-backup-sa` under `spec:`. Use the same
indentation as the surrounding fields. Place near the top of `spec` (right
after `description` / `imageName` / `instances`, before `bootstrap` —
follow the live cluster's field order if you want to be tidy:
`kubectl get cluster knoe-db -o yaml`).
Add an inline comment:
```yaml
# Run cluster pods as cnpg-backup-sa (annotated for Workload Identity to
# the cnpg-backup@... GCP SA). This is how barman-cloud authenticates
# to gs://knoe-0-backups/ without a static key. Requires CNPG v1.29+.
# The SA is provisioned by etc/init_cnpg_gke.sh § "Apply ServiceAccount
# + annotate with WI".
serviceAccountName: cnpg-backup-sa
```
### 3.3 — RoleBinding subjects (decide path)
Two paths, depending on what CNPG v1.29 actually does at cluster
creation time. Test both before deciding.
**Path A — operator does the right thing automatically.** With
`spec.serviceAccountName: cnpg-backup-sa` set from the start, CNPG v1.29+
auto-creates `knoe-db` and `knoe-db-barman-cloud` RoleBindings with
`cnpg-backup-sa` as the (sole) subject. If true: nothing more to do in the
script — the live two-subject state is just a historical artifact of
having patched after-the-fact, and a fresh install would have a single
subject.
**Path B — operator still creates with a default SA name.** If CNPG
auto-creates the RoleBindings with subject `knoe-db` (the default SA name
matching the cluster name) regardless of the explicit
`serviceAccountName`, then the script needs to patch them. Add a section
after cluster apply:
```bash
# ── 11. Ensure RoleBindings reference cnpg-backup-sa ─────────────────────────
log "Ensuring CNPG-managed RoleBindings include cnpg-backup-sa ..."
for rb in "${CNPG_CLUSTER_NAME}" "${CNPG_CLUSTER_NAME}-barman-cloud"; do
# Wait briefly for CNPG to create them on first install
for _ in 1 2 3 4 5; do
kubectl -n "$CNPG_NAMESPACE" get rolebinding "$rb" >/dev/null 2>&1 && break
sleep 2
done
if kubectl -n "$CNPG_NAMESPACE" get rolebinding "$rb" >/dev/null 2>&1; then
if ! kubectl -n "$CNPG_NAMESPACE" get rolebinding "$rb" -o jsonpath='{.subjects[*].name}' \
| grep -qw cnpg-backup-sa; then
kubectl -n "$CNPG_NAMESPACE" patch rolebinding "$rb" --type='json' -p="$(cat <<'EOF'
[{"op":"add","path":"/subjects/-","value":{"kind":"ServiceAccount","name":"cnpg-backup-sa","namespace":"NAMESPACE_PLACEHOLDER"}}]
EOF
)" --patch-file=/dev/stdin 2>/dev/null || true
fi
fi
done
```
(Sketch — finalize the syntax. The `NAMESPACE_PLACEHOLDER` would be
substituted; `--patch-file` doesn't accept `-` in all kubectl versions, so
you may need a temp file.)
**How to decide A vs B**: spin up a throwaway k3d cluster (or a CNPG-only
test namespace on `knoe-dev-cnpg-0` if you can without disturbing
`knoe-db-0`), apply a minimal `Cluster` resource with
`serviceAccountName: cnpg-backup-sa`, observe the RoleBinding subjects
that CNPG auto-creates. If the SA you specified is the only subject:
Path A. Otherwise: Path B.
If you can't easily test, lean toward Path B (the patch is idempotent and
costs nothing if the operator already did the right thing).
## 4. Verification
Without running against the live cluster:
1. **Manifest renders cleanly:**
```bash
ARTIFACT_REGISTRY=us-west3-docker.pkg.dev/plenary-truck-485623-p7/knoe-system \
KNOE_DB_IMAGE_TAG=18-055 \
envsubst '${ARTIFACT_REGISTRY} ${KNOE_DB_IMAGE_TAG}' < deploy/gcp/gke/knoe-db.yaml \
| yq '.spec.serviceAccountName'
# Expected: cnpg-backup-sa
```
2. **Operator version is 1.29+:**
```bash
grep -E 'cnpg-[0-9]' etc/init_cnpg_gke.sh
# Expected: cnpg-1.29.0.yaml (or newer)
```
3. **`bash -n etc/init_cnpg_gke.sh`** clean.
4. **(Optional) Dry-run apply against k3d** if you have one running. The
cluster won't reach Healthy in k3d (no Workload Identity), but
`kubectl get cluster -o yaml` should show `spec.serviceAccountName:
cnpg-backup-sa` set.
## 5. Out of scope
- Don't touch the live cluster. This is purely script + manifest work.
- Don't change the `cnpg-backup-sa` annotation pattern in
`deploy/gcp/gke/knoe-db-backup-gcs.yaml` — that's how the SA gets the
WI annotation, separate concern.
- The `patch_garage_cross_cluster.sh` defects are queue item **#6** with
its own brief. If your changes here surface a discrepancy with that
brief, flag it but don't try to fix both in one commit.
- The PodMonitor manual-management migration (queue #13) doesn't interact
with this brief.
## 6. Commit shape
Likely two commits:
1. `chore(deploy): add serviceAccountName: cnpg-backup-sa to knoe-db cluster spec`
2. `fix(init): bump CNPG operator to v1.29 + ensure RoleBindings include cnpg-backup-sa`
Or one combined commit titled
`fix(init): wire cnpg-backup-sa as cluster SA + bump CNPG to v1.29 (drift R8)`.
## 7. Definition of done
- [ ] `etc/init_cnpg_gke.sh` references CNPG operator v1.29.0+ with a
comment explaining why (the SA field requirement).
- [ ] `deploy/gcp/gke/knoe-db.yaml` carries
`spec.serviceAccountName: cnpg-backup-sa` with explanatory comment.
- [ ] RoleBinding handling: either documented as auto-correct (Path A,
with a one-liner test result included in the commit message), OR
scripted patch (Path B).
- [ ] `bash -n` passes; manifest renders.
- [ ] `docs/TODO.md` queue item #7 + drift R8 row archived to **Done**.

277
docs/plans/junie/13-podmonitor-manual-management.md Normal file
View File

@ -0,0 +1,277 @@
# Junie brief — Queue #13: migrate off CNPG-deprecated `enablePodMonitor` + `podMonitorRelabelings`
> **Self-contained brief.** Manage the PodMonitor as a sibling resource
> instead of through CNPG operator fields. Closes a deprecation warning
> we get on every `kubectl patch` of the cluster spec.
---
## 1. Why
CNPG operator (running v1.29.0 on `knoe-dev-cnpg-0`) emits these warnings
on every patch of `Cluster.spec.monitoring`:
```
Warning: spec.monitoring.enablePodMonitor is deprecated and will be removed
in a future release. Please migrate to manually managing your PodMonitor
resources. Set this field to false and create a PodMonitor resource for
your cluster as described in the documentation.
Warning: spec.monitoring.podMonitorRelabelings is deprecated and will be
removed in a future release. Please migrate to manually managing your
PodMonitor resources with custom relabeling configurations.
```
Migration is straightforward: write the PodMonitor as a sibling YAML doc,
apply it alongside the Cluster, set `enablePodMonitor: false` (or remove
the whole `monitoring` block once that's also been deprecated). No
functional change — just lifts the relabel config out of the CNPG-managed
field and into a standard prometheus-operator resource.
The cnpg-grafana dashboard relies on the `cluster` label; **breaking the
relabel breaks the dashboard**, so verify carefully.
## 2. Live state to match
```bash
DB=gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0
kubectl --context=$DB -n knoe-db-0 get podmonitor knoe-db -o yaml
```
The CNPG operator currently auto-creates this PodMonitor. Relevant fields:
```yaml
spec:
namespaceSelector: {}
podMetricsEndpoints:
- port: metrics
relabelings:
- action: replace
sourceLabels: [__meta_kubernetes_pod_label_cnpg_io_cluster]
targetLabel: cluster
selector:
matchLabels:
cnpg.io/cluster: knoe-db
cnpg.io/podRole: instance
```
You'll write this manifest into the repo, lose the CNPG-managed metadata
(labels/annotations/ownerReferences) since you own it now, and apply it
yourself.
## 3. What changes
### 3.1 — New file: `deploy/gcp/gke/knoe-db-podmonitor.yaml`
```yaml
---
# Manually-managed PodMonitor for the CNPG `knoe-db` cluster.
#
# Migrated 2026-05-XX from the (now-deprecated) CNPG fields
# `Cluster.spec.monitoring.enablePodMonitor` +
# `Cluster.spec.monitoring.podMonitorRelabelings`. Same scrape spec, same
# relabel config — now living as a first-class resource we control.
#
# The selector matches every CNPG instance pod via the operator-applied
# labels `cnpg.io/cluster=knoe-db` + `cnpg.io/podRole=instance`. The
# `metrics` port is the postgres-exporter (sidecar) on the CNPG pod.
#
# The relabel injects a `cluster` label on every scraped sample, sourced
# from the `cnpg.io/cluster` pod label. The cnpg-grafana dashboards filter
# every panel by `cluster=$cluster`; without this relabel only the
# operator-collector metrics carry the cluster label and 83 of 85 panels
# render empty.
#
# Apply alongside `knoe-db.yaml` from etc/init_cnpg_gke.sh.
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: knoe-db
namespace: knoe-db-0
labels:
# Match what kube-prometheus-stack discovers via Helm-managed Prometheus
# serviceMonitorSelector / podMonitorSelector. kps's defaults match
# release=kps; verify in the live cluster:
# kubectl -n monitoring get prometheus -o yaml | grep -A2 podMonitorSelector
# If the selector requires a specific label, set it here.
release: kps
spec:
namespaceSelector: {}
selector:
matchLabels:
cnpg.io/cluster: knoe-db
cnpg.io/podRole: instance
podMetricsEndpoints:
- port: metrics
relabelings:
- action: replace
sourceLabels:
- __meta_kubernetes_pod_label_cnpg_io_cluster
targetLabel: cluster
```
**Verify the `release: kps` label trick.** Inspect the live Prometheus's
`podMonitorSelector` and ensure your manifest matches it. The current
auto-created PodMonitor doesn't carry `release: kps` (it uses
`app.kubernetes.io/managed-by: cloudnative-pg`), but it still gets
discovered — meaning the kps Prometheus must be configured with
`podMonitorSelector: {}` (any) or `podMonitorSelectorNilUsesHelmValues:
false` (which is what `monitoring/kps-cnpg-values.yaml` sets). Confirm
this and document in the manifest comment whether the `release: kps`
label is actually required.
### 3.2 — `deploy/gcp/gke/knoe-db.yaml`
Replace the entire `monitoring:` block:
```yaml
monitoring:
# NOTE: enablePodMonitor + podMonitorRelabelings are CNPG-deprecated and
# will be removed in a future release. Migration path: manually manage
# the PodMonitor resource. Tracked in docs/TODO.md as a follow-up.
enablePodMonitor: true
podMonitorRelabelings:
- sourceLabels: ["__meta_kubernetes_pod_label_cnpg_io_cluster"]
targetLabel: cluster
action: replace
```
with:
```yaml
monitoring:
# PodMonitor is managed manually as a sibling resource:
# deploy/gcp/gke/knoe-db-podmonitor.yaml. The CNPG-managed alternative
# (enablePodMonitor + podMonitorRelabelings) is deprecated.
enablePodMonitor: false
```
### 3.3 — `etc/init_cnpg_gke.sh`
Find the section that applies `knoe-db.yaml` (search for
`render_cnpg_manifest "$storage_class" "$tmp_manifest"`). After the
cluster apply succeeds (and after the cluster reaches a working storage
class), apply the PodMonitor:
```bash
# ── 11. Apply the manually-managed PodMonitor ────────────────────────────────
log "Applying PodMonitor for $CNPG_CLUSTER_NAME ..."
kubectl apply -f "$GKE_MANIFEST_DIR/knoe-db-podmonitor.yaml"
```
Place this AFTER the cluster apply completes (since the PodMonitor doesn't
depend on the cluster being healthy — it just needs the namespace) but
BEFORE any verification waits.
Also: when CNPG-managed PodMonitor was set, the operator auto-deleted it
when `enablePodMonitor` flipped to false. Verify what happens during the
transition on a live cluster: when you change `enablePodMonitor: true →
false` AND apply your manual PodMonitor with the same name in the same
namespace, does CNPG try to delete your manual one? Based on
`ownerReferences` on the auto-created PodMonitor, the answer should be
"the auto-created one gets deleted by garbage collection when the
controller flag flips, and your manually-applied one (no ownerReference)
survives". Test this on a non-prod cluster before claiming victory in the
commit message.
### 3.4 — Cross-mode parity (k3s + k8s/min)
Both `deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml` and
`k8s/knoe/knoe-db.yaml` have `monitoring.enablePodMonitor: false` already
— they don't run the dashboard, so no PodMonitor at all today. **Leave
those modes alone for now.** They'll need their own PodMonitor manifest if
we ever wire monitoring there, but that's separate work. Add a comment to
each pointing at the GKE manifest as the reference if/when needed.
## 4. Verification
End-to-end — must pass before merging:
1. **Apply the new manifest doesn't 404:**
```bash
kubectl --dry-run=server -f deploy/gcp/gke/knoe-db-podmonitor.yaml apply
```
2. **Cluster spec change doesn't break operator:**
```bash
yq '.spec.monitoring' deploy/gcp/gke/knoe-db.yaml
# Expected: only enablePodMonitor: false (no podMonitorRelabelings)
```
3. **Live transition test (the scary one).** On the live cluster, do the
transition AND verify dashboard still works:
```bash
DB=gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0
APP=gke_plenary-truck-485623-p7_us-west3_knoe-dev-0
# 1. Apply the new manual PodMonitor
kubectl --context=$DB apply -f deploy/gcp/gke/knoe-db-podmonitor.yaml
# 2. Verify it's there
kubectl --context=$DB -n knoe-db-0 get podmonitor
# 3. Apply the cluster spec change (flips enablePodMonitor false)
kubectl --context=$DB apply -f deploy/gcp/gke/knoe-db.yaml
# (or however the script renders + applies it)
# 4. Confirm only ONE PodMonitor remains (the manual one) — the
# CNPG-owned one should garbage-collect once the field flips
kubectl --context=$DB -n knoe-db-0 get podmonitor
# 5. Wait 30s and confirm DB-cluster Prometheus still has scrape targets:
kubectl --context=$APP -n monitoring exec sts/kps-grafana -c grafana -- \
wget -qO- "http://admin:$(kubectl --context=$APP -n monitoring get secret kps-grafana -o jsonpath='{.data.admin-password}' | base64 -d)@localhost:3000/api/datasources/proxy/uid/P5531627C358300FE/api/v1/targets?state=active" \
| python3 -c 'import sys,json; t=json.load(sys.stdin)["data"]["activeTargets"]; print(f"{len(t)} active targets, postgres exporters: {sum(1 for x in t if \"9187\" in x[\"scrapeUrl\"])}")'
# Expected: ≥3 postgres exporters active (our 3 CNPG pods)
# 6. Refresh the cnpg-grafana dashboard, confirm panels still populate
```
4. **Rollback path documented.** If step 3.5 shows dropped targets, the
rollback is to flip `enablePodMonitor: true` and delete the manual
PodMonitor. Note this in the commit message.
## 5. Out of scope
- Any work on `monitoring/kps-cnpg-values.yaml` itself.
`podMonitorSelectorNilUsesHelmValues: false` stays as is.
- Cross-mode PodMonitor (k3s/k8s/min) — defer.
- The `cluster=knoe-db` label vs. dynamic templating — currently hardcoded
to `knoe-db`. If a second CNPG cluster ever lands, the manual PodMonitor
needs to be parameterized. Leave a TODO comment but don't solve.
- Don't touch the cnpg-grafana dashboard transforms
(`monitoring/cnpg-dashboard-transforms.yaml`) — those are unrelated.
## 6. Commit shape
```
feat(monitoring): manage knoe-db PodMonitor manually (drop CNPG-deprecated fields)
CNPG operator emits deprecation warnings on every cluster.spec.monitoring
patch:
- enablePodMonitor will be removed in a future release
- podMonitorRelabelings will be removed in a future release
Migrate to a manually-managed PodMonitor:
+ deploy/gcp/gke/knoe-db-podmonitor.yaml (NEW — same spec as auto-created)
~ deploy/gcp/gke/knoe-db.yaml (enablePodMonitor: true → false,
podMonitorRelabelings dropped)
~ etc/init_cnpg_gke.sh (apply PodMonitor after cluster)
Verified live: cnpg-grafana dashboard panels render unchanged after the
flip; 3 postgres-exporter targets active in cnpg-prometheus.
Closes queue item #13 in docs/TODO.md.
```
## 7. Definition of done
- [ ] `deploy/gcp/gke/knoe-db-podmonitor.yaml` created, content verified
against live PodMonitor spec.
- [ ] `deploy/gcp/gke/knoe-db.yaml` no longer contains
`podMonitorRelabelings`; `enablePodMonitor: false` with comment.
- [ ] `etc/init_cnpg_gke.sh` applies the new manifest after cluster apply.
- [ ] Live cnpg-grafana dashboard verified after the transition: panels
populate, 3 postgres-exporter targets in Prometheus.
- [ ] No new deprecation warnings on `kubectl apply` of the cluster manifest.
- [ ] `docs/TODO.md` queue item #13 archived to **Done** with verification
notes.

249
docs/plans/junie/15-remove-dead-dashboard-consumer.md Normal file
View File

@ -0,0 +1,249 @@
# Junie brief — Queue #15: remove dead `DASHBOARD` consumer + `basicauth_credentials` from supabase Kong chart
> **Self-contained brief.** Pure cleanup. The basic-auth plugin block was
> removed in commit [`25f1b2e`](#) (oauth2-proxy → Kong fronting); the
> consumer + credentials it gated have no enforcer. **Watch the OpenAI key
> trap** — it lives in the same `secret.dashboard` block in
> `values.yaml`, so don't blanket-delete the section.
---
## 1. Why
Before commit [`25f1b2e`](#) (2026-04-30, oauth2-proxy fronting), Studio
was gated by Kong's `basic-auth` plugin against credentials in
`secret.dashboard.{username,password}`. That gate was replaced by
oauth2-proxy + Google OAuth, so the `basic-auth` plugin reference was
removed from the Kong route definition.
Three artifacts of the old gate remain:
1. **Kong consumer named `DASHBOARD`** in
`supabase/helm/knoe-supabase/templates/kong/config.yaml` — has no
plugin enforcing it. Dead.
2. **`basicauth_credentials` block** in the same file — provides
credentials for a consumer no plugin is checking. Dead.
3. **`DASHBOARD_USERNAME` / `DASHBOARD_PASSWORD` env vars** flowing through
the Kong deployment + the wrapper-script's envsubst. Dead.
The `secret.dashboard.{username,password}` values in `values.yaml`
themselves are unused after this cleanup, but the **`secret.dashboard.openAiApiKey`
field is NOT dead** — it's consumed by
`templates/studio/deployment.yaml:92` to populate `OPENAI_API_KEY` in the
Studio pod env. **Don't delete the whole `secret.dashboard` block.**
Restructure carefully so the OpenAI key still has a home.
## 2. The four artifacts to remove
### Artifact 1 — `supabase/helm/knoe-supabase/templates/kong/config.yaml`
**Lines 19-20** (the wrapper-script's sed replacements):
```yaml
-e "s|\${DASHBOARD_USERNAME}|${DASHBOARD_USERNAME}|" \
-e "s|\${DASHBOARD_PASSWORD}|${DASHBOARD_PASSWORD}|" \
```
Delete both lines.
**Lines 30-32** (the consumer):
```yaml
{{- if .Values.secret.dashboard }}
- username: DASHBOARD
{{- end }}
```
Delete all three lines (the `if`-end pair around the consumer).
**Lines 44-49** (the credentials block):
```yaml
{{- if .Values.secret.dashboard }}
basicauth_credentials:
- consumer: DASHBOARD
username: ${DASHBOARD_USERNAME}
password: ${DASHBOARD_PASSWORD}
{{- end }}
```
Delete all six lines.
### Artifact 2 — `supabase/helm/knoe-supabase/templates/kong/deployment.yaml`
**Lines 74-92** (the two env-var blocks): both `DASHBOARD_USERNAME` and
`DASHBOARD_PASSWORD` env-from-secret references. They're both wrapped in
`{{- if .Values.secret.dashboard }} ... {{- end }}`. Delete the whole
conditional block (both env vars, the wrapper, the inner conditional for
`secretRef` vs default secret).
After deletion, verify the surrounding env list still parses (Kong
deployment manifest must still be a valid Pod spec).
### Artifact 3 — `supabase/helm/knoe-supabase/values.yaml`
**Lines 104-118** (the `secret.dashboard` section):
```yaml
## Studio dashboard credentials
##
dashboard:
username: "supabase"
password: "this_password_is_insecure_and_should_be_updated"
openAiApiKey: "key_super_secret"
## Reference to existing secret
# secretRef: ""
## Map to actual keys inside secretRef if they differ
# secretRefKey:
# username: username
# password: password
# openAiApiKey: openAiApiKey
```
**Restructure** — don't delete. Two paths:
**Path A (recommended): rename the section to `openai`.** Remove
`username`/`password`. Rename `dashboard:` to `openai:`. Update the one
consumer (`templates/studio/deployment.yaml:92`) to read from
`.Values.secret.openai.apiKey` (renamed from `openAiApiKey`). Update the
secretRefKey mapping similarly. Cleanest.
```yaml
## OpenAI API key — passed to Studio's AI Assist sidebar.
## Optional; if absent, the AI Assist panel renders an error toast but
## the rest of Studio works fine. We default to a placeholder string so
## helm template doesn't fail; replace via secretRef in production.
openai:
apiKey: "key_super_secret"
# secretRef: ""
# secretRefKey:
# apiKey: apiKey
```
**Path B (minimal): drop username/password only.** Keep the section name
`dashboard:` since renaming touches more files. End up with a `dashboard:`
section that holds only `openAiApiKey`. Slightly confusing name vs.
contents, but the diff is smaller.
Pick Path A unless the touch surface scares you. The `if .Values.secret.dashboard`
conditional in the kong-deployment env block goes away in Path A
naturally; in Path B, replace those `if`s with explicit
`if .Values.secret.dashboard.username` / `.password` checks before
removing the env vars (since the block-level check used to mean "any
field set"; now it must be field-specific).
### Artifact 4 — `supabase/helm/knoe-supabase/templates/studio/deployment.yaml`
If you take Path A above, update the env-var binding at line 92:
```yaml
- name: OPENAI_API_KEY
valueFrom:
secretKeyRef:
{{- if .Values.secret.openai.secretRef }}
name: {{ .Values.secret.openai.secretRef }}
key: {{ .Values.secret.openai.secretRefKey.apiKey | default "apiKey" }}
{{- else }}
name: {{ include "supabase.secret.openai" . }}
key: apiKey
{{- end }}
```
(rename `dashboard` → `openai`, `openAiApiKey` → `apiKey`).
Also rename or re-source the helper template `supabase.secret.dashboard`
(if it exists in `_helpers.tpl`) to `supabase.secret.openai`. If the
helper doesn't exist (the grep earlier returned no match), skip.
### Artifact 5 — `supabase/helm/knoe-supabase/ci/example.yaml`
Has a parallel `secret.dashboard.openAiApiKey` for chart CI. Update to
match Path A or B.
## 3. Don't break
- **OpenAI API key flow**: pre-and-post-cleanup the Studio pod must
receive `OPENAI_API_KEY` from a Secret, identically. Verify with
`helm template` before vs. after.
- **Existing live deployments using `secretRef`** — if any production
deploy uses `secret.dashboard.secretRef` to point at an existing K8s
Secret with `username`/`password`/`openAiApiKey` keys, that secret
needs renaming too. Grep for `secret.dashboard.secretRef` in any
real values overrides (not just the chart defaults). If found,
ASK before proceeding.
## 4. Verification
1. **`helm template` the chart and diff for OPENAI_API_KEY:**
```bash
helm template supabase/helm/knoe-supabase \
| grep -A 6 'name: OPENAI_API_KEY'
```
Expected: env var still binds to a Secret. The Secret name + key may
differ from before (if you renamed); the **flow** must still resolve.
2. **`helm template` the chart and confirm DASHBOARD is gone:**
```bash
helm template supabase/helm/knoe-supabase | grep -i DASHBOARD
```
Expected: no output.
3. **`helm lint`** the chart:
```bash
helm lint supabase/helm/knoe-supabase
```
Expected: clean.
4. **No stale references in the rest of the repo:**
```bash
grep -rni DASHBOARD_USERNAME supabase/ k8s/ deploy/ etc/
grep -rni DASHBOARD_PASSWORD supabase/ k8s/ deploy/ etc/
```
Expected: no output (or only docs explaining the old flow that you
then update / remove).
5. **OpenAI key still works in Studio**: hard to verify without rolling
out, but a `helm template` showing the env var still bound is
sufficient for chart correctness. Live re-roll of supabase-studio is
out of scope; flag it as a manual follow-up if anything changes about
the secret name.
## 5. Out of scope
- **Don't roll out the chart change to live `knoe-dev-0`.** Chart
cleanup is code-only; the current live deployment uses the OLD chart
(with `secret.dashboard`), and migrating production secret structure
is a separate decision.
- The Studio fork (queue item #12) — totally separate.
- Any other Kong route changes — leave the file alone outside the
surgical lines listed.
## 6. Commit shape
If Path A:
```
chore(supabase): remove dead DASHBOARD consumer + basic-auth credentials
The basic-auth plugin was retired in commit 25f1b2e (oauth2-proxy
took over Studio gating via Google OAuth). Three artifacts of the
old gate remained as no-op config:
- Kong consumer named DASHBOARD
- basicauth_credentials block
- DASHBOARD_USERNAME / DASHBOARD_PASSWORD env in kong deployment
All removed.
The secret.dashboard.openAiApiKey field — the only LIVE consumer
of secret.dashboard — was migrated to secret.openai.apiKey. Studio's
OPENAI_API_KEY env binding updated; helm template diff confirms
the same Secret-source flow.
Closes queue item #15 in docs/TODO.md.
```
If Path B: simpler subject — `chore(supabase): drop dead DASHBOARD username/password`.
## 7. Definition of done
- [ ] Four artifacts removed from `kong/config.yaml`,
`kong/deployment.yaml`, `values.yaml`, `ci/example.yaml`.
- [ ] OpenAI key still resolves through `helm template`.
- [ ] `helm lint` clean.
- [ ] `grep -rni DASHBOARD_USERNAME` returns nothing in repo.
- [ ] `docs/TODO.md` queue item #15 archived to **Done**.

66
docs/plans/junie/README.md Normal file
View File

@ -0,0 +1,66 @@
# Junie briefs
Self-contained, single-task briefs for [Junie](https://www.jetbrains.com/junie/)
to consume from inside the IntelliJ IDE. Each file is a hand-off — Junie
reads the brief cold (no shared chat history), implements the change, and
opens an MR.
## What lives here vs. `../`
The `..` parent directory holds **architectural plans** — multi-section
documents covering an initiative's strategy, schema, and component design.
They outlive the implementation and become the architectural reference
once shipped.
This subdirectory holds **work briefs** — smaller, more tactical, scoped
to a single MR's worth of changes. They reference the master TODO index
(`../../TODO.md`) and contain enough context for Junie to land the change
without escalating questions.
## Brief shape (convention)
Each brief follows this skeleton:
1. **Why** — one or two paragraphs of context, including the trigger event
if any (e.g. "the 4/28 14:00 UTC outage").
2. **What changes** — concrete file paths, line numbers, before/after where
useful. Don't make Junie re-derive the change.
3. **Verification** — runnable commands or `helm template` diffs that
confirm the change. Each brief ends with a **Definition of done**
checklist.
4. **Out of scope** — explicit fences. The TODO is interconnected; without
this section briefs creep.
5. **Commit shape** — proposed commit message + structure.
## Status tracking
Every brief here corresponds to a numbered item in
[`../../TODO.md`](../../TODO.md) (or has an explicit "doesn't yet exist
in TODO" note). When Junie lands a brief:
1. Move the corresponding item from the TODO ranked queue into the
**Done** section with date + commit ref (or the convention used by
the rest of the file).
2. Don't delete the brief from this directory — it stays as the design
record.
## Current briefs (as of 2026-05-01)
| File | Queue # | Subject |
|---|---|---|
| [`02-k3s-prole-rename.md`](02-k3s-prole-rename.md) | #2 | Rename k3s `prole-*.yaml` → `knoe-*.yaml` (kustomize is broken) |
| [`06-patch-garage-script-fixes.md`](06-patch-garage-script-fixes.md) | #6 | Three defects in `scripts/patch_garage_cross_cluster.sh` |
| [`07-init-cnpg-gke-sa-wiring.md`](07-init-cnpg-gke-sa-wiring.md) | #7 | Wire `cnpg-backup-sa` into CNPG cluster spec; bump operator to v1.29 |
| [`13-podmonitor-manual-management.md`](13-podmonitor-manual-management.md) | #13 | Migrate off CNPG-deprecated `enablePodMonitor` + `podMonitorRelabelings` |
| [`15-remove-dead-dashboard-consumer.md`](15-remove-dead-dashboard-consumer.md) | #15 | Remove dead Kong DASHBOARD consumer + `basicauth_credentials` |
## How a session fires off a batch
The driving session (Claude Code, Cowork+Code, or a human) writes the
briefs into this directory and points Junie at one or more of them. Junie
reads the brief, implements, runs the verification checklist, opens an
MR. Each brief is independent — Junie can take them in any order, or in
parallel across separate IDE sessions.
The brief is the contract. If something is unclear, the brief is buggy
and should be edited before Junie continues.