docs(plans): k3d-mirror-of-GKE plan + Phase 1 brief for Junie

The next development thread is knoe-auth Phase 2 OIDC iteration, which just landed on main but has no fast inner-loop. Instead of bouncing edits through the GKE deploy chain, we want a laptop-resident model where knoe-auth runs from the IDE / mvn against real Postgres + KDC brought up in k3d. Two new docs: docs/plans/k3d-gke-mirror.md Architectural plan. Top-to-bottom rationale for what the k3d model should mirror, what we deliberately skip (supabase, kong, oauth2-proxy, Studio — all unnecessary for knoe-auth's surfaces), and the phasing. Captures the three scope decisions taken 2026-05-02 with the user: - knoe-auth runs on the laptop, not as a pod (faster inner loop) - stack scope is CNPG + KDC only - single-replica CNPG, realm KNOE.LOCAL (distinct from KNOE.DEV) Out-of-scope for Phase 1 listed explicitly so reviewer expectations match scope: SPNEGO E2E, image-build-and-load, supabase, oauth2-proxy, OidcCodeService DB persistence. docs/plans/junie/k3d-knoe-auth-dev-loop.md Phase 1 brief. Concrete deliverables: - k8s/knoe/knoe-kdc-{deployment,service,configmap,pvc,init-job}.yaml - etc/init_knoe_auth.sh --mode k3d flag (or sibling script) - Makefile: k3d-knoe-{up,pf,down} - etc/krb5.local.conf (with udp_preference_limit=1 for kubectl port-forward UDP flakiness on macOS) - docs/local-dev-knoe-auth.md (engineer-facing) - scripts/k3d-knoe-smoke.sh Definition of done: a fresh-clone laptop can `make k3d-knoe-up` + `make k3d-knoe-pf` + `mvn -pl authority spring-boot:run` and hit /health, /.well-known/openid-configuration in <8 minutes. Index updates: docs/plans/README.md — k3d-gke-mirror.md row added docs/plans/junie/README.md — split into Active / Shipped; k3d brief listed under Active; the 5 shipped-2026-05-02 briefs moved to Shipped with commit refs. docs/TODO.md In-progress — k3d brief now top of "In progress"; the previously in-flight "Phase 2 pg_oauth in install/deploy.sh" item demoted to "paused" (resume after the local dev loop lands). No code changes; all docs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-09-23 12:03:59 +00:00 · 2026-05-02 04:45:28 -07:00 · 2026-05-02 04:45:28 -07:00 · 6d484ef13e
commit 6d484ef13e
parent 232981c18c
5 changed files with 595 additions and 7 deletions
--- a/docs/TODO.md
+++ b/docs/TODO.md
@ -13,7 +13,9 @@ The Kanban "Now" section at top is the only place this doc imposes structure. Ev
 ## Now (Kanban)
 ### In progress
- **Phase 2: pg_oauth in install/deploy.sh + k3d build** — assigned to Junie. The Phase 1 work (per-engineer roles, external LB at `pg.0.knoe.dev`, cert SANs, pg_hba tightening, [`docs/db-access.md`](db-access.md)) is committed. Phase 2 replaces the SCRAM password mechanism with PG18 native OAUTHBEARER, wires it into `install.sh` / `supabase/deploy.sh` so a fresh deploy lands with the OIDC mechanism active by default, and ports the Service + cluster cert pattern to k3d. Connection target stays at `pg.0.knoe.dev`; only the credential issuance changes (libpq Device Flow → Google Bearer token). Plan reference: `~/.claude/plans/we-re-continuing-work-on-happy-toast.md`.
+- **k3d-mirror-of-GKE Phase 1: laptop dev loop for knoe-auth** — assigned to Junie. Stand up the smallest k3d-resident stack (CNPG single-replica + standalone KDC, realm `KNOE.LOCAL`) that lets a host-side knoe-auth (via `mvn spring-boot:run` / IntelliJ) iterate against real Postgres + Kerberos. Architectural plan: [`docs/plans/k3d-gke-mirror.md`](plans/k3d-gke-mirror.md). Phase 1 brief: [`docs/plans/junie/k3d-knoe-auth-dev-loop.md`](plans/junie/k3d-knoe-auth-dev-loop.md). Deliverables: `k8s/knoe/knoe-kdc-*.yaml` manifests, `etc/init_knoe_auth.sh` `--mode k3d` flag, `make k3d-knoe-{up,pf,down}` targets, `etc/krb5.local.conf`, `docs/local-dev-knoe-auth.md`, `scripts/k3d-knoe-smoke.sh`. Out of scope (Phase 2+): SPNEGO E2E from browsers, knoe-auth-as-pod image builds, supabase stack on k3d, `OidcCodeService` DB persistence.
 - **Phase 2: pg_oauth in install/deploy.sh + k3d build** — *previously assigned to Junie; paused.* The Phase 1 work (per-engineer roles, external LB at `pg.0.knoe.dev`, cert SANs, pg_hba tightening, [`docs/db-access.md`](db-access.md)) is committed. Phase 2 replaces the SCRAM password mechanism with PG18 native OAUTHBEARER, wires it into `install.sh` / `supabase/deploy.sh` so a fresh deploy lands with the OIDC mechanism active by default, and ports the Service + cluster cert pattern to k3d. Connection target stays at `pg.0.knoe.dev`; only the credential issuance changes (libpq Device Flow → Google Bearer token). Park while the k3d dev loop above lands; resume once knoe-auth Phase 2 OIDC dev has somewhere to run locally.
 ### Up next
 *(empty — pull from the ranked queue below)*
--- a/docs/plans/README.md
+++ b/docs/plans/README.md
@ -19,6 +19,7 @@ If a sentence in a plan assumes knowledge you don't have, that's a bug in the pl
 | --- | --- | --- |
 | [`knoe-auth-round-1.md`](knoe-auth-round-1.md) | Identity backbone for the platform: MIT Kerberos KDC + invite-anchored web enrollment + TOTP 2FA. Round 1 of N. **Shipped.** | Before touching `authority/`, `etc/init_kdc.sh`, `etc/init_knoe_users.sh`, anything in `deploy/gcp/gke/knoe-auth-*` or `knoe-kdc-*`, or the `knoe.*` database schema. |
 | [`deployment-modes.md`](deployment-modes.md) | Four-mode installer (`min` / `k3d` / `k3s` / `gke`) with a welcome-screen mode selector and a `min`-mode fast-path through the wizard. **Shipped in Phase 0.** | Before touching `knoe/ui/screens/welcome.py`, `knoe/ui/screens/navigation.py`, or adding any new wizard screen. |
 | [`k3d-gke-mirror.md`](k3d-gke-mirror.md) | Laptop-resident model of the GKE platform — minimum-viable k3d stack (CNPG + KDC) that lets host-side knoe-auth iterate against real Postgres + Kerberos. **Active. Phase 1 in flight (Junie).** | Before touching `k8s/knoe/knoe-kdc-*`, the `make k3d-knoe-*` targets, or `docs/local-dev-knoe-auth.md`. Phase 1 brief at [`junie/k3d-knoe-auth-dev-loop.md`](junie/k3d-knoe-auth-dev-loop.md). |
 | [`customer-deploy-resync.md`](customer-deploy-resync.md) | Original plan to converge `~/dev/prole` onto `knoe-db/main` as a customer-deploy branch. **Dormant** — prole rebrand merged into `main`; no separate prole working tree currently under development. | Only if you're considering activating a per-customer branching workflow. |
 | [`junie/`](junie/README.md) (subdirectory) | Self-contained, single-task **work briefs** for Junie to consume from inside the IDE. Tactical, single-MR scope, paired against numbered items in `../TODO.md`. | When you want to hand a discrete task to Junie or audit what's been queued. |
 | [`../TODO.md`](../TODO.md) (sibling) | Master TODO index — single source of truth for unfinished work, including reality-vs-intent gaps flagged in `CLAUDE.md`/`AGENTS.md`. | Before picking up any task, to see what's already on the queue. |
--- a/docs/plans/junie/README.md
+++ b/docs/plans/junie/README.md
@ -44,15 +44,23 @@ in TODO" note). When Junie lands a brief:
 2. Don't delete the brief from this directory — it stays as the design
   record.
-## Current briefs (as of 2026-05-01)
+## Current briefs (as of 2026-05-02)
 ### Active (in flight)
 | File | Tracked at | Subject |
 |---|---|---|
 | [`k3d-knoe-auth-dev-loop.md`](k3d-knoe-auth-dev-loop.md) | TODO §"In progress"; parent [`../k3d-gke-mirror.md`](../k3d-gke-mirror.md) | Phase 1 of the k3d-mirror-of-GKE plan: stand up CNPG + KDC in k3d so host-side knoe-auth can iterate against real Postgres + Kerberos |
 ### Shipped (kept as design record)
 | File | Queue # | Subject |
 |---|---|---|
-| [`02-k3s-prole-rename.md`](02-k3s-prole-rename.md) | #2 | Rename k3s `prole-*.yaml` → `knoe-*.yaml` (kustomize is broken) |
+| [`02-k3s-prole-rename.md`](02-k3s-prole-rename.md) | #2 | Rename k3s `prole-*.yaml` → `knoe-*.yaml` (kustomize is broken). **Shipped 2026-05-02 (commit `fb7e8b7`).** |
-| [`06-patch-garage-script-fixes.md`](06-patch-garage-script-fixes.md) | #6 | Three defects in `scripts/patch_garage_cross_cluster.sh` |
+| [`06-patch-garage-script-fixes.md`](06-patch-garage-script-fixes.md) | #6 | Three defects in `scripts/patch_garage_cross_cluster.sh`. **Shipped 2026-05-02 (commits `34a25dd` + `5d17325`).** |
-| [`07-init-cnpg-gke-sa-wiring.md`](07-init-cnpg-gke-sa-wiring.md) | #7 | Wire `cnpg-backup-sa` into CNPG cluster spec; bump operator to v1.29 |
+| [`07-init-cnpg-gke-sa-wiring.md`](07-init-cnpg-gke-sa-wiring.md) | #7 | Wire `cnpg-backup-sa` into CNPG cluster spec; bump operator to v1.29. **Shipped 2026-05-02 (commit `c3fae73`).** |
-| [`13-podmonitor-manual-management.md`](13-podmonitor-manual-management.md) | #13 | Migrate off CNPG-deprecated `enablePodMonitor` + `podMonitorRelabelings` |
+| [`13-podmonitor-manual-management.md`](13-podmonitor-manual-management.md) | #13 | Migrate off CNPG-deprecated `enablePodMonitor` + `podMonitorRelabelings`. **Shipped 2026-05-02 (commit `c3fae73`).** |
-| [`15-remove-dead-dashboard-consumer.md`](15-remove-dead-dashboard-consumer.md) | #15 | Remove dead Kong DASHBOARD consumer + `basicauth_credentials` |
+| [`15-remove-dead-dashboard-consumer.md`](15-remove-dead-dashboard-consumer.md) | #15 | Remove dead Kong DASHBOARD consumer + `basicauth_credentials`. **Shipped 2026-05-02 (commit `c3fae73`).** |
 ## How a session fires off a batch
--- a/docs/plans/junie/k3d-knoe-auth-dev-loop.md
+++ b/docs/plans/junie/k3d-knoe-auth-dev-loop.md
@ -0,0 +1,319 @@
 # Junie brief — Phase 1 of k3d-mirror-of-GKE: laptop dev loop for knoe-auth
 > **Self-contained brief.** Reference: parent architectural plan at
 > [`../k3d-gke-mirror.md`](../k3d-gke-mirror.md). Read that for the
 > "why" and the multi-phase shape. This brief delivers Phase 1 only.
 ---
 ## 1. Why
 knoe-auth Phase 2 OIDC code is on `main` but no fast inner-loop
 exists for iterating on it. Hitting the GKE cluster on every change
 is slow. Running unit tests with testcontainers covers a lot but
 doesn't exercise the real DB schema, Kerberos KDC, or the actual HTTP
 surface against a Postgres+JDBC stack.
 This brief wires up a **laptop-resident dev loop**: bring up CNPG +
 KDC in k3d, port-forward 5432/88/464 to localhost, and let the
 engineer run knoe-auth from their IDE/`mvn spring-boot:run` against
 those local ports. Edit Java → re-run → see the change at
 `http://localhost:8080`, with a real PostgreSQL and a real Kerberos
 backing it.
 knoe-auth itself stays on the host. Image-build-and-load into k3d is
 a separate later phase (and is in fact mostly orthogonal to this
 work — Phase 1 finishes when the host loop is solid).
 ## 2. What you produce
 Concrete deliverables, all in one MR:
 ### 2.1 — KDC manifest set for k3d
 New files (mirroring the auth-side filename pattern under `k8s/knoe/`):
 - `k8s/knoe/knoe-kdc-deployment.yaml` — single-replica `Deployment`
  running `krb5kdc` + `kadmind`. Realm `KNOE.LOCAL`. Use a
  Debian-derived MIT Kerberos image; whatever the GKE sidecar image
  uses is fine to reuse. **Set `realm = KNOE.LOCAL`** explicitly —
  do NOT inherit from the existing `KNOE.DEV` GKE config.
 - `k8s/knoe/knoe-kdc-service.yaml` — ClusterIP service exposing 88
  (TCP+UDP) and 464 (TCP+UDP — kpasswd).
 - `k8s/knoe/knoe-kdc-configmap.yaml` — `krb5.conf` + `kdc.conf` with
  realm `KNOE.LOCAL` and a `[realms] KNOE.LOCAL = { kdc = knoe-kdc:88 ... }`
  block.
 - `k8s/knoe/knoe-kdc-pvc.yaml` — small PVC (1 Gi) for the KDC
  database `/var/lib/krb5kdc/principal`. Local-path provisioner
  (k3d default).
 - `k8s/knoe/knoe-kdc-init-job.yaml` — one-shot `Job` that runs
  `kdb5_util create -s -P <stash>` (database init) and then
  `kadmin.local addprinc -randkey admin/admin@KNOE.LOCAL` plus a
  developer principal. Idempotent (skip create if DB exists). Runs
  once on first apply; subsequent apply is a no-op via a guard
  inside the script.
 Reference existing GKE `kdc.conf` and `krb5.conf` shapes from
 [`deploy/gcp/gke/knoe-kdc-configmap.yaml`](../../deploy/gcp/gke/knoe-kdc-configmap.yaml).
 Substitute the realm, drop GCE-specific bits.
 ### 2.2 — Init-script entry point for k3d
 Either:
 - **(preferred)** Extend `etc/init_knoe_auth.sh` with a `--mode k3d`
  flag that swaps the `APP_CTX` default to `k3d-knoe` (or whatever
  `kubectl config current-context` returns from the k3d-up flow),
  swaps the realm to `KNOE.LOCAL`, and skips the GCP/Workload
  Identity steps (no GCP project on a laptop). Same `schema` /
  `invite` / `status` subcommands work.
 - **(or)** Write a thin `etc/init_knoe_auth_k3d.sh` that wraps the
  existing one with the right env vars set.
 Pick whichever results in less duplication. The existing script's
 `schema` subcommand already runs the SQL via `kubectl exec` on the
 CNPG primary — that path works in k3d unchanged.
 ### 2.3 — Make targets
 Three new targets in the top-level `Makefile`:
 - **`make k3d-knoe-up`** — provisions the local k3d cluster (if not
  present), installs the CNPG operator, applies
  `k8s/knoe/knoe-db.yaml` (single-replica per the architectural
  plan §3), waits for `Cluster in healthy state`, runs the
  knoe-kdc init job, applies the KDC Deployment + Service, runs
  `etc/init_knoe_auth.sh schema` to seed `knoe.*` tables.
 - **`make k3d-knoe-pf`** — opens three port-forwards in the
  foreground:
    - `5432` → `service/knoe-db-rw -n knoe-db-0`
    - `88`/`464` → `service/knoe-kdc -n knoe-system` (TCP and UDP both)
  - Prints the JDBC URL and the env vars the engineer should
    `export` (e.g. `KRB5_CONFIG=$PWD/etc/krb5.local.conf`,
    `KNOE_DB_PASSWORD=...`).
  - On ^C, cleans up all three forwards.
 - **`make k3d-knoe-down`** — tears down the k3d cluster (or just
  the namespaces if you want incremental cleanup).
 The cluster name should be `k3d-knoe` (so the kubeconfig context is
 `k3d-k3d-knoe`, matching k3d's prefix convention). Reuse anything in
 `scripts/` that already drives k3d if it's there.
 ### 2.4 — Engineer-side krb5 config
 A new file `etc/krb5.local.conf` checked into the repo:
 ```ini
 [libdefaults]
  default_realm = KNOE.LOCAL
  rdns = false
  forwardable = true
  udp_preference_limit = 1   # workaround for kubectl port-forward UDP flakiness; force TCP
 [realms]
  KNOE.LOCAL = {
    kdc = localhost:88
    admin_server = localhost:749
    default_domain = local
  }
 [domain_realm]
  .local = KNOE.LOCAL
  localhost = KNOE.LOCAL
 ```
 Engineer's runtime: `export KRB5_CONFIG=$PWD/etc/krb5.local.conf` and
 the JVM picks it up automatically.
 ### 2.5 — Engineer-facing doc
 New `docs/local-dev-knoe-auth.md`. Sections:
 1. **Prerequisites** — k3d, kubectl, Docker Desktop running, JDK 21,
   Maven. Brew one-liner.
 2. **One-time setup** — `make k3d-knoe-up`. Wait ~3 minutes. Verify
   `kubectl --context k3d-k3d-knoe -n knoe-db-0 get cluster knoe-db`
   shows `Healthy`.
 3. **Daily loop** — open two terminals.
   - Terminal A: `make k3d-knoe-pf` (leave running).
   - Terminal B: `export KRB5_CONFIG=$PWD/etc/krb5.local.conf` then
     `mvn -pl authority spring-boot:run`.
 4. **Verify the loop** — `curl localhost:8080/health` returns `ok`;
   `curl localhost:8080/.well-known/openid-configuration` returns
   the OIDC discovery JSON; `psql
   "postgresql://postgres:$(make show-db-password)@localhost:5432/knoe-db?sslmode=require"`
   connects.
 5. **`kinit` for SPNEGO testing** — `kinit -k -t <keytab>
   admin/admin@KNOE.LOCAL` (full SPNEGO E2E is Phase 2; document
   the expected limitations: `curl --negotiate -u : http://localhost:8080/`
   may or may not fully round-trip on macOS depending on
   `udp_preference_limit`).
 6. **IntelliJ run configuration** — paste-ready text or
   screenshot. Module `authority`, main class
   `dev.knoe.auth.KnoeAuthApplication`, env vars
   `KRB5_CONFIG=…/etc/krb5.local.conf`,
   `KNOE_DB_PASSWORD=…`.
 7. **Reset** — `make k3d-knoe-down && make k3d-knoe-up`. Time: ~3 min.
 ### 2.6 — Smoke-test script
 `scripts/k3d-knoe-smoke.sh` — bash, runs after `make k3d-knoe-up`
 to confirm the loop is intact:
 ```bash
 #!/usr/bin/env bash
 set -euo pipefail
 ctx=k3d-k3d-knoe
 ns_db=knoe-db-0
 ns_sys=knoe-system
 # 1. CNPG cluster healthy
 phase=$(kubectl --context=$ctx -n $ns_db get cluster knoe-db -o jsonpath='{.status.phase}')
 [[ "$phase" == "Cluster in healthy state" ]] || { echo "CNPG: $phase"; exit 1; }
 # 2. knoe.* schema present
 tables=$(kubectl --context=$ctx -n $ns_db exec sts/knoe-db-1 -c postgres -- \
  psql -U postgres -tAc "SELECT count(*) FROM information_schema.tables WHERE table_schema='knoe'")
 [[ "$tables" -ge 6 ]] || { echo "knoe.* tables: $tables (expected ≥6)"; exit 1; }
 # 3. KDC has the seed admin principal
 kubectl --context=$ctx -n $ns_sys exec deploy/knoe-kdc -- \
  kadmin.local listprincs | grep -q 'admin/admin@KNOE.LOCAL' \
  || { echo "missing admin principal"; exit 1; }
 echo "k3d-knoe-smoke: PASS"
 ```
 CI integration is out of scope for Phase 1 — this is engineer-run.
 ## 3. Don't break
 - The existing GKE knoe-auth deployment must continue to work
  unchanged. If you touch `etc/init_knoe_auth.sh`, the GKE
  invocation must remain the default behavior (no `--mode` flag =
  current GKE behavior).
 - The k3s manifests under `deploy/opentofu/k3s/manifests/knoe/`
  must NOT be modified. The k3s deploy mode targets a different
  use case (on-prem VM); leave it alone.
 - The CNPG cluster manifest at `k8s/knoe/knoe-db.yaml` must stay
  the canonical k3d/min Postgres definition. If you need it to
  behave differently for k3d (e.g. single-replica), make the change
  configurable, not destructive — `_strip_k3d_synology_blocks` in
  `knoe/core/ops/cloudnative_pg.py` is the existing precedent for
  k3d-specific manifest manipulation.
 ## 4. Verification (Definition of done)
 A new engineer, freshly cloning the repo, completes this in ≤ 8
 minutes on a MacBook Air:
 ```bash
 git clone <repo> && cd knoe-db
 make k3d-knoe-up                            # 3-5 min
 make k3d-knoe-pf &                          # foreground; backgrounded for this script
 sleep 3
 scripts/k3d-knoe-smoke.sh                   # PASS
 export KRB5_CONFIG=$PWD/etc/krb5.local.conf
 mvn -pl authority spring-boot:run &         # backgrounded for the verify
 sleep 30
 curl -fsS http://localhost:8080/health      # ok
 curl -fsS http://localhost:8080/.well-known/openid-configuration | jq .issuer
 # → "http://localhost:8080"
 ```
 Tick all of:
 - [ ] `make k3d-knoe-up` exits 0 on a fresh laptop with k3d/kubectl
  installed.
 - [ ] `kubectl --context=k3d-k3d-knoe -n knoe-db-0 get cluster knoe-db`
  reports `Healthy`.
 - [ ] `kubectl --context=k3d-k3d-knoe -n knoe-system get deploy knoe-kdc`
  reports `1/1 ready`.
 - [ ] `scripts/k3d-knoe-smoke.sh` reports `PASS`.
 - [ ] Host-side `psql` connects via `localhost:5432` after
  `make k3d-knoe-pf` is running.
 - [ ] `kinit admin/admin@KNOE.LOCAL` from the host succeeds.
 - [ ] `mvn -pl authority spring-boot:run` starts cleanly with
  `KRB5_CONFIG=etc/krb5.local.conf` set; `/health` returns `ok`;
  `/.well-known/openid-configuration` returns valid JSON.
 - [ ] `make k3d-knoe-down && make k3d-knoe-up` is idempotent.
 - [ ] `docs/local-dev-knoe-auth.md` exists and is accurate.
 - [ ] GKE deploy mode unchanged: `git diff main -- deploy/gcp/gke/`
  shows no changes; the existing init scripts behave identically
  when invoked the same way they were before.
 ## 5. Out of scope
 Captured in the parent plan §6, but call out the highest-friction
 deferrals here so reviewer expectations are right:
 - **SPNEGO E2E from a host browser** — requires a service principal
  for `HTTP/localhost@KNOE.LOCAL` and a keytab the host knoe-auth
  process can read. Phase 1 stops at "KDC reachable, principals
  exist, `kinit` works." Phase 2 makes browser SPNEGO actually
  authenticate.
 - **knoe-auth as a pod inside k3d** — image-build-and-load. Phase 3.
  Don't take this on now.
 - **Supabase / Studio / Kong / oauth2-proxy on k3d** — all explicit
  Phase 4+.
 - **OIDC code persistence in DB** — `OidcCodeService` uses an
  in-memory `ConcurrentHashMap` today. That's a real Phase 2.5 gap
  but it's the same gap on GKE; not k3d-specific. Don't fold it
  into this brief.
 ## 6. Commit shape
 Single commit unless the deliverables genuinely separate into two
 clean halves (e.g. infra + docs). Suggested message:
 ```
 feat(k3d): laptop dev loop for knoe-auth — CNPG + KDC + port-forward
 Brings up the smallest k3d-resident stack that lets a host-side
 knoe-auth (run via mvn spring-boot:run or IntelliJ) iterate against
 real Postgres + Kerberos. Closes Phase 1 of the k3d-gke-mirror plan.
 Scope:
  - k8s/knoe/knoe-kdc-{deployment,service,configmap,pvc,init-job}.yaml
    NEW; standalone KDC, realm KNOE.LOCAL (distinct from KNOE.DEV).
  - etc/init_knoe_auth.sh: --mode k3d flag, swaps realm + skips
    GCP-specific steps. GKE behavior unchanged when flag absent.
  - Makefile: k3d-knoe-up, k3d-knoe-pf, k3d-knoe-down.
  - etc/krb5.local.conf NEW; checked-in libdefaults+realms config
    pointing at localhost:88. udp_preference_limit=1 to dodge
    kubectl port-forward UDP flakiness.
  - docs/local-dev-knoe-auth.md NEW; one-time setup + daily loop.
  - scripts/k3d-knoe-smoke.sh NEW; bash sanity script.
 Verified by following docs/local-dev-knoe-auth.md from a fresh
 clone: end-to-end in <8 min, knoe-auth at localhost:8080 hits
 real DB + KDC; OIDC discovery returns valid JSON.
 Out of scope (parent plan docs/plans/k3d-gke-mirror.md §6):
  - SPNEGO from host browsers
  - knoe-auth-as-pod (image build/load)
  - Supabase stack
  - OidcCodeService DB persistence
 Closes Phase 1; Phase 2+ briefs filed as needed.
 ```
 ## 7. Notes for reading
 - The user is happy to **decide between two paths if you surface
  them clearly** — e.g. "extend `init_knoe_auth.sh` with `--mode
  k3d` vs write a sibling `init_knoe_auth_k3d.sh`". Pick the one
  that minimizes drift and explain why in the commit.
 - The user runs macOS. UDP port-forward through kubectl on Mac has
  historically been finicky; setting `udp_preference_limit = 1` in
  `krb5.local.conf` (force TCP) is the recommended workaround and is
  why it's in the example above.
 - If you run into resource ceiling issues on the laptop (Docker
  Desktop OOM, k3d node not ready), document the workaround in the
  engineer-side doc rather than working around it in the manifests.
 - If the existing Python orchestrator
  (`knoe.core.ops.cloudnative_pg`) already handles the
  k3d-CNPG-bring-up cleanly, leverage it rather than writing a
  shell wrapper from scratch. The Make target can call into it via
  `python -m knoe.core.ops.cloudnative_pg ...` if the entrypoints
  exist; if they don't, a bash wrapper is fine.
--- a/docs/plans/k3d-gke-mirror.md
+++ b/docs/plans/k3d-gke-mirror.md
@ -0,0 +1,258 @@
 # k3d-mirror-of-GKE — local platform model for knoe-auth dev
 > **Active plan. Phase 1 in flight (Junie).** Strategic context for a
 > running initiative; reads top-to-bottom. Phase 1 brief at
 > [`junie/k3d-knoe-auth-dev-loop.md`](junie/k3d-knoe-auth-dev-loop.md).
 ---
 ## 1. Why
 knoe-auth Phase 2 (OIDC provider — discovery, authorize, token, userinfo,
 JWKS, RS256 signing) just landed on `main` (via the merge that brought
 `claude/crazy-bose-fec256` back in). It builds; it hasn't been deployed
 to the cluster yet. The next development steps — wiring OAuth2 code
 exchange to a persistent table (Phase 2.5), the
 ProvisioningWorker→GitLab/Gitea/CNPG dispatch surface, future Round 2
 work — all benefit from a tight inner-loop that doesn't require:
 - a GKE cluster to push to,
 - a deploy pipeline to wait on,
 - a real Google OAuth tenant to bounce through, or
 - the full supabase + kong + oauth2-proxy + studio stack just to test
  what knoe-auth does.
 What we want is a **laptop dev loop**: edit Java, run from IntelliJ
 (or `mvn spring-boot:run`), break-pointing freely, against a real
 PostgreSQL with the canonical `knoe.*` schema and a real Kerberos KDC
 the SPNEGO codepath can talk to.
 The four-mode installer we already ship has `k3d` as one of those modes
 (see [`deployment-modes.md`](deployment-modes.md)). It brings up CNPG
 today. It does **not** yet bring up the bits knoe-auth needs alongside.
 This plan closes that gap deliberately, in phases.
 ## 2. Where the modes stand today
 | Mode | DB | knoe-auth | KDC | Supabase | Studio | OAuth2-proxy |
 |---|---|---|---|---|---|---|
 | `gke` | CNPG (3-replica) | ✅ deployed | ✅ sidecar | ✅ | ✅ | ✅ Google OIDC |
 | `k3s` | CNPG (single) | ✅ deployed (manifests at `deploy/opentofu/k3s/manifests/knoe/knoe-auth-*`) | ✅ kdc-configmap | partial | partial | ❌ |
 | `k3d` | ✅ CNPG via `k8s/knoe/knoe-db.yaml` | ❌ no manifests | ❌ no manifests | ❌ | ❌ | ❌ |
 | `min` | ❌ no Postgres | ❌ | ❌ | ❌ | ❌ | ❌ |
 The k3s mode has the auth-stack manifests but is targeted at the
 on-prem-VM deploy pattern (talks to a single-node k3s on a customer
 machine), not the laptop-dev-loop pattern. Reusing them for k3d is
 attractive but the wiring assumptions differ enough (LB types,
 ingress class, service-account identity) that copy-and-adapt is
 easier than refactor-to-share.
 ## 3. Strategic shape
 The **goal** is not a full GKE mirror — that would mean Studio + Kong
 + supabase + oauth2-proxy + everything, and the laptop fan would not
 be happy. The goal is **the smallest k3d-resident stack that lets a
 host-resident knoe-auth talk to it end-to-end**.
 Decisions (set 2026-05-02 with the user):
 1. **knoe-auth runs on the host, not in the cluster.** `mvn
   spring-boot:run` (or IntelliJ Run Configuration) starts the process
   on the laptop. It connects to the k3d cluster's DB and KDC via
   `kubectl port-forward`. This gives:
   - hot reload via Spring DevTools,
   - direct debugger attach,
   - immediate stack traces in the same terminal as the build,
   - no image-build-and-load cycle on every change.
   Trade-off: this isn't testing the deployed-pod-shaped behaviour
   (mounted secrets, SA identity, sidecar networking). That's
   covered by the GKE deploy when it's time to go live, and a future
   "in-cluster smoke" Phase 3 brief if we want pre-merge checks too.
 2. **Stack scope is CNPG + KDC only.** No supabase, no Kong, no
   oauth2-proxy, no Studio, no garage/MinIO. knoe-auth doesn't
   depend on any of those for its own surfaces (`/health`,
   `/auth/enroll/*`, `/auth/admin/*`, OIDC discovery/authorize/
   token/userinfo/jwks). The ProvisioningWorker reaches OUT to
   GitLab/Gitea/CNPG — those are mockable or deferrable for dev.
 3. **Single-replica CNPG.** Three-replica is GKE-shaped for
   availability; on a laptop it's wasteful. One instance, the same
   PG18 image, same `postInitTemplateSQL` schema bootstrap, same
   `knoe.*` tables.
 4. **Kerberos realm `KNOE.LOCAL`.** Distinct from production
   `KNOE.DEV` so credentials don't accidentally cross-talk. Pre-seed
   one admin principal and one developer principal so the engineer
   can `kinit` immediately.
 5. **Plain `sslmode=require` for libpq from host.** Production uses
   `verify-full` against the CNPG-issued cert. Locally that needs the
   CA chain and a SAN for `localhost`, which adds yarn-ball complexity.
   `require` keeps TLS on the wire without the cert plumbing. (Engineers
   can opt into `verify-full` later by mounting the CA cert; not
   gated on Phase 1.)
 ## 4. Architecture (Phase 1 target)
 ```
   ┌─────────────────────────────────────── Laptop ───────────────────────────────────────┐
   │                                                                                       │
   │   IntelliJ / mvn spring-boot:run                                                      │
   │             │                                                                         │
   │             ▼                                                                         │
   │   knoe-auth   :8080  ──[ http ]──▶  curl / browser tests                             │
   │             │                                                                         │
   │             │  jdbc:postgresql://localhost:5432/knoe-db                                │
   │             │  KRB5_CONFIG=./etc/krb5.local.conf                                      │
   │             │  KDC at  localhost:88                                                    │
   │             │                                                                         │
   │             ▼                                                                         │
   │   ┌─── kubectl port-forward ────┐                                                    │
   │   │   localhost:5432 → svc/knoe-db-rw:5432                                            │
   │   │   localhost:88   → svc/knoe-kdc:88  (TCP + UDP)                                   │
   │   │   localhost:464  → svc/knoe-kdc:464 (kpasswd)                                     │
   │   └────────────────┬─────────────┘                                                   │
   └────────────────────┼─────────────────────────────────────────────────────────────────┘
                        │
                        ▼
   ┌─────────────────────────────────── k3d cluster (single node) ──────────────────────────┐
   │                                                                                          │
   │   ┌──── ns: knoe-db-0 ─────────────┐         ┌──── ns: knoe-system ─────┐                │
   │   │                                 │         │                            │                │
   │   │   CNPG operator                 │         │   knoe-kdc Deployment      │                │
   │   │   knoe-db CNPG Cluster (1 inst) │         │   (krb5kdc + kadmind,      │                │
   │   │     → schema knoe.* via         │         │    realm KNOE.LOCAL,       │                │
   │   │       postInitTemplateSQL       │         │    pre-seeded principals)  │                │
   │   │                                 │         │                            │                │
   │   └─────────────────────────────────┘         └────────────────────────────┘                │
   │                                                                                          │
   └──────────────────────────────────────────────────────────────────────────────────────────┘
 ```
 ## 5. Phase 1 scope (Junie brief)
 Brief: [`junie/k3d-knoe-auth-dev-loop.md`](junie/k3d-knoe-auth-dev-loop.md).
 Deliverables:
 1. **CNPG-on-k3d known-good** — a single-replica CNPG cluster with
   the canonical `knoe.*` schema, brought up via a Make target or
   short shell script (`make k3d-knoe-up` style). Reuses existing
   `k8s/knoe/knoe-db.yaml` with the `_strip_k3d_synology_blocks` flow
   the Python orchestrator already runs.
 2. **KDC manifest for k3d** — a `Deployment` + `Service` + `ConfigMap`
   under `k8s/knoe/knoe-kdc-*.yaml` (parallel naming to
   `knoe-auth-*`) with realm `KNOE.LOCAL` and a pre-seeded admin
   principal. Init script (`etc/init_knoe_kdc_k3d.sh` or similar)
   adds the principal via `kadmin.local` once the pod is Ready.
 3. **Port-forward helper** — a `make k3d-knoe-pf` target (or shell
   script) that opens 5432, 88, 464 to localhost, prints the JDBC
   URL and the `KRB5_CONFIG` env var the engineer should `export`,
   and stays in the foreground until ^C. Background mode optional.
 4. **Engineer doc** — `docs/local-dev-knoe-auth.md`: one-time setup,
   daily loop (`make k3d-knoe-up && make k3d-knoe-pf`), how to
   `kinit`, how to point IntelliJ at it, how to reset.
 5. **Smoke test** — a shell script that, given the port-forward up,
   verifies: psql connects + `\dt knoe.*` shows ≥6 tables;
   `kadmin.local listprincs` includes the seed admin; from the host,
   `mvn -pl authority test` passes; `mvn -pl authority spring-boot:run`
   starts cleanly and `curl localhost:8080/health` returns `ok` and
   `curl localhost:8080/.well-known/openid-configuration` returns
   valid JSON with the right issuer.
 Done = a 5-minute path from `git pull` → editing
 `OidcTokenController.java` → re-run → see the change reflected at
 `localhost:8080/oauth2/token`, with the DB and KDC backing it real.
 ## 6. Out of scope for Phase 1 (potential later phases)
 - **Phase 2 — KDC: full SPNEGO E2E** including a host-side
  `firefox`/`curl` that does negotiate against `localhost:8080`.
  Needs `KRB5_CONFIG`, an SPN like `HTTP/localhost@KNOE.LOCAL`, a
  keytab file mounted into knoe-auth's runtime. Phase 1 stops at the
  KDC being reachable; Phase 2 makes SPNEGO actually work.
 - **Phase 3 — image-build-and-load smoke** so engineers can run
  `make k3d-knoe-deploy` and see knoe-auth as a pod in the cluster
  before pushing to GKE. Useful as a pre-merge gate; not required
  for daily dev.
 - **Phase 4 — supabase stack on k3d** for end-to-end SDK testing
  against a local DB. Heavy. Only if/when someone needs it.
 - **Phase 5 — oauth2-proxy + Google OIDC on localhost** for testing
  the Studio gate flow without going to GKE. Even heavier; needs a
  Google OAuth client with `localhost` redirect URLs registered.
  Probably never worth it.
 - **Phase 2.5 (separate track) — persistent OAuth2 code/session
  tables.** Currently `OidcCodeService` uses an in-memory
  `ConcurrentHashMap`, which means codes don't survive a knoe-auth
  restart. Not blocked by k3d work but useful to have once dev
  iteration is fast enough that engineers exercise the flow often.
  Tracked separately; would touch `authority/src/main/.../session/`
  and add migrations under `authority/src/main/resources/db/migration/`.
 ## 7. Risks / open questions
 - **Resource ceiling on a MacBook.** A single-replica CNPG plus KDC
  is light (~600 MB RAM combined), but k3d itself + Docker Desktop
  + IntelliJ + Spring Boot can climb. Document a "low-RAM mode" if
  it becomes a real problem; don't speculate now.
 - **k3d's Docker networking and `kubectl port-forward` UDP.** UDP
  port-forward (port 88 for Kerberos UDP, 464 for kpasswd) works in
  recent kubectl versions but has historically been finicky. If it
  fails, fall back to TCP-only Kerberos (`tcp_only` in `krb5.conf`).
  Phase 1 should set this in the engineer-side `krb5.local.conf`
  preemptively.
 - **CNPG operator CRD installation in fresh k3d.** The `make k3d-up`
  flow today runs `init_cnpg_gke.sh` for GKE, but there's no
  symmetric `init_cnpg_k3d.sh`. Either: (a) Junie writes a thin one
  that just `kubectl apply -f`s the CNPG release manifest with no
  GCP/IAM steps, or (b) the existing Python orchestrator
  `knoe.core.ops.cloudnative_pg` already handles k3d (per the
  survey, it does — `_strip_k3d_synology_blocks` is in there).
  Verify which during Phase 1.
 - **Schema bootstrap.** The GKE flow seeds `knoe.*` via
  `etc/init_knoe_auth.sh` running SQL against the CNPG primary
  through `kubectl exec`. The same approach works in k3d; the
  brief should reuse the script (with a `KUBECONTEXT` swap), not
  duplicate it.
 ## 8. Verification (when Phase 1 is "done")
 A new engineer, freshly cloning the repo on a Mac:
 ```bash
 git clone <repo> && cd knoe-db
 make k3d-knoe-up                # provisions cluster, CNPG, KDC, schema
 make k3d-knoe-pf &              # port-forward in background
 mvn -pl authority spring-boot:run
 # in another terminal:
 curl http://localhost:8080/health
 # → ok
 curl http://localhost:8080/.well-known/openid-configuration
 # → JSON, issuer "http://localhost:8080"
 ```
 Total elapsed time on a fresh laptop: ≤ 8 minutes. The engineer can
 now edit `authority/src/main/java/dev/knoe/auth/...` and see changes
 without leaving their editor.
 ## 9. Glossary
 - **k3d** — `k3s` running inside a Docker container. Lightweight
  Kubernetes for laptops; no VM needed.
 - **CNPG** — `cloudnative-pg`, a PostgreSQL Kubernetes operator. We
  run a single instance in k3d, three in GKE.
 - **KDC** — Key Distribution Center, the Kerberos protocol's auth
  server. SPNEGO browsers/clients fetch tickets from it; knoe-auth
  validates SPNEGO challenges using these.
 - **KNOE.LOCAL** — the Kerberos realm name we use for laptop dev.
  Distinct from `KNOE.DEV` (production) so a stray ticket from one
  doesn't authenticate you on the other.
 - **SPNEGO** — `Negotiate` HTTP auth scheme. Carries a Kerberos
  ticket inside an HTTP header.
 - **Phase 2.5** — colloquial name for the gap between Phase 2 OIDC
  (in-memory codes) and a fully persistent OIDC provider (DB-backed
  codes/sessions). Not formally numbered; will get a name when work
  starts.