From 6d484ef13e48b90d00a010cb8403e6bfe416edf5 Mon Sep 17 00:00:00 2001 From: chrisfu Date: Sat, 2 May 2026 04:45:28 -0700 Subject: [PATCH] docs(plans): k3d-mirror-of-GKE plan + Phase 1 brief for Junie MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The next development thread is knoe-auth Phase 2 OIDC iteration, which just landed on main but has no fast inner-loop. Instead of bouncing edits through the GKE deploy chain, we want a laptop-resident model where knoe-auth runs from the IDE / mvn against real Postgres + KDC brought up in k3d. Two new docs: docs/plans/k3d-gke-mirror.md Architectural plan. Top-to-bottom rationale for what the k3d model should mirror, what we deliberately skip (supabase, kong, oauth2-proxy, Studio — all unnecessary for knoe-auth's surfaces), and the phasing. Captures the three scope decisions taken 2026-05-02 with the user: - knoe-auth runs on the laptop, not as a pod (faster inner loop) - stack scope is CNPG + KDC only - single-replica CNPG, realm KNOE.LOCAL (distinct from KNOE.DEV) Out-of-scope for Phase 1 listed explicitly so reviewer expectations match scope: SPNEGO E2E, image-build-and-load, supabase, oauth2-proxy, OidcCodeService DB persistence. docs/plans/junie/k3d-knoe-auth-dev-loop.md Phase 1 brief. Concrete deliverables: - k8s/knoe/knoe-kdc-{deployment,service,configmap,pvc,init-job}.yaml - etc/init_knoe_auth.sh --mode k3d flag (or sibling script) - Makefile: k3d-knoe-{up,pf,down} - etc/krb5.local.conf (with udp_preference_limit=1 for kubectl port-forward UDP flakiness on macOS) - docs/local-dev-knoe-auth.md (engineer-facing) - scripts/k3d-knoe-smoke.sh Definition of done: a fresh-clone laptop can `make k3d-knoe-up` + `make k3d-knoe-pf` + `mvn -pl authority spring-boot:run` and hit /health, /.well-known/openid-configuration in <8 minutes. Index updates: docs/plans/README.md — k3d-gke-mirror.md row added docs/plans/junie/README.md — split into Active / Shipped; k3d brief listed under Active; the 5 shipped-2026-05-02 briefs moved to Shipped with commit refs. docs/TODO.md In-progress — k3d brief now top of "In progress"; the previously in-flight "Phase 2 pg_oauth in install/deploy.sh" item demoted to "paused" (resume after the local dev loop lands). No code changes; all docs. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/TODO.md | 4 +- docs/plans/README.md | 1 + docs/plans/junie/README.md | 20 +- docs/plans/junie/k3d-knoe-auth-dev-loop.md | 319 +++++++++++++++++++++ docs/plans/k3d-gke-mirror.md | 258 +++++++++++++++++ 5 files changed, 595 insertions(+), 7 deletions(-) create mode 100644 docs/plans/junie/k3d-knoe-auth-dev-loop.md create mode 100644 docs/plans/k3d-gke-mirror.md diff --git a/docs/TODO.md b/docs/TODO.md index a8337c5..136bc43 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -13,7 +13,9 @@ The Kanban "Now" section at top is the only place this doc imposes structure. Ev ## Now (Kanban) ### In progress -- **Phase 2: pg_oauth in install/deploy.sh + k3d build** — assigned to Junie. The Phase 1 work (per-engineer roles, external LB at `pg.0.knoe.dev`, cert SANs, pg_hba tightening, [`docs/db-access.md`](db-access.md)) is committed. Phase 2 replaces the SCRAM password mechanism with PG18 native OAUTHBEARER, wires it into `install.sh` / `supabase/deploy.sh` so a fresh deploy lands with the OIDC mechanism active by default, and ports the Service + cluster cert pattern to k3d. Connection target stays at `pg.0.knoe.dev`; only the credential issuance changes (libpq Device Flow → Google Bearer token). Plan reference: `~/.claude/plans/we-re-continuing-work-on-happy-toast.md`. +- **k3d-mirror-of-GKE Phase 1: laptop dev loop for knoe-auth** — assigned to Junie. Stand up the smallest k3d-resident stack (CNPG single-replica + standalone KDC, realm `KNOE.LOCAL`) that lets a host-side knoe-auth (via `mvn spring-boot:run` / IntelliJ) iterate against real Postgres + Kerberos. Architectural plan: [`docs/plans/k3d-gke-mirror.md`](plans/k3d-gke-mirror.md). Phase 1 brief: [`docs/plans/junie/k3d-knoe-auth-dev-loop.md`](plans/junie/k3d-knoe-auth-dev-loop.md). Deliverables: `k8s/knoe/knoe-kdc-*.yaml` manifests, `etc/init_knoe_auth.sh` `--mode k3d` flag, `make k3d-knoe-{up,pf,down}` targets, `etc/krb5.local.conf`, `docs/local-dev-knoe-auth.md`, `scripts/k3d-knoe-smoke.sh`. Out of scope (Phase 2+): SPNEGO E2E from browsers, knoe-auth-as-pod image builds, supabase stack on k3d, `OidcCodeService` DB persistence. + +- **Phase 2: pg_oauth in install/deploy.sh + k3d build** — *previously assigned to Junie; paused.* The Phase 1 work (per-engineer roles, external LB at `pg.0.knoe.dev`, cert SANs, pg_hba tightening, [`docs/db-access.md`](db-access.md)) is committed. Phase 2 replaces the SCRAM password mechanism with PG18 native OAUTHBEARER, wires it into `install.sh` / `supabase/deploy.sh` so a fresh deploy lands with the OIDC mechanism active by default, and ports the Service + cluster cert pattern to k3d. Connection target stays at `pg.0.knoe.dev`; only the credential issuance changes (libpq Device Flow → Google Bearer token). Park while the k3d dev loop above lands; resume once knoe-auth Phase 2 OIDC dev has somewhere to run locally. ### Up next *(empty — pull from the ranked queue below)* diff --git a/docs/plans/README.md b/docs/plans/README.md index 51229eb..5b2b34f 100644 --- a/docs/plans/README.md +++ b/docs/plans/README.md @@ -19,6 +19,7 @@ If a sentence in a plan assumes knowledge you don't have, that's a bug in the pl | --- | --- | --- | | [`knoe-auth-round-1.md`](knoe-auth-round-1.md) | Identity backbone for the platform: MIT Kerberos KDC + invite-anchored web enrollment + TOTP 2FA. Round 1 of N. **Shipped.** | Before touching `authority/`, `etc/init_kdc.sh`, `etc/init_knoe_users.sh`, anything in `deploy/gcp/gke/knoe-auth-*` or `knoe-kdc-*`, or the `knoe.*` database schema. | | [`deployment-modes.md`](deployment-modes.md) | Four-mode installer (`min` / `k3d` / `k3s` / `gke`) with a welcome-screen mode selector and a `min`-mode fast-path through the wizard. **Shipped in Phase 0.** | Before touching `knoe/ui/screens/welcome.py`, `knoe/ui/screens/navigation.py`, or adding any new wizard screen. | +| [`k3d-gke-mirror.md`](k3d-gke-mirror.md) | Laptop-resident model of the GKE platform — minimum-viable k3d stack (CNPG + KDC) that lets host-side knoe-auth iterate against real Postgres + Kerberos. **Active. Phase 1 in flight (Junie).** | Before touching `k8s/knoe/knoe-kdc-*`, the `make k3d-knoe-*` targets, or `docs/local-dev-knoe-auth.md`. Phase 1 brief at [`junie/k3d-knoe-auth-dev-loop.md`](junie/k3d-knoe-auth-dev-loop.md). | | [`customer-deploy-resync.md`](customer-deploy-resync.md) | Original plan to converge `~/dev/prole` onto `knoe-db/main` as a customer-deploy branch. **Dormant** — prole rebrand merged into `main`; no separate prole working tree currently under development. | Only if you're considering activating a per-customer branching workflow. | | [`junie/`](junie/README.md) (subdirectory) | Self-contained, single-task **work briefs** for Junie to consume from inside the IDE. Tactical, single-MR scope, paired against numbered items in `../TODO.md`. | When you want to hand a discrete task to Junie or audit what's been queued. | | [`../TODO.md`](../TODO.md) (sibling) | Master TODO index — single source of truth for unfinished work, including reality-vs-intent gaps flagged in `CLAUDE.md`/`AGENTS.md`. | Before picking up any task, to see what's already on the queue. | diff --git a/docs/plans/junie/README.md b/docs/plans/junie/README.md index 034c109..4c57cdd 100644 --- a/docs/plans/junie/README.md +++ b/docs/plans/junie/README.md @@ -44,15 +44,23 @@ in TODO" note). When Junie lands a brief: 2. Don't delete the brief from this directory — it stays as the design record. -## Current briefs (as of 2026-05-01) +## Current briefs (as of 2026-05-02) + +### Active (in flight) + +| File | Tracked at | Subject | +|---|---|---| +| [`k3d-knoe-auth-dev-loop.md`](k3d-knoe-auth-dev-loop.md) | TODO §"In progress"; parent [`../k3d-gke-mirror.md`](../k3d-gke-mirror.md) | Phase 1 of the k3d-mirror-of-GKE plan: stand up CNPG + KDC in k3d so host-side knoe-auth can iterate against real Postgres + Kerberos | + +### Shipped (kept as design record) | File | Queue # | Subject | |---|---|---| -| [`02-k3s-prole-rename.md`](02-k3s-prole-rename.md) | #2 | Rename k3s `prole-*.yaml` → `knoe-*.yaml` (kustomize is broken) | -| [`06-patch-garage-script-fixes.md`](06-patch-garage-script-fixes.md) | #6 | Three defects in `scripts/patch_garage_cross_cluster.sh` | -| [`07-init-cnpg-gke-sa-wiring.md`](07-init-cnpg-gke-sa-wiring.md) | #7 | Wire `cnpg-backup-sa` into CNPG cluster spec; bump operator to v1.29 | -| [`13-podmonitor-manual-management.md`](13-podmonitor-manual-management.md) | #13 | Migrate off CNPG-deprecated `enablePodMonitor` + `podMonitorRelabelings` | -| [`15-remove-dead-dashboard-consumer.md`](15-remove-dead-dashboard-consumer.md) | #15 | Remove dead Kong DASHBOARD consumer + `basicauth_credentials` | +| [`02-k3s-prole-rename.md`](02-k3s-prole-rename.md) | #2 | Rename k3s `prole-*.yaml` → `knoe-*.yaml` (kustomize is broken). **Shipped 2026-05-02 (commit `fb7e8b7`).** | +| [`06-patch-garage-script-fixes.md`](06-patch-garage-script-fixes.md) | #6 | Three defects in `scripts/patch_garage_cross_cluster.sh`. **Shipped 2026-05-02 (commits `34a25dd` + `5d17325`).** | +| [`07-init-cnpg-gke-sa-wiring.md`](07-init-cnpg-gke-sa-wiring.md) | #7 | Wire `cnpg-backup-sa` into CNPG cluster spec; bump operator to v1.29. **Shipped 2026-05-02 (commit `c3fae73`).** | +| [`13-podmonitor-manual-management.md`](13-podmonitor-manual-management.md) | #13 | Migrate off CNPG-deprecated `enablePodMonitor` + `podMonitorRelabelings`. **Shipped 2026-05-02 (commit `c3fae73`).** | +| [`15-remove-dead-dashboard-consumer.md`](15-remove-dead-dashboard-consumer.md) | #15 | Remove dead Kong DASHBOARD consumer + `basicauth_credentials`. **Shipped 2026-05-02 (commit `c3fae73`).** | ## How a session fires off a batch diff --git a/docs/plans/junie/k3d-knoe-auth-dev-loop.md b/docs/plans/junie/k3d-knoe-auth-dev-loop.md new file mode 100644 index 0000000..406c873 --- /dev/null +++ b/docs/plans/junie/k3d-knoe-auth-dev-loop.md @@ -0,0 +1,319 @@ +# Junie brief — Phase 1 of k3d-mirror-of-GKE: laptop dev loop for knoe-auth + +> **Self-contained brief.** Reference: parent architectural plan at +> [`../k3d-gke-mirror.md`](../k3d-gke-mirror.md). Read that for the +> "why" and the multi-phase shape. This brief delivers Phase 1 only. + +--- + +## 1. Why + +knoe-auth Phase 2 OIDC code is on `main` but no fast inner-loop +exists for iterating on it. Hitting the GKE cluster on every change +is slow. Running unit tests with testcontainers covers a lot but +doesn't exercise the real DB schema, Kerberos KDC, or the actual HTTP +surface against a Postgres+JDBC stack. + +This brief wires up a **laptop-resident dev loop**: bring up CNPG + +KDC in k3d, port-forward 5432/88/464 to localhost, and let the +engineer run knoe-auth from their IDE/`mvn spring-boot:run` against +those local ports. Edit Java → re-run → see the change at +`http://localhost:8080`, with a real PostgreSQL and a real Kerberos +backing it. + +knoe-auth itself stays on the host. Image-build-and-load into k3d is +a separate later phase (and is in fact mostly orthogonal to this +work — Phase 1 finishes when the host loop is solid). + +## 2. What you produce + +Concrete deliverables, all in one MR: + +### 2.1 — KDC manifest set for k3d + +New files (mirroring the auth-side filename pattern under `k8s/knoe/`): + +- `k8s/knoe/knoe-kdc-deployment.yaml` — single-replica `Deployment` + running `krb5kdc` + `kadmind`. Realm `KNOE.LOCAL`. Use a + Debian-derived MIT Kerberos image; whatever the GKE sidecar image + uses is fine to reuse. **Set `realm = KNOE.LOCAL`** explicitly — + do NOT inherit from the existing `KNOE.DEV` GKE config. +- `k8s/knoe/knoe-kdc-service.yaml` — ClusterIP service exposing 88 + (TCP+UDP) and 464 (TCP+UDP — kpasswd). +- `k8s/knoe/knoe-kdc-configmap.yaml` — `krb5.conf` + `kdc.conf` with + realm `KNOE.LOCAL` and a `[realms] KNOE.LOCAL = { kdc = knoe-kdc:88 ... }` + block. +- `k8s/knoe/knoe-kdc-pvc.yaml` — small PVC (1 Gi) for the KDC + database `/var/lib/krb5kdc/principal`. Local-path provisioner + (k3d default). +- `k8s/knoe/knoe-kdc-init-job.yaml` — one-shot `Job` that runs + `kdb5_util create -s -P ` (database init) and then + `kadmin.local addprinc -randkey admin/admin@KNOE.LOCAL` plus a + developer principal. Idempotent (skip create if DB exists). Runs + once on first apply; subsequent apply is a no-op via a guard + inside the script. + +Reference existing GKE `kdc.conf` and `krb5.conf` shapes from +[`deploy/gcp/gke/knoe-kdc-configmap.yaml`](../../deploy/gcp/gke/knoe-kdc-configmap.yaml). +Substitute the realm, drop GCE-specific bits. + +### 2.2 — Init-script entry point for k3d + +Either: + +- **(preferred)** Extend `etc/init_knoe_auth.sh` with a `--mode k3d` + flag that swaps the `APP_CTX` default to `k3d-knoe` (or whatever + `kubectl config current-context` returns from the k3d-up flow), + swaps the realm to `KNOE.LOCAL`, and skips the GCP/Workload + Identity steps (no GCP project on a laptop). Same `schema` / + `invite` / `status` subcommands work. +- **(or)** Write a thin `etc/init_knoe_auth_k3d.sh` that wraps the + existing one with the right env vars set. + +Pick whichever results in less duplication. The existing script's +`schema` subcommand already runs the SQL via `kubectl exec` on the +CNPG primary — that path works in k3d unchanged. + +### 2.3 — Make targets + +Three new targets in the top-level `Makefile`: + +- **`make k3d-knoe-up`** — provisions the local k3d cluster (if not + present), installs the CNPG operator, applies + `k8s/knoe/knoe-db.yaml` (single-replica per the architectural + plan §3), waits for `Cluster in healthy state`, runs the + knoe-kdc init job, applies the KDC Deployment + Service, runs + `etc/init_knoe_auth.sh schema` to seed `knoe.*` tables. +- **`make k3d-knoe-pf`** — opens three port-forwards in the + foreground: + - `5432` → `service/knoe-db-rw -n knoe-db-0` + - `88`/`464` → `service/knoe-kdc -n knoe-system` (TCP and UDP both) + - Prints the JDBC URL and the env vars the engineer should + `export` (e.g. `KRB5_CONFIG=$PWD/etc/krb5.local.conf`, + `KNOE_DB_PASSWORD=...`). + - On ^C, cleans up all three forwards. +- **`make k3d-knoe-down`** — tears down the k3d cluster (or just + the namespaces if you want incremental cleanup). + +The cluster name should be `k3d-knoe` (so the kubeconfig context is +`k3d-k3d-knoe`, matching k3d's prefix convention). Reuse anything in +`scripts/` that already drives k3d if it's there. + +### 2.4 — Engineer-side krb5 config + +A new file `etc/krb5.local.conf` checked into the repo: + +```ini +[libdefaults] + default_realm = KNOE.LOCAL + rdns = false + forwardable = true + udp_preference_limit = 1 # workaround for kubectl port-forward UDP flakiness; force TCP + +[realms] + KNOE.LOCAL = { + kdc = localhost:88 + admin_server = localhost:749 + default_domain = local + } + +[domain_realm] + .local = KNOE.LOCAL + localhost = KNOE.LOCAL +``` + +Engineer's runtime: `export KRB5_CONFIG=$PWD/etc/krb5.local.conf` and +the JVM picks it up automatically. + +### 2.5 — Engineer-facing doc + +New `docs/local-dev-knoe-auth.md`. Sections: + +1. **Prerequisites** — k3d, kubectl, Docker Desktop running, JDK 21, + Maven. Brew one-liner. +2. **One-time setup** — `make k3d-knoe-up`. Wait ~3 minutes. Verify + `kubectl --context k3d-k3d-knoe -n knoe-db-0 get cluster knoe-db` + shows `Healthy`. +3. **Daily loop** — open two terminals. + - Terminal A: `make k3d-knoe-pf` (leave running). + - Terminal B: `export KRB5_CONFIG=$PWD/etc/krb5.local.conf` then + `mvn -pl authority spring-boot:run`. +4. **Verify the loop** — `curl localhost:8080/health` returns `ok`; + `curl localhost:8080/.well-known/openid-configuration` returns + the OIDC discovery JSON; `psql + "postgresql://postgres:$(make show-db-password)@localhost:5432/knoe-db?sslmode=require"` + connects. +5. **`kinit` for SPNEGO testing** — `kinit -k -t + admin/admin@KNOE.LOCAL` (full SPNEGO E2E is Phase 2; document + the expected limitations: `curl --negotiate -u : http://localhost:8080/` + may or may not fully round-trip on macOS depending on + `udp_preference_limit`). +6. **IntelliJ run configuration** — paste-ready text or + screenshot. Module `authority`, main class + `dev.knoe.auth.KnoeAuthApplication`, env vars + `KRB5_CONFIG=…/etc/krb5.local.conf`, + `KNOE_DB_PASSWORD=…`. +7. **Reset** — `make k3d-knoe-down && make k3d-knoe-up`. Time: ~3 min. + +### 2.6 — Smoke-test script + +`scripts/k3d-knoe-smoke.sh` — bash, runs after `make k3d-knoe-up` +to confirm the loop is intact: + +```bash +#!/usr/bin/env bash +set -euo pipefail +ctx=k3d-k3d-knoe +ns_db=knoe-db-0 +ns_sys=knoe-system + +# 1. CNPG cluster healthy +phase=$(kubectl --context=$ctx -n $ns_db get cluster knoe-db -o jsonpath='{.status.phase}') +[[ "$phase" == "Cluster in healthy state" ]] || { echo "CNPG: $phase"; exit 1; } + +# 2. knoe.* schema present +tables=$(kubectl --context=$ctx -n $ns_db exec sts/knoe-db-1 -c postgres -- \ + psql -U postgres -tAc "SELECT count(*) FROM information_schema.tables WHERE table_schema='knoe'") +[[ "$tables" -ge 6 ]] || { echo "knoe.* tables: $tables (expected ≥6)"; exit 1; } + +# 3. KDC has the seed admin principal +kubectl --context=$ctx -n $ns_sys exec deploy/knoe-kdc -- \ + kadmin.local listprincs | grep -q 'admin/admin@KNOE.LOCAL' \ + || { echo "missing admin principal"; exit 1; } + +echo "k3d-knoe-smoke: PASS" +``` + +CI integration is out of scope for Phase 1 — this is engineer-run. + +## 3. Don't break + +- The existing GKE knoe-auth deployment must continue to work + unchanged. If you touch `etc/init_knoe_auth.sh`, the GKE + invocation must remain the default behavior (no `--mode` flag = + current GKE behavior). +- The k3s manifests under `deploy/opentofu/k3s/manifests/knoe/` + must NOT be modified. The k3s deploy mode targets a different + use case (on-prem VM); leave it alone. +- The CNPG cluster manifest at `k8s/knoe/knoe-db.yaml` must stay + the canonical k3d/min Postgres definition. If you need it to + behave differently for k3d (e.g. single-replica), make the change + configurable, not destructive — `_strip_k3d_synology_blocks` in + `knoe/core/ops/cloudnative_pg.py` is the existing precedent for + k3d-specific manifest manipulation. + +## 4. Verification (Definition of done) + +A new engineer, freshly cloning the repo, completes this in ≤ 8 +minutes on a MacBook Air: + +```bash +git clone && cd knoe-db +make k3d-knoe-up # 3-5 min +make k3d-knoe-pf & # foreground; backgrounded for this script +sleep 3 +scripts/k3d-knoe-smoke.sh # PASS +export KRB5_CONFIG=$PWD/etc/krb5.local.conf +mvn -pl authority spring-boot:run & # backgrounded for the verify +sleep 30 +curl -fsS http://localhost:8080/health # ok +curl -fsS http://localhost:8080/.well-known/openid-configuration | jq .issuer +# → "http://localhost:8080" +``` + +Tick all of: + +- [ ] `make k3d-knoe-up` exits 0 on a fresh laptop with k3d/kubectl + installed. +- [ ] `kubectl --context=k3d-k3d-knoe -n knoe-db-0 get cluster knoe-db` + reports `Healthy`. +- [ ] `kubectl --context=k3d-k3d-knoe -n knoe-system get deploy knoe-kdc` + reports `1/1 ready`. +- [ ] `scripts/k3d-knoe-smoke.sh` reports `PASS`. +- [ ] Host-side `psql` connects via `localhost:5432` after + `make k3d-knoe-pf` is running. +- [ ] `kinit admin/admin@KNOE.LOCAL` from the host succeeds. +- [ ] `mvn -pl authority spring-boot:run` starts cleanly with + `KRB5_CONFIG=etc/krb5.local.conf` set; `/health` returns `ok`; + `/.well-known/openid-configuration` returns valid JSON. +- [ ] `make k3d-knoe-down && make k3d-knoe-up` is idempotent. +- [ ] `docs/local-dev-knoe-auth.md` exists and is accurate. +- [ ] GKE deploy mode unchanged: `git diff main -- deploy/gcp/gke/` + shows no changes; the existing init scripts behave identically + when invoked the same way they were before. + +## 5. Out of scope + +Captured in the parent plan §6, but call out the highest-friction +deferrals here so reviewer expectations are right: + +- **SPNEGO E2E from a host browser** — requires a service principal + for `HTTP/localhost@KNOE.LOCAL` and a keytab the host knoe-auth + process can read. Phase 1 stops at "KDC reachable, principals + exist, `kinit` works." Phase 2 makes browser SPNEGO actually + authenticate. +- **knoe-auth as a pod inside k3d** — image-build-and-load. Phase 3. + Don't take this on now. +- **Supabase / Studio / Kong / oauth2-proxy on k3d** — all explicit + Phase 4+. +- **OIDC code persistence in DB** — `OidcCodeService` uses an + in-memory `ConcurrentHashMap` today. That's a real Phase 2.5 gap + but it's the same gap on GKE; not k3d-specific. Don't fold it + into this brief. + +## 6. Commit shape + +Single commit unless the deliverables genuinely separate into two +clean halves (e.g. infra + docs). Suggested message: + +``` +feat(k3d): laptop dev loop for knoe-auth — CNPG + KDC + port-forward + +Brings up the smallest k3d-resident stack that lets a host-side +knoe-auth (run via mvn spring-boot:run or IntelliJ) iterate against +real Postgres + Kerberos. Closes Phase 1 of the k3d-gke-mirror plan. + +Scope: + - k8s/knoe/knoe-kdc-{deployment,service,configmap,pvc,init-job}.yaml + NEW; standalone KDC, realm KNOE.LOCAL (distinct from KNOE.DEV). + - etc/init_knoe_auth.sh: --mode k3d flag, swaps realm + skips + GCP-specific steps. GKE behavior unchanged when flag absent. + - Makefile: k3d-knoe-up, k3d-knoe-pf, k3d-knoe-down. + - etc/krb5.local.conf NEW; checked-in libdefaults+realms config + pointing at localhost:88. udp_preference_limit=1 to dodge + kubectl port-forward UDP flakiness. + - docs/local-dev-knoe-auth.md NEW; one-time setup + daily loop. + - scripts/k3d-knoe-smoke.sh NEW; bash sanity script. + +Verified by following docs/local-dev-knoe-auth.md from a fresh +clone: end-to-end in <8 min, knoe-auth at localhost:8080 hits +real DB + KDC; OIDC discovery returns valid JSON. + +Out of scope (parent plan docs/plans/k3d-gke-mirror.md §6): + - SPNEGO from host browsers + - knoe-auth-as-pod (image build/load) + - Supabase stack + - OidcCodeService DB persistence + +Closes Phase 1; Phase 2+ briefs filed as needed. +``` + +## 7. Notes for reading + +- The user is happy to **decide between two paths if you surface + them clearly** — e.g. "extend `init_knoe_auth.sh` with `--mode + k3d` vs write a sibling `init_knoe_auth_k3d.sh`". Pick the one + that minimizes drift and explain why in the commit. +- The user runs macOS. UDP port-forward through kubectl on Mac has + historically been finicky; setting `udp_preference_limit = 1` in + `krb5.local.conf` (force TCP) is the recommended workaround and is + why it's in the example above. +- If you run into resource ceiling issues on the laptop (Docker + Desktop OOM, k3d node not ready), document the workaround in the + engineer-side doc rather than working around it in the manifests. +- If the existing Python orchestrator + (`knoe.core.ops.cloudnative_pg`) already handles the + k3d-CNPG-bring-up cleanly, leverage it rather than writing a + shell wrapper from scratch. The Make target can call into it via + `python -m knoe.core.ops.cloudnative_pg ...` if the entrypoints + exist; if they don't, a bash wrapper is fine. diff --git a/docs/plans/k3d-gke-mirror.md b/docs/plans/k3d-gke-mirror.md new file mode 100644 index 0000000..66a19f7 --- /dev/null +++ b/docs/plans/k3d-gke-mirror.md @@ -0,0 +1,258 @@ +# k3d-mirror-of-GKE — local platform model for knoe-auth dev + +> **Active plan. Phase 1 in flight (Junie).** Strategic context for a +> running initiative; reads top-to-bottom. Phase 1 brief at +> [`junie/k3d-knoe-auth-dev-loop.md`](junie/k3d-knoe-auth-dev-loop.md). + +--- + +## 1. Why + +knoe-auth Phase 2 (OIDC provider — discovery, authorize, token, userinfo, +JWKS, RS256 signing) just landed on `main` (via the merge that brought +`claude/crazy-bose-fec256` back in). It builds; it hasn't been deployed +to the cluster yet. The next development steps — wiring OAuth2 code +exchange to a persistent table (Phase 2.5), the +ProvisioningWorker→GitLab/Gitea/CNPG dispatch surface, future Round 2 +work — all benefit from a tight inner-loop that doesn't require: + +- a GKE cluster to push to, +- a deploy pipeline to wait on, +- a real Google OAuth tenant to bounce through, or +- the full supabase + kong + oauth2-proxy + studio stack just to test + what knoe-auth does. + +What we want is a **laptop dev loop**: edit Java, run from IntelliJ +(or `mvn spring-boot:run`), break-pointing freely, against a real +PostgreSQL with the canonical `knoe.*` schema and a real Kerberos KDC +the SPNEGO codepath can talk to. + +The four-mode installer we already ship has `k3d` as one of those modes +(see [`deployment-modes.md`](deployment-modes.md)). It brings up CNPG +today. It does **not** yet bring up the bits knoe-auth needs alongside. +This plan closes that gap deliberately, in phases. + +## 2. Where the modes stand today + +| Mode | DB | knoe-auth | KDC | Supabase | Studio | OAuth2-proxy | +|---|---|---|---|---|---|---| +| `gke` | CNPG (3-replica) | ✅ deployed | ✅ sidecar | ✅ | ✅ | ✅ Google OIDC | +| `k3s` | CNPG (single) | ✅ deployed (manifests at `deploy/opentofu/k3s/manifests/knoe/knoe-auth-*`) | ✅ kdc-configmap | partial | partial | ❌ | +| `k3d` | ✅ CNPG via `k8s/knoe/knoe-db.yaml` | ❌ no manifests | ❌ no manifests | ❌ | ❌ | ❌ | +| `min` | ❌ no Postgres | ❌ | ❌ | ❌ | ❌ | ❌ | + +The k3s mode has the auth-stack manifests but is targeted at the +on-prem-VM deploy pattern (talks to a single-node k3s on a customer +machine), not the laptop-dev-loop pattern. Reusing them for k3d is +attractive but the wiring assumptions differ enough (LB types, +ingress class, service-account identity) that copy-and-adapt is +easier than refactor-to-share. + +## 3. Strategic shape + +The **goal** is not a full GKE mirror — that would mean Studio + Kong ++ supabase + oauth2-proxy + everything, and the laptop fan would not +be happy. The goal is **the smallest k3d-resident stack that lets a +host-resident knoe-auth talk to it end-to-end**. + +Decisions (set 2026-05-02 with the user): + +1. **knoe-auth runs on the host, not in the cluster.** `mvn + spring-boot:run` (or IntelliJ Run Configuration) starts the process + on the laptop. It connects to the k3d cluster's DB and KDC via + `kubectl port-forward`. This gives: + - hot reload via Spring DevTools, + - direct debugger attach, + - immediate stack traces in the same terminal as the build, + - no image-build-and-load cycle on every change. + + Trade-off: this isn't testing the deployed-pod-shaped behaviour + (mounted secrets, SA identity, sidecar networking). That's + covered by the GKE deploy when it's time to go live, and a future + "in-cluster smoke" Phase 3 brief if we want pre-merge checks too. + +2. **Stack scope is CNPG + KDC only.** No supabase, no Kong, no + oauth2-proxy, no Studio, no garage/MinIO. knoe-auth doesn't + depend on any of those for its own surfaces (`/health`, + `/auth/enroll/*`, `/auth/admin/*`, OIDC discovery/authorize/ + token/userinfo/jwks). The ProvisioningWorker reaches OUT to + GitLab/Gitea/CNPG — those are mockable or deferrable for dev. + +3. **Single-replica CNPG.** Three-replica is GKE-shaped for + availability; on a laptop it's wasteful. One instance, the same + PG18 image, same `postInitTemplateSQL` schema bootstrap, same + `knoe.*` tables. + +4. **Kerberos realm `KNOE.LOCAL`.** Distinct from production + `KNOE.DEV` so credentials don't accidentally cross-talk. Pre-seed + one admin principal and one developer principal so the engineer + can `kinit` immediately. + +5. **Plain `sslmode=require` for libpq from host.** Production uses + `verify-full` against the CNPG-issued cert. Locally that needs the + CA chain and a SAN for `localhost`, which adds yarn-ball complexity. + `require` keeps TLS on the wire without the cert plumbing. (Engineers + can opt into `verify-full` later by mounting the CA cert; not + gated on Phase 1.) + +## 4. Architecture (Phase 1 target) + +``` + ┌─────────────────────────────────────── Laptop ───────────────────────────────────────┐ + │ │ + │ IntelliJ / mvn spring-boot:run │ + │ │ │ + │ ▼ │ + │ knoe-auth :8080 ──[ http ]──▶ curl / browser tests │ + │ │ │ + │ │ jdbc:postgresql://localhost:5432/knoe-db │ + │ │ KRB5_CONFIG=./etc/krb5.local.conf │ + │ │ KDC at localhost:88 │ + │ │ │ + │ ▼ │ + │ ┌─── kubectl port-forward ────┐ │ + │ │ localhost:5432 → svc/knoe-db-rw:5432 │ + │ │ localhost:88 → svc/knoe-kdc:88 (TCP + UDP) │ + │ │ localhost:464 → svc/knoe-kdc:464 (kpasswd) │ + │ └────────────────┬─────────────┘ │ + └────────────────────┼─────────────────────────────────────────────────────────────────┘ + │ + ▼ + ┌─────────────────────────────────── k3d cluster (single node) ──────────────────────────┐ + │ │ + │ ┌──── ns: knoe-db-0 ─────────────┐ ┌──── ns: knoe-system ─────┐ │ + │ │ │ │ │ │ + │ │ CNPG operator │ │ knoe-kdc Deployment │ │ + │ │ knoe-db CNPG Cluster (1 inst) │ │ (krb5kdc + kadmind, │ │ + │ │ → schema knoe.* via │ │ realm KNOE.LOCAL, │ │ + │ │ postInitTemplateSQL │ │ pre-seeded principals) │ │ + │ │ │ │ │ │ + │ └─────────────────────────────────┘ └────────────────────────────┘ │ + │ │ + └──────────────────────────────────────────────────────────────────────────────────────────┘ +``` + +## 5. Phase 1 scope (Junie brief) + +Brief: [`junie/k3d-knoe-auth-dev-loop.md`](junie/k3d-knoe-auth-dev-loop.md). + +Deliverables: + +1. **CNPG-on-k3d known-good** — a single-replica CNPG cluster with + the canonical `knoe.*` schema, brought up via a Make target or + short shell script (`make k3d-knoe-up` style). Reuses existing + `k8s/knoe/knoe-db.yaml` with the `_strip_k3d_synology_blocks` flow + the Python orchestrator already runs. +2. **KDC manifest for k3d** — a `Deployment` + `Service` + `ConfigMap` + under `k8s/knoe/knoe-kdc-*.yaml` (parallel naming to + `knoe-auth-*`) with realm `KNOE.LOCAL` and a pre-seeded admin + principal. Init script (`etc/init_knoe_kdc_k3d.sh` or similar) + adds the principal via `kadmin.local` once the pod is Ready. +3. **Port-forward helper** — a `make k3d-knoe-pf` target (or shell + script) that opens 5432, 88, 464 to localhost, prints the JDBC + URL and the `KRB5_CONFIG` env var the engineer should `export`, + and stays in the foreground until ^C. Background mode optional. +4. **Engineer doc** — `docs/local-dev-knoe-auth.md`: one-time setup, + daily loop (`make k3d-knoe-up && make k3d-knoe-pf`), how to + `kinit`, how to point IntelliJ at it, how to reset. +5. **Smoke test** — a shell script that, given the port-forward up, + verifies: psql connects + `\dt knoe.*` shows ≥6 tables; + `kadmin.local listprincs` includes the seed admin; from the host, + `mvn -pl authority test` passes; `mvn -pl authority spring-boot:run` + starts cleanly and `curl localhost:8080/health` returns `ok` and + `curl localhost:8080/.well-known/openid-configuration` returns + valid JSON with the right issuer. + +Done = a 5-minute path from `git pull` → editing +`OidcTokenController.java` → re-run → see the change reflected at +`localhost:8080/oauth2/token`, with the DB and KDC backing it real. + +## 6. Out of scope for Phase 1 (potential later phases) + +- **Phase 2 — KDC: full SPNEGO E2E** including a host-side + `firefox`/`curl` that does negotiate against `localhost:8080`. + Needs `KRB5_CONFIG`, an SPN like `HTTP/localhost@KNOE.LOCAL`, a + keytab file mounted into knoe-auth's runtime. Phase 1 stops at the + KDC being reachable; Phase 2 makes SPNEGO actually work. +- **Phase 3 — image-build-and-load smoke** so engineers can run + `make k3d-knoe-deploy` and see knoe-auth as a pod in the cluster + before pushing to GKE. Useful as a pre-merge gate; not required + for daily dev. +- **Phase 4 — supabase stack on k3d** for end-to-end SDK testing + against a local DB. Heavy. Only if/when someone needs it. +- **Phase 5 — oauth2-proxy + Google OIDC on localhost** for testing + the Studio gate flow without going to GKE. Even heavier; needs a + Google OAuth client with `localhost` redirect URLs registered. + Probably never worth it. +- **Phase 2.5 (separate track) — persistent OAuth2 code/session + tables.** Currently `OidcCodeService` uses an in-memory + `ConcurrentHashMap`, which means codes don't survive a knoe-auth + restart. Not blocked by k3d work but useful to have once dev + iteration is fast enough that engineers exercise the flow often. + Tracked separately; would touch `authority/src/main/.../session/` + and add migrations under `authority/src/main/resources/db/migration/`. + +## 7. Risks / open questions + +- **Resource ceiling on a MacBook.** A single-replica CNPG plus KDC + is light (~600 MB RAM combined), but k3d itself + Docker Desktop + + IntelliJ + Spring Boot can climb. Document a "low-RAM mode" if + it becomes a real problem; don't speculate now. +- **k3d's Docker networking and `kubectl port-forward` UDP.** UDP + port-forward (port 88 for Kerberos UDP, 464 for kpasswd) works in + recent kubectl versions but has historically been finicky. If it + fails, fall back to TCP-only Kerberos (`tcp_only` in `krb5.conf`). + Phase 1 should set this in the engineer-side `krb5.local.conf` + preemptively. +- **CNPG operator CRD installation in fresh k3d.** The `make k3d-up` + flow today runs `init_cnpg_gke.sh` for GKE, but there's no + symmetric `init_cnpg_k3d.sh`. Either: (a) Junie writes a thin one + that just `kubectl apply -f`s the CNPG release manifest with no + GCP/IAM steps, or (b) the existing Python orchestrator + `knoe.core.ops.cloudnative_pg` already handles k3d (per the + survey, it does — `_strip_k3d_synology_blocks` is in there). + Verify which during Phase 1. +- **Schema bootstrap.** The GKE flow seeds `knoe.*` via + `etc/init_knoe_auth.sh` running SQL against the CNPG primary + through `kubectl exec`. The same approach works in k3d; the + brief should reuse the script (with a `KUBECONTEXT` swap), not + duplicate it. + +## 8. Verification (when Phase 1 is "done") + +A new engineer, freshly cloning the repo on a Mac: + +```bash +git clone && cd knoe-db +make k3d-knoe-up # provisions cluster, CNPG, KDC, schema +make k3d-knoe-pf & # port-forward in background +mvn -pl authority spring-boot:run +# in another terminal: +curl http://localhost:8080/health +# → ok +curl http://localhost:8080/.well-known/openid-configuration +# → JSON, issuer "http://localhost:8080" +``` + +Total elapsed time on a fresh laptop: ≤ 8 minutes. The engineer can +now edit `authority/src/main/java/dev/knoe/auth/...` and see changes +without leaving their editor. + +## 9. Glossary + +- **k3d** — `k3s` running inside a Docker container. Lightweight + Kubernetes for laptops; no VM needed. +- **CNPG** — `cloudnative-pg`, a PostgreSQL Kubernetes operator. We + run a single instance in k3d, three in GKE. +- **KDC** — Key Distribution Center, the Kerberos protocol's auth + server. SPNEGO browsers/clients fetch tickets from it; knoe-auth + validates SPNEGO challenges using these. +- **KNOE.LOCAL** — the Kerberos realm name we use for laptop dev. + Distinct from `KNOE.DEV` (production) so a stray ticket from one + doesn't authenticate you on the other. +- **SPNEGO** — `Negotiate` HTTP auth scheme. Carries a Kerberos + ticket inside an HTTP header. +- **Phase 2.5** — colloquial name for the gap between Phase 2 OIDC + (in-memory codes) and a fully persistent OIDC provider (DB-backed + codes/sessions). Not formally numbered; will get a name when work + starts.