docs(plans): file Phase 3 brief — knoe-auth as a pod inside k3d

Phase 1 (host loop) and the Phase 2 OIDC sandbox are shipped; this brief queues Phase 3 of k3d-mirror-of-GKE: build the knoe-auth image, k3d-import, run as a pod inside the cluster. Pre-merge smoke loop — not the daily inner loop. Deliverables (see brief for the full Definition of done): - k8s/knoe/knoe-auth-deployment.yaml (NEW; sibling of GKE manifest) - scripts/k3d-knoe-deploy.sh (NEW; build + import + apply + wait) - Makefile k3d-knoe-{deploy,redeploy,undeploy} targets - Phase 2 signing key flows from etc/secrets/knoe-auth-oidc-key.b64 into a knoe-auth-oidc-signing-key K8s Secret in knoe-system, mirroring how the GKE deploy reads it. Index updates: docs/plans/junie/README.md — k3d Phase 3 in Active row docs/TODO.md §"In progress" — promotes the brief to top docs/plans/k3d-gke-mirror.md — banner + §6 Phase 3 entry flipped from "out of scope" to "in flight"
2026-09-23 10:13:58 +00:00 · 2026-05-02 13:15:29 -07:00 · 2026-05-02 13:15:29 -07:00 · 2b36add592
commit 2b36add592
parent 903f84f200
4 changed files with 303 additions and 10 deletions
--- a/docs/TODO.md
+++ b/docs/TODO.md
@ -13,8 +13,7 @@ The Kanban "Now" section at top is the only place this doc imposes structure. Ev
 ## Now (Kanban)

 ### In progress
-
-*(none — all items shipped or paused)*
+- **k3d-mirror-of-GKE Phase 3 — knoe-auth as a pod inside k3d** — assigned to Junie. Pre-merge smoke loop: build the `knoe-auth:latest` image (using the `make docker-build-auth` target shipped in commit `903f84f`), import it into the k3d cluster (`k3d image import`), apply a k3d-flavoured `Deployment` manifest (sibling of `deploy/gcp/gke/knoe-auth-deployment.yaml`, but with `imagePullPolicy: Never` and realm `KNOE.LOCAL`), and run the keytab-bootstrap initContainer + KDC sidecar pattern in-cluster. New `make k3d-knoe-{deploy,redeploy,undeploy}` targets. Phase 2 OIDC signing key flows from `etc/secrets/knoe-auth-oidc-key.b64` into a K8s Secret (`knoe-auth-oidc-signing-key` in `knoe-system`) so the in-cluster pod gets it the same way GKE does. Brief: [`docs/plans/junie/k3d-knoe-auth-pod-deploy.md`](plans/junie/k3d-knoe-auth-pod-deploy.md). Architectural plan: [`docs/plans/k3d-gke-mirror.md`](plans/k3d-gke-mirror.md) §6 Phase 3.

 ### Paused
 - **pg_oauth in install/deploy.sh + k3d build** — *previously assigned to Junie; paused.* The Phase 1 work (per-engineer roles, external LB at `pg.0.knoe.dev`, cert SANs, pg_hba tightening, [`docs/db-access.md`](db-access.md)) is committed. This phase replaces the SCRAM password mechanism with PG18 native OAUTHBEARER, wires it into `install.sh` / `supabase/deploy.sh` so a fresh deploy lands with the OIDC mechanism active by default, and ports the Service + cluster cert pattern to k3d. Connection target stays at `pg.0.knoe.dev`; only the credential issuance changes (libpq Device Flow → knoe-auth Bearer token). OIDC issuer now deployed to GKE (2026-05-02) — pg_oauth can resume.
--- a/docs/plans/junie/README.md
+++ b/docs/plans/junie/README.md
@ -48,7 +48,9 @@ in TODO" note). When Junie lands a brief:

 ### Active (in flight)

-*(none — all briefs shipped)*
+| File | Tracked at | Subject |
+|---|---|---|
+| [`k3d-knoe-auth-pod-deploy.md`](k3d-knoe-auth-pod-deploy.md) | TODO §"In progress"; parent [`../k3d-gke-mirror.md`](../k3d-gke-mirror.md) §6 Phase 3 | Phase 3 of k3d-mirror-of-GKE: build the knoe-auth image, `k3d image import`, run as a pod inside the cluster. Pre-merge smoke loop with `make k3d-knoe-{deploy,redeploy,undeploy}`. |

 ### Shipped (kept as design record)

--- a/docs/plans/junie/k3d-knoe-auth-pod-deploy.md
+++ b/docs/plans/junie/k3d-knoe-auth-pod-deploy.md
@ -0,0 +1,289 @@
+# Junie brief — Phase 3 of k3d-mirror-of-GKE: knoe-auth as a pod inside k3d
+
+> **Self-contained brief.** Reference: parent architectural plan at
+> [`../k3d-gke-mirror.md`](../k3d-gke-mirror.md), §6 Phase 3. Phase 1 (CNPG +
+> KDC up; knoe-auth from host) and the Phase 2 OIDC k3d sandbox are both
+> shipped; chrisfu LOGIN role + `knoe_developer` group are seeded on every
+> `make k3d-knoe-up`. The next loop is **knoe-auth running as a pod inside
+> the k3d cluster**, exercising the deployed-pod shape (image pull,
+> mounted secrets, sidecar wiring) without pushing to GKE.
+
+---
+
+## 1. Why
+
+Today's daily loop runs knoe-auth from the laptop (`mvn spring-boot:run`)
+against k3d's CNPG + KDC. Fast inner loop, but it skips the parts that
+matter on production:
+
+- Image build correctness — does `authority/Dockerfile.app` actually
+  produce a runnable container? (Was added in commit `903f84f` for queue
+  #3 but never run end-to-end yet.)
+- Mounted-secret behaviour — does the pod read `KNOE_AUTH_OIDC_SIGNING_KEY`
+  from a K8s Secret, the way GKE does, vs. an env-var the engineer set
+  in their shell?
+- Sidecar networking — does the keytab-bootstrap initContainer + KDC
+  sidecar wiring in `deploy/gcp/gke/knoe-auth-deployment.yaml` work
+  unchanged on a single-node k3d cluster, or are there GKE-isms that
+  need a k3d variant?
+
+This brief delivers a `make k3d-knoe-deploy` target that wraps:
+
+1. Build the knoe-auth Docker image locally
+2. Import it into the k3d cluster
+3. Apply a k3d-flavoured deployment manifest
+4. Wait for the pod to be ready
+5. Print verification commands (port-forward + curl /health + /jwks.json)
+
+It's a **pre-merge gate**, not a daily inner loop. Engineers who care
+about catching deploy-shape regressions before pushing to GKE will run
+it; the host-side `mvn` loop stays the default for code iteration.
+
+## 2. What you produce
+
+Single-MR scope, all under `k8s/knoe/`, `scripts/`, and `Makefile`:
+
+### 2.1 — `k8s/knoe/knoe-auth-deployment.yaml` (NEW)
+
+A k3d-flavoured `Deployment` adapted from
+[`deploy/gcp/gke/knoe-auth-deployment.yaml`](../../deploy/gcp/gke/knoe-auth-deployment.yaml).
+Differences from the GKE version (handle each):
+
+| Concern | GKE | k3d |
+|---|---|---|
+| Image | `us-west3-docker.pkg.dev/.../knoe-auth:latest` (registry pull) | `knoe-auth:latest` with `imagePullPolicy: Never` (loaded via `k3d image import`) |
+| KDC sidecar realm | `KNOE.DEV` | `KNOE.LOCAL` (matches `etc/krb5.local.conf` and `application-k3d.yml`) |
+| Workload Identity | KSA → GSA bindings for GCP secret access | Not applicable; secrets are in-cluster |
+| Resource requests/limits | GKE node-aware (1Gi/1) | Drop or shrink (k3d single node has finite RAM) |
+| Probes | GKE-tuned timings | Keep; same Spring Boot endpoints |
+| OIDC signing key | from `knoe-auth-oidc` secret (1Password-fed) | from `knoe-auth-oidc-signing-key` secret created by the new init step (see 2.2) |
+| OIDC issuer | `https://api.knoe.dev/auth` | `http://knoe-auth.knoe-system.svc.cluster.local:8080` (in-cluster) **OR** keep the localhost-port-forward issuer `http://localhost:8080` and document the trade-off |
+
+The two-issuer choice deserves a comment in the manifest. Pick one and
+pin it; the other path is a future option.
+
+Don't try to make this manifest a copy of the GKE one — they'll diverge
+naturally and that's fine. Keep it readable.
+
+### 2.2 — Phase 2 OIDC signing key as a k3d Secret
+
+Today the host-side loop reads `etc/secrets/knoe-auth-oidc-key.b64` and
+exports `KNOE_AUTH_OIDC_SIGNING_KEY` as an env var. The in-cluster pod
+needs the same value as a K8s Secret.
+
+Add a step to `scripts/k3d-knoe-deploy.sh` (NEW — see 2.3) that:
+
+1. Runs `bash etc/gen_oidc_signing_key.sh` (idempotent, will be a no-op
+   if the key already exists from `make k3d-knoe-up`).
+2. Creates/updates a Secret in `knoe-system`:
+
+```bash
+kubectl --context=k3d-k3d-knoe -n knoe-system create secret generic \
+  knoe-auth-oidc-signing-key \
+  --from-literal=signingKey="$(cat etc/secrets/knoe-auth-oidc-key.b64)" \
+  --dry-run=client -o yaml | kubectl --context=k3d-k3d-knoe apply -f -
+```
+
+The deployment manifest (2.1) references this Secret via `valueFrom.secretKeyRef`.
+
+### 2.3 — `scripts/k3d-knoe-deploy.sh` (NEW)
+
+The orchestrator the new Make target calls. Sequence:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+# 1. Verify cluster + namespace exist (fail loudly if make k3d-knoe-up not run)
+# 2. Build the image (call into existing `make docker-build-auth`)
+# 3. k3d image import knoe-auth:latest -c knoe   # name from K3D_CLUSTER_NAME default
+# 4. Ensure OIDC signing-key Secret exists (gen + kubectl apply per 2.2)
+# 5. Apply k8s/knoe/knoe-auth-deployment.yaml
+# 6. kubectl rollout status deployment/knoe-auth -n knoe-system --timeout=180s
+# 7. Print port-forward instructions and the verification curls
+```
+
+Keep the script ≤ 100 lines; lean on existing functions/helpers when
+possible. The `make k3d-knoe-up` script's structure is a fine reference.
+
+### 2.4 — `Makefile` additions
+
+Three new targets:
+
+- `k3d-knoe-deploy` — calls `scripts/k3d-knoe-deploy.sh`. Depends
+  implicitly on `make k3d-knoe-up` having been run first; surface a
+  clear error message if the cluster context isn't reachable.
+- `k3d-knoe-redeploy` — convenience: rebuild image + re-import + restart
+  the deployment. Useful inner loop after the first `k3d-knoe-deploy`.
+  Implementation: `make docker-build-auth && k3d image import …` then
+  `kubectl rollout restart deployment/knoe-auth`.
+- `k3d-knoe-undeploy` — cleanup: `kubectl delete -f
+  k8s/knoe/knoe-auth-deployment.yaml`. Doesn't tear down the cluster
+  (use `k3d-knoe-down` for that).
+
+Update `make help` so the new targets appear under the existing
+"k3d dev loop targets" header.
+
+### 2.5 — Documentation
+
+Update [`docs/local-dev-knoe-auth.md`](../../docs/local-dev-knoe-auth.md)
+with a new section (insert before the existing "Reset / rebuild loop"
+section):
+
+```
+## Pre-merge smoke: knoe-auth as a pod inside k3d
+
+When you want to verify the deployed-pod shape (image build, mounted
+secrets, sidecar wiring) before pushing to GKE — typically after
+schema changes, env-var changes, or before opening a deploy MR:
+
+    make k3d-knoe-deploy
+
+This builds the image, imports it into the cluster, and applies the
+deployment.  After it succeeds:
+
+    kubectl --context=k3d-k3d-knoe -n knoe-system port-forward \
+      deployment/knoe-auth 8090:8080
+
+(Use 8090 to avoid colliding with your host-side `mvn spring-boot:run` on
+8080 if both are running.)
+
+    curl http://localhost:8090/health
+    curl http://localhost:8090/.well-known/openid-configuration
+
+Iteration:  `make k3d-knoe-redeploy`  (rebuild + restart)
+Cleanup:    `make k3d-knoe-undeploy`  (drops the deployment; cluster stays)
+```
+
+Update [`docs/knoe-system.md`](../../docs/knoe-system.md) §2 status table:
+add a row "Phase 3 in-cluster pod deploy" → **Shipped** once this brief
+lands.
+
+Update [`docs/plans/k3d-gke-mirror.md`](../k3d-gke-mirror.md) §6 — Phase 3
+flips from "out of scope" to "shipped" with a back-link to this brief.
+
+## 3. Don't break
+
+- The host-side `mvn spring-boot:run` daily loop must keep working.
+  Don't change `application-k3d.yml`, `etc/krb5.local.conf`, or the
+  port-forward script in ways that affect the host loop. The in-cluster
+  pod uses the same configmap-and-env-var inputs but in a different
+  arrangement.
+- The GKE-side deployment manifest at `deploy/gcp/gke/knoe-auth-deployment.yaml`
+  is **not** to be touched by this brief. The k3d manifest is a sibling,
+  not a replacement.
+- The KDC pod and its init Job (`k8s/knoe/knoe-kdc-*.yaml`) stay
+  unchanged. The knoe-auth pod talks to it via the existing
+  `service/knoe-kdc.knoe-system.svc.cluster.local` ClusterIP service.
+
+## 4. Verification (Definition of done)
+
+A fresh laptop:
+
+```bash
+git pull
+make k3d-knoe-up                                     # 3-5 min, includes chrisfu seed
+make k3d-knoe-deploy                                 # NEW — builds, imports, applies, waits
+# → ends with: knoe-auth deployment ready (1/1)
+
+kubectl --context=k3d-k3d-knoe -n knoe-system port-forward \
+  deployment/knoe-auth 8090:8080 &
+sleep 3
+
+curl -fsS http://localhost:8090/health
+# → {"status":"ok"}
+
+curl -fsS http://localhost:8090/.well-known/openid-configuration | jq .issuer
+# → "http://knoe-auth.knoe-system.svc.cluster.local:8080"  (or http://localhost:8080
+#    depending on your 2.1 issuer-pinning choice)
+
+curl -fsS http://localhost:8090/jwks.json | jq '.keys[0].kty'
+# → "RSA"
+
+# Iteration loop:
+make k3d-knoe-redeploy                                # ~30s
+# Verify the change you made is reflected:
+curl -fsS http://localhost:8090/.well-known/openid-configuration
+```
+
+Tick all of:
+
+- [ ] `make k3d-knoe-deploy` exits 0 on a fresh laptop with `make k3d-knoe-up`
+  already run.
+- [ ] `kubectl --context=k3d-k3d-knoe -n knoe-system get deploy knoe-auth`
+  reports `1/1 ready`.
+- [ ] Port-forwarded `/health`, `/.well-known/openid-configuration`,
+  `/jwks.json` all return 200 with the expected payloads.
+- [ ] `make k3d-knoe-redeploy` triggers a rolling restart and a fresh
+  build-and-import.
+- [ ] `make k3d-knoe-undeploy` removes the deployment without touching
+  the rest of the cluster.
+- [ ] Host-side `mvn -pl authority spring-boot:run -Dspring-boot.run.profiles=k3d`
+  still works (port 8080 free unless you're also running the in-cluster
+  pod's port-forward there).
+
+## 5. Out of scope
+
+- **SPNEGO E2E from a host browser** — needs a service principal for
+  `HTTP/localhost@KNOE.LOCAL` and a keytab the host knoe-auth or
+  in-cluster knoe-auth can read; that's still parent-plan §6 Phase 2 of
+  k3d-gke-mirror (different "Phase 2" from the OIDC one — yes the
+  numbering's noisy).
+- **Supabase stack on k3d** — parent §6 Phase 4. Not blocked by this
+  brief but separately worthwhile.
+- **`OidcCodeService` DB persistence** — the in-memory ConcurrentHashMap
+  is a known gap, same on GKE; tracked separately.
+- **Pushing knoe-auth to a GKE deploy** — Phase 2 OIDC GKE deploy was
+  Junie's separate brief (`docs/plans/junie/phase2-oidc-gke-deploy.md`).
+
+## 6. Commit shape
+
+One MR. Suggested message:
+
+```
+feat(k3d): knoe-auth as a pod inside k3d (k3d-gke-mirror Phase 3)
+
+Adds a pre-merge smoke loop: build the knoe-auth image, import it into
+the k3d cluster, run it as a pod with the keytab-bootstrap initContainer
+ KDC sidecar pattern. Lets engineers exercise deployed-pod-shape
+behaviour (image build, mounted-secret reads, sidecar networking)
+without pushing to GKE.
+
+Scope:
+  - k8s/knoe/knoe-auth-deployment.yaml NEW; k3d-flavoured (image pull
+    policy Never, realm KNOE.LOCAL, no Workload Identity).
+  - scripts/k3d-knoe-deploy.sh NEW; wraps build + import + secret-create
+    + apply + rollout-wait.
+  - Makefile: k3d-knoe-deploy, k3d-knoe-redeploy, k3d-knoe-undeploy targets.
+  - docs/local-dev-knoe-auth.md: new "Pre-merge smoke" section.
+  - docs/knoe-system.md, docs/plans/k3d-gke-mirror.md: status flips.
+
+Verified per the brief's Definition of done.
+
+Out of scope:
+  - SPNEGO E2E from host browsers (k3d-mirror Phase 2)
+  - Supabase stack on k3d (Phase 4)
+  - OidcCodeService DB persistence
+
+Closes Phase 3 of docs/plans/k3d-gke-mirror.md.
+```
+
+## 7. Notes for reading
+
+- The image build path lives in commit `903f84f`: `make docker-build-auth`
+  produces `knoe-auth:latest` locally; `make docker-push-auth` pushes to
+  Artifact Registry. **For Phase 3 you don't push** — you `k3d image
+  import` instead, and `imagePullPolicy: Never` on the deployment so
+  Kubernetes doesn't try to pull from a registry.
+- The OIDC signing key file `etc/secrets/knoe-auth-oidc-key.b64` is
+  already gitignored. Don't accidentally bake the key into the image
+  layer; mount via Secret only.
+- If the rolling restart in `k3d-knoe-redeploy` doesn't pick up a new
+  image (Kubernetes can be sticky about identical image tags), an
+  `imagePullPolicy: Never` + tag-rotation pattern (`:latest` →
+  `:dev-$(date +%s)`) inside the redeploy script may help. Optional —
+  only fix if you hit the issue.
+- If the cluster has insufficient resources to schedule both knoe-auth
+  and CNPG simultaneously, drop the resource requests/limits on the
+  knoe-auth deployment to `requests: {}` (k3d single-node has finite
+  capacity). Document if you do.
--- a/docs/plans/k3d-gke-mirror.md
+++ b/docs/plans/k3d-gke-mirror.md
@ -1,8 +1,10 @@
 # k3d-mirror-of-GKE — local platform model for knoe-auth dev

-> **Active plan. Phase 1 in flight (Junie).** Strategic context for a
-> running initiative; reads top-to-bottom. Phase 1 brief at
-> [`junie/k3d-knoe-auth-dev-loop.md`](junie/k3d-knoe-auth-dev-loop.md).
+> **Active plan. Phase 1 + Phase 2-OIDC-sandbox shipped 2026-05-02;
+> Phase 3 in flight (Junie).** Strategic context for a running initiative;
+> reads top-to-bottom. Brief refs:
+> [Phase 1 (host loop, shipped)](junie/k3d-knoe-auth-dev-loop.md) ·
+> [Phase 3 (in-cluster pod, in flight)](junie/k3d-knoe-auth-pod-deploy.md).

 ---

@ -174,10 +176,11 @@ Done = a 5-minute path from `git pull` → editing
  Needs `KRB5_CONFIG`, an SPN like `HTTP/localhost@KNOE.LOCAL`, a
  keytab file mounted into knoe-auth's runtime. Phase 1 stops at the
  KDC being reachable; Phase 2 makes SPNEGO actually work.
- **Phase 3 — image-build-and-load smoke** so engineers can run
-  `make k3d-knoe-deploy` and see knoe-auth as a pod in the cluster
-  before pushing to GKE. Useful as a pre-merge gate; not required
-  for daily dev.
+- **Phase 3 — image-build-and-load smoke** ([brief](junie/k3d-knoe-auth-pod-deploy.md),
+  in flight 2026-05-02). Engineers run `make k3d-knoe-deploy` and see
+  knoe-auth as a pod in the cluster, exercising image build, mounted
+  secrets, and KDC-sidecar wiring without pushing to GKE. Pre-merge
+  gate; not required for daily dev.
 - **Phase 4 — supabase stack on k3d** for end-to-end SDK testing
  against a local DB. Heavy. Only if/when someone needs it.
 - **Phase 5 — oauth2-proxy + Google OIDC on localhost** for testing