prole/docs/completed/k3d-knoe-auth-dev-loop.md

Junie brief — Phase 1 of k3d-mirror-of-GKE: laptop dev loop for knoe-auth

Self-contained brief. Reference: parent architectural plan at ../k3d-gke-mirror.md. Read that for the "why" and the multi-phase shape. This brief delivers Phase 1 only.

---

1. Why

knoe-auth Phase 2 OIDC code is on main but no fast inner-loop exists for iterating on it. Hitting the GKE cluster on every change is slow. Running unit tests with testcontainers covers a lot but doesn't exercise the real DB schema, Kerberos KDC, or the actual HTTP surface against a Postgres+JDBC stack.

This brief wires up a laptop-resident dev loop: bring up CNPG + KDC in k3d, port-forward 5432/88/464 to localhost, and let the engineer run knoe-auth from their IDE/mvn spring-boot:run against those local ports. Edit Java → re-run → see the change at http://localhost:8080, with a real PostgreSQL and a real Kerberos backing it.

knoe-auth itself stays on the host. Image-build-and-load into k3d is a separate later phase (and is in fact mostly orthogonal to this work — Phase 1 finishes when the host loop is solid).

2. What you produce

Concrete deliverables, all in one MR:

2.1 — KDC manifest set for k3d

New files (mirroring the auth-side filename pattern under k8s/knoe/):

• k8s/knoe/knoe-kdc-deployment.yaml — single-replica Deployment running krb5kdc + kadmind. Realm KNOE.LOCAL. Use a Debian-derived MIT Kerberos image; whatever the GKE sidecar image uses is fine to reuse. Set realm = KNOE.LOCAL explicitly — do NOT inherit from the existing KNOE.DEV GKE config.
• k8s/knoe/knoe-kdc-service.yaml — ClusterIP service exposing 88 (TCP+UDP) and 464 (TCP+UDP — kpasswd).
• k8s/knoe/knoe-kdc-configmap.yaml — krb5.conf + kdc.conf with realm KNOE.LOCAL and a [realms] KNOE.LOCAL = { kdc = knoe-kdc:88 ... } block.
• k8s/knoe/knoe-kdc-pvc.yaml — small PVC (1 Gi) for the KDC database /var/lib/krb5kdc/principal. Local-path provisioner (k3d default).
• k8s/knoe/knoe-kdc-init-job.yaml — one-shot Job that runs kdb5_util create -s -P <stash> (database init) and then kadmin.local addprinc -randkey admin/admin@KNOE.LOCAL plus a developer principal. Idempotent (skip create if DB exists). Runs once on first apply; subsequent apply is a no-op via a guard inside the script.

Reference existing GKE kdc.conf and krb5.conf shapes from deploy/gcp/gke/knoe-kdc-configmap.yaml. Substitute the realm, drop GCE-specific bits.

2.2 — Init-script entry point for k3d

Either:

• (preferred) Extend etc/init_knoe_auth.sh with a --mode k3d flag that swaps the APP_CTX default to k3d-knoe (or whatever kubectl config current-context returns from the k3d-up flow), swaps the realm to KNOE.LOCAL, and skips the GCP/Workload Identity steps (no GCP project on a laptop). Same schema / invite / status subcommands work.
• (or) Write a thin etc/init_knoe_auth_k3d.sh that wraps the existing one with the right env vars set.

Pick whichever results in less duplication. The existing script's schema subcommand already runs the SQL via kubectl exec on the CNPG primary — that path works in k3d unchanged.

2.3 — Make targets

Three new targets in the top-level Makefile:

• make k3d-knoe-up — provisions the local k3d cluster (if not present), installs the CNPG operator, applies k8s/knoe/knoe-db.yaml (single-replica per the architectural plan §3), waits for Cluster in healthy state, runs the knoe-kdc init job, applies the KDC Deployment + Service, runs etc/init_knoe_auth.sh schema to seed knoe.* tables.
• make k3d-knoe-pf — opens three port-forwards in the foreground:
  • 5432 → service/knoe-db-rw -n knoe-db-0
  • 88/464 → service/knoe-kdc -n knoe-system (TCP and UDP both)
  • Prints the JDBC URL and the env vars the engineer should export (e.g. KRB5_CONFIG=$PWD/etc/krb5.local.conf, KNOE_DB_PASSWORD=...).
  • On ^C, cleans up all three forwards.
• make k3d-knoe-down — tears down the k3d cluster (or just the namespaces if you want incremental cleanup).

The cluster name should be k3d-knoe (so the kubeconfig context is k3d-k3d-knoe, matching k3d's prefix convention). Reuse anything in scripts/ that already drives k3d if it's there.

2.4 — Engineer-side krb5 config

A new file etc/krb5.local.conf checked into the repo:

[libdefaults]
  default_realm = KNOE.LOCAL
  rdns = false
  forwardable = true
  udp_preference_limit = 1   # workaround for kubectl port-forward UDP flakiness; force TCP

[realms]
  KNOE.LOCAL = {
    kdc = localhost:88
    admin_server = localhost:749
    default_domain = local
  }

[domain_realm]
  .local = KNOE.LOCAL
  localhost = KNOE.LOCAL

Engineer's runtime: export KRB5_CONFIG=$PWD/etc/krb5.local.conf and the JVM picks it up automatically.

2.5 — Engineer-facing doc

New docs/local-dev-knoe-auth.md. Sections:

1. Prerequisites — k3d, kubectl, Docker Desktop running, JDK 21, Maven. Brew one-liner.
2. One-time setup — make k3d-knoe-up. Wait ~3 minutes. Verify kubectl --context k3d-k3d-knoe -n knoe-db-0 get cluster knoe-db shows Healthy.
3. Daily loop — open two terminals.
   • Terminal A: make k3d-knoe-pf (leave running).
   • Terminal B: export KRB5_CONFIG=$PWD/etc/krb5.local.conf then mvn -pl authority spring-boot:run.
4. Verify the loop — curl localhost:8080/health returns ok; curl localhost:8080/.well-known/openid-configuration returns the OIDC discovery JSON; psql "postgresql://postgres:$(make show-db-password)@localhost:5432/knoe-db?sslmode=require" connects.
5. kinit for SPNEGO testing — kinit -k -t <keytab> admin/admin@KNOE.LOCAL (full SPNEGO E2E is Phase 2; document the expected limitations: curl --negotiate -u : http://localhost:8080/ may or may not fully round-trip on macOS depending on udp_preference_limit).
6. IntelliJ run configuration — paste-ready text or screenshot. Module authority, main class dev.knoe.auth.KnoeAuthApplication, env vars KRB5_CONFIG=…/etc/krb5.local.conf, KNOE_DB_PASSWORD=….
7. Reset — make k3d-knoe-down && make k3d-knoe-up. Time: ~3 min.

2.6 — Smoke-test script

scripts/k3d-knoe-smoke.sh — bash, runs after make k3d-knoe-up to confirm the loop is intact:

#!/usr/bin/env bash
set -euo pipefail
ctx=k3d-k3d-knoe
ns_db=knoe-db-0
ns_sys=knoe-system

# 1. CNPG cluster healthy
phase=$(kubectl --context=$ctx -n $ns_db get cluster knoe-db -o jsonpath='{.status.phase}')
[[ "$phase" == "Cluster in healthy state" ]] || { echo "CNPG: $phase"; exit 1; }

# 2. knoe.* schema present
tables=$(kubectl --context=$ctx -n $ns_db exec sts/knoe-db-1 -c postgres -- \
  psql -U postgres -tAc "SELECT count(*) FROM information_schema.tables WHERE table_schema='knoe'")
[[ "$tables" -ge 6 ]] || { echo "knoe.* tables: $tables (expected ≥6)"; exit 1; }

# 3. KDC has the seed admin principal
kubectl --context=$ctx -n $ns_sys exec deploy/knoe-kdc -- \
  kadmin.local listprincs | grep -q 'admin/admin@KNOE.LOCAL' \
  || { echo "missing admin principal"; exit 1; }

echo "k3d-knoe-smoke: PASS"

CI integration is out of scope for Phase 1 — this is engineer-run.

3. Don't break

• The existing GKE knoe-auth deployment must continue to work unchanged. If you touch etc/init_knoe_auth.sh, the GKE invocation must remain the default behavior (no --mode flag = current GKE behavior).
• The k3s manifests under deploy/opentofu/k3s/manifests/knoe/ must NOT be modified. The k3s deploy mode targets a different use case (on-prem VM); leave it alone.
• The CNPG cluster manifest at k8s/knoe/knoe-db.yaml must stay the canonical k3d/min Postgres definition. If you need it to behave differently for k3d (e.g. single-replica), make the change configurable, not destructive — _strip_k3d_synology_blocks in knoe/core/ops/cloudnative_pg.py is the existing precedent for k3d-specific manifest manipulation.

4. Verification (Definition of done)

A new engineer, freshly cloning the repo, completes this in ≤ 8 minutes on a MacBook Air:

git clone <repo> && cd knoe-db
make k3d-knoe-up                            # 3-5 min
make k3d-knoe-pf &                          # foreground; backgrounded for this script
sleep 3
scripts/k3d-knoe-smoke.sh                   # PASS
export KRB5_CONFIG=$PWD/etc/krb5.local.conf
mvn -pl authority spring-boot:run &         # backgrounded for the verify
sleep 30
curl -fsS http://localhost:8080/health      # ok
curl -fsS http://localhost:8080/.well-known/openid-configuration | jq .issuer
# → "http://localhost:8080"

Tick all of:

• make k3d-knoe-up exits 0 on a fresh laptop with k3d/kubectl installed.
• kubectl --context=k3d-k3d-knoe -n knoe-db-0 get cluster knoe-db reports Healthy.
• kubectl --context=k3d-k3d-knoe -n knoe-system get deploy knoe-kdc reports 1/1 ready.
• scripts/k3d-knoe-smoke.sh reports PASS.
• Host-side psql connects via localhost:5432 after make k3d-knoe-pf is running.
• kinit admin/admin@KNOE.LOCAL from the host succeeds.
• mvn -pl authority spring-boot:run starts cleanly with KRB5_CONFIG=etc/krb5.local.conf set; /health returns ok; /.well-known/openid-configuration returns valid JSON.
• make k3d-knoe-down && make k3d-knoe-up is idempotent.
• docs/local-dev-knoe-auth.md exists and is accurate.
• GKE deploy mode unchanged: git diff main -- deploy/gcp/gke/ shows no changes; the existing init scripts behave identically when invoked the same way they were before.

5. Out of scope

Captured in the parent plan §6, but call out the highest-friction deferrals here so reviewer expectations are right:

• SPNEGO E2E from a host browser — requires a service principal for HTTP/localhost@KNOE.LOCAL and a keytab the host knoe-auth process can read. Phase 1 stops at "KDC reachable, principals exist, kinit works." Phase 2 makes browser SPNEGO actually authenticate.
• knoe-auth as a pod inside k3d — image-build-and-load. Phase 3. Don't take this on now.
• Supabase / Studio / Kong / oauth2-proxy on k3d — all explicit Phase 4+.
• OIDC code persistence in DB — OidcCodeService uses an in-memory ConcurrentHashMap today. That's a real Phase 2.5 gap but it's the same gap on GKE; not k3d-specific. Don't fold it into this brief.

6. Commit shape

Single commit unless the deliverables genuinely separate into two clean halves (e.g. infra + docs). Suggested message:

feat(k3d): laptop dev loop for knoe-auth — CNPG + KDC + port-forward

Brings up the smallest k3d-resident stack that lets a host-side
knoe-auth (run via mvn spring-boot:run or IntelliJ) iterate against
real Postgres + Kerberos. Closes Phase 1 of the k3d-gke-mirror plan.

Scope:
  - k8s/knoe/knoe-kdc-{deployment,service,configmap,pvc,init-job}.yaml
    NEW; standalone KDC, realm KNOE.LOCAL (distinct from KNOE.DEV).
  - etc/init_knoe_auth.sh: --mode k3d flag, swaps realm + skips
    GCP-specific steps. GKE behavior unchanged when flag absent.
  - Makefile: k3d-knoe-up, k3d-knoe-pf, k3d-knoe-down.
  - etc/krb5.local.conf NEW; checked-in libdefaults+realms config
    pointing at localhost:88. udp_preference_limit=1 to dodge
    kubectl port-forward UDP flakiness.
  - docs/local-dev-knoe-auth.md NEW; one-time setup + daily loop.
  - scripts/k3d-knoe-smoke.sh NEW; bash sanity script.

Verified by following docs/local-dev-knoe-auth.md from a fresh
clone: end-to-end in <8 min, knoe-auth at localhost:8080 hits
real DB + KDC; OIDC discovery returns valid JSON.

Out of scope (parent plan docs/plans/k3d-gke-mirror.md §6):
  - SPNEGO from host browsers
  - knoe-auth-as-pod (image build/load)
  - Supabase stack
  - OidcCodeService DB persistence

Closes Phase 1; Phase 2+ briefs filed as needed.

7. Notes for reading

• The user is happy to decide between two paths if you surface them clearly — e.g. "extend init_knoe_auth.sh with --mode k3d vs write a sibling init_knoe_auth_k3d.sh". Pick the one that minimizes drift and explain why in the commit.
• The user runs macOS. UDP port-forward through kubectl on Mac has historically been finicky; setting udp_preference_limit = 1 in krb5.local.conf (force TCP) is the recommended workaround and is why it's in the example above.
• If you run into resource ceiling issues on the laptop (Docker Desktop OOM, k3d node not ready), document the workaround in the engineer-side doc rather than working around it in the manifests.
• If the existing Python orchestrator (knoe.core.ops.cloudnative_pg) already handles the k3d-CNPG-bring-up cleanly, leverage it rather than writing a shell wrapper from scratch. The Make target can call into it via python -m knoe.core.ops.cloudnative_pg ... if the entrypoints exist; if they don't, a bash wrapper is fine.