prole/docs/completed/k3d-knoe-auth-dev-loop.md

# Junie brief — Phase 1 of k3d-mirror-of-GKE: laptop dev loop for knoe-auth

> **Self-contained brief.** Reference: parent architectural plan at
> [`../k3d-gke-mirror.md`](../k3d-gke-mirror.md). Read that for the
> "why" and the multi-phase shape. This brief delivers Phase 1 only.

---

## 1. Why

knoe-auth Phase 2 OIDC code is on `main` but no fast inner-loop
exists for iterating on it. Hitting the GKE cluster on every change
is slow. Running unit tests with testcontainers covers a lot but
doesn't exercise the real DB schema, Kerberos KDC, or the actual HTTP
surface against a Postgres+JDBC stack.

This brief wires up a **laptop-resident dev loop**: bring up CNPG +
KDC in k3d, port-forward 5432/88/464 to localhost, and let the
engineer run knoe-auth from their IDE/`mvn spring-boot:run` against
those local ports. Edit Java → re-run → see the change at
`http://localhost:8080`, with a real PostgreSQL and a real Kerberos
backing it.

knoe-auth itself stays on the host. Image-build-and-load into k3d is
a separate later phase (and is in fact mostly orthogonal to this
work — Phase 1 finishes when the host loop is solid).

## 2. What you produce

Concrete deliverables, all in one MR:

### 2.1 — KDC manifest set for k3d

New files (mirroring the auth-side filename pattern under `k8s/knoe/`):

- `k8s/knoe/knoe-kdc-deployment.yaml` — single-replica `Deployment`
  running `krb5kdc` + `kadmind`. Realm `KNOE.LOCAL`. Use a
  Debian-derived MIT Kerberos image; whatever the GKE sidecar image
  uses is fine to reuse. **Set `realm = KNOE.LOCAL`** explicitly —
  do NOT inherit from the existing `KNOE.DEV` GKE config.
- `k8s/knoe/knoe-kdc-service.yaml` — ClusterIP service exposing 88
  (TCP+UDP) and 464 (TCP+UDP — kpasswd).
- `k8s/knoe/knoe-kdc-configmap.yaml` — `krb5.conf` + `kdc.conf` with
  realm `KNOE.LOCAL` and a `[realms] KNOE.LOCAL = { kdc = knoe-kdc:88 ... }`
  block.
- `k8s/knoe/knoe-kdc-pvc.yaml` — small PVC (1 Gi) for the KDC
  database `/var/lib/krb5kdc/principal`. Local-path provisioner
  (k3d default).
- `k8s/knoe/knoe-kdc-init-job.yaml` — one-shot `Job` that runs
  `kdb5_util create -s -P <stash>` (database init) and then
  `kadmin.local addprinc -randkey admin/admin@KNOE.LOCAL` plus a
  developer principal. Idempotent (skip create if DB exists). Runs
  once on first apply; subsequent apply is a no-op via a guard
  inside the script.

Reference existing GKE `kdc.conf` and `krb5.conf` shapes from
[`deploy/gcp/gke/knoe-kdc-configmap.yaml`](../../deploy/gcp/gke/knoe-kdc-configmap.yaml).
Substitute the realm, drop GCE-specific bits.

### 2.2 — Init-script entry point for k3d

Either:

- **(preferred)** Extend `etc/init_knoe_auth.sh` with a `--mode k3d`
  flag that swaps the `APP_CTX` default to `k3d-knoe` (or whatever
  `kubectl config current-context` returns from the k3d-up flow),
  swaps the realm to `KNOE.LOCAL`, and skips the GCP/Workload
  Identity steps (no GCP project on a laptop). Same `schema` /
  `invite` / `status` subcommands work.
- **(or)** Write a thin `etc/init_knoe_auth_k3d.sh` that wraps the
  existing one with the right env vars set.

Pick whichever results in less duplication. The existing script's
`schema` subcommand already runs the SQL via `kubectl exec` on the
CNPG primary — that path works in k3d unchanged.

### 2.3 — Make targets

Three new targets in the top-level `Makefile`:

- **`make k3d-knoe-up`** — provisions the local k3d cluster (if not
  present), installs the CNPG operator, applies
  `k8s/knoe/knoe-db.yaml` (single-replica per the architectural
  plan §3), waits for `Cluster in healthy state`, runs the
  knoe-kdc init job, applies the KDC Deployment + Service, runs
  `etc/init_knoe_auth.sh schema` to seed `knoe.*` tables.
- **`make k3d-knoe-pf`** — opens three port-forwards in the
  foreground:
    - `5432` → `service/knoe-db-rw -n knoe-db-0`
    - `88`/`464` → `service/knoe-kdc -n knoe-system` (TCP and UDP both)
  - Prints the JDBC URL and the env vars the engineer should
    `export` (e.g. `KRB5_CONFIG=$PWD/etc/krb5.local.conf`,
    `KNOE_DB_PASSWORD=...`).
  - On ^C, cleans up all three forwards.
- **`make k3d-knoe-down`** — tears down the k3d cluster (or just
  the namespaces if you want incremental cleanup).

The cluster name should be `k3d-knoe` (so the kubeconfig context is
`k3d-k3d-knoe`, matching k3d's prefix convention). Reuse anything in
`scripts/` that already drives k3d if it's there.

### 2.4 — Engineer-side krb5 config

A new file `etc/krb5.local.conf` checked into the repo:

```ini
[libdefaults]
  default_realm = KNOE.LOCAL
  rdns = false
  forwardable = true
  udp_preference_limit = 1   # workaround for kubectl port-forward UDP flakiness; force TCP

[realms]
  KNOE.LOCAL = {
    kdc = localhost:88
    admin_server = localhost:749
    default_domain = local
  }

[domain_realm]
  .local = KNOE.LOCAL
  localhost = KNOE.LOCAL
```

Engineer's runtime: `export KRB5_CONFIG=$PWD/etc/krb5.local.conf` and
the JVM picks it up automatically.

### 2.5 — Engineer-facing doc

New `docs/local-dev-knoe-auth.md`. Sections:

1. **Prerequisites** — k3d, kubectl, Docker Desktop running, JDK 21,
   Maven. Brew one-liner.
2. **One-time setup** — `make k3d-knoe-up`. Wait ~3 minutes. Verify
   `kubectl --context k3d-k3d-knoe -n knoe-db-0 get cluster knoe-db`
   shows `Healthy`.
3. **Daily loop** — open two terminals.
   - Terminal A: `make k3d-knoe-pf` (leave running).
   - Terminal B: `export KRB5_CONFIG=$PWD/etc/krb5.local.conf` then
     `mvn -pl authority spring-boot:run`.
4. **Verify the loop** — `curl localhost:8080/health` returns `ok`;
   `curl localhost:8080/.well-known/openid-configuration` returns
   the OIDC discovery JSON; `psql
   "postgresql://postgres:$(make show-db-password)@localhost:5432/knoe-db?sslmode=require"`
   connects.
5. **`kinit` for SPNEGO testing** — `kinit -k -t <keytab>
   admin/admin@KNOE.LOCAL` (full SPNEGO E2E is Phase 2; document
   the expected limitations: `curl --negotiate -u : http://localhost:8080/`
   may or may not fully round-trip on macOS depending on
   `udp_preference_limit`).
6. **IntelliJ run configuration** — paste-ready text or
   screenshot. Module `authority`, main class
   `dev.knoe.auth.KnoeAuthApplication`, env vars
   `KRB5_CONFIG=…/etc/krb5.local.conf`,
   `KNOE_DB_PASSWORD=…`.
7. **Reset** — `make k3d-knoe-down && make k3d-knoe-up`. Time: ~3 min.

### 2.6 — Smoke-test script

`scripts/k3d-knoe-smoke.sh` — bash, runs after `make k3d-knoe-up`
to confirm the loop is intact:

```bash
#!/usr/bin/env bash
set -euo pipefail
ctx=k3d-k3d-knoe
ns_db=knoe-db-0
ns_sys=knoe-system

# 1. CNPG cluster healthy
phase=$(kubectl --context=$ctx -n $ns_db get cluster knoe-db -o jsonpath='{.status.phase}')
[[ "$phase" == "Cluster in healthy state" ]] || { echo "CNPG: $phase"; exit 1; }

# 2. knoe.* schema present
tables=$(kubectl --context=$ctx -n $ns_db exec sts/knoe-db-1 -c postgres -- \
  psql -U postgres -tAc "SELECT count(*) FROM information_schema.tables WHERE table_schema='knoe'")
[[ "$tables" -ge 6 ]] || { echo "knoe.* tables: $tables (expected ≥6)"; exit 1; }

# 3. KDC has the seed admin principal
kubectl --context=$ctx -n $ns_sys exec deploy/knoe-kdc -- \
  kadmin.local listprincs | grep -q 'admin/admin@KNOE.LOCAL' \
  || { echo "missing admin principal"; exit 1; }

echo "k3d-knoe-smoke: PASS"
```

CI integration is out of scope for Phase 1 — this is engineer-run.

## 3. Don't break

- The existing GKE knoe-auth deployment must continue to work
  unchanged. If you touch `etc/init_knoe_auth.sh`, the GKE
  invocation must remain the default behavior (no `--mode` flag =
  current GKE behavior).
- The k3s manifests under `deploy/opentofu/k3s/manifests/knoe/`
  must NOT be modified. The k3s deploy mode targets a different
  use case (on-prem VM); leave it alone.
- The CNPG cluster manifest at `k8s/knoe/knoe-db.yaml` must stay
  the canonical k3d/min Postgres definition. If you need it to
  behave differently for k3d (e.g. single-replica), make the change
  configurable, not destructive — `_strip_k3d_synology_blocks` in
  `knoe/core/ops/cloudnative_pg.py` is the existing precedent for
  k3d-specific manifest manipulation.

## 4. Verification (Definition of done)

A new engineer, freshly cloning the repo, completes this in ≤ 8
minutes on a MacBook Air:

```bash
git clone <repo> && cd knoe-db
make k3d-knoe-up                            # 3-5 min
make k3d-knoe-pf &                          # foreground; backgrounded for this script
sleep 3
scripts/k3d-knoe-smoke.sh                   # PASS
export KRB5_CONFIG=$PWD/etc/krb5.local.conf
mvn -pl authority spring-boot:run &         # backgrounded for the verify
sleep 30
curl -fsS http://localhost:8080/health      # ok
curl -fsS http://localhost:8080/.well-known/openid-configuration | jq .issuer
# → "http://localhost:8080"
```

Tick all of:

- [ ] `make k3d-knoe-up` exits 0 on a fresh laptop with k3d/kubectl
  installed.
- [ ] `kubectl --context=k3d-k3d-knoe -n knoe-db-0 get cluster knoe-db`
  reports `Healthy`.
- [ ] `kubectl --context=k3d-k3d-knoe -n knoe-system get deploy knoe-kdc`
  reports `1/1 ready`.
- [ ] `scripts/k3d-knoe-smoke.sh` reports `PASS`.
- [ ] Host-side `psql` connects via `localhost:5432` after
  `make k3d-knoe-pf` is running.
- [ ] `kinit admin/admin@KNOE.LOCAL` from the host succeeds.
- [ ] `mvn -pl authority spring-boot:run` starts cleanly with
  `KRB5_CONFIG=etc/krb5.local.conf` set; `/health` returns `ok`;
  `/.well-known/openid-configuration` returns valid JSON.
- [ ] `make k3d-knoe-down && make k3d-knoe-up` is idempotent.
- [ ] `docs/local-dev-knoe-auth.md` exists and is accurate.
- [ ] GKE deploy mode unchanged: `git diff main -- deploy/gcp/gke/`
  shows no changes; the existing init scripts behave identically
  when invoked the same way they were before.

## 5. Out of scope

Captured in the parent plan §6, but call out the highest-friction
deferrals here so reviewer expectations are right:

- **SPNEGO E2E from a host browser** — requires a service principal
  for `HTTP/localhost@KNOE.LOCAL` and a keytab the host knoe-auth
  process can read. Phase 1 stops at "KDC reachable, principals
  exist, `kinit` works." Phase 2 makes browser SPNEGO actually
  authenticate.
- **knoe-auth as a pod inside k3d** — image-build-and-load. Phase 3.
  Don't take this on now.
- **Supabase / Studio / Kong / oauth2-proxy on k3d** — all explicit
  Phase 4+.
- **OIDC code persistence in DB** — `OidcCodeService` uses an
  in-memory `ConcurrentHashMap` today. That's a real Phase 2.5 gap
  but it's the same gap on GKE; not k3d-specific. Don't fold it
  into this brief.

## 6. Commit shape

Single commit unless the deliverables genuinely separate into two
clean halves (e.g. infra + docs). Suggested message:

```
feat(k3d): laptop dev loop for knoe-auth — CNPG + KDC + port-forward

Brings up the smallest k3d-resident stack that lets a host-side
knoe-auth (run via mvn spring-boot:run or IntelliJ) iterate against
real Postgres + Kerberos. Closes Phase 1 of the k3d-gke-mirror plan.

Scope:
  - k8s/knoe/knoe-kdc-{deployment,service,configmap,pvc,init-job}.yaml
    NEW; standalone KDC, realm KNOE.LOCAL (distinct from KNOE.DEV).
  - etc/init_knoe_auth.sh: --mode k3d flag, swaps realm + skips
    GCP-specific steps. GKE behavior unchanged when flag absent.
  - Makefile: k3d-knoe-up, k3d-knoe-pf, k3d-knoe-down.
  - etc/krb5.local.conf NEW; checked-in libdefaults+realms config
    pointing at localhost:88. udp_preference_limit=1 to dodge
    kubectl port-forward UDP flakiness.
  - docs/local-dev-knoe-auth.md NEW; one-time setup + daily loop.
  - scripts/k3d-knoe-smoke.sh NEW; bash sanity script.

Verified by following docs/local-dev-knoe-auth.md from a fresh
clone: end-to-end in <8 min, knoe-auth at localhost:8080 hits
real DB + KDC; OIDC discovery returns valid JSON.

Out of scope (parent plan docs/plans/k3d-gke-mirror.md §6):
  - SPNEGO from host browsers
  - knoe-auth-as-pod (image build/load)
  - Supabase stack
  - OidcCodeService DB persistence

Closes Phase 1; Phase 2+ briefs filed as needed.
```

## 7. Notes for reading

- The user is happy to **decide between two paths if you surface
  them clearly** — e.g. "extend `init_knoe_auth.sh` with `--mode
  k3d` vs write a sibling `init_knoe_auth_k3d.sh`". Pick the one
  that minimizes drift and explain why in the commit.
- The user runs macOS. UDP port-forward through kubectl on Mac has
  historically been finicky; setting `udp_preference_limit = 1` in
  `krb5.local.conf` (force TCP) is the recommended workaround and is
  why it's in the example above.
- If you run into resource ceiling issues on the laptop (Docker
  Desktop OOM, k3d node not ready), document the workaround in the
  engineer-side doc rather than working around it in the manifests.
- If the existing Python orchestrator
  (`knoe.core.ops.cloudnative_pg`) already handles the
  k3d-CNPG-bring-up cleanly, leverage it rather than writing a
  shell wrapper from scratch. The Make target can call into it via
  `python -m knoe.core.ops.cloudnative_pg ...` if the entrypoints
  exist; if they don't, a bash wrapper is fine.