prole/docs/plans/junie/k3d-knoe-auth-pod-deploy.md

# Junie brief — Phase 3 of k3d-mirror-of-GKE: knoe-auth as a pod inside k3d

> **Self-contained brief.** Reference: parent architectural plan at
> [`../k3d-gke-mirror.md`](../k3d-gke-mirror.md), §6 Phase 3. Phase 1 (CNPG +
> KDC up; knoe-auth from host) and the Phase 2 OIDC k3d sandbox are both
> shipped; chrisfu LOGIN role + `knoe_developer` group are seeded on every
> `make k3d-knoe-up`. The next loop is **knoe-auth running as a pod inside
> the k3d cluster**, exercising the deployed-pod shape (image pull,
> mounted secrets, sidecar wiring) without pushing to GKE.

---

## 1. Why

Today's daily loop runs knoe-auth from the laptop (`mvn spring-boot:run`)
against k3d's CNPG + KDC. Fast inner loop, but it skips the parts that
matter on production:

- Image build correctness — does `authority/Dockerfile.app` actually
  produce a runnable container? (Was added in commit `903f84f` for queue
  #3 but never run end-to-end yet.)
- Mounted-secret behaviour — does the pod read `KNOE_AUTH_OIDC_SIGNING_KEY`
  from a K8s Secret, the way GKE does, vs. an env-var the engineer set
  in their shell?
- Sidecar networking — does the keytab-bootstrap initContainer + KDC
  sidecar wiring in `deploy/gcp/gke/knoe-auth-deployment.yaml` work
  unchanged on a single-node k3d cluster, or are there GKE-isms that
  need a k3d variant?

This brief delivers a `make k3d-knoe-deploy` target that wraps:

1. Build the knoe-auth Docker image locally
2. Import it into the k3d cluster
3. Apply a k3d-flavoured deployment manifest
4. Wait for the pod to be ready
5. Print verification commands (port-forward + curl /health + /jwks.json)

It's a **pre-merge gate**, not a daily inner loop. Engineers who care
about catching deploy-shape regressions before pushing to GKE will run
it; the host-side `mvn` loop stays the default for code iteration.

## 2. What you produce

Single-MR scope, all under `k8s/knoe/`, `scripts/`, and `Makefile`:

### 2.1 — `k8s/knoe/knoe-auth-deployment.yaml` (NEW)

A k3d-flavoured `Deployment` adapted from
[`deploy/gcp/gke/knoe-auth-deployment.yaml`](../../deploy/gcp/gke/knoe-auth-deployment.yaml).
Differences from the GKE version (handle each):

| Concern | GKE | k3d |
|---|---|---|
| Image | `us-west3-docker.pkg.dev/.../knoe-auth:latest` (registry pull) | `knoe-auth:latest` with `imagePullPolicy: Never` (loaded via `k3d image import`) |
| KDC sidecar realm | `KNOE.DEV` | `KNOE.LOCAL` (matches `etc/krb5.local.conf` and `application-k3d.yml`) |
| Workload Identity | KSA → GSA bindings for GCP secret access | Not applicable; secrets are in-cluster |
| Resource requests/limits | GKE node-aware (1Gi/1) | Drop or shrink (k3d single node has finite RAM) |
| Probes | GKE-tuned timings | Keep; same Spring Boot endpoints |
| OIDC signing key | from `knoe-auth-oidc` secret (1Password-fed) | from `knoe-auth-oidc-signing-key` secret created by the new init step (see 2.2) |
| OIDC issuer | `https://api.knoe.dev/auth` | `http://knoe-auth.knoe-system.svc.cluster.local:8080` (in-cluster) **OR** keep the localhost-port-forward issuer `http://localhost:8080` and document the trade-off |

The two-issuer choice deserves a comment in the manifest. Pick one and
pin it; the other path is a future option.

Don't try to make this manifest a copy of the GKE one — they'll diverge
naturally and that's fine. Keep it readable.

### 2.2 — Phase 2 OIDC signing key as a k3d Secret

Today the host-side loop reads `etc/secrets/knoe-auth-oidc-key.b64` and
exports `KNOE_AUTH_OIDC_SIGNING_KEY` as an env var. The in-cluster pod
needs the same value as a K8s Secret.

Add a step to `scripts/k3d-knoe-deploy.sh` (NEW — see 2.3) that:

1. Runs `bash etc/gen_oidc_signing_key.sh` (idempotent, will be a no-op
   if the key already exists from `make k3d-knoe-up`).
2. Creates/updates a Secret in `knoe-system`:

```bash
kubectl --context=k3d-k3d-knoe -n knoe-system create secret generic \
  knoe-auth-oidc-signing-key \
  --from-literal=signingKey="$(cat etc/secrets/knoe-auth-oidc-key.b64)" \
  --dry-run=client -o yaml | kubectl --context=k3d-k3d-knoe apply -f -
```

The deployment manifest (2.1) references this Secret via `valueFrom.secretKeyRef`.

### 2.3 — `scripts/k3d-knoe-deploy.sh` (NEW)

The orchestrator the new Make target calls. Sequence:

```bash
#!/usr/bin/env bash
set -euo pipefail

# 1. Verify cluster + namespace exist (fail loudly if make k3d-knoe-up not run)
# 2. Build the image (call into existing `make docker-build-auth`)
# 3. k3d image import knoe-auth:latest -c knoe   # name from K3D_CLUSTER_NAME default
# 4. Ensure OIDC signing-key Secret exists (gen + kubectl apply per 2.2)
# 5. Apply k8s/knoe/knoe-auth-deployment.yaml
# 6. kubectl rollout status deployment/knoe-auth -n knoe-system --timeout=180s
# 7. Print port-forward instructions and the verification curls
```

Keep the script ≤ 100 lines; lean on existing functions/helpers when
possible. The `make k3d-knoe-up` script's structure is a fine reference.

### 2.4 — `Makefile` additions

Three new targets:

- `k3d-knoe-deploy` — calls `scripts/k3d-knoe-deploy.sh`. Depends
  implicitly on `make k3d-knoe-up` having been run first; surface a
  clear error message if the cluster context isn't reachable.
- `k3d-knoe-redeploy` — convenience: rebuild image + re-import + restart
  the deployment. Useful inner loop after the first `k3d-knoe-deploy`.
  Implementation: `make docker-build-auth && k3d image import …` then
  `kubectl rollout restart deployment/knoe-auth`.
- `k3d-knoe-undeploy` — cleanup: `kubectl delete -f
  k8s/knoe/knoe-auth-deployment.yaml`. Doesn't tear down the cluster
  (use `k3d-knoe-down` for that).

Update `make help` so the new targets appear under the existing
"k3d dev loop targets" header.

### 2.5 — Documentation

Update [`docs/local-dev-knoe-auth.md`](../../docs/local-dev-knoe-auth.md)
with a new section (insert before the existing "Reset / rebuild loop"
section):

```
## Pre-merge smoke: knoe-auth as a pod inside k3d

When you want to verify the deployed-pod shape (image build, mounted
secrets, sidecar wiring) before pushing to GKE — typically after
schema changes, env-var changes, or before opening a deploy MR:

    make k3d-knoe-deploy

This builds the image, imports it into the cluster, and applies the
deployment.  After it succeeds:

    kubectl --context=k3d-k3d-knoe -n knoe-system port-forward \
      deployment/knoe-auth 8090:8080

(Use 8090 to avoid colliding with your host-side `mvn spring-boot:run` on
8080 if both are running.)

    curl http://localhost:8090/health
    curl http://localhost:8090/.well-known/openid-configuration

Iteration:  `make k3d-knoe-redeploy`  (rebuild + restart)
Cleanup:    `make k3d-knoe-undeploy`  (drops the deployment; cluster stays)
```

Update [`docs/knoe-system.md`](../../docs/knoe-system.md) §2 status table:
add a row "Phase 3 in-cluster pod deploy" → **Shipped** once this brief
lands.

Update [`docs/plans/k3d-gke-mirror.md`](../k3d-gke-mirror.md) §6 — Phase 3
flips from "out of scope" to "shipped" with a back-link to this brief.

## 3. Don't break

- The host-side `mvn spring-boot:run` daily loop must keep working.
  Don't change `application-k3d.yml`, `etc/krb5.local.conf`, or the
  port-forward script in ways that affect the host loop. The in-cluster
  pod uses the same configmap-and-env-var inputs but in a different
  arrangement.
- The GKE-side deployment manifest at `deploy/gcp/gke/knoe-auth-deployment.yaml`
  is **not** to be touched by this brief. The k3d manifest is a sibling,
  not a replacement.
- The KDC pod and its init Job (`k8s/knoe/knoe-kdc-*.yaml`) stay
  unchanged. The knoe-auth pod talks to it via the existing
  `service/knoe-kdc.knoe-system.svc.cluster.local` ClusterIP service.

## 4. Verification (Definition of done)

A fresh laptop:

```bash
git pull
make k3d-knoe-up                                     # 3-5 min, includes chrisfu seed
make k3d-knoe-deploy                                 # NEW — builds, imports, applies, waits
# → ends with: knoe-auth deployment ready (1/1)

kubectl --context=k3d-k3d-knoe -n knoe-system port-forward \
  deployment/knoe-auth 8090:8080 &
sleep 3

curl -fsS http://localhost:8090/health
# → {"status":"ok"}

curl -fsS http://localhost:8090/.well-known/openid-configuration | jq .issuer
# → "http://knoe-auth.knoe-system.svc.cluster.local:8080"  (or http://localhost:8080
#    depending on your 2.1 issuer-pinning choice)

curl -fsS http://localhost:8090/jwks.json | jq '.keys[0].kty'
# → "RSA"

# Iteration loop:
make k3d-knoe-redeploy                                # ~30s
# Verify the change you made is reflected:
curl -fsS http://localhost:8090/.well-known/openid-configuration
```

Tick all of:

- [ ] `make k3d-knoe-deploy` exits 0 on a fresh laptop with `make k3d-knoe-up`
  already run.
- [ ] `kubectl --context=k3d-k3d-knoe -n knoe-system get deploy knoe-auth`
  reports `1/1 ready`.
- [ ] Port-forwarded `/health`, `/.well-known/openid-configuration`,
  `/jwks.json` all return 200 with the expected payloads.
- [ ] `make k3d-knoe-redeploy` triggers a rolling restart and a fresh
  build-and-import.
- [ ] `make k3d-knoe-undeploy` removes the deployment without touching
  the rest of the cluster.
- [ ] Host-side `mvn -pl authority spring-boot:run -Dspring-boot.run.profiles=k3d`
  still works (port 8080 free unless you're also running the in-cluster
  pod's port-forward there).

## 5. Out of scope

- **SPNEGO E2E from a host browser** — needs a service principal for
  `HTTP/localhost@KNOE.LOCAL` and a keytab the host knoe-auth or
  in-cluster knoe-auth can read; that's still parent-plan §6 Phase 2 of
  k3d-gke-mirror (different "Phase 2" from the OIDC one — yes the
  numbering's noisy).
- **Supabase stack on k3d** — parent §6 Phase 4. Not blocked by this
  brief but separately worthwhile.
- **`OidcCodeService` DB persistence** — the in-memory ConcurrentHashMap
  is a known gap, same on GKE; tracked separately.
- **Pushing knoe-auth to a GKE deploy** — Phase 2 OIDC GKE deploy was
  Junie's separate brief (`docs/plans/junie/phase2-oidc-gke-deploy.md`).

## 6. Commit shape

One MR. Suggested message:

```
feat(k3d): knoe-auth as a pod inside k3d (k3d-gke-mirror Phase 3)

Adds a pre-merge smoke loop: build the knoe-auth image, import it into
the k3d cluster, run it as a pod with the keytab-bootstrap initContainer
+ KDC sidecar pattern. Lets engineers exercise deployed-pod-shape
behaviour (image build, mounted-secret reads, sidecar networking)
without pushing to GKE.

Scope:
  - k8s/knoe/knoe-auth-deployment.yaml NEW; k3d-flavoured (image pull
    policy Never, realm KNOE.LOCAL, no Workload Identity).
  - scripts/k3d-knoe-deploy.sh NEW; wraps build + import + secret-create
    + apply + rollout-wait.
  - Makefile: k3d-knoe-deploy, k3d-knoe-redeploy, k3d-knoe-undeploy targets.
  - docs/local-dev-knoe-auth.md: new "Pre-merge smoke" section.
  - docs/knoe-system.md, docs/plans/k3d-gke-mirror.md: status flips.

Verified per the brief's Definition of done.

Out of scope:
  - SPNEGO E2E from host browsers (k3d-mirror Phase 2)
  - Supabase stack on k3d (Phase 4)
  - OidcCodeService DB persistence

Closes Phase 3 of docs/plans/k3d-gke-mirror.md.
```

## 7. Notes for reading

- The image build path lives in commit `903f84f`: `make docker-build-auth`
  produces `knoe-auth:latest` locally; `make docker-push-auth` pushes to
  Artifact Registry. **For Phase 3 you don't push** — you `k3d image
  import` instead, and `imagePullPolicy: Never` on the deployment so
  Kubernetes doesn't try to pull from a registry.
- The OIDC signing key file `etc/secrets/knoe-auth-oidc-key.b64` is
  already gitignored. Don't accidentally bake the key into the image
  layer; mount via Secret only.
- If the rolling restart in `k3d-knoe-redeploy` doesn't pick up a new
  image (Kubernetes can be sticky about identical image tags), an
  `imagePullPolicy: Never` + tag-rotation pattern (`:latest` →
  `:dev-$(date +%s)`) inside the redeploy script may help. Optional —
  only fix if you hit the issue.
- If the cluster has insufficient resources to schedule both knoe-auth
  and CNPG simultaneously, drop the resource requests/limits on the
  knoe-auth deployment to `requests: {}` (k3d single-node has finite
  capacity). Document if you do.