prole/docs/plans/junie/k3d-knoe-auth-pod-deploy.md

Junie brief — Phase 3 of k3d-mirror-of-GKE: knoe-auth as a pod inside k3d

Self-contained brief. Reference: parent architectural plan at ../k3d-gke-mirror.md, §6 Phase 3. Phase 1 (CNPG + KDC up; knoe-auth from host) and the Phase 2 OIDC k3d sandbox are both shipped; chrisfu LOGIN role + knoe_developer group are seeded on every make k3d-knoe-up. The next loop is knoe-auth running as a pod inside the k3d cluster, exercising the deployed-pod shape (image pull, mounted secrets, sidecar wiring) without pushing to GKE.

---

1. Why

Today's daily loop runs knoe-auth from the laptop (mvn spring-boot:run) against k3d's CNPG + KDC. Fast inner loop, but it skips the parts that matter on production:

• Image build correctness — does authority/Dockerfile.app actually produce a runnable container? (Was added in commit 903f84f for queue #3 but never run end-to-end yet.)
• Mounted-secret behaviour — does the pod read KNOE_AUTH_OIDC_SIGNING_KEY from a K8s Secret, the way GKE does, vs. an env-var the engineer set in their shell?
• Sidecar networking — does the keytab-bootstrap initContainer + KDC sidecar wiring in deploy/gcp/gke/knoe-auth-deployment.yaml work unchanged on a single-node k3d cluster, or are there GKE-isms that need a k3d variant?

This brief delivers a make k3d-knoe-deploy target that wraps:

1. Build the knoe-auth Docker image locally
2. Import it into the k3d cluster
3. Apply a k3d-flavoured deployment manifest
4. Wait for the pod to be ready
5. Print verification commands (port-forward + curl /health + /jwks.json)

It's a pre-merge gate, not a daily inner loop. Engineers who care about catching deploy-shape regressions before pushing to GKE will run it; the host-side mvn loop stays the default for code iteration.

2. What you produce

Single-MR scope, all under k8s/knoe/, scripts/, and Makefile:

2.1 — k8s/knoe/knoe-auth-deployment.yaml (NEW)

A k3d-flavoured Deployment adapted from deploy/gcp/gke/knoe-auth-deployment.yaml. Differences from the GKE version (handle each):

Concern	GKE	k3d
Image	us-west3-docker.pkg.dev/.../knoe-auth:latest (registry pull)	knoe-auth:latest with imagePullPolicy: Never (loaded via k3d image import)
KDC sidecar realm	KNOE.DEV	KNOE.LOCAL (matches etc/krb5.local.conf and application-k3d.yml)
Workload Identity	KSA → GSA bindings for GCP secret access	Not applicable; secrets are in-cluster
Resource requests/limits	GKE node-aware (1Gi/1)	Drop or shrink (k3d single node has finite RAM)
Probes	GKE-tuned timings	Keep; same Spring Boot endpoints
OIDC signing key	from knoe-auth-oidc secret (1Password-fed)	from knoe-auth-oidc-signing-key secret created by the new init step (see 2.2)
OIDC issuer	https://api.knoe.dev/auth	http://knoe-auth.knoe-system.svc.cluster.local:8080 (in-cluster) OR keep the localhost-port-forward issuer http://localhost:8080 and document the trade-off

The two-issuer choice deserves a comment in the manifest. Pick one and pin it; the other path is a future option.

Don't try to make this manifest a copy of the GKE one — they'll diverge naturally and that's fine. Keep it readable.

2.2 — Phase 2 OIDC signing key as a k3d Secret

Today the host-side loop reads etc/secrets/knoe-auth-oidc-key.b64 and exports KNOE_AUTH_OIDC_SIGNING_KEY as an env var. The in-cluster pod needs the same value as a K8s Secret.

Add a step to scripts/k3d-knoe-deploy.sh (NEW — see 2.3) that:

1. Runs bash etc/gen_oidc_signing_key.sh (idempotent, will be a no-op if the key already exists from make k3d-knoe-up).
2. Creates/updates a Secret in knoe-system:

kubectl --context=k3d-k3d-knoe -n knoe-system create secret generic \
  knoe-auth-oidc-signing-key \
  --from-literal=signingKey="$(cat etc/secrets/knoe-auth-oidc-key.b64)" \
  --dry-run=client -o yaml | kubectl --context=k3d-k3d-knoe apply -f -

The deployment manifest (2.1) references this Secret via valueFrom.secretKeyRef.

2.3 — scripts/k3d-knoe-deploy.sh (NEW)

The orchestrator the new Make target calls. Sequence:

#!/usr/bin/env bash
set -euo pipefail

# 1. Verify cluster + namespace exist (fail loudly if make k3d-knoe-up not run)
# 2. Build the image (call into existing `make docker-build-auth`)
# 3. k3d image import knoe-auth:latest -c knoe   # name from K3D_CLUSTER_NAME default
# 4. Ensure OIDC signing-key Secret exists (gen + kubectl apply per 2.2)
# 5. Apply k8s/knoe/knoe-auth-deployment.yaml
# 6. kubectl rollout status deployment/knoe-auth -n knoe-system --timeout=180s
# 7. Print port-forward instructions and the verification curls

Keep the script ≤ 100 lines; lean on existing functions/helpers when possible. The make k3d-knoe-up script's structure is a fine reference.

2.4 — Makefile additions

Three new targets:

• k3d-knoe-deploy — calls scripts/k3d-knoe-deploy.sh. Depends implicitly on make k3d-knoe-up having been run first; surface a clear error message if the cluster context isn't reachable.
• k3d-knoe-redeploy — convenience: rebuild image + re-import + restart the deployment. Useful inner loop after the first k3d-knoe-deploy. Implementation: make docker-build-auth && k3d image import … then kubectl rollout restart deployment/knoe-auth.
• k3d-knoe-undeploy — cleanup: kubectl delete -f k8s/knoe/knoe-auth-deployment.yaml. Doesn't tear down the cluster (use k3d-knoe-down for that).

Update make help so the new targets appear under the existing "k3d dev loop targets" header.

2.5 — Documentation

Update docs/local-dev-knoe-auth.md with a new section (insert before the existing "Reset / rebuild loop" section):

## Pre-merge smoke: knoe-auth as a pod inside k3d

When you want to verify the deployed-pod shape (image build, mounted
secrets, sidecar wiring) before pushing to GKE — typically after
schema changes, env-var changes, or before opening a deploy MR:

    make k3d-knoe-deploy

This builds the image, imports it into the cluster, and applies the
deployment.  After it succeeds:

    kubectl --context=k3d-k3d-knoe -n knoe-system port-forward \
      deployment/knoe-auth 8090:8080

(Use 8090 to avoid colliding with your host-side `mvn spring-boot:run` on
8080 if both are running.)

    curl http://localhost:8090/health
    curl http://localhost:8090/.well-known/openid-configuration

Iteration:  `make k3d-knoe-redeploy`  (rebuild + restart)
Cleanup:    `make k3d-knoe-undeploy`  (drops the deployment; cluster stays)

Update docs/knoe-system.md §2 status table: add a row "Phase 3 in-cluster pod deploy" → Shipped once this brief lands.

Update docs/plans/k3d-gke-mirror.md §6 — Phase 3 flips from "out of scope" to "shipped" with a back-link to this brief.

3. Don't break

• The host-side mvn spring-boot:run daily loop must keep working. Don't change application-k3d.yml, etc/krb5.local.conf, or the port-forward script in ways that affect the host loop. The in-cluster pod uses the same configmap-and-env-var inputs but in a different arrangement.
• The GKE-side deployment manifest at deploy/gcp/gke/knoe-auth-deployment.yaml is not to be touched by this brief. The k3d manifest is a sibling, not a replacement.
• The KDC pod and its init Job (k8s/knoe/knoe-kdc-*.yaml) stay unchanged. The knoe-auth pod talks to it via the existing service/knoe-kdc.knoe-system.svc.cluster.local ClusterIP service.

4. Verification (Definition of done)

A fresh laptop:

git pull
make k3d-knoe-up                                     # 3-5 min, includes chrisfu seed
make k3d-knoe-deploy                                 # NEW — builds, imports, applies, waits
# → ends with: knoe-auth deployment ready (1/1)

kubectl --context=k3d-k3d-knoe -n knoe-system port-forward \
  deployment/knoe-auth 8090:8080 &
sleep 3

curl -fsS http://localhost:8090/health
# → {"status":"ok"}

curl -fsS http://localhost:8090/.well-known/openid-configuration | jq .issuer
# → "http://knoe-auth.knoe-system.svc.cluster.local:8080"  (or http://localhost:8080
#    depending on your 2.1 issuer-pinning choice)

curl -fsS http://localhost:8090/jwks.json | jq '.keys[0].kty'
# → "RSA"

# Iteration loop:
make k3d-knoe-redeploy                                # ~30s
# Verify the change you made is reflected:
curl -fsS http://localhost:8090/.well-known/openid-configuration

Tick all of:

• make k3d-knoe-deploy exits 0 on a fresh laptop with make k3d-knoe-up already run.
• kubectl --context=k3d-k3d-knoe -n knoe-system get deploy knoe-auth reports 1/1 ready.
• Port-forwarded /health, /.well-known/openid-configuration, /jwks.json all return 200 with the expected payloads.
• make k3d-knoe-redeploy triggers a rolling restart and a fresh build-and-import.
• make k3d-knoe-undeploy removes the deployment without touching the rest of the cluster.
• Host-side mvn -pl authority spring-boot:run -Dspring-boot.run.profiles=k3d still works (port 8080 free unless you're also running the in-cluster pod's port-forward there).

5. Out of scope

• SPNEGO E2E from a host browser — needs a service principal for HTTP/localhost@KNOE.LOCAL and a keytab the host knoe-auth or in-cluster knoe-auth can read; that's still parent-plan §6 Phase 2 of k3d-gke-mirror (different "Phase 2" from the OIDC one — yes the numbering's noisy).
• Supabase stack on k3d — parent §6 Phase 4. Not blocked by this brief but separately worthwhile.
• OidcCodeService DB persistence — the in-memory ConcurrentHashMap is a known gap, same on GKE; tracked separately.
• Pushing knoe-auth to a GKE deploy — Phase 2 OIDC GKE deploy was Junie's separate brief (docs/plans/junie/phase2-oidc-gke-deploy.md).

6. Commit shape

One MR. Suggested message:

feat(k3d): knoe-auth as a pod inside k3d (k3d-gke-mirror Phase 3)

Adds a pre-merge smoke loop: build the knoe-auth image, import it into
the k3d cluster, run it as a pod with the keytab-bootstrap initContainer
+ KDC sidecar pattern. Lets engineers exercise deployed-pod-shape
behaviour (image build, mounted-secret reads, sidecar networking)
without pushing to GKE.

Scope:
  - k8s/knoe/knoe-auth-deployment.yaml NEW; k3d-flavoured (image pull
    policy Never, realm KNOE.LOCAL, no Workload Identity).
  - scripts/k3d-knoe-deploy.sh NEW; wraps build + import + secret-create
    + apply + rollout-wait.
  - Makefile: k3d-knoe-deploy, k3d-knoe-redeploy, k3d-knoe-undeploy targets.
  - docs/local-dev-knoe-auth.md: new "Pre-merge smoke" section.
  - docs/knoe-system.md, docs/plans/k3d-gke-mirror.md: status flips.

Verified per the brief's Definition of done.

Out of scope:
  - SPNEGO E2E from host browsers (k3d-mirror Phase 2)
  - Supabase stack on k3d (Phase 4)
  - OidcCodeService DB persistence

Closes Phase 3 of docs/plans/k3d-gke-mirror.md.

7. Notes for reading

• The image build path lives in commit 903f84f: make docker-build-auth produces knoe-auth:latest locally; make docker-push-auth pushes to Artifact Registry. For Phase 3 you don't push — you k3d image import instead, and imagePullPolicy: Never on the deployment so Kubernetes doesn't try to pull from a registry.
• The OIDC signing key file etc/secrets/knoe-auth-oidc-key.b64 is already gitignored. Don't accidentally bake the key into the image layer; mount via Secret only.
• If the rolling restart in k3d-knoe-redeploy doesn't pick up a new image (Kubernetes can be sticky about identical image tags), an imagePullPolicy: Never + tag-rotation pattern (:latest → :dev-$(date +%s)) inside the redeploy script may help. Optional — only fix if you hit the issue.
• If the cluster has insufficient resources to schedule both knoe-auth and CNPG simultaneously, drop the resource requests/limits on the knoe-auth deployment to requests: {} (k3d single-node has finite capacity). Document if you do.