prole/docs/plans/k3d-gke-mirror.md

k3d-mirror-of-GKE — local platform model for knoe-auth dev

Active plan. Phase 1 in flight (Junie). Strategic context for a running initiative; reads top-to-bottom. Phase 1 brief at junie/k3d-knoe-auth-dev-loop.md.

---

1. Why

knoe-auth Phase 2 (OIDC provider — discovery, authorize, token, userinfo, JWKS, RS256 signing) just landed on main (via the merge that brought claude/crazy-bose-fec256 back in). It builds; it hasn't been deployed to the cluster yet. The next development steps — wiring OAuth2 code exchange to a persistent table (Phase 2.5), the ProvisioningWorker→GitLab/Gitea/CNPG dispatch surface, future Round 2 work — all benefit from a tight inner-loop that doesn't require:

• a GKE cluster to push to,
• a deploy pipeline to wait on,
• a real Google OAuth tenant to bounce through, or
• the full supabase + kong + oauth2-proxy + studio stack just to test what knoe-auth does.

What we want is a laptop dev loop: edit Java, run from IntelliJ (or mvn spring-boot:run), break-pointing freely, against a real PostgreSQL with the canonical knoe.* schema and a real Kerberos KDC the SPNEGO codepath can talk to.

The four-mode installer we already ship has k3d as one of those modes (see deployment-modes.md). It brings up CNPG today. It does not yet bring up the bits knoe-auth needs alongside. This plan closes that gap deliberately, in phases.

2. Where the modes stand today

Mode	DB	knoe-auth	KDC	Supabase	Studio	OAuth2-proxy
gke	CNPG (3-replica)	✅ deployed	✅ sidecar	✅	✅	✅ Google OIDC
k3s	CNPG (single)	✅ deployed (manifests at deploy/opentofu/k3s/manifests/knoe/knoe-auth-*)	✅ kdc-configmap	partial	partial	❌
k3d	✅ CNPG via k8s/knoe/knoe-db.yaml	❌ no manifests	❌ no manifests	❌	❌	❌
min	❌ no Postgres	❌	❌	❌	❌	❌

The k3s mode has the auth-stack manifests but is targeted at the on-prem-VM deploy pattern (talks to a single-node k3s on a customer machine), not the laptop-dev-loop pattern. Reusing them for k3d is attractive but the wiring assumptions differ enough (LB types, ingress class, service-account identity) that copy-and-adapt is easier than refactor-to-share.

3. Strategic shape

The goal is not a full GKE mirror — that would mean Studio + Kong

• supabase + oauth2-proxy + everything, and the laptop fan would not be happy. The goal is the smallest k3d-resident stack that lets a host-resident knoe-auth talk to it end-to-end.

Decisions (set 2026-05-02 with the user):

1. knoe-auth runs on the host, not in the cluster. mvn spring-boot:run (or IntelliJ Run Configuration) starts the process on the laptop. It connects to the k3d cluster's DB and KDC via kubectl port-forward. This gives:

   • hot reload via Spring DevTools,
   • direct debugger attach,
   • immediate stack traces in the same terminal as the build,
   • no image-build-and-load cycle on every change.

   Trade-off: this isn't testing the deployed-pod-shaped behaviour (mounted secrets, SA identity, sidecar networking). That's covered by the GKE deploy when it's time to go live, and a future "in-cluster smoke" Phase 3 brief if we want pre-merge checks too.

2. Stack scope is CNPG + KDC only. No supabase, no Kong, no oauth2-proxy, no Studio, no garage/MinIO. knoe-auth doesn't depend on any of those for its own surfaces (/health, /auth/enroll/*, /auth/admin/*, OIDC discovery/authorize/ token/userinfo/jwks). The ProvisioningWorker reaches OUT to GitLab/Gitea/CNPG — those are mockable or deferrable for dev.

3. Single-replica CNPG. Three-replica is GKE-shaped for availability; on a laptop it's wasteful. One instance, the same PG18 image, same postInitTemplateSQL schema bootstrap, same knoe.* tables.

4. Kerberos realm KNOE.LOCAL. Distinct from production KNOE.DEV so credentials don't accidentally cross-talk. Pre-seed one admin principal and one developer principal so the engineer can kinit immediately.

5. Plain sslmode=require for libpq from host. Production uses verify-full against the CNPG-issued cert. Locally that needs the CA chain and a SAN for localhost, which adds yarn-ball complexity. require keeps TLS on the wire without the cert plumbing. (Engineers can opt into verify-full later by mounting the CA cert; not gated on Phase 1.)

4. Architecture (Phase 1 target)

   ┌─────────────────────────────────────── Laptop ───────────────────────────────────────┐
   │                                                                                       │
   │   IntelliJ / mvn spring-boot:run                                                      │
   │             │                                                                         │
   │             ▼                                                                         │
   │   knoe-auth   :8080  ──[ http ]──▶  curl / browser tests                             │
   │             │                                                                         │
   │             │  jdbc:postgresql://localhost:5432/knoe-db                                │
   │             │  KRB5_CONFIG=./etc/krb5.local.conf                                      │
   │             │  KDC at  localhost:88                                                    │
   │             │                                                                         │
   │             ▼                                                                         │
   │   ┌─── kubectl port-forward ────┐                                                    │
   │   │   localhost:5432 → svc/knoe-db-rw:5432                                            │
   │   │   localhost:88   → svc/knoe-kdc:88  (TCP + UDP)                                   │
   │   │   localhost:464  → svc/knoe-kdc:464 (kpasswd)                                     │
   │   └────────────────┬─────────────┘                                                   │
   └────────────────────┼─────────────────────────────────────────────────────────────────┘
                        │
                        ▼
   ┌─────────────────────────────────── k3d cluster (single node) ──────────────────────────┐
   │                                                                                          │
   │   ┌──── ns: knoe-db-0 ─────────────┐         ┌──── ns: knoe-system ─────┐                │
   │   │                                 │         │                            │                │
   │   │   CNPG operator                 │         │   knoe-kdc Deployment      │                │
   │   │   knoe-db CNPG Cluster (1 inst) │         │   (krb5kdc + kadmind,      │                │
   │   │     → schema knoe.* via         │         │    realm KNOE.LOCAL,       │                │
   │   │       postInitTemplateSQL       │         │    pre-seeded principals)  │                │
   │   │                                 │         │                            │                │
   │   └─────────────────────────────────┘         └────────────────────────────┘                │
   │                                                                                          │
   └──────────────────────────────────────────────────────────────────────────────────────────┘

5. Phase 1 scope (Junie brief)

Brief: junie/k3d-knoe-auth-dev-loop.md.

Deliverables:

1. CNPG-on-k3d known-good — a single-replica CNPG cluster with the canonical knoe.* schema, brought up via a Make target or short shell script (make k3d-knoe-up style). Reuses existing k8s/knoe/knoe-db.yaml with the _strip_k3d_synology_blocks flow the Python orchestrator already runs.
2. KDC manifest for k3d — a Deployment + Service + ConfigMap under k8s/knoe/knoe-kdc-*.yaml (parallel naming to knoe-auth-*) with realm KNOE.LOCAL and a pre-seeded admin principal. Init script (etc/init_knoe_kdc_k3d.sh or similar) adds the principal via kadmin.local once the pod is Ready.
3. Port-forward helper — a make k3d-knoe-pf target (or shell script) that opens 5432, 88, 464 to localhost, prints the JDBC URL and the KRB5_CONFIG env var the engineer should export, and stays in the foreground until ^C. Background mode optional.
4. Engineer doc — docs/local-dev-knoe-auth.md: one-time setup, daily loop (make k3d-knoe-up && make k3d-knoe-pf), how to kinit, how to point IntelliJ at it, how to reset.
5. Smoke test — a shell script that, given the port-forward up, verifies: psql connects + \dt knoe.* shows ≥6 tables; kadmin.local listprincs includes the seed admin; from the host, mvn -pl authority test passes; mvn -pl authority spring-boot:run starts cleanly and curl localhost:8080/health returns ok and curl localhost:8080/.well-known/openid-configuration returns valid JSON with the right issuer.

Done = a 5-minute path from git pull → editing OidcTokenController.java → re-run → see the change reflected at localhost:8080/oauth2/token, with the DB and KDC backing it real.

6. Out of scope for Phase 1 (potential later phases)

• Phase 2 — KDC: full SPNEGO E2E including a host-side firefox/curl that does negotiate against localhost:8080. Needs KRB5_CONFIG, an SPN like HTTP/localhost@KNOE.LOCAL, a keytab file mounted into knoe-auth's runtime. Phase 1 stops at the KDC being reachable; Phase 2 makes SPNEGO actually work.
• Phase 3 — image-build-and-load smoke so engineers can run make k3d-knoe-deploy and see knoe-auth as a pod in the cluster before pushing to GKE. Useful as a pre-merge gate; not required for daily dev.
• Phase 4 — supabase stack on k3d for end-to-end SDK testing against a local DB. Heavy. Only if/when someone needs it.
• Phase 5 — oauth2-proxy + Google OIDC on localhost for testing the Studio gate flow without going to GKE. Even heavier; needs a Google OAuth client with localhost redirect URLs registered. Probably never worth it.
• Phase 2.5 (separate track) — persistent OAuth2 code/session tables. Currently OidcCodeService uses an in-memory ConcurrentHashMap, which means codes don't survive a knoe-auth restart. Not blocked by k3d work but useful to have once dev iteration is fast enough that engineers exercise the flow often. Tracked separately; would touch authority/src/main/.../session/ and add migrations under authority/src/main/resources/db/migration/.

7. Risks / open questions

• Resource ceiling on a MacBook. A single-replica CNPG plus KDC is light (~600 MB RAM combined), but k3d itself + Docker Desktop
  • IntelliJ + Spring Boot can climb. Document a "low-RAM mode" if it becomes a real problem; don't speculate now.
• k3d's Docker networking and kubectl port-forward UDP. UDP port-forward (port 88 for Kerberos UDP, 464 for kpasswd) works in recent kubectl versions but has historically been finicky. If it fails, fall back to TCP-only Kerberos (tcp_only in krb5.conf). Phase 1 should set this in the engineer-side krb5.local.conf preemptively.
• CNPG operator CRD installation in fresh k3d. The make k3d-up flow today runs init_cnpg_gke.sh for GKE, but there's no symmetric init_cnpg_k3d.sh. Either: (a) Junie writes a thin one that just kubectl apply -fs the CNPG release manifest with no GCP/IAM steps, or (b) the existing Python orchestrator knoe.core.ops.cloudnative_pg already handles k3d (per the survey, it does — _strip_k3d_synology_blocks is in there). Verify which during Phase 1.
• Schema bootstrap. The GKE flow seeds knoe.* via etc/init_knoe_auth.sh running SQL against the CNPG primary through kubectl exec. The same approach works in k3d; the brief should reuse the script (with a KUBECONTEXT swap), not duplicate it.

8. Verification (when Phase 1 is "done")

A new engineer, freshly cloning the repo on a Mac:

git clone <repo> && cd knoe-db
make k3d-knoe-up                # provisions cluster, CNPG, KDC, schema
make k3d-knoe-pf &              # port-forward in background
mvn -pl authority spring-boot:run
# in another terminal:
curl http://localhost:8080/health
# → ok
curl http://localhost:8080/.well-known/openid-configuration
# → JSON, issuer "http://localhost:8080"

Total elapsed time on a fresh laptop: ≤ 8 minutes. The engineer can now edit authority/src/main/java/dev/knoe/auth/... and see changes without leaving their editor.

9. Glossary

• k3d — k3s running inside a Docker container. Lightweight Kubernetes for laptops; no VM needed.
• CNPG — cloudnative-pg, a PostgreSQL Kubernetes operator. We run a single instance in k3d, three in GKE.
• KDC — Key Distribution Center, the Kerberos protocol's auth server. SPNEGO browsers/clients fetch tickets from it; knoe-auth validates SPNEGO challenges using these.
• KNOE.LOCAL — the Kerberos realm name we use for laptop dev. Distinct from KNOE.DEV (production) so a stray ticket from one doesn't authenticate you on the other.
• SPNEGO — Negotiate HTTP auth scheme. Carries a Kerberos ticket inside an HTTP header.
• Phase 2.5 — colloquial name for the gap between Phase 2 OIDC (in-memory codes) and a fully persistent OIDC provider (DB-backed codes/sessions). Not formally numbered; will get a name when work starts.