mirror of
https://github.com/dredx/prole.git
synced 2026-09-24 18:34:31 +00:00
2b36add592
Phase 1 (host loop) and the Phase 2 OIDC sandbox are shipped; this
brief queues Phase 3 of k3d-mirror-of-GKE: build the knoe-auth image,
k3d-import, run as a pod inside the cluster. Pre-merge smoke loop —
not the daily inner loop.
Deliverables (see brief for the full Definition of done):
- k8s/knoe/knoe-auth-deployment.yaml (NEW; sibling of GKE manifest)
- scripts/k3d-knoe-deploy.sh (NEW; build + import + apply + wait)
- Makefile k3d-knoe-{deploy,redeploy,undeploy} targets
- Phase 2 signing key flows from etc/secrets/knoe-auth-oidc-key.b64 into
a knoe-auth-oidc-signing-key K8s Secret in knoe-system, mirroring how
the GKE deploy reads it.
Index updates:
docs/plans/junie/README.md — k3d Phase 3 in Active row
docs/TODO.md §"In progress" — promotes the brief to top
docs/plans/k3d-gke-mirror.md — banner + §6 Phase 3 entry
flipped from "out of scope"
to "in flight"
262 lines
15 KiB
Markdown
262 lines
15 KiB
Markdown
# k3d-mirror-of-GKE — local platform model for knoe-auth dev
|
|
|
|
> **Active plan. Phase 1 + Phase 2-OIDC-sandbox shipped 2026-05-02;
|
|
> Phase 3 in flight (Junie).** Strategic context for a running initiative;
|
|
> reads top-to-bottom. Brief refs:
|
|
> [Phase 1 (host loop, shipped)](junie/k3d-knoe-auth-dev-loop.md) ·
|
|
> [Phase 3 (in-cluster pod, in flight)](junie/k3d-knoe-auth-pod-deploy.md).
|
|
|
|
---
|
|
|
|
## 1. Why
|
|
|
|
knoe-auth Phase 2 (OIDC provider — discovery, authorize, token, userinfo,
|
|
JWKS, RS256 signing) just landed on `main` (via the merge that brought
|
|
`claude/crazy-bose-fec256` back in). It builds; it hasn't been deployed
|
|
to the cluster yet. The next development steps — wiring OAuth2 code
|
|
exchange to a persistent table (Phase 2.5), the
|
|
ProvisioningWorker→GitLab/Gitea/CNPG dispatch surface, future Round 2
|
|
work — all benefit from a tight inner-loop that doesn't require:
|
|
|
|
- a GKE cluster to push to,
|
|
- a deploy pipeline to wait on,
|
|
- a real Google OAuth tenant to bounce through, or
|
|
- the full supabase + kong + oauth2-proxy + studio stack just to test
|
|
what knoe-auth does.
|
|
|
|
What we want is a **laptop dev loop**: edit Java, run from IntelliJ
|
|
(or `mvn spring-boot:run`), break-pointing freely, against a real
|
|
PostgreSQL with the canonical `knoe.*` schema and a real Kerberos KDC
|
|
the SPNEGO codepath can talk to.
|
|
|
|
The four-mode installer we already ship has `k3d` as one of those modes
|
|
(see [`deployment-modes.md`](deployment-modes.md)). It brings up CNPG
|
|
today. It does **not** yet bring up the bits knoe-auth needs alongside.
|
|
This plan closes that gap deliberately, in phases.
|
|
|
|
## 2. Where the modes stand today
|
|
|
|
| Mode | DB | knoe-auth | KDC | Supabase | Studio | OAuth2-proxy |
|
|
|---|---|---|---|---|---|---|
|
|
| `gke` | CNPG (3-replica) | ✅ deployed | ✅ sidecar | ✅ | ✅ | ✅ Google OIDC |
|
|
| `k3s` | CNPG (single) | ✅ deployed (manifests at `deploy/opentofu/k3s/manifests/knoe/knoe-auth-*`) | ✅ kdc-configmap | partial | partial | ❌ |
|
|
| `k3d` | ✅ CNPG via `k8s/knoe/knoe-db.yaml` | ❌ no manifests | ❌ no manifests | ❌ | ❌ | ❌ |
|
|
| `min` | ❌ no Postgres | ❌ | ❌ | ❌ | ❌ | ❌ |
|
|
|
|
The k3s mode has the auth-stack manifests but is targeted at the
|
|
on-prem-VM deploy pattern (talks to a single-node k3s on a customer
|
|
machine), not the laptop-dev-loop pattern. Reusing them for k3d is
|
|
attractive but the wiring assumptions differ enough (LB types,
|
|
ingress class, service-account identity) that copy-and-adapt is
|
|
easier than refactor-to-share.
|
|
|
|
## 3. Strategic shape
|
|
|
|
The **goal** is not a full GKE mirror — that would mean Studio + Kong
|
|
+ supabase + oauth2-proxy + everything, and the laptop fan would not
|
|
be happy. The goal is **the smallest k3d-resident stack that lets a
|
|
host-resident knoe-auth talk to it end-to-end**.
|
|
|
|
Decisions (set 2026-05-02 with the user):
|
|
|
|
1. **knoe-auth runs on the host, not in the cluster.** `mvn
|
|
spring-boot:run` (or IntelliJ Run Configuration) starts the process
|
|
on the laptop. It connects to the k3d cluster's DB and KDC via
|
|
`kubectl port-forward`. This gives:
|
|
- hot reload via Spring DevTools,
|
|
- direct debugger attach,
|
|
- immediate stack traces in the same terminal as the build,
|
|
- no image-build-and-load cycle on every change.
|
|
|
|
Trade-off: this isn't testing the deployed-pod-shaped behaviour
|
|
(mounted secrets, SA identity, sidecar networking). That's
|
|
covered by the GKE deploy when it's time to go live, and a future
|
|
"in-cluster smoke" Phase 3 brief if we want pre-merge checks too.
|
|
|
|
2. **Stack scope is CNPG + KDC only.** No supabase, no Kong, no
|
|
oauth2-proxy, no Studio, no garage/MinIO. knoe-auth doesn't
|
|
depend on any of those for its own surfaces (`/health`,
|
|
`/auth/enroll/*`, `/auth/admin/*`, OIDC discovery/authorize/
|
|
token/userinfo/jwks). The ProvisioningWorker reaches OUT to
|
|
GitLab/Gitea/CNPG — those are mockable or deferrable for dev.
|
|
|
|
3. **Single-replica CNPG.** Three-replica is GKE-shaped for
|
|
availability; on a laptop it's wasteful. One instance, the same
|
|
PG18 image, same `postInitTemplateSQL` schema bootstrap, same
|
|
`knoe.*` tables.
|
|
|
|
4. **Kerberos realm `KNOE.LOCAL`.** Distinct from production
|
|
`KNOE.DEV` so credentials don't accidentally cross-talk. Pre-seed
|
|
one admin principal and one developer principal so the engineer
|
|
can `kinit` immediately.
|
|
|
|
5. **Plain `sslmode=require` for libpq from host.** Production uses
|
|
`verify-full` against the CNPG-issued cert. Locally that needs the
|
|
CA chain and a SAN for `localhost`, which adds yarn-ball complexity.
|
|
`require` keeps TLS on the wire without the cert plumbing. (Engineers
|
|
can opt into `verify-full` later by mounting the CA cert; not
|
|
gated on Phase 1.)
|
|
|
|
## 4. Architecture (Phase 1 target)
|
|
|
|
```
|
|
┌─────────────────────────────────────── Laptop ───────────────────────────────────────┐
|
|
│ │
|
|
│ IntelliJ / mvn spring-boot:run │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ knoe-auth :8080 ──[ http ]──▶ curl / browser tests │
|
|
│ │ │
|
|
│ │ jdbc:postgresql://localhost:5432/knoe-db │
|
|
│ │ KRB5_CONFIG=./etc/krb5.local.conf │
|
|
│ │ KDC at localhost:88 │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌─── kubectl port-forward ────┐ │
|
|
│ │ localhost:5432 → svc/knoe-db-rw:5432 │
|
|
│ │ localhost:88 → svc/knoe-kdc:88 (TCP + UDP) │
|
|
│ │ localhost:464 → svc/knoe-kdc:464 (kpasswd) │
|
|
│ └────────────────┬─────────────┘ │
|
|
└────────────────────┼─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────── k3d cluster (single node) ──────────────────────────┐
|
|
│ │
|
|
│ ┌──── ns: knoe-db-0 ─────────────┐ ┌──── ns: knoe-system ─────┐ │
|
|
│ │ │ │ │ │
|
|
│ │ CNPG operator │ │ knoe-kdc Deployment │ │
|
|
│ │ knoe-db CNPG Cluster (1 inst) │ │ (krb5kdc + kadmind, │ │
|
|
│ │ → schema knoe.* via │ │ realm KNOE.LOCAL, │ │
|
|
│ │ postInitTemplateSQL │ │ pre-seeded principals) │ │
|
|
│ │ │ │ │ │
|
|
│ └─────────────────────────────────┘ └────────────────────────────┘ │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## 5. Phase 1 scope (Junie brief)
|
|
|
|
Brief: [`junie/k3d-knoe-auth-dev-loop.md`](junie/k3d-knoe-auth-dev-loop.md).
|
|
|
|
Deliverables:
|
|
|
|
1. **CNPG-on-k3d known-good** — a single-replica CNPG cluster with
|
|
the canonical `knoe.*` schema, brought up via a Make target or
|
|
short shell script (`make k3d-knoe-up` style). Reuses existing
|
|
`k8s/knoe/knoe-db.yaml` with the `_strip_k3d_synology_blocks` flow
|
|
the Python orchestrator already runs.
|
|
2. **KDC manifest for k3d** — a `Deployment` + `Service` + `ConfigMap`
|
|
under `k8s/knoe/knoe-kdc-*.yaml` (parallel naming to
|
|
`knoe-auth-*`) with realm `KNOE.LOCAL` and a pre-seeded admin
|
|
principal. Init script (`etc/init_knoe_kdc_k3d.sh` or similar)
|
|
adds the principal via `kadmin.local` once the pod is Ready.
|
|
3. **Port-forward helper** — a `make k3d-knoe-pf` target (or shell
|
|
script) that opens 5432, 88, 464 to localhost, prints the JDBC
|
|
URL and the `KRB5_CONFIG` env var the engineer should `export`,
|
|
and stays in the foreground until ^C. Background mode optional.
|
|
4. **Engineer doc** — `docs/local-dev-knoe-auth.md`: one-time setup,
|
|
daily loop (`make k3d-knoe-up && make k3d-knoe-pf`), how to
|
|
`kinit`, how to point IntelliJ at it, how to reset.
|
|
5. **Smoke test** — a shell script that, given the port-forward up,
|
|
verifies: psql connects + `\dt knoe.*` shows ≥6 tables;
|
|
`kadmin.local listprincs` includes the seed admin; from the host,
|
|
`mvn -pl authority test` passes; `mvn -pl authority spring-boot:run`
|
|
starts cleanly and `curl localhost:8080/health` returns `ok` and
|
|
`curl localhost:8080/.well-known/openid-configuration` returns
|
|
valid JSON with the right issuer.
|
|
|
|
Done = a 5-minute path from `git pull` → editing
|
|
`OidcTokenController.java` → re-run → see the change reflected at
|
|
`localhost:8080/oauth2/token`, with the DB and KDC backing it real.
|
|
|
|
## 6. Out of scope for Phase 1 (potential later phases)
|
|
|
|
- **Phase 2 — KDC: full SPNEGO E2E** including a host-side
|
|
`firefox`/`curl` that does negotiate against `localhost:8080`.
|
|
Needs `KRB5_CONFIG`, an SPN like `HTTP/localhost@KNOE.LOCAL`, a
|
|
keytab file mounted into knoe-auth's runtime. Phase 1 stops at the
|
|
KDC being reachable; Phase 2 makes SPNEGO actually work.
|
|
- **Phase 3 — image-build-and-load smoke** ([brief](junie/k3d-knoe-auth-pod-deploy.md),
|
|
in flight 2026-05-02). Engineers run `make k3d-knoe-deploy` and see
|
|
knoe-auth as a pod in the cluster, exercising image build, mounted
|
|
secrets, and KDC-sidecar wiring without pushing to GKE. Pre-merge
|
|
gate; not required for daily dev.
|
|
- **Phase 4 — supabase stack on k3d** for end-to-end SDK testing
|
|
against a local DB. Heavy. Only if/when someone needs it.
|
|
- **Phase 5 — oauth2-proxy + Google OIDC on localhost** for testing
|
|
the Studio gate flow without going to GKE. Even heavier; needs a
|
|
Google OAuth client with `localhost` redirect URLs registered.
|
|
Probably never worth it.
|
|
- **Phase 2.5 (separate track) — persistent OAuth2 code/session
|
|
tables.** Currently `OidcCodeService` uses an in-memory
|
|
`ConcurrentHashMap`, which means codes don't survive a knoe-auth
|
|
restart. Not blocked by k3d work but useful to have once dev
|
|
iteration is fast enough that engineers exercise the flow often.
|
|
Tracked separately; would touch `authority/src/main/.../session/`
|
|
and add migrations under `authority/src/main/resources/db/migration/`.
|
|
|
|
## 7. Risks / open questions
|
|
|
|
- **Resource ceiling on a MacBook.** A single-replica CNPG plus KDC
|
|
is light (~600 MB RAM combined), but k3d itself + Docker Desktop
|
|
+ IntelliJ + Spring Boot can climb. Document a "low-RAM mode" if
|
|
it becomes a real problem; don't speculate now.
|
|
- **k3d's Docker networking and `kubectl port-forward` UDP.** UDP
|
|
port-forward (port 88 for Kerberos UDP, 464 for kpasswd) works in
|
|
recent kubectl versions but has historically been finicky. If it
|
|
fails, fall back to TCP-only Kerberos (`tcp_only` in `krb5.conf`).
|
|
Phase 1 should set this in the engineer-side `krb5.local.conf`
|
|
preemptively.
|
|
- **CNPG operator CRD installation in fresh k3d.** The `make k3d-up`
|
|
flow today runs `init_cnpg_gke.sh` for GKE, but there's no
|
|
symmetric `init_cnpg_k3d.sh`. Either: (a) Junie writes a thin one
|
|
that just `kubectl apply -f`s the CNPG release manifest with no
|
|
GCP/IAM steps, or (b) the existing Python orchestrator
|
|
`knoe.core.ops.cloudnative_pg` already handles k3d (per the
|
|
survey, it does — `_strip_k3d_synology_blocks` is in there).
|
|
Verify which during Phase 1.
|
|
- **Schema bootstrap.** The GKE flow seeds `knoe.*` via
|
|
`etc/init_knoe_auth.sh` running SQL against the CNPG primary
|
|
through `kubectl exec`. The same approach works in k3d; the
|
|
brief should reuse the script (with a `KUBECONTEXT` swap), not
|
|
duplicate it.
|
|
|
|
## 8. Verification (when Phase 1 is "done")
|
|
|
|
A new engineer, freshly cloning the repo on a Mac:
|
|
|
|
```bash
|
|
git clone <repo> && cd knoe-db
|
|
make k3d-knoe-up # provisions cluster, CNPG, KDC, schema
|
|
make k3d-knoe-pf & # port-forward in background
|
|
mvn -pl authority spring-boot:run
|
|
# in another terminal:
|
|
curl http://localhost:8080/health
|
|
# → ok
|
|
curl http://localhost:8080/.well-known/openid-configuration
|
|
# → JSON, issuer "http://localhost:8080"
|
|
```
|
|
|
|
Total elapsed time on a fresh laptop: ≤ 8 minutes. The engineer can
|
|
now edit `authority/src/main/java/dev/knoe/auth/...` and see changes
|
|
without leaving their editor.
|
|
|
|
## 9. Glossary
|
|
|
|
- **k3d** — `k3s` running inside a Docker container. Lightweight
|
|
Kubernetes for laptops; no VM needed.
|
|
- **CNPG** — `cloudnative-pg`, a PostgreSQL Kubernetes operator. We
|
|
run a single instance in k3d, three in GKE.
|
|
- **KDC** — Key Distribution Center, the Kerberos protocol's auth
|
|
server. SPNEGO browsers/clients fetch tickets from it; knoe-auth
|
|
validates SPNEGO challenges using these.
|
|
- **KNOE.LOCAL** — the Kerberos realm name we use for laptop dev.
|
|
Distinct from `KNOE.DEV` (production) so a stray ticket from one
|
|
doesn't authenticate you on the other.
|
|
- **SPNEGO** — `Negotiate` HTTP auth scheme. Carries a Kerberos
|
|
ticket inside an HTTP header.
|
|
- **Phase 2.5** — colloquial name for the gap between Phase 2 OIDC
|
|
(in-memory codes) and a fully persistent OIDC provider (DB-backed
|
|
codes/sessions). Not formally numbered; will get a name when work
|
|
starts.
|