From f1450605c290d56bb93e8a791ae52bd0794d60ad Mon Sep 17 00:00:00 2001 From: chrisfu Date: Mon, 11 May 2026 16:19:14 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20knoe-auth=20=E2=80=94=20prole.org=20dep?= =?UTF-8?q?loyment=20overlay?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Companion to the canonical architecture doc at ~/dev/knoe-db/docs/knoe-auth.md. Documents what's specific to the prole.org homelab deployment: - Two upstream identity layers (Samba AD on myrddin for PROLE.ORG Kerberos, Google Workspace knoey.com for OIDC) - Three deploy targets (k3s on myrddin/merlin/gandalf, k3d for engineer dev loop, GKE for production API endpoint) - Hostname map (myrddin/morgana/zinfandel/db.prole.org/ git.prole.org/api.knoe.dev/db.knoe.dev/etc.) - Cross-realm Kerberos trust setup (KNOE.LOCAL ↔ PROLE.ORG) via infrastructure/playbooks/kerberos_trust_setup.yml — RC4 pin explanation, kvno sync notes - Gitea SPNEGO at git.prole.org with Mermaid flow diagram - Supabase Studio status (db.prole.org Google OAuth working; db.knoe.dev knoe-auth OIDC blocked on pg_oauth image) - Engineer onboarding checklist (prole-specific steps) - Known fragility / current incidents table Mermaid sequence diagram covers the cross-realm flow end-to-end (workstation → Samba KDC → MIT KDC → service ticket), distinct from the canonical doc's generic version. References the matching Junie briefs in knoe-db for the open fragility items (pg_oauth-image-install, init-user-schemas-heredoc-fix, deploy-pipeline-restore). Co-Authored-By: Claude Sonnet 4.6 --- docs/knoe-auth.md | 306 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 306 insertions(+) create mode 100644 docs/knoe-auth.md diff --git a/docs/knoe-auth.md b/docs/knoe-auth.md new file mode 100644 index 0000000..dd0f120 --- /dev/null +++ b/docs/knoe-auth.md @@ -0,0 +1,306 @@ +# knoe-auth — prole.org deployment overlay + +> **Customer-specific overlay** of the canonical architecture +> documented in [`~/dev/knoe-db/docs/knoe-auth.md`](https://github.com/dredx/knoe-db/blob/main/docs/knoe-auth.md). +> Read the canonical doc first for components, flows, and intent. +> This file documents what's specific to the **`prole.org` homelab +> deployment** — hostnames, IdP wiring, Kerberos realm names, cross-realm +> trust setup, current state of each integration. +> +> Filed 2026-05-11. Related: [`db-access.md`](db-access.md), +> [`knoe-auth-phase-2.md`](knoe-auth-phase-2.md), +> [`oidc-setup.md`](oidc-setup.md), +> [`local-dev-knoe-auth.md`](local-dev-knoe-auth.md). + +--- + +## 1. Identity sources (what's authoritative) + +The prole deployment has **two upstream identity layers**, each +authoritative for a different population: + +| IdP | Authoritative for | Realm / Domain | +|---|---|---| +| **Samba AD on myrddin** | Homelab humans + workstation logins | Kerberos realm `PROLE.ORG`; SMB workgroup `PROLE` | +| **Google Workspace (knoey.com)** | Engineers, OIDC sign-ins, email-based identity | `@knoey.com` | + +Both feed into `knoe-auth` as upstream IdPs: + +- Samba AD via **Kerberos cross-realm trust** (KNOE.LOCAL ↔ PROLE.ORG) +- Google Workspace via **OIDC** (knoe-auth registers as a Google OAuth client) + +A single human (chrisfu, ron, …) typically has both: +`chrisfu@PROLE.ORG` (workstation Kerberos identity) and +`chrisfu@knoey.com` (Google Workspace / OIDC identity). The +`preferred_username` claim canonicalizes them to the same short-name +(`chrisfu`) so downstream services see one identity. + +## 2. Deployments + +Two clusters in the prole homelab, plus the managed GKE deployment +for production: + +| Cluster | Where | Role | knoe-auth URL | +|---|---|---|---| +| **k3s** (`prole-service-cluster`) | myrddin + merlin + gandalf | Homelab production — Gitea, Supabase, OpenBao, Grafana, Garage | `https://auth.prole.org` (planned) / `http://knoe-auth.knoe-system.svc.cluster.local:8080` (in-cluster) | +| **k3d** (`k3d-knoe`) | Engineer laptop | Dev loop — knoe-auth Phase 3 shipped | `http://localhost:8090` (host port-forward) | +| **GKE** (`knoe-dev-0` + `knoe-dev-cnpg-0`) | `plenary-truck-485623-p7` | Production OIDC issuer for the broader `knoe.dev` platform | `https://api.knoe.dev/auth` | + +A workstation in the prole homelab typically uses **the GKE-hosted +issuer** for OIDC (because that's where Google Workspace OAuth client +registration lives) and **the k3s in-cluster KDC** for Kerberos +(because that's where the Samba AD trust lives). Both are addressable +externally via their respective ingresses. + +## 3. Hostname map + +| Hostname | Resolves to | What lives there | +|---|---|---| +| `myrddin.prole.org` | LAN IP | Samba AD DC, k3s server node, Kerberos KDC for `PROLE.ORG` | +| `merlin.prole.org`, `gandalf.prole.org` | LAN IPs | k3s agent nodes | +| `morgana.prole.org`, `zinfandel.prole.org` | LAN IPs | Engineer workstations | +| `git.prole.org` | k3s Traefik | Gitea + SPNEGO proxy | +| `db.prole.org` | k3s Traefik | Supabase Studio (current: Google OAuth via oauth2-proxy) | +| `auth.prole.org` | k3s Traefik (planned) | knoe-auth OIDC issuer (homelab path) | +| `svc.prole.org` | k3s Traefik | Kong API gateway for internal services | +| `api.knoe.dev` | GKE GCE LB | knoe-auth OIDC issuer (production path) | +| `db.knoe.dev` | GKE GCE LB (planned) | Supabase Studio + knoe-auth OIDC (per [`per-user-supabase-workspace.md`](../docs/plans/junie/per-user-supabase-workspace.md)) | +| `db.0.knoe.dev` | GKE GCE LB | Supabase Studio (current; Google OAuth via oauth2-proxy) — to be aliased + deprecated when `db.knoe.dev` lands | +| `pg.0.knoe.dev` | GKE TCP LB | CNPG external endpoint (port 5432) | + +## 4. Cross-realm Kerberos trust (PROLE.ORG ↔ KNOE.LOCAL) + +This is the prole-specific glue that lets a workstation user with a +PROLE.ORG TGT obtain a KNOE.LOCAL service ticket without re-typing +a password. + +### 4.1 Setup + +Provisioned by the Ansible playbook +[`infrastructure/playbooks/kerberos_trust_setup.yml`](../infrastructure/playbooks/kerberos_trust_setup.yml) +(landed in commits `5cece40`..`ad1eced` on 2026-05-10/11). Reads the +shared `trust_shared_password` from the k8s Secret +`knoe-system/knoe-kdc-secrets` and writes it both: + +- **On myrddin (Samba side)** — creates user `krbtgt_KNOE.LOCAL` in + Samba with the shared password, sets UPN to + `krbtgt/KNOE.LOCAL@PROLE.ORG` (the Kerberos-canonical form), pins + `msDS-SupportedEncryptionTypes = 4` (RC4 only). +- **In-cluster MIT KDC** — `etc/init_kdc.sh` creates both + `krbtgt/KNOE.LOCAL@PROLE.ORG` and `krbtgt/PROLE.ORG@KNOE.LOCAL` + locally via `kadmin.local addprinc -e arcfour-hmac:normal` with the + matching shared password. + +RC4 enforcement on both sides is critical — AES key derivation uses a +salt (Samba salts as `+UPN`, MIT salts as +`+`), so the same password produces +different AES keys on each side. RC4 has no salt, keys converge. + +### 4.2 What works today + +Verified 2026-05-10 from morgana (workstation) and myrddin (DC): + +``` +$ kdestroy && kinit chrisfu@PROLE.ORG +$ kvno krbtgt/KNOE.LOCAL@PROLE.ORG +krbtgt/KNOE.LOCAL@PROLE.ORG: kvno = 3 +$ klist +Ticket cache: FILE:/tmp/krb5cc_1000 +Default principal: chrisfu@PROLE.ORG + krbtgt/PROLE.ORG@PROLE.ORG (local TGT) + krbtgt/KNOE.LOCAL@PROLE.ORG (cross-realm referral) +``` + +Workstation Kerberos config is provisioned by +[`infrastructure/playbooks/workstation_kerberos.yml`](../infrastructure/playbooks/workstation_kerberos.yml) +(landed in earlier commits). It deploys `/etc/krb5.conf` with both +realms defined, plus Chrome/Edge managed-policy `AuthServerAllowlist` +so the browser will offer Kerberos SPNEGO for `*.prole.org`. + +### 4.3 Cross-realm flow + +```mermaid +sequenceDiagram + autonumber + participant WS as Workstation
(morgana, PROLE.ORG-joined) + participant SambaKDC as Samba KDC
(myrddin, PROLE.ORG) + participant MITKDC as MIT KDC
(in-cluster, KNOE.LOCAL) + participant Svc as Kerberized service
(HTTP/git.prole.org) + + WS->>SambaKDC: AS_REQ chrisfu@PROLE.ORG + SambaKDC-->>WS: TGT (krbtgt/PROLE.ORG@PROLE.ORG) + Note over WS: User opens https://git.prole.org
(realm of svc = PROLE.ORG for SPNEGO,
but pretend = KNOE.LOCAL for illustration) + WS->>SambaKDC: TGS_REQ for krbtgt/KNOE.LOCAL@PROLE.ORG + Note right of SambaKDC: Samba's krbtgt_KNOE.LOCAL user
has the shared RC4 key + SambaKDC-->>WS: cross-realm TGT
(enc with shared key, kvno N) + WS->>MITKDC: TGS_REQ for HTTP/@KNOE.LOCAL + Note right of MITKDC: Decrypts inbound TGT
with matching RC4 key (kvno N) + MITKDC-->>WS: service ticket + WS->>Svc: HTTP/2 + Authorization: Negotiate + Svc-->>WS: 200 + identity = chrisfu (PROLE.ORG) +``` + +Today's only Kerberized service is **git.prole.org (Gitea SPNEGO)** — +that one is `HTTP/git.prole.org@PROLE.ORG`, so the cross-realm +mechanic isn't actually exercised in production yet. The cross-realm +trust exists for **future services that live in the cluster (KNOE.LOCAL)** +and want to accept workstation TGTs from PROLE.ORG without forcing the +operator to maintain two parallel user databases. + +### 4.4 Operational notes + +- The `kerberos_trust_setup.yml` playbook is **idempotent**. + Re-running it rotates the password (matches the cluster Secret) + and bumps the kvno on the Samba side. If the cluster Secret is + rotated, re-run the playbook to sync. +- MIT side kvno must match Samba side kvno; tonight's debug session + (2026-05-11) walked through what happens when they drift. +- **pg_tde Transparent Data Encryption** is in + `shared_preload_libraries` on the in-cluster Postgres. If you ever + patch the cluster spec to remove it, the next pod restart will fail + to open the encrypted data directory. Don't. + +## 5. Gitea SPNEGO at `git.prole.org` + +Shipped 2026-05-09. Workflow: + +```mermaid +sequenceDiagram + autonumber + participant Br as Browser
(PROLE.ORG-joined, Kerberos TGT) + participant Trf as k3s Traefik + participant SpnegoProxy as gitea-spnego-proxy + participant Gitea as Gitea + + Br->>Trf: GET https://git.prole.org/ + Trf->>SpnegoProxy: pass-through + SpnegoProxy-->>Br: 401 + WWW-Authenticate: Negotiate + Br->>SambaKDC: get HTTP/git.prole.org service ticket + Br->>SpnegoProxy: GET / + Authorization: Negotiate + SpnegoProxy->>SpnegoProxy: validate SPNEGO with keytab
(HTTP/git.prole.org@PROLE.ORG) + SpnegoProxy->>Gitea: GET / + X-WEBAUTH-USER: chrisfu + Note over Gitea: ENABLE_REVERSE_PROXY_AUTHENTICATION=true
auto-provisions user on first sign-in + Gitea-->>Br: 200 (signed-in as chrisfu) +``` + +Components: + +- `gitea-spnego-proxy` Deployment (Go service, port 4000) with a + `HTTP/git.prole.org@PROLE.ORG` keytab mounted from a k8s Secret +- Gitea env `ENABLE_REVERSE_PROXY_AUTHENTICATION=true` + + `REVERSE_PROXY_AUTHENTICATION_HEADER=X-WEBAUTH-USER` +- Traefik IngressRoute routing `git.prole.org` → proxy → Gitea +- Gitea Helm-deployed; PVC on local-path SC; values managed in + `gitea/deploy.sh` + +For password fallback (1Password-stored), each engineer has a Gitea +recovery password set via `etc/init_knoe_users.sh:gitea_api_set_password` +(landed today after several false-positive-success rounds — see commits +`5d545d2`, `d5f6e8f`, `4f23b63`). + +## 6. Supabase Studio web SSO + +Two endpoints, two states. + +### 6.1 `db.prole.org` (k3s, current) + +Live since 2026-04-30. **Google OAuth** via `oauth2-proxy` with +`--email-domain=knoey.com`. Backed by the k3s Supabase deployment. +Auth flow is the standard oauth2-proxy → Kong → Studio chain +documented in the canonical doc. + +Future plan: keep `db.prole.org` on Google OAuth as the homelab path +(prole workstations don't have a domain-joined identity that maps +to knoey.com naturally). A separate Kerberos-gated path can be +layered in later if needed. + +### 6.2 `db.knoe.dev` (GKE, planned) + +Brief: [`docs/plans/junie/per-user-supabase-workspace.md`](../docs/plans/junie/per-user-supabase-workspace.md). +Target state — swap oauth2-proxy + GoTrue from Google OAuth to +**knoe-auth OIDC** as the upstream, so engineers sign in once +(via knoe-auth) and land in their per-user Postgres schema. + +**Blocked on:** [`pg_oauth-image-install.md`](https://github.com/dredx/knoe-db/blob/main/docs/plans/junie/pg_oauth-image-install.md) +— the knoe-db image doesn't yet contain the pg_oauth extension, so +the Studio → Postgres step can't validate JWTs. + +## 7. Engineer psql access + +Two paths active. + +### 7.1 SCRAM (Phase 1; current default) + +Per-engineer LOGIN role with a strong password rotated through +1Password (`knoey` vault). Works today, scales poorly. See +[`db-access.md`](db-access.md) §Phase 1. + +### 7.2 OAUTHBEARER via Device Flow (Phase 2; shipped, blocked) + +Brief: [`pg-oauth-resume.md`](plans/junie/pg-oauth-resume.md). Shipped +to source (commit `eef729a`) on 2026-05-11 with full spec changes to +`deploy/gcp/gke/knoe-db.yaml` and `etc/init_cnpg_gke.sh`, but the +matching pg_oauth extension was never installed in the knoe-db +Docker image. The cluster crashed on rollout attempt +(2026-05-11 evening); rolled back via `kubectl patch` to remove +`pg_oauth` from `shared_preload_libraries`. **Live cluster spec +currently diverged from git** — will reconcile when +[`pg_oauth-image-install.md`](https://github.com/dredx/knoe-db/blob/main/docs/plans/junie/pg_oauth-image-install.md) +lands. + +## 8. Engineer onboarding (prole-specific steps) + +1. **knoey.com Google Workspace** — invite `@knoey.com` + (existing operator with admin role does this in Google Admin). +2. **Samba AD** — provision the workstation account on myrddin: + ```bash + sudo samba-tool user create + sudo samba-tool group addmembers prole-engineers + ``` +3. **knoe-auth user table** — `etc/init_knoe_users.sh provision_user + @knoey.com '' developer` (or + use the lower-level `etc/onboard_engineer.sh`). +4. **Postgres role** — same per-engineer SCRAM role flow as + [`db-access.md`](db-access.md) Phase 1. After + `init_user_schemas_gke.sh` (or local k3s equivalent) lands the + per-user schema brief, also add to the allowlist there. +5. **1Password access** — share the `knoey` vault. +6. **Workstation join (optional)** — if they'll log into a homelab + workstation, run `realm join PROLE.ORG` plus the Ansible + `workstation_kerberos.yml` playbook so their `/etc/krb5.conf` and + browser policies are set up for SPNEGO. + +## 9. Known fragility / current incidents + +| Issue | First seen | State | +|---|---|---| +| pg_oauth extension missing from knoe-db image | 2026-05-11 | Open. Cluster spec rolled back; awaiting [`pg_oauth-image-install.md`](https://github.com/dredx/knoe-db/blob/main/docs/plans/junie/pg_oauth-image-install.md). | +| `init_user_schemas_gke.sh` silent no-op | 2026-05-11 | Open. Manually patched chrisfu + ron tonight; brief filed at [`init-user-schemas-heredoc-fix.md`](https://github.com/dredx/knoe-db/blob/main/docs/plans/junie/init-user-schemas-heredoc-fix.md). | +| `deploy.sh` calls non-existent `knoe.deploy_pipeline` module | Since rebrand 2026 commit `55b6a6a` | Open. Brief: [`deploy-pipeline-restore.md`](https://github.com/dredx/knoe-db/blob/main/docs/plans/junie/deploy-pipeline-restore.md). | +| MagicMock leaking into `conf/*.cfg` despite shipped fix | 2026-05-11 (after `a220400`) | Open. Original brief `todo-1-cfg-save-path-bug.md` is shipped but doesn't catch this shape; needs reopen. | +| KDC cross-realm trust needs a re-run after cluster Secret rotation | Always | Documented; idempotent playbook handles it. | +| Gitea admin-promotion PATCH resets password on SPNEGO users | 2026-05-10 | Fixed (commits `5d545d2`, `d5f6e8f`, `4f23b63`). | + +## 10. Reference + +| File / dir | Role | +|---|---| +| `infrastructure/playbooks/kerberos_trust_setup.yml` | Cross-realm trust on the Samba side | +| `infrastructure/playbooks/workstation_kerberos.yml` | Workstation `/etc/krb5.conf` + browser policy | +| `etc/init_knoe_users.sh` | User-table + Gitea + Kerberos principal provisioning | +| `etc/init_kdc.sh` | In-cluster MIT KDC + cross-realm krbtgts | +| `etc/init_1password.sh` | 1Password preflight + `knoey` vault bootstrap | +| `gitea/deploy.sh` | Gitea Helm install + SPNEGO proxy + reverse-proxy auth env | +| `k8s/knoe/gitea-spnego-proxy.yaml` | gitea-spnego-proxy Deployment + keytab Secret mount | +| `deploy/opentofu/k3s/manifests/knoe/` | k3s manifests (mirror of knoe-db's `deploy/gcp/gke/` + `k8s/knoe/`) | +| `~/.vault_pass` | Ansible vault password file (operator-local, not in repo) | +| `inventory/group_vars/ad_dc/vault.yml` | Samba admin password (Ansible vault) | +| `inventory/group_vars/all/vault_k3s.yml` | k3s join token + secrets | + +## 11. See also + +- [`~/dev/knoe-db/docs/knoe-auth.md`](https://github.com/dredx/knoe-db/blob/main/docs/knoe-auth.md) — canonical architecture +- [`db-access.md`](db-access.md) — engineer psql access (Phase 1 + 2) +- [`knoe-auth-phase-2.md`](knoe-auth-phase-2.md) — OIDC issuer Phase 2 design +- [`local-dev-knoe-auth.md`](local-dev-knoe-auth.md) — k3d dev loop for the knoe-auth pod +- [`oidc-setup.md`](oidc-setup.md) — Google OAuth client registration for oauth2-proxy + Grafana