From c9591a5da2208d0aaf90346f082a091da88b09e3 Mon Sep 17 00:00:00 2001 From: chrisfu Date: Wed, 3 Jun 2026 02:59:07 -0700 Subject: [PATCH] docs(runbook): stage db.prole.org GSSAPI auth for the airgap lane MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Complete, review-ready runbook to light up Kerberos/GSSAPI on db.prole.org so `kn db --service prole.org` authenticates with a local PROLE.ORG ticket — the airgap data lane (no password, no knoe-auth, no internet). Nothing applied. Delivers: - Samba SPN + keytab export steps (myrddin): postgres/db.prole.org@PROLE.ORG - k8s keytab secret (knoe-db-gss-keytab-secret.example.yaml template) - exact knoe-db.yaml diff: krb_server_keyfile, hostgssenc pg_hba (include_realm=0 → role), projectedVolumeTemplate keytab mount, chrisfu/ron managed roles - apply + end-to-end verify (kinit → kn db --service prole.org) - failure-mode triage, rollback, follow-ups Hooks into the manifest's existing placeholders (knoe-db.yaml lines 38 + 45, which already note 'Kerberos disabled on k3s'). Image is already --with-gssapi. Flags the PROLE.ORG (Kerberos) vs PROLE.LOCAL (knoe.user default) discrepancy to reconcile. Grants intentionally deferred — connect succeeds, reads gated. Co-Authored-By: Claude Opus 4.8 --- .../knoe-db-gss-keytab-secret.example.yaml | 26 +++ docs/runbooks/db-prole-gssapi.md | 175 ++++++++++++++++++ 2 files changed, 201 insertions(+) create mode 100644 deploy/opentofu/k3s/manifests/knoe/knoe-db-gss-keytab-secret.example.yaml create mode 100644 docs/runbooks/db-prole-gssapi.md diff --git a/deploy/opentofu/k3s/manifests/knoe/knoe-db-gss-keytab-secret.example.yaml b/deploy/opentofu/k3s/manifests/knoe/knoe-db-gss-keytab-secret.example.yaml new file mode 100644 index 0000000..500870c --- /dev/null +++ b/deploy/opentofu/k3s/manifests/knoe/knoe-db-gss-keytab-secret.example.yaml @@ -0,0 +1,26 @@ +# knoe-db-gss-keytab — Kerberos service keytab for db.prole.org GSSAPI auth. +# +# Holds the keytab for the Postgres service principal so libpq GSSAPI clients +# (e.g. `kn db --service prole.org`) can authenticate with a local PROLE.ORG +# ticket — no password, fully airgap-capable. +# +# DO NOT commit the real keytab. This is a template; create the live secret from +# the keytab exported on myrddin (see docs/runbooks/db-prole-gssapi.md): +# +# sudo samba-tool domain exportkeytab /tmp/pg.keytab \ +# --principal=postgres/db.prole.org@PROLE.ORG +# kubectl --context=prole-service-cluster -n knoe create secret generic \ +# knoe-db-gss-keytab --from-file=postgres.keytab=/tmp/pg.keytab +# rm -f /tmp/pg.keytab # shred the keytab off disk +# +# The CNPG cluster mounts this via spec.projectedVolumeTemplate at +# /projected/postgres.keytab and points krb_server_keyfile there. +apiVersion: v1 +kind: Secret +metadata: + name: knoe-db-gss-keytab + namespace: knoe +type: Opaque +data: + # base64 of the binary keytab for postgres/db.prole.org@PROLE.ORG + postgres.keytab: diff --git a/docs/runbooks/db-prole-gssapi.md b/docs/runbooks/db-prole-gssapi.md new file mode 100644 index 0000000..f6da724 --- /dev/null +++ b/docs/runbooks/db-prole-gssapi.md @@ -0,0 +1,175 @@ +# Runbook — GSSAPI (Kerberos) auth on db.prole.org + +**Status:** staged for review — nothing here is applied yet. +**Goal:** let `kn db --service prole.org` authenticate to `db.prole.org` with a +local PROLE.ORG Kerberos ticket via libpq GSSAPI. No password, no knoe-auth, no +outbound network — the airgap data lane. + +## Why GSSAPI (not OAUTHBEARER) for the airgap lane + +The cloud lane (`db.0.knoe.dev`) uses OAUTHBEARER + pg_knoe_auth against +knoe-auth's JWKS — that needs network to the issuer. The airgap lane's KDC +(`myrddin.prole.org`, realm `PROLE.ORG`) is **on the LAN**, and the workstation +already holds a `PROLE.ORG` TGT. libpq's native GSSAPI uses that ticket +directly, so the DB lane works with zero internet. `kn db` already selects this +lane for `--service prole.org` (commit `a06ddad`); this runbook lights up the +server side. + +## Preconditions (verify first) + +- **Realm:** workstation `default_realm = PROLE.ORG`, `kdc = myrddin.prole.org` + (confirmed in `/etc/krb5.conf`). Note: `knoe-db.yaml` `knoe.user.realm` + defaults to `PROLE.LOCAL` — that's a stale app default, **not** the Kerberos + realm. Use `PROLE.ORG` throughout. (Worth reconciling separately.) +- **Image:** `registry.prole.org/knoe-system/knoe-db:18-*` is built + `--with-gssapi` (confirmed via `pg_config --configure`). No image change. +- **DNS:** `db.prole.org` resolves on the LAN to the k3s LB (10.0.0.x). The + Postgres SPN host (`postgres/db.prole.org`) must match what clients connect to. + +--- + +## Step 1 — Samba: create the service principal + export the keytab (on myrddin) + +```bash +ssh myrddin.prole.org +# A dedicated service account holds the SPN (cleaner than the machine account): +sudo samba-tool user create knoe-db-pg --random-password \ + --description="Postgres GSSAPI service principal for db.prole.org" +sudo samba-tool spn add postgres/db.prole.org knoe-db-pg + +# Export a keytab for exactly that principal: +sudo samba-tool domain exportkeytab /tmp/pg.keytab \ + --principal=postgres/db.prole.org@PROLE.ORG +sudo chown "$USER" /tmp/pg.keytab + +# Sanity: the keytab should list postgres/db.prole.org@PROLE.ORG +klist -k /tmp/pg.keytab +``` + +Copy `/tmp/pg.keytab` to where you run `kubectl`, then **shred it on myrddin**: +`shred -u /tmp/pg.keytab`. + +## Step 2 — Create the k8s keytab secret + +```bash +kubectl --context=prole-service-cluster -n knoe create secret generic \ + knoe-db-gss-keytab --from-file=postgres.keytab=/tmp/pg.keytab +shred -u /tmp/pg.keytab # remove the keytab from disk +``` + +(Template: `manifests/knoe/knoe-db-gss-keytab-secret.example.yaml`.) + +## Step 3 — Patch the CNPG cluster (`manifests/knoe/knoe-db.yaml`) + +Three edits. **Diff:** + +```yaml + postgresql: + parameters: + shared_buffers: 256MB + pg_stat_statements.max: '10000' + pg_stat_statements.track: all +- # krb_server_keyfile omitted — Kerberos disabled on k3s (no keytab secret) ++ # Kerberos: keytab mounted via projectedVolumeTemplate at /projected ++ krb_server_keyfile: '/projected/postgres.keytab' + shared_preload_libraries: + - pg_stat_statements + - pg_tde + pg_hba: + - local all postgres trust + - local all knoe scram-sha-256 +- # SCRAM auth — Kerberos/GSS rules omitted (kerberos_enabled=false on k3s) ++ # GSSAPI: PROLE.ORG ticket → role (include_realm=0 strips @PROLE.ORG). ++ # Placed before the catch-all scram rules so gss is tried first. ++ - hostgssenc knoe-db chrisfu 0.0.0.0/0 gss include_realm=0 krb_realm=PROLE.ORG ++ - hostgssenc knoe-db ron 0.0.0.0/0 gss include_realm=0 krb_realm=PROLE.ORG + - host all postgres all scram-sha-256 + - host knoe knoe-db all scram-sha-256 + - host all all all scram-sha-256 + - hostssl knoe knoe-db all scram-sha-256 +``` + +Mount the keytab (CNPG `projectedVolumeTemplate` lands sources under `/projected`): + +```yaml +spec: + # … existing fields … + projectedVolumeTemplate: + sources: + - secret: + name: knoe-db-gss-keytab + items: + - key: postgres.keytab + path: postgres.keytab +``` + +Add the login roles (under `managed.roles`, mirroring `admin`): + +```yaml + managed: + roles: + - name: chrisfu + ensure: present + login: true + comment: "Kerberos GSSAPI identity (PROLE.ORG)" + - name: ron + ensure: present + login: true + comment: "Kerberos GSSAPI identity (PROLE.ORG)" +``` + +> Grants are deliberately omitted — these roles can log in but read nothing +> until granted. Decide schema access separately (mirrors the cloud lane, where +> a connect succeeds but `permission denied for schema …` until granted). + +## Step 4 — Apply + +```bash +# Whichever applies this cluster (ArgoCD app-of-apps, or direct): +kubectl --context=prole-service-cluster apply \ + -f deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml +# CNPG reconciles pg_hba + the projected keytab without a full restart; +# a config reload is enough. Watch: +kubectl --context=prole-service-cluster -n knoe get cluster knoe-db -w +``` + +## Step 5 — Verify the airgap lane end to end + +```bash +kinit chrisfu@PROLE.ORG # local TGT, no internet +kn db --service prole.org -c "select current_user, session_user" +# → chrisfu chrisfu (GSSAPI, no password) +``` + +If it fails, check the Postgres log for the GSSAPI reason: +```bash +POD=$(kubectl --context=prole-service-cluster -n knoe get pods \ + -l cnpg.io/instanceRole=primary -o jsonpath='{.items[0].metadata.name}') +kubectl --context=prole-service-cluster -n knoe logs "$POD" -c postgres --tail=40 \ + | grep -iE "gss|krb|authentication" +``` + +Common causes: +- *"GSSAPI ... no credentials"* → keytab principal ≠ `postgres/db.prole.org@PROLE.ORG` + (re-export, confirm `klist -k`). +- *server falls back to scram ("no password supplied")* → the `hostgssenc` rule + didn't match (db/user/order), or `gssencmode` not negotiated. +- *clock skew* → Kerberos needs workstation and KDC within ~5 min; check NTP. + +--- + +## Rollback + +Remove the two `hostgssenc` lines and `krb_server_keyfile`, re-apply. The secret +and roles are inert without the pg_hba rules. Delete the secret/roles if desired. + +## Notes / follow-ups + +- **PROLE.ORG vs PROLE.LOCAL** in `knoe.user.realm` — reconcile so the app's + notion of realm matches Kerberos. +- **Grants** — define what `chrisfu`/`ron` (and future group roles) may read. + The cloud lane carries a `groups` claim; the airgap lane can map AD group + membership to roles for parity. +- **Same source of truth** — once this lands, an airgap project + its local MCP + both point at `db.prole.org`, exactly as a cloud project + MCP share + `db.0.knoe.dev`.