docs(runbook): stage db.prole.org GSSAPI auth for the airgap lane

Complete, review-ready runbook to light up Kerberos/GSSAPI on db.prole.org so
`kn db --service prole.org` authenticates with a local PROLE.ORG ticket — the
airgap data lane (no password, no knoe-auth, no internet).

Nothing applied. Delivers:
- Samba SPN + keytab export steps (myrddin): postgres/db.prole.org@PROLE.ORG
- k8s keytab secret (knoe-db-gss-keytab-secret.example.yaml template)
- exact knoe-db.yaml diff: krb_server_keyfile, hostgssenc pg_hba (include_realm=0
  → role), projectedVolumeTemplate keytab mount, chrisfu/ron managed roles
- apply + end-to-end verify (kinit → kn db --service prole.org)
- failure-mode triage, rollback, follow-ups

Hooks into the manifest's existing placeholders (knoe-db.yaml lines 38 + 45,
which already note 'Kerberos disabled on k3s'). Image is already --with-gssapi.

Flags the PROLE.ORG (Kerberos) vs PROLE.LOCAL (knoe.user default) discrepancy
to reconcile. Grants intentionally deferred — connect succeeds, reads gated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

deploy/opentofu/k3s/manifests/knoe/knoe-db-gss-keytab-secret.example.yaml

@@ -0,0 +1,26 @@
+# knoe-db-gss-keytab — Kerberos service keytab for db.prole.org GSSAPI auth.
+#
+# Holds the keytab for the Postgres service principal so libpq GSSAPI clients
+# (e.g. `kn db --service prole.org`) can authenticate with a local PROLE.ORG
+# ticket — no password, fully airgap-capable.
+#
+# DO NOT commit the real keytab. This is a template; create the live secret from
+# the keytab exported on myrddin (see docs/runbooks/db-prole-gssapi.md):
+#
+#   sudo samba-tool domain exportkeytab /tmp/pg.keytab \
+#       --principal=postgres/db.prole.org@PROLE.ORG
+#   kubectl --context=prole-service-cluster -n knoe create secret generic \
+#     knoe-db-gss-keytab --from-file=postgres.keytab=/tmp/pg.keytab
+#   rm -f /tmp/pg.keytab        # shred the keytab off disk
+#
+# The CNPG cluster mounts this via spec.projectedVolumeTemplate at
+# /projected/postgres.keytab and points krb_server_keyfile there.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: knoe-db-gss-keytab
+  namespace: knoe
+type: Opaque
+data:
+  # base64 of the binary keytab for postgres/db.prole.org@PROLE.ORG
+  postgres.keytab: <BASE64_KEYTAB_HERE>

docs/runbooks/db-prole-gssapi.md

@@ -0,0 +1,175 @@
+# Runbook — GSSAPI (Kerberos) auth on db.prole.org
+
+**Status:** staged for review — nothing here is applied yet.
+**Goal:** let `kn db --service prole.org` authenticate to `db.prole.org` with a
+local PROLE.ORG Kerberos ticket via libpq GSSAPI. No password, no knoe-auth, no
+outbound network — the airgap data lane.
+
+## Why GSSAPI (not OAUTHBEARER) for the airgap lane
+
+The cloud lane (`db.0.knoe.dev`) uses OAUTHBEARER + pg_knoe_auth against
+knoe-auth's JWKS — that needs network to the issuer. The airgap lane's KDC
+(`myrddin.prole.org`, realm `PROLE.ORG`) is **on the LAN**, and the workstation
+already holds a `PROLE.ORG` TGT. libpq's native GSSAPI uses that ticket
+directly, so the DB lane works with zero internet. `kn db` already selects this
+lane for `--service prole.org` (commit `a06ddad`); this runbook lights up the
+server side.
+
+## Preconditions (verify first)
+
+- **Realm:** workstation `default_realm = PROLE.ORG`, `kdc = myrddin.prole.org`
+  (confirmed in `/etc/krb5.conf`). Note: `knoe-db.yaml` `knoe.user.realm`
+  defaults to `PROLE.LOCAL` — that's a stale app default, **not** the Kerberos
+  realm. Use `PROLE.ORG` throughout. (Worth reconciling separately.)
+- **Image:** `registry.prole.org/knoe-system/knoe-db:18-*` is built
+  `--with-gssapi` (confirmed via `pg_config --configure`). No image change.
+- **DNS:** `db.prole.org` resolves on the LAN to the k3s LB (10.0.0.x). The
+  Postgres SPN host (`postgres/db.prole.org`) must match what clients connect to.
+
+---
+
+## Step 1 — Samba: create the service principal + export the keytab (on myrddin)
+
+```bash
+ssh myrddin.prole.org
+# A dedicated service account holds the SPN (cleaner than the machine account):
+sudo samba-tool user create knoe-db-pg --random-password \
+  --description="Postgres GSSAPI service principal for db.prole.org"
+sudo samba-tool spn add postgres/db.prole.org knoe-db-pg
+
+# Export a keytab for exactly that principal:
+sudo samba-tool domain exportkeytab /tmp/pg.keytab \
+  --principal=postgres/db.prole.org@PROLE.ORG
+sudo chown "$USER" /tmp/pg.keytab
+
+# Sanity: the keytab should list postgres/db.prole.org@PROLE.ORG
+klist -k /tmp/pg.keytab
+```
+
+Copy `/tmp/pg.keytab` to where you run `kubectl`, then **shred it on myrddin**:
+`shred -u /tmp/pg.keytab`.
+
+## Step 2 — Create the k8s keytab secret
+
+```bash
+kubectl --context=prole-service-cluster -n knoe create secret generic \
+  knoe-db-gss-keytab --from-file=postgres.keytab=/tmp/pg.keytab
+shred -u /tmp/pg.keytab     # remove the keytab from disk
+```
+
+(Template: `manifests/knoe/knoe-db-gss-keytab-secret.example.yaml`.)
+
+## Step 3 — Patch the CNPG cluster (`manifests/knoe/knoe-db.yaml`)
+
+Three edits. **Diff:**
+
+```yaml
+  postgresql:
+    parameters:
+      shared_buffers: 256MB
+      pg_stat_statements.max: '10000'
+      pg_stat_statements.track: all
+-     # krb_server_keyfile omitted — Kerberos disabled on k3s (no keytab secret)
++     # Kerberos: keytab mounted via projectedVolumeTemplate at /projected
++     krb_server_keyfile: '/projected/postgres.keytab'
+    shared_preload_libraries:
+      - pg_stat_statements
+      - pg_tde
+    pg_hba:
+      - local all postgres trust
+      - local all knoe scram-sha-256
+-     # SCRAM auth — Kerberos/GSS rules omitted (kerberos_enabled=false on k3s)
++     # GSSAPI: PROLE.ORG ticket → role (include_realm=0 strips @PROLE.ORG).
++     # Placed before the catch-all scram rules so gss is tried first.
++     - hostgssenc knoe-db chrisfu 0.0.0.0/0 gss include_realm=0 krb_realm=PROLE.ORG
++     - hostgssenc knoe-db ron     0.0.0.0/0 gss include_realm=0 krb_realm=PROLE.ORG
+      - host all postgres all scram-sha-256
+      - host knoe knoe-db all scram-sha-256
+      - host all all all scram-sha-256
+      - hostssl knoe knoe-db all scram-sha-256
+```
+
+Mount the keytab (CNPG `projectedVolumeTemplate` lands sources under `/projected`):
+
+```yaml
+spec:
+  # … existing fields …
+  projectedVolumeTemplate:
+    sources:
+      - secret:
+          name: knoe-db-gss-keytab
+          items:
+            - key: postgres.keytab
+              path: postgres.keytab
+```
+
+Add the login roles (under `managed.roles`, mirroring `admin`):
+
+```yaml
+  managed:
+    roles:
+      - name: chrisfu
+        ensure: present
+        login: true
+        comment: "Kerberos GSSAPI identity (PROLE.ORG)"
+      - name: ron
+        ensure: present
+        login: true
+        comment: "Kerberos GSSAPI identity (PROLE.ORG)"
+```
+
+> Grants are deliberately omitted — these roles can log in but read nothing
+> until granted. Decide schema access separately (mirrors the cloud lane, where
+> a connect succeeds but `permission denied for schema …` until granted).
+
+## Step 4 — Apply
+
+```bash
+# Whichever applies this cluster (ArgoCD app-of-apps, or direct):
+kubectl --context=prole-service-cluster apply \
+  -f deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml
+# CNPG reconciles pg_hba + the projected keytab without a full restart;
+# a config reload is enough. Watch:
+kubectl --context=prole-service-cluster -n knoe get cluster knoe-db -w
+```
+
+## Step 5 — Verify the airgap lane end to end
+
+```bash
+kinit chrisfu@PROLE.ORG                       # local TGT, no internet
+kn db --service prole.org -c "select current_user, session_user"
+# → chrisfu  chrisfu      (GSSAPI, no password)
+```
+
+If it fails, check the Postgres log for the GSSAPI reason:
+```bash
+POD=$(kubectl --context=prole-service-cluster -n knoe get pods \
+  -l cnpg.io/instanceRole=primary -o jsonpath='{.items[0].metadata.name}')
+kubectl --context=prole-service-cluster -n knoe logs "$POD" -c postgres --tail=40 \
+  | grep -iE "gss|krb|authentication"
+```
+
+Common causes:
+- *"GSSAPI ... no credentials"* → keytab principal ≠ `postgres/db.prole.org@PROLE.ORG`
+  (re-export, confirm `klist -k`).
+- *server falls back to scram ("no password supplied")* → the `hostgssenc` rule
+  didn't match (db/user/order), or `gssencmode` not negotiated.
+- *clock skew* → Kerberos needs workstation and KDC within ~5 min; check NTP.
+
+---
+
+## Rollback
+
+Remove the two `hostgssenc` lines and `krb_server_keyfile`, re-apply. The secret
+and roles are inert without the pg_hba rules. Delete the secret/roles if desired.
+
+## Notes / follow-ups
+
+- **PROLE.ORG vs PROLE.LOCAL** in `knoe.user.realm` — reconcile so the app's
+  notion of realm matches Kerberos.
+- **Grants** — define what `chrisfu`/`ron` (and future group roles) may read.
+  The cloud lane carries a `groups` claim; the airgap lane can map AD group
+  membership to roles for parity.
+- **Same source of truth** — once this lands, an airgap project + its local MCP
+  both point at `db.prole.org`, exactly as a cloud project + MCP share
+  `db.0.knoe.dev`.