prole/docs/runbooks/db-prole-gssapi.md

Runbook — GSSAPI (Kerberos) auth on db.prole.org

Status: staged for review — nothing here is applied yet. Goal: let kn db --service prole.org authenticate to db.prole.org with a local PROLE.ORG Kerberos ticket via libpq GSSAPI. No password, no knoe-auth, no outbound network — the airgap data lane.

Why GSSAPI (not OAUTHBEARER) for the airgap lane

The cloud lane (db.0.knoe.dev) uses OAUTHBEARER + pg_knoe_auth against knoe-auth's JWKS — that needs network to the issuer. The airgap lane's KDC (myrddin.prole.org, realm PROLE.ORG) is on the LAN, and the workstation already holds a PROLE.ORG TGT. libpq's native GSSAPI uses that ticket directly, so the DB lane works with zero internet. kn db already selects this lane for --service prole.org (commit a06ddad); this runbook lights up the server side.

Preconditions (verify first)

• Realm: workstation default_realm = PROLE.ORG, kdc = myrddin.prole.org (confirmed in /etc/krb5.conf). Note: knoe-db.yaml knoe.user.realm defaults to PROLE.LOCAL — that's a stale app default, not the Kerberos realm. Use PROLE.ORG throughout. (Worth reconciling separately.)
• Image: registry.prole.org/knoe-system/knoe-db:18-* is built --with-gssapi (confirmed via pg_config --configure). No image change.
• DNS: db.prole.org resolves on the LAN to the k3s LB (10.0.0.x). The Postgres SPN host (postgres/db.prole.org) must match what clients connect to.

---

Step 1 — Samba: create the service principal + export the keytab (on myrddin)

ssh myrddin.prole.org
# A dedicated service account holds the SPN (cleaner than the machine account):
sudo samba-tool user create knoe-db-pg --random-password \
  --description="Postgres GSSAPI service principal for db.prole.org"
sudo samba-tool spn add postgres/db.prole.org knoe-db-pg

# Export a keytab for exactly that principal:
sudo samba-tool domain exportkeytab /tmp/pg.keytab \
  --principal=postgres/db.prole.org@PROLE.ORG
sudo chown "$USER" /tmp/pg.keytab

# Sanity: the keytab should list postgres/db.prole.org@PROLE.ORG
klist -k /tmp/pg.keytab

Copy /tmp/pg.keytab to where you run kubectl, then shred it on myrddin: shred -u /tmp/pg.keytab.

Step 2 — Create the k8s keytab secret

kubectl --context=prole-service-cluster -n knoe create secret generic \
  knoe-db-gss-keytab --from-file=postgres.keytab=/tmp/pg.keytab
shred -u /tmp/pg.keytab     # remove the keytab from disk

(Template: manifests/knoe/knoe-db-gss-keytab-secret.example.yaml.)

Step 3 — Patch the CNPG cluster (manifests/knoe/knoe-db.yaml)

Three edits. Diff:

  postgresql:
    parameters:
      shared_buffers: 256MB
      pg_stat_statements.max: '10000'
      pg_stat_statements.track: all
-     # krb_server_keyfile omitted — Kerberos disabled on k3s (no keytab secret)
+     # Kerberos: keytab mounted via projectedVolumeTemplate at /projected
+     krb_server_keyfile: '/projected/postgres.keytab'
    shared_preload_libraries:
      - pg_stat_statements
      - pg_tde
    pg_hba:
      - local all postgres trust
      - local all knoe scram-sha-256
-     # SCRAM auth — Kerberos/GSS rules omitted (kerberos_enabled=false on k3s)
+     # GSSAPI: PROLE.ORG ticket → role (include_realm=0 strips @PROLE.ORG).
+     # Placed before the catch-all scram rules so gss is tried first.
+     - hostgssenc knoe-db chrisfu 0.0.0.0/0 gss include_realm=0 krb_realm=PROLE.ORG
+     - hostgssenc knoe-db ron     0.0.0.0/0 gss include_realm=0 krb_realm=PROLE.ORG
      - host all postgres all scram-sha-256
      - host knoe knoe-db all scram-sha-256
      - host all all all scram-sha-256
      - hostssl knoe knoe-db all scram-sha-256

Mount the keytab (CNPG projectedVolumeTemplate lands sources under /projected):

spec:
  # … existing fields …
  projectedVolumeTemplate:
    sources:
      - secret:
          name: knoe-db-gss-keytab
          items:
            - key: postgres.keytab
              path: postgres.keytab

Add the login roles (under managed.roles, mirroring admin):

  managed:
    roles:
      - name: chrisfu
        ensure: present
        login: true
        comment: "Kerberos GSSAPI identity (PROLE.ORG)"
      - name: ron
        ensure: present
        login: true
        comment: "Kerberos GSSAPI identity (PROLE.ORG)"

Grants are deliberately omitted — these roles can log in but read nothing until granted. Decide schema access separately (mirrors the cloud lane, where a connect succeeds but permission denied for schema … until granted).

Step 4 — Apply

# Whichever applies this cluster (ArgoCD app-of-apps, or direct):
kubectl --context=prole-service-cluster apply \
  -f deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml
# CNPG reconciles pg_hba + the projected keytab without a full restart;
# a config reload is enough. Watch:
kubectl --context=prole-service-cluster -n knoe get cluster knoe-db -w

Step 5 — Verify the airgap lane end to end

kinit chrisfu@PROLE.ORG                       # local TGT, no internet
kn db --service prole.org -c "select current_user, session_user"
# → chrisfu  chrisfu      (GSSAPI, no password)

If it fails, check the Postgres log for the GSSAPI reason:

POD=$(kubectl --context=prole-service-cluster -n knoe get pods \
  -l cnpg.io/instanceRole=primary -o jsonpath='{.items[0].metadata.name}')
kubectl --context=prole-service-cluster -n knoe logs "$POD" -c postgres --tail=40 \
  | grep -iE "gss|krb|authentication"

Common causes:

• "GSSAPI ... no credentials" → keytab principal ≠ postgres/db.prole.org@PROLE.ORG (re-export, confirm klist -k).
• server falls back to scram ("no password supplied") → the hostgssenc rule didn't match (db/user/order), or gssencmode not negotiated.
• clock skew → Kerberos needs workstation and KDC within ~5 min; check NTP.

---

Rollback

Remove the two hostgssenc lines and krb_server_keyfile, re-apply. The secret and roles are inert without the pg_hba rules. Delete the secret/roles if desired.

Notes / follow-ups

• PROLE.ORG vs PROLE.LOCAL in knoe.user.realm — reconcile so the app's notion of realm matches Kerberos.
• Grants — define what chrisfu/ron (and future group roles) may read. The cloud lane carries a groups claim; the airgap lane can map AD group membership to roles for parity.
• Same source of truth — once this lands, an airgap project + its local MCP both point at db.prole.org, exactly as a cloud project + MCP share db.0.knoe.dev.