prole/docs/knoe-system.md

knoe-system — knoe-auth deployment and k3d TDD guide

Single reference for the knoe-auth identity service: what it does, how it is deployed on the GKE knoe-db stack, and how to spin up a local k3d cluster for test-driven development.

---

1. What knoe-auth is

knoe-auth is the identity system for the knoe.dev platform. It decides who you are, what you can access, and how new contributors come on board.

Round 1 (shipped) — Durable Kerberos identity + contributor enrollment. A Spring Boot service (authority/) wraps a KDC sidecar. Engineers enroll via an invite OTP → Google corroboration → TOTP setup flow; the result is a Kerberos principal (alice@KNOE.DEV) and a knoe.user row in PostgreSQL. Downstream provisioning (GitLab, Gitea, CNPG roles) is queued automatically.

Phase 2 (planned, not yet implemented) — knoe-auth becomes a full OIDC provider (issuer: https://api.knoe.dev/auth). Services that currently trust Google directly will trust knoe-auth instead; knoe-auth federates upstream to Google. See docs/knoe-auth-phase-2.md and the paused pg_oauth item in docs/TODO.md.

Trust model (Round 1)

The invite OTP is the trust anchor. Google sign-in is corroboration. TOTP is the ongoing second factor. knoe.dev never pre-configures trust with any external Google Workspace — any verified Google account works once invited.

---

2. Work-in-progress status

Item	Status	Notes
Round 1 Kerberos enrollment	Shipped	Operational on GKE
k3d dev loop (Phase 1 brief)	Shipped	make k3d-knoe-up
Phase 2 OIDC provider — k3d setup	Shipped	application-k3d.yml + etc/gen_oidc_signing_key.sh; OIDC discovery / JWKS / token endpoints exercise locally. See docs/local-dev-knoe-auth.md §"Daily loop".
Phase 2 OIDC provider — GKE deploy	Shipped	KNOE_AUTH_OIDC_ENABLED=true set; Kong /auth route added; signing key wired via knoe-auth-oidc secret (1Password). Run ./etc/init_knoe_auth.sh initialize to create the secret.
pg_oauth (PG18 OAUTHBEARER)	Ready	OIDC issuer shipped to GKE; pg_oauth can resume
Round 1.5 OpenBao transit-key encryption	Later	UserProvisioningService.java:84 TODO

---

3. GKE deployment — knoe-auth on knoe-db

Clusters

Resource	Cluster	Namespace
knoe-auth pod (Spring Boot + KDC sidecar)	knoe-dev-0 (app cluster)	knoe-system
PostgreSQL (CNPG knoe-db)	knoe-dev-cnpg-0 (DB cluster)	knoe-db-0

knoe-auth connects to PostgreSQL over the internal service knoe-db-rw.knoe-db-0.svc.cluster.local:5432 (cross-cluster via VPC-native routing; both clusters share the same VPC in us-west3).

Key manifests

File	Purpose
deploy/gcp/gke/knoe-auth-deployment.yaml	Deployment: KDC sidecar + keytab-bootstrap initContainer + Spring Boot container. Realm KNOE.DEV. Image: knoe-auth:latest.
deploy/gcp/gke/knoe-kdc-configmap.yaml	krb5.conf + kdc.conf for realm KNOE.DEV.
deploy/gcp/gke/knoe-kdc-secrets.yaml	Master key + admin password. Production values come from OpenBao; file is a placeholder only — no real secrets committed.
deploy/gcp/gke/knoe-auth-google-oidc-secret.example.yaml	GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET template.
deploy/gcp/gke/workload-identity.yaml	KSA↔GSA bindings for GCP secret access.
deploy/gcp/gke/knoe-db.yaml	CNPG Cluster manifest (3-replica, GCS barman backup).

Init script

etc/init_knoe_auth.sh provisions the knoe.* schema and seeds initial state. Run it after applying the manifests:

# GKE (default mode — requires kubectl context pointing at knoe-dev-0)
./etc/init_knoe_auth.sh schema
./etc/init_knoe_auth.sh status

Subcommands: schema (create/migrate tables), status (print principal + table counts), invite <email> (issue an enrollment invite).

Environment variables (Spring Boot container)

Variable	Purpose
KNOE_KDC_REALM	Kerberos realm (default KNOE.DEV)
KNOE_KERBEROS_SERVICE_PRINCIPAL	Service principal for SPNEGO (HTTP/auth.knoe.dev@KNOE.DEV)
KNOE_AUTH_BASE_URL	Public base URL (https://api.knoe.dev/auth)
KNOE_AUTH_COOKIE_DOMAIN	Cookie scope (knoe.dev)
KNOE_AUTH_SESSION_SECRET	HMAC key for session JWTs
KNOE_AUTH_OIDC_ENABLED	false until Phase 2 ships
KNOE_AUTH_OIDC_ISSUER	Phase 2 issuer URL
KNOE_AUTH_OIDC_SIGNING_KEY	Base64 PKCS#8 RS256 private key (Phase 2)
KNOE_AUTH_GOOGLE_CLIENT_ID	Upstream Google OAuth client
KNOE_AUTH_GOOGLE_CLIENT_SECRET	Upstream Google OAuth secret

Apply sequence (fresh GKE deploy)

# 1. Ensure kubectl context is knoe-dev-0
kubectl config use-context gke_plenary-truck-485623-p7_us-west3_knoe-dev-0

# 2. Apply KDC config + secrets (secrets must be pre-populated from OpenBao)
kubectl apply -f deploy/gcp/gke/knoe-kdc-configmap.yaml
kubectl apply -f deploy/gcp/gke/knoe-kdc-secrets.yaml   # populate first!

# 3. Apply the deployment
kubectl apply -f deploy/gcp/gke/knoe-auth-deployment.yaml

# 4. Wait for rollout
kubectl -n knoe-system rollout status deployment/knoe-auth

# 5. Seed schema (runs against the DB cluster via the pod's psql)
./etc/init_knoe_auth.sh schema
./etc/init_knoe_auth.sh status

---

4. k3d local dev loop (TDD)

Run knoe-auth from IntelliJ or mvn spring-boot:run against a real PostgreSQL + Kerberos KDC, both in a local k3d cluster. No GKE, no 1Password, no Google OAuth tenant required.

Full details: docs/local-dev-knoe-auth.md. Quick reference below.

Prerequisites (install once)

brew install k3d kubectl maven openjdk@21
# Docker Desktop must be running

Verify:

k3d version        # ≥ 5.x
kubectl version --client
mvn -version       # Maven 3.x, Java 21
docker info        # must not error

One-time cluster setup

make k3d-knoe-up

Takes 3–5 minutes. Creates a single-node k3d cluster (k3d-knoe), installs the CNPG operator (v1.29.0), applies a single-replica knoe-db PostgreSQL cluster, deploys the KDC (realm KNOE.LOCAL), seeds the knoe.* schema, and seeds the knoe_developer group role + per-engineer LOGIN role chrisfu (password chrisfu-dev, member of knoe_developer) so the host can connect as chrisfu@knoey.com over the port-forward.

Verify:

kubectl --context=k3d-k3d-knoe -n knoe-db-0 get cluster knoe-db
# → knoe-db   Cluster in healthy state

make k3d-knoe-smoke   # full smoke check

Daily dev loop

# Terminal 1 — keep port-forwards alive
make k3d-knoe-pf
# Opens: localhost:5432 → PostgreSQL, localhost:88 → KDC (TCP), localhost:464 → kpasswd

# Terminal 2 — get a Kerberos ticket
export KRB5_CONFIG=$(pwd)/etc/krb5.local.conf
kinit developer@KNOE.LOCAL          # password: developer
klist                                # confirm ticket

# Terminal 3 — run knoe-auth
cd authority
mvn spring-boot:run \
  -Dspring-boot.run.jvmArguments="-Djava.security.krb5.conf=$(pwd)/../etc/krb5.local.conf" \
  -Dspring-boot.run.profiles=k3d

# Verify
curl http://localhost:8080/health    # → {"status":"ok"}

IntelliJ run config

Add a Spring Boot run configuration for KnoeAuthApplication with:

• VM options: -Djava.security.krb5.conf=$PROJECT_DIR$/etc/krb5.local.conf
• Active profiles: k3d
• Working directory: $MODULE_WORKING_DIR$

Kerberos realm differences

	k3d (local)	GKE (production)
Realm	KNOE.LOCAL	KNOE.DEV
KDC address	localhost:88 (port-forwarded)	knoe-kdc.knoe-system.svc
Admin principal	admin/admin@KNOE.LOCAL	OpenBao-managed
Developer principal	developer@KNOE.LOCAL (password: developer)	per-engineer invite flow
krb5 config	etc/krb5.local.conf	pod-mounted from knoe-kdc-configmap.yaml

etc/krb5.local.conf sets udp_preference_limit = 1 to force TCP — required because kubectl port-forward does not proxy UDP on macOS.

Tear down

make k3d-knoe-down   # deletes the k3d cluster entirely

Smoke check details

scripts/k3d-knoe-smoke.sh verifies:

1. CNPG cluster knoe-db is in Cluster in healthy state
2. At least 6 knoe.* tables exist in PostgreSQL
3. KDC Deployment is 1/1 Ready
4. admin/admin@KNOE.LOCAL principal exists in the KDC database

---

5. Schema overview

The knoe.* schema lives in the CNPG PostgreSQL cluster (knoe-db-0 namespace, database knoe). Core tables:

Table	Purpose
knoe.user	Canonical user record (id, username, email, created_at)
knoe.identity	Per-provider identity rows (google, kerberos) linked to knoe.user
knoe.invitation	Pending invites; OTP stored as bcrypt hash
knoe.totp_credential	TOTP secrets per user (AES-GCM encrypted at rest; Round 1.5 will add OpenBao transit-key envelope)
knoe.knobject	Provisioned resources (GitLab project, Gitea repo, CNPG role, …)
knoe.access_grant	User↔knobject grants
knoe.provisioning_job	Async job queue polled by ProvisioningWorker

Schema is applied/migrated by etc/init_knoe_auth.sh schema in both GKE and k3d modes.

---

6. Source map

Java application (authority/)

File	Responsibility
KnoeAuthApplication.java	@SpringBootApplication entry point
HealthController.java	GET /health
web/LoginController.java	Form-login + SPNEGO challenge
web/VerifyController.java	Token-verify endpoint for downstream services
session/SessionTokenService.java	HMAC-SHA256 JWT cookies
kerberos/KerberosSpnegoService.java	SPNEGO challenge/response
kerberos/KadminClient.java	Shells out to kadmin.local to addprinc/cpw (input sanitized)
enroll/EnrollmentController.java	Enrollment web flow (OTP → Google → TOTP → complete)
enroll/InviteService.java	Invite CRUD, OTP bcrypt hashing, rate limiting
enroll/UserProvisioningService.java	Transactional orchestrator: inserts rows, calls KadminClient, queues jobs
admin/AdminController.java	POST /auth/admin/invites, GET /auth/admin/users — SPNEGO + admin-role gated
provisioning/ProvisioningWorker.java	@Scheduled poller for knoe.provisioning_job WHERE status='pending'
config/AuthProperties.java	Typed binding for knoe.auth.* config keys

k3d manifests (k8s/knoe/)

File	Purpose
knoe-db.yaml	CNPG Cluster (single-replica, local-path storage)
knoe-kdc-configmap.yaml	KDC krb5.conf + kdc.conf, realm KNOE.LOCAL
knoe-kdc-deployment.yaml	KDC Deployment (debian:bookworm-slim + MIT Kerberos)
knoe-kdc-service.yaml	ClusterIP service for KDC (ports 88, 464)
knoe-kdc-pvc.yaml	PVC for KDC database persistence
knoe-kdc-init-job.yaml	One-shot Job: creates admin/admin + developer principals

---

7. Open work items

• pg_oauth (Phase 2 pg_hba) — Ready to resume. Replaces SCRAM with PG18 native OAUTHBEARER; wires into install.sh / supabase/deploy.sh. OIDC issuer now reachable at https://api.knoe.dev/auth — pg_oauth can proceed.

• Round 1.5 OpenBao transit-key encryption — UserProvisioningService.java:84 has the TODO. TOTP secrets currently have PG-level encryption only.