chrisfu/prole
1
0
Fork 0
mirror of https://github.com/dredx/prole.git synced 2026-09-23 11:03:59 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

docs(plans): add platform architecture plans — deployment-modes, knoe-auth round 1

Browse Source 
Adds docs/plans/ as the canonical engineering reference for completed and
in-flight initiatives. Written for jr/mid engineers who have not seen the
repo before — each plan starts with strategic context and links to existing
code before asking for changes.

  README.md            index, audience, and status conventions
  deployment-modes.md  four-mode installer (min/k3d/k3s/gke), welcome-screen
                       mode selector, min-mode fast-path. Status: shipped.
  knoe-auth-round-1.md Kerberos KNOE.DEV realm, invite-OTP enrollment,
                       Google corroboration, TOTP 2FA. Status: operational.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
chrisfu 2026-04-27 14:07:00 -07:00
parent d3520d4837
commit 636a2795cb
3 changed files with 703 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

45
docs/plans/README.md
View File

@ -1,24 +1,43 @@
# Plans (prole — customer deploy)
# Plans
This directory holds **plans specific to the prole.org customer deployment** of the knoe.dev platform.
Engineering plans for the **knoe.dev** platform. Each document scopes one initiative — its strategic context, design, schema, and the components it adds — and is intended to outlive the implementation work itself: when the work ships, the plan stays as the architectural reference.
## Who this is for
You are a junior or mid-level engineer who has been invited to contribute to knoe.dev. These documents are written for you. They assume:
- You are comfortable reading code and skimming a Maven `pom.xml`.
- You have used Kubernetes at least casually (`kubectl get pods`, helm charts).
- You may **not** have prior Kerberos, OIDC, or CNPG experience — relevant terms are defined inline or in the glossary section of each plan.
- You have not seen this repo before. Each plan starts with the strategic context and links to existing code before it asks you to write any.
If a sentence in a plan assumes knowledge you don't have, that's a bug in the plan — open an issue.
## What lives here
| File | Purpose |
| --- | --- |
| [`customer-deploy-resync.md`](customer-deploy-resync.md) | The active plan to converge this working tree onto `knoe-db/main` as a long-lived customer branch. |
| File | Initiative | When you should read it |
| --- | --- | --- |
| [`knoe-auth-round-1.md`](knoe-auth-round-1.md) | Identity backbone for the platform: MIT Kerberos KDC + invite-anchored web enrollment + TOTP 2FA. Round 1 of N. **Shipped.** | Before touching `authority/`, `etc/init_kdc.sh`, `etc/init_knoe_users.sh`, anything in `deploy/gcp/gke/knoe-auth-*` or `knoe-kdc-*`, or the `knoe.*` database schema. |
| [`deployment-modes.md`](deployment-modes.md) | Four-mode installer (`min` / `k3d` / `k3s` / `gke`) with a welcome-screen mode selector and a `min`-mode fast-path through the wizard. **Shipped in Phase 0.** | Before touching `knoe/ui/screens/welcome.py`, `knoe/ui/screens/navigation.py`, or adding any new wizard screen. |
## What does **not** live here
Each plan follows the same shape: **Context → How it's wired (file references) → Architecture → Schema / API → Step-by-step → Verification → Glossary.**
Platform-level plans live upstream in `~/dev/knoe-db/docs/plans/`:
## Status conventions
- [`knoe-auth-round-1.md`](../../../knoe-db/docs/plans/knoe-auth-round-1.md) — Kerberos identity + invite-OTP enrollment + TOTP. Architectural reference.
- [`deployment-modes.md`](../../../knoe-db/docs/plans/deployment-modes.md) — Four-mode installer + welcome-screen selector. Architectural reference.
A plan in this directory has one of three statuses, declared at the top:
(Paths above are relative to your `~/dev/` root. If you're reading this on the file server, navigate to the `knoe-db` checkout.)
- **Active plan. Not yet implemented.** — the design is agreed, the code isn't there yet. Read deliverables top-to-bottom.
- **Implemented and operational. Architectural reference.** — the work shipped. The doc explains how it works and *why*; cross-references point at real files. Read the context and the rationale; jump to specific sections when you have a question.
- **Archived.** — superseded by a later round or a different approach. Lives in `archive/` with a one-line "see X instead" pointer.
## Why the split
## Customer deploys
`prole` is being converged into a customer-deploy branch in `knoe-db`, not a separate fork. Platform docs travel with the platform code (in `knoe-db`); customer-deploy-specific docs travel with the customer branch (here). When the customer branch is published in `knoe-db`, this directory will move with it.
The knoe.dev platform supports per-customer deployments — currently the most active is `prole.org`. The convention is **branches in this repo**, not separate forks. Customer-specific divergence (config, branding, on-prem manifests, kubeconfig handling) lives on a branch named after the customer; platform changes always land on `main` and customer branches rebase or merge from main on a regular cadence. See the customer's own `docs/plans/` for resync notes — for example, `~/dev/prole/docs/plans/customer-deploy-resync.md` if you have that working tree checked out.
See the upstream `~/dev/knoe-db/docs/plans/README.md` for the full conventions on writing plans, plan statuses, and how customer deploys are structured.
## How to propose a new plan
1. Copy `knoe-auth-round-1.md` as a template — the section structure is the convention.
2. Lead with **why** (one or two paragraphs). Half the value of a plan is forcing the author to articulate the motivation.
3. Cite real files with paths relative to the repo root. If you reference something that doesn't exist yet, label it **(net-new)** so a reader doesn't go hunting.
4. End with a **Verification** section — a short checklist a reviewer can run to decide whether the plan is done, or — once shipped — that the implementation still matches the spec.
5. Open an MR. Plans are reviewed like code.

188
docs/plans/deployment-modes.md Normal file
View File

@ -0,0 +1,188 @@
# Deployment Modes — Welcome Screen + Wizard Fast-Path
**Status:** Implemented and operational. Architectural reference. Welcome-screen mode selector and `min`-mode fast-path shipped in Phase 0.
**Owner:** chrisfu
**Audience:** Jr/mid engineer onboarding to knoe.dev. Familiarity with Tk/tkinter helps but is not required.
---
## 1. Context
The knoe-db installer (`./install.sh` / `./knoe.sh install`, driven by the Tk UI in `knoe/ui/screens/`) supports four ways to deploy the platform. Different audiences, different hardware, but **the same architecture underneath**.
Earlier installer revisions assumed the engineer already knew which mode they wanted, presented a welcome screen that was a wall of text, and shipped a sidebar tab strip (`k3d` / `k3s` / `k8s`) that didn't reliably refresh state across screens. Phase 0 replaced all of that with a first-screen mode selector that teaches the engineer what each mode *is* before asking them to choose, and added a `min`-mode fast-path that skips wizard screens irrelevant to a single-container deployment.
### The four modes
| Mode | What it is | Who it's for | Required tools |
| --- | --- | --- | --- |
| **min** | A single `knoe-db` container under `containerd` — no Kubernetes, no Docker. The Spring Boot app runs locally on the host. | Engineers running a Spring app on a laptop and wanting a CNPG-compatible Postgres without a full cluster. | Homebrew, 1Password |
| **k3d** | A Docker-based local k3s cluster. Full CNPG + optional Supabase / ArgoCD / Gitea / GitLab. | Local development with the full platform. | Docker, Homebrew, 1Password |
| **k3s** | A multi-node k3s cluster on real hardware. Full Kerberos auth stack, Garage S3, monitoring. Mirrors the GKE shape. | Homelab contributors with their own hardware (e.g. a 3-node Pi cluster). | k3sup, Homebrew, 1Password |
| **gke** | The production dual-cluster on Google Cloud (`knoe-dev-0` + `knoe-cnpg-0` in `us-west3`). | Production. | gcloud, kubectl, 1Password |
The architectural commitment matters: **all four modes are scaled-down mirrors of the GKE shape.** Not parallel implementations of the same idea, not divergent forks — the same components composed at different scales. CNPG runs in all four. The Spring Boot `authority` service runs in all four. `knoe.user` lives in the same schema in all four. The only thing that changes is what platform the components run on and which optional services are enabled.
### Mode → cluster_env mapping
| `cluster_env` | `KNOE_MODE` | Target |
| --- | --- | --- |
| `dev` | `k3d` | Local K3d cluster |
| `service` | `k3s` | On-prem K3s cluster |
| `prod` | `k8s` (alias `gke`) | GKE (or other cloud) |
| `min` | `min` | Local containerd (no Kubernetes) |
The conversion lives in `knoe/core/env.py` (`_deployment_mode_from_env()`).
### Config file mapping
`knoe/knoe_conf.py` maps environments to config files under `conf/`:
| Mode | Config file |
| --- | --- |
| `min` | `conf/min.cfg` |
| `k3d` (`dev`) | `conf/k3d.cfg` |
| `k3s` (`service`) | `conf/k3s.cfg` |
| `gke` (`prod`) | `conf/gke.cfg` |
Config is layered: env-specific file overrides base. `KNOE_CONF` env var or `conf/service/` subdirs point to the active config.
---
## 2. How it's wired — file map
| Component | Location | What's there |
| --- | --- | --- |
| Welcome screen with mode selector | `knoe/ui/screens/welcome.py` | `WelcomeScreenMixin._render_welcome_page()` renders the four mode cards. Click handler updates `deployment_mode` and `cluster_env`, redraws card borders, re-evaluates the Next gate. |
| Sidebar nav | `knoe/ui/screens/navigation.py` | `_create_sidebar_nav()` no longer renders the old `k3d/k3s/k8s` tab strip — the welcome card selector replaces it. `_set_deployment_mode` is still called from the welcome handler. |
| Mode/state initialization | `knoe/ui/screens/__init__.py` | `nav_items` list, `deployment_mode = tk.StringVar(...)`, `cluster_env = tk.StringVar(...)`. |
| Wizard transitions | `knoe/ui/screens/navigation.py` `on_next` / `on_prev` | Min-mode fast-path skips screens that don't apply (network scan, cluster init, Kerberos, ArgoCD, Supabase, common services). |
| Update-footer Next-button gate | `knoe/ui/screens/navigation.py` `update_footer` | Welcome screen disables Next until a mode is selected. |
| Min-mode bootstrap | `etc/init_min.sh` | Idempotent: ensures Homebrew + containerd, pulls and runs the `knoe-db` container locally, runs schema bootstrap, prints connection info. Invoked from `init_scripts` step in min mode. |
| Min-mode config | `conf/min.cfg` | Modeled on `conf/k3d.cfg`. Just what `init_min.sh` needs. |
| Min-mode lifecycle | `knoe.sh` (`start` / `stop` / `restart` subcommands) | Manages the local containerd `knoe-db` instance for users who don't want to run the wizard every time. |
---
## 3. UI layout — welcome screen
```
┌──────────────────────────────────────────────────────────────────┐
│ │
│ knoe.dev │
│ infrastructure.auto() │
│ │
│ Welcome │
│ ─────── │
│ Knoe.DB sets up a production-grade PostgreSQL cluster with │
│ Kerberos authentication. Pick the mode that matches your │
│ hardware — we'll walk you through every step. │
│ │
│ ┌────────────┬────────────┬────────────┬────────────┐ │
│ │ min │ k3d │ k3s │ gke │ │
│ │ Just a │ Local │ Homelab │ Production│ │
│ │ Database │ Cluster │ │ │ │
│ │ One knoe-db│ k3s-in- │ Multi-node │ Dual GKE │ │
│ │ container │ Docker │ k3s on real│ clusters on│ │
│ │ via │ cluster on │ hardware. │ Google │ │
│ │ containerd.│ your Mac. │ Full │ Cloud. │ │
│ │ No K8s. │ Full CNPG │ Kerberos │ CNPG + │ │
│ │ Perfect for│ database — │ auth stack,│ GCS backups│ │
│ │ a Spring │ add │ Garage S3, │ + Workload │ │
│ │ app on your│ Supabase, │ monitoring.│ Identity. │ │
│ │ laptop. │ ArgoCD, │ │ │ │
│ │ │ Gitea, or │ │ │ │
│ │ │ GitLab. │ │ │ │
│ │ Homebrew + │ Docker + │ k3sup + │ gcloud + │ │
│ │ 1Password │ Homebrew + │ Homebrew + │ 1Password │ │
│ │ │ 1Password │ 1Password │ │ │
│ └────────────┴────────────┴────────────┴────────────┘ │
│ │
│ You'll need: Docker · Homebrew · 1Password │
│ │
│ [ Next → ] │
└──────────────────────────────────────────────────────────────────┘
```
Behavior:
- The four cards are clickable.
- The selected card gets a colored border in the mode's accent color (`_MODE_COLORS`: `min` purple, `k3d` blue, `k3s` green, `gke` orange).
- The "You'll need:" line below the cards updates to reflect the selected mode's tools.
- The Next button is **disabled** until a mode is selected.
- Selecting a card also sets `cluster_env` via the mode-to-cluster_env map.
- On Next, mode is persisted to `knoe_cfg_data["Global"]["DEPLOYMENT_MODE"]`.
---
## 4. Wizard transitions — `min`-mode fast-path
### Default flow (k3d / k3s / gke)
```
welcome → network_scan → env_setup → init_cluster → cluster_nodes →
init_password → init_scripts → kerberos_config → knoe_users →
argocd_config → gitops_config → supabase_config → common_services →
… → security
```
### Min-mode flow
```
welcome → dependencies → environment → init_password → init_scripts → security
```
Skipped in min: `network_scan`, `init_cluster`, `cluster_nodes`, `kerberos_config`, `knoe_users`, `argocd_config`, `gitops_config`, `supabase_config`, `common_services`. None of them apply when there's no Kubernetes cluster and no Kerberos KDC. `init_scripts` invokes `etc/init_min.sh` instead of the cluster-mode init scripts.
The fast-path is implemented as a `_min_mode_next` dictionary in `knoe/ui/screens/navigation.py` consulted from `on_next` / `on_prev`. The override is only active when `_is_min_mode()` returns true.
---
## 5. Verification
Run after any change touching the wizard or mode logic.
1. `./install.sh` opens. The welcome screen shows four mode cards. The sidebar has **no** k3d/k3s/k8s tab strip.
2. With no mode selected, the Next button is disabled.
3. Click the **min** card. Next becomes enabled. Click Next.
- The wizard skips the network-scan screen.
- It skips `init_cluster` and `cluster_nodes`.
- It skips `kerberos_config`, `knoe_users`, `argocd_config`, `gitops_config`, `supabase_config`, `common_services`.
- It runs `init_min.sh` at the `init_scripts` step.
- It lands on the `security` screen.
- Going back through `on_prev` walks the same trimmed sequence in reverse without hitting any skipped screens.
4. Click the **k3d** card on a fresh run. The wizard follows the existing k3d flow exactly as before.
5. `python3 -c "from knoe.ui.screens import KnoeInstaller"` — no import errors.
6. After completing min mode, `nerdctl ps` shows the `knoe-db` container running locally and `psql "${printed_conn_string}" -c '\dt knoe.*'` lists the `knoe.user` table.
7. `./knoe.sh stop` and `./knoe.sh start` cleanly stop/restart the local containerd instance.
---
## 6. Out of scope
- A `min`-mode equivalent of the full Kerberos auth stack. Min is "just a database". Auth in min is a follow-up.
- Migrating an existing `k3d` deployment to `min` (or any other mode-to-mode migration). Each mode is an independent target.
- A unified package format that produces all four mode-specific installers from one `pyinstaller` build. The current per-mode build is fine.
- Letting one wizard run install *multiple* modes (e.g. set up `k3d` and `gke` from the same session). One mode per run.
---
## 7. Glossary
**k3s** — A lightweight Kubernetes distribution by Rancher Labs. Single binary, runs comfortably on a Raspberry Pi. The same project, two delivery shapes: `k3d` is k3s wrapped in Docker; bare k3s is k3s on actual hardware.
**k3d** — A wrapper that runs k3s clusters inside Docker containers. Useful for local dev because it gives you a "real" Kubernetes cluster without having to provision VMs.
**GKE (Google Kubernetes Engine)** — Google Cloud's managed Kubernetes. We use Standard for `knoe-dev-0` and `knoe-cnpg-0`. See `CLAUDE.md` for cluster topology.
**containerd** — A container runtime. Speaks the OCI image and runtime specs. Both Docker and Kubernetes use containerd under the hood. In `min` mode we talk to it directly via `nerdctl`, skipping Docker entirely.
**nerdctl** — A Docker-CLI-compatible client for `containerd`. If you know `docker run`, you know `nerdctl run`.
**CNPG (CloudNativePG)** — The Postgres operator we use for `knoe-db`. See `CLAUDE.md` for which cluster runs it in production.
**`cluster_env`** — A label the installer carries internally that picks which `conf/*.cfg` file to read and which downstream config branches to take. Maps from `deployment_mode` per the table in §1. Don't reuse this name for new variables — it's already overloaded enough.
**Tk / tkinter** — Python's standard GUI toolkit. The installer is built on it. Canvas-based rendering means we draw text and shapes directly rather than using Tk's widget hierarchy for the welcome screen.
**Wizard / pid** — The installer is a multi-screen wizard. Each screen has a "page id" (`pid`) — `welcome`, `network_scan`, `init_cluster`, etc. `nav_items` is the ordered list; `on_next` / `on_prev` decide transitions based on the current `pid` and the user's state.

483
docs/plans/knoe-auth-round-1.md Normal file
View File

@ -0,0 +1,483 @@
# knoe-auth Round 1 — Durable Kerberos Identity + Contributor Onboarding
**Status:** Implemented and operational. Architectural reference.
**Owner:** chrisfu
**Audience:** Jr/mid engineer onboarding to knoe.dev. No prior Kerberos or OIDC experience assumed.
---
## 1. Context
`knoe-auth` is the identity system for the knoe.dev platform — the thing that decides who you are, what you can access, and how new contributors come on board. It is a long-term, multi-round initiative. This document is the architectural reference for **Round 1**, which is shipped.
### The chicken-and-egg problem
To build a sophisticated identity provider safely, the team needs to be able to authenticate themselves against *something* trustworthy in the meantime. We needed a working auth store *before* we could safely build the better one.
Round 1 leans on a tool that has been doing this job for thirty-five years: **MIT Kerberos**. It is unfashionable but well-understood, cryptographically sound, and we already had it running in our k3s homelab cluster. We extended it to GKE, wrapped a small enrollment portal around it, and use that to onboard contributing engineers while Round 2 (a full OIDC provider) is being built.
### Why Kerberos
Kerberos *principals* are durable, DNS-like identifiers. `chrisfu@KNOE.DEV` is a stable cryptographic identity that survives any change to our web stack, our database, or our cloud provider. Once a principal exists in the KDC, it can issue tickets that any kerberized service trusts — Spring Boot via SPNEGO, Postgres via `gss` auth, SSH, NFS — without each of those services needing its own user table.
That's the asset Round 1 builds on.
### Two independent Google Workspaces — do not conflate
| Workspace | Role |
| --- | --- |
| `knoe.dev` | Internal Google Workspace for the platform. Has zero pre-knowledge of any contributor's home org. |
| `prole.org` | Workspace of the first contributing engineer's organization. Independently operated. |
`knoe.dev` does **not** trust `prole.org` as a domain. `prole.org` is just one engineer's email provider, no different from `gmail.com` or any other workspace a future contributor might use. Trust between knoe.dev and a new contributor is bootstrapped by the **invite**, not by the contributor's home Google domain.
### Trust model — the most important paragraph in this document
When a new engineer enrolls, the trust sequence is:
1. **An admin sends an invite to a specific email address or phone number.** That contact channel — and only that channel — is the trust anchor. The admin's choice of who to invite **is** the policy.
2. **The engineer proves they control that contact** by entering a one-time password (OTP) delivered to it. Until the OTP verifies, no further steps are possible.
3. **Only after the OTP gate** is the engineer offered a Google sign-in to *corroborate* their identity. The Google sign-in is welcomed in *after* trust is already established by the invite — it is not the source of trust.
4. **TOTP** (the rotating six-digit code from Google Authenticator / Authy) is set up as the ongoing 2FA credential.
5. **Only then** is a Kerberos principal minted, a `knoe.user` row inserted, and downstream provisioning jobs (GitLab account, Gitea account) queued.
This means knoe.dev never needs to pre-configure trust with any external workspace. The OAuth2 app does **not** restrict by Google `hd` (hosted domain) — any verified Google account works. The contributor's home domain is *recorded* for audit (`knoe.identity.provider_hd`) but never used to gate access.
If you remember nothing else: **the invite OTP is the trust anchor. Google is corroboration. TOTP is the ongoing factor.**
---
## 2. How it's wired — file map
The actual files that implement Round 1. Verify with `git ls-files` before assuming any of the below has rotted.
### Java application — `authority/`
The Spring Boot service that implements the enrollment flow, the admin API, and SPNEGO-protected endpoints. Multi-module Maven build (`authority/pom.xml`); the application package is `org.prole.authority` (kept as-is across the prole→knoe rebrand for compatibility).
| File | Responsibility |
| --- | --- |
| `authority/src/main/java/org/prole/authority/KnoeAuthApplication.java` | `@SpringBootApplication` entry point. |
| `authority/src/main/java/org/prole/authority/HealthController.java` | `/health` endpoint. |
| `authority/src/main/java/org/prole/authority/web/LoginController.java` | Form-login + SPNEGO challenge for browsers without a ticket. |
| `authority/src/main/java/org/prole/authority/web/VerifyController.java` | Token-verify endpoint for downstream services. |
| `authority/src/main/java/org/prole/authority/session/SessionTokenService.java` | Issues HMAC-SHA256 JWT cookies after successful auth. |
| `authority/src/main/java/org/prole/authority/session/SessionUser.java` | Authenticated principal carried in the security context. |
| `authority/src/main/java/org/prole/authority/user/PrincipalNormalizer.java` | Strips realm/instance from a Kerberos principal (`alice/admin@KNOE.DEV` → `alice`). |
| `authority/src/main/java/org/prole/authority/kerberos/KerberosSpnegoService.java` | SPNEGO challenge/response handling. |
| `authority/src/main/java/org/prole/authority/kerberos/KerberosPasswordService.java` | Password-style auth fallback for browsers that can't do SPNEGO. |
| `authority/src/main/java/org/prole/authority/kerberos/KadminClient.java` | Shells out to `kadmin.local` (in the KDC sidecar) to `addprinc` and `cpw`. **Sanitizes input.** |
| `authority/src/main/java/org/prole/authority/enroll/EnrollmentController.java` | Web endpoints: `GET /auth/enroll`, `POST /auth/enroll/verify-otp`, `POST /auth/enroll/identity/start`, `GET /auth/enroll/google-callback`, `GET /auth/enroll/totp`, `POST /auth/enroll/totp/verify`, `POST /auth/enroll/complete`. |
| `authority/src/main/java/org/prole/authority/enroll/InviteService.java` | CRUD + validation against `knoe.invitation`. OTP hashing (bcrypt) and rate limiting (3 attempts). |
| `authority/src/main/java/org/prole/authority/enroll/GoogleOAuthService.java` | Exchange OAuth2 code → ID token, validate `email_verified`, return a `GoogleIdentity` record. **No `hd` allowlist.** |
| `authority/src/main/java/org/prole/authority/enroll/TotpService.java` | Generate TOTP secret, produce `otpauth://` URI, verify codes. |
| `authority/src/main/java/org/prole/authority/enroll/UserProvisioningService.java` | Transactional orchestrator: inserts user/identity/totp rows, calls `KadminClient`, queues provisioning jobs. |
| `authority/src/main/java/org/prole/authority/admin/AdminController.java` | `POST /auth/admin/invites`, `GET /auth/admin/users`, `POST /auth/admin/grants`. SPNEGO + admin-role gated. |
| `authority/src/main/java/org/prole/authority/admin/KnobjectService.java` | CRUD on `knoe.knobject` and `knoe.access_grant`; enqueues `provisioning_job` rows. |
| `authority/src/main/java/org/prole/authority/provisioning/ProvisioningWorker.java` | `@Scheduled` poller for `knoe.provisioning_job WHERE status = 'pending'`. Dispatches to GitLab/Gitea/CNPG. |
| `authority/src/main/java/org/prole/authority/config/AuthProperties.java` | Typed binding for `knoe.auth.*` keys. |
| `authority/src/main/java/org/prole/authority/config/KerberosProperties.java` | Typed binding for `knoe.auth.kerberos.*` keys. |
Tests for the above live under `authority/src/test/java/org/prole/authority/` — notably `web/VerifyControllerTest.java` and `session/SessionTokenServiceTest.java`.
### Kubernetes manifests — GKE
| File | Purpose |
| --- | --- |
| `deploy/gcp/gke/knoe-kdc-configmap.yaml` | `krb5.conf` + `kdc.conf` for the `KNOE.DEV` realm. |
| `deploy/gcp/gke/knoe-kdc-secrets.yaml` | Master key + admin password references. Production values come from OpenBao. |
| `deploy/gcp/gke/knoe-auth-deployment.yaml` | KDC sidecar + Spring Boot pod. Service principal `HTTP/auth.knoe.dev@KNOE.DEV`. |
| `deploy/gcp/gke/knoe-auth-google-oidc-secret.example.yaml` | Templated Secret with `GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET` placeholders. Real values are not committed. |
| `deploy/gcp/gke/workload-identity.yaml` | KSA↔GSA bindings for any GCP-backed secret access. |
### Kubernetes manifests — k3s (homelab and customer deploys)
| File | Purpose |
| --- | --- |
| `deploy/opentofu/k3s/manifests/prole/prole-kdc-configmap.yaml` | The `PROLE.LOCAL` realm KDC for the homelab cluster. The pattern the GKE configmap was modeled on. |
| `deploy/opentofu/k3s/manifests/prole/prole-kdc-secrets.example.yaml` | Templated secrets for the same. |
| `deploy/opentofu/k3s/manifests/prole/prole-auth-deployment.yaml` | KDC + Spring Boot for the homelab. |
| `deploy/opentofu/k3s/manifests/prole/prole-auth-kerberos-configmap.yaml` | `krb5.conf` for the auth pod's Kerberos client. |
### Bootstrap scripts — `etc/`
| File | Purpose |
| --- | --- |
| `etc/init_kdc.sh` | Provisions the in-cluster KDC. Idempotent. Read this end-to-end before writing anything that interacts with the KDC. |
| `etc/init_knoe_users.sh` | Creates `knoe.user`, `knoe.user_role`, and (per Round 1) the additional auth tables; seeds initial principals via `kadmin.local`. |
| `etc/init_kerberos.sh` | Cluster-wide krb5.conf wiring for kerberized services (Postgres, etc.). |
---
## 3. Architecture
### The invite-to-enrolled flow at a glance
```
Invite URL
https://auth.knoe.dev/enroll?token=<uuid>
│
├─ Step 1: Enter OTP (delivered to invite email/phone)
│ ─ Trust anchor. Without this, no further steps.
│
├─ Step 2: Pick a username, link with Google (any account, any hd)
│ ─ Corroboration. Records provider_sub + provider_hd for audit.
│
├─ Step 3: Scan QR with authenticator app, verify TOTP code
│ ─ Sets up the ongoing 2FA factor.
│
└─ Step 4: System provisions:
├─ Kerberos principal: <username>@KNOE.DEV
├─ knoe.user row + knoe.identity link to Google subject
├─ knoe.totp_credential row (encrypted secret)
└─ Async queue: GitLab account, Gitea account, …
```
### Cluster topology
Round 1 lives in the GKE app cluster (`knoe-dev-0`) in the `knoe-system` namespace. The database stays where it already is — the dedicated CNPG cluster `knoe-cnpg-0`. See `CLAUDE.md` for the cluster layout and storage-quota rules.
```
knoe-dev-0 / knoe-system namespace:
┌─ knoe-auth (Spring Boot) ─────────────────────────────────────────┐
│ /health │
│ /auth/login, /auth/spnego, /auth/verify │
│ /auth/enroll/* (invite → OTP → Google → TOTP → provision) │
│ /auth/admin/* (create invites, manage knobjects) │
└────────────────────────────────────────────────────────────────────┘
│ kadmin.local calls (KDC is a sidecar in the same pod)
▼
┌─ knoe-kdc (MIT Kerberos, KNOE.DEV realm) ─────────────────────────┐
│ Container pattern from prole-kdc-configmap.yaml │
│ Realm: KNOE.DEV │
│ Cross-realm trust with PROLE.LOCAL: deferred to Round 2 │
└────────────────────────────────────────────────────────────────────┘
│
▼
┌─ CNPG / knoe-db (in knoe-cnpg-0) ─────────────────────────────────┐
│ knoe.user (base table) │
│ knoe.user_role (base table) │
│ knoe.invitation (Round 1) │
│ knoe.identity (Round 1 — Google sub → knoe user) │
│ knoe.totp_credential (Round 1 — encrypted TOTP secret + backups) │
│ knoe.knobject (Round 1 — platform resources) │
│ knoe.access_grant (Round 1 — user → knobject grants) │
│ knoe.provisioning_job (Round 1 — async outbox) │
└────────────────────────────────────────────────────────────────────┘
```
The KDC runs as a sidecar in the same pod as the Spring Boot app. They share the pod network namespace, so `kadmin.local` calls reach the KDC over loopback — no Kubernetes Service required between them. This is the same pattern the k3s deployment uses.
---
## 4. Schema
The `knoe.*` schema lives in CNPG (`knoe-cnpg-0` namespace `knoe-db-0`). Base tables (`knoe.user`, `knoe.user_role`) are created by `etc/init_knoe_users.sh` lines 482–510. Round 1 added the six tables below; they are created by the same script later in its run.
```sql
-- ────────────────────────────────────────────────────────────────────
-- knoe.invitation — admin creates one of these per invited engineer.
-- The (contact, otp_hash) pair IS the trust anchor for that engineer.
-- ────────────────────────────────────────────────────────────────────
CREATE TABLE IF NOT EXISTS knoe.invitation (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
token TEXT NOT NULL UNIQUE, -- URL token (long, random)
contact TEXT NOT NULL, -- email or phone the invite was sent to
contact_type TEXT NOT NULL DEFAULT 'email',-- 'email' | 'sms'
name_hint TEXT, -- optional display-name hint from admin
otp_hash TEXT NOT NULL, -- bcrypt of the 6-digit OTP
otp_expires_at TIMESTAMPTZ NOT NULL, -- short TTL (10 min)
otp_attempts INT NOT NULL DEFAULT 0, -- max 3 before invalidation
otp_verified_at TIMESTAMPTZ, -- set when OTP passes — gate for steps 2-4
created_by TEXT NOT NULL, -- admin username
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
expires_at TIMESTAMPTZ NOT NULL, -- invite URL TTL (72h)
used_at TIMESTAMPTZ, -- set at step-4 completion
used_by TEXT -- knoe username after use
);
-- ────────────────────────────────────────────────────────────────────
-- knoe.identity — external identity corroborations.
-- Round 1 only writes Google rows here; future providers reuse the table.
-- ────────────────────────────────────────────────────────────────────
CREATE TABLE IF NOT EXISTS knoe.identity (
id SERIAL PRIMARY KEY,
user_id INT NOT NULL REFERENCES knoe.user(id) ON DELETE CASCADE,
provider TEXT NOT NULL, -- 'google'
provider_sub TEXT NOT NULL, -- Google subject ID (stable per user)
provider_email TEXT,
provider_hd TEXT, -- 'prole.org' | 'gmail.com' | NULL — audit only
verified_at TIMESTAMPTZ NOT NULL,
UNIQUE(provider, provider_sub)
);
-- ────────────────────────────────────────────────────────────────────
-- knoe.totp_credential — the rotating 2FA factor for ongoing logins.
-- ────────────────────────────────────────────────────────────────────
CREATE TABLE IF NOT EXISTS knoe.totp_credential (
user_id INT PRIMARY KEY REFERENCES knoe.user(id) ON DELETE CASCADE,
secret TEXT NOT NULL, -- AES-GCM encrypted, key in OpenBao
verified_at TIMESTAMPTZ, -- NULL until first successful verification
backup_codes TEXT[], -- bcrypt-hashed one-time recovery codes
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- ────────────────────────────────────────────────────────────────────
-- knoe.knobject — platform-managed resources (a "knobbed object",
-- something an admin can hand to a user).
-- ────────────────────────────────────────────────────────────────────
CREATE TABLE IF NOT EXISTS knoe.knobject (
id SERIAL PRIMARY KEY,
type TEXT NOT NULL, -- 'gitea_repo' | 'gitlab_project' | 'cnpg_role' | 'openbao_policy'
name TEXT NOT NULL,
platform_id TEXT, -- external identifier on target platform
metadata JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE(type, name)
);
-- ────────────────────────────────────────────────────────────────────
-- knoe.access_grant — user ← knobject with role.
-- ────────────────────────────────────────────────────────────────────
CREATE TABLE IF NOT EXISTS knoe.access_grant (
id SERIAL PRIMARY KEY,
user_id INT NOT NULL REFERENCES knoe.user(id),
knobject_id INT NOT NULL REFERENCES knoe.knobject(id),
role TEXT NOT NULL, -- 'owner' | 'developer' | 'viewer'
granted_by TEXT NOT NULL,
granted_at TIMESTAMPTZ NOT NULL DEFAULT now(),
revoked_at TIMESTAMPTZ,
UNIQUE(user_id, knobject_id)
);
-- ────────────────────────────────────────────────────────────────────
-- knoe.provisioning_job — async outbox.
-- A worker bean inside knoe-auth polls this table.
-- ────────────────────────────────────────────────────────────────────
CREATE TABLE IF NOT EXISTS knoe.provisioning_job (
id SERIAL PRIMARY KEY,
user_id INT NOT NULL REFERENCES knoe.user(id),
job_type TEXT NOT NULL, -- 'create_gitlab_user' | 'create_gitea_user' | 'grant_cnpg_role'
status TEXT NOT NULL DEFAULT 'pending', -- 'pending' | 'running' | 'done' | 'failed'
payload JSONB,
result JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
```
---
## 5. Configuration
### `authority/.../application.properties` — relevant keys
```properties
knoe.auth.google.client-id=${GOOGLE_CLIENT_ID}
knoe.auth.google.client-secret=${GOOGLE_CLIENT_SECRET}
knoe.auth.google.redirect-uri=https://auth.knoe.dev/auth/enroll/google-callback
# No allowed-domains list. Trust is established by invite OTP, not by the
# developer's home Google domain. provider_hd is recorded in knoe.identity
# for audit, never used for access control.
knoe.auth.enroll.invite-ttl-hours=72
knoe.auth.enroll.otp-ttl-minutes=10
knoe.auth.enroll.otp-max-attempts=3
knoe.auth.enroll.totp-issuer=knoe.dev
knoe.auth.provisioning.poll-interval-ms=10000
```
### `pom.xml` — relevant dependencies
```xml
<!-- TOTP -->
<dependency>
<groupId>dev.samstevens.totp</groupId>
<artifactId>totp-spring-boot-starter</artifactId>
<version>1.7.1</version>
</dependency>
<!-- Google OAuth2 client -->
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
<version>2.4.0</version>
</dependency>
```
### Google OAuth2 app
Configured by hand in the GCP console under project `plenary-truck-485623-p7`.
- **Authorized redirect URIs:**
- `https://auth.knoe.dev/auth/enroll/google-callback` (Round 1 enrollment)
- `https://git.knoe.dev/...` (Gitea OIDC, future round)
- `https://git.prole.org/...` (GitLab OIDC, future round)
- **No `hd=` restriction** at the OAuth app level. Deliberate.
- Client ID and secret stored in a Kubernetes Secret modeled on `deploy/gcp/gke/knoe-auth-google-oidc-secret.example.yaml`.
---
## 6. Step-by-step enrollment flow
This is what the engineer being onboarded actually experiences, what the server actually does, and where the trust transitions live.
### Step 0 — Admin creates the invite
```
POST /auth/admin/invites
Body: { contact: "chrisfu@prole.org", contact_type: "email", name_hint: "Chris Fu" }
Server:
generate invite_token (UUID v4)
generate otp (6-digit numeric, 10-min TTL)
hash otp with bcrypt → otp_hash
insert into knoe.invitation
send email to chrisfu@prole.org:
Subject: "You've been invited to knoe.dev"
Body: invite URL + "Your verification code: 847291"
```
The OTP is delivered through the **same channel** as the invite URL. Both arrive in the engineer's inbox. The OTP is *not* sent to a side channel — its purpose is to prove possession of that inbox.
### Step 1 — Engineer proves contact ownership (the trust gate)
```
GET /auth/enroll?token=<uuid>
→ InviteService validates token is not expired/used
→ renders landing.html with the OTP entry form
POST /auth/enroll/verify-otp { otp: "847291" }
→ InviteService compares bcrypt(otp) with otp_hash, checks expiry
→ on success: set otp_verified_at=now(), advance session to step 2
→ on failure: increment otp_attempts; if >= 3, invalidate the invite and return 403
```
After this step, the session is gated. The remaining steps are unreachable without a verified OTP. **This is where the trust transition happens.**
### Step 2 — Engineer picks a username and links Google
```
GET /auth/enroll/identity
→ renders identity.html — username field + display-name field +
"Sign in with Google" button
POST /auth/enroll/identity/start { username: "chrisfu", display_name: "Chris Fu" }
→ store proposed username + display name in session
→ redirect to Google OAuth2 authorize URL
(state=<session_id>, nonce=<random>, NO hd parameter)
GET /auth/enroll/google-callback?code=<code>&state=<session_id>
→ GoogleOAuthService exchanges code for ID token
→ validates id_token.email_verified == true (REQUIRED)
→ records provider_sub, provider_email, provider_hd
(provider_hd is whatever Google reports — prole.org, gmail.com, etc.)
→ stores GoogleIdentity in session, advances to step 3
```
This step is **corroboration**, not authorization. The engineer's home Google workspace is not a trust source. We accept any verified Google account and record which domain it came from for audit purposes.
### Step 3 — Engineer sets up TOTP
```
GET /auth/enroll/totp
→ TotpService.generateSecret() — 160-bit base32 secret
→ store the encrypted secret in the session (NOT yet in the DB)
→ render totp-setup.html with:
• a QR code encoding otpauth://totp/knoe.dev:<username>?secret=...&issuer=knoe.dev
• the 16-character manual key for users with no QR scanner
• "Open Google Authenticator / Authy and scan this code"
POST /auth/enroll/totp/verify { code: "123456" }
→ TotpService.verify(sessionSecret, code) — validates within ±1 30-second window
→ on success: advance to step 4
→ on failure: re-render with the same secret (don't rotate yet)
```
### Step 4 — System provisions
```
POST /auth/enroll/complete
→ UserProvisioningService.provision(session) runs in a single transaction:
1. INSERT INTO knoe.user (username, realm='KNOE.DEV', email, display_name)
2. INSERT INTO knoe.identity (provider='google', sub, email, hd)
3. INSERT INTO knoe.totp_credential (AES-encrypted secret, verified_at=now())
4. KadminClient.addPrincipal("<username>@KNOE.DEV")
5. UPDATE knoe.invitation SET used_at=now(), used_by=<username>
6. INSERT INTO knoe.provisioning_job (job_type='create_gitea_user', payload={...})
7. INSERT INTO knoe.provisioning_job (job_type='create_gitlab_user', payload={...})
→ render complete.html with the engineer's new username and a "what happens next"
summary (their dev environment is being set up async, they'll get a follow-up email).
```
### Properties of this flow you can rely on
- The OTP delivery channel is the identity proof. If the OTP arrives, the engineer owns that inbox.
- knoe.dev never trusted `prole.org`. It trusted the admin's choice to send the invite to a `prole.org` address. Different thing.
- The Google link captures the engineer's home workspace as audit data, but does not gate access.
- TOTP becomes the ongoing 2FA factor. The Google sign-in is a one-time corroboration; logins after enrollment use Kerberos + TOTP.
- `knoe.identity.provider_hd` records the home domain without pre-judging it.
---
## 7. Verification
Run this end-to-end after any change touching the auth/Kerberos surface.
1. `kubectl -n knoe-system get pods` — `knoe-auth` and `knoe-kdc` both Running.
2. `curl https://auth.knoe.dev/health` returns 200.
3. Hit `POST /auth/admin/invites` from an admin SPNEGO session, receive an invite URL.
4. Open the enrollment URL in a fresh browser, complete all four steps using a real Google account in a non-`knoe.dev` workspace (e.g. `gmail.com`).
5. `psql … -c "SELECT username, realm FROM knoe.user WHERE username = '<test_user>';"` returns the row.
6. `kinit <test_user>@KNOE.DEV` from a machine that trusts the realm — succeeds.
7. `psql … -c "SELECT job_type, status FROM knoe.provisioning_job WHERE user_id = (SELECT id FROM knoe.user WHERE username = '<test_user>');"` shows `create_gitea_user` and `create_gitlab_user` rows.
8. After the polling interval, those rows transition to `status = 'done'` and the corresponding accounts exist on the platforms.
---
## 8. Out of scope for Round 1
Real, named items the team has discussed. They are **not** in Round 1 — when the team gets to them, each becomes its own plan in this directory.
- Cross-realm trust between `KNOE.DEV` and `PROLE.LOCAL` (so a `chrisfu@PROLE.LOCAL` ticket can talk to a `KNOE.DEV` service). Round 2.
- A full OIDC provider hosted by knoe-auth, replacing the dependence on Google for downstream services. Round 2.
- SSO into kerberized Postgres roles (`gss` auth) so `knoe.user` rows map directly to database principals.
- A "knobject inspector" admin UI. Right now the admin API is JSON-only.
- Phone/SMS-based OTP delivery. Round 1 covers email; the `contact_type` column is already present on `knoe.invitation` so adding SMS later is additive.
- Self-service password rotation, recovery flows, and deactivation. Admin-only for Round 1.
---
## 9. Glossary
**KDC** — Key Distribution Center. The Kerberos server. Holds the master key for the realm; issues TGTs (ticket-granting tickets) and service tickets.
**Kerberos principal** — A named identity in a realm. Format: `name@REALM` (or `service/host@REALM`). Example: `chrisfu@KNOE.DEV`. Long-lived, cryptographic, decoupled from any particular service's user table.
**Realm** — A Kerberos administrative domain. Independent KDCs each own their own realm. `PROLE.LOCAL` and `KNOE.DEV` are two realms; cross-realm trust is configured separately.
**Keytab** — A file containing one or more principals' long-term keys, used by services to authenticate to the KDC without an interactive password. Spring Boot reads its service principal's keytab at startup.
**SPNEGO** — Simple and Protected GSSAPI Negotiation Mechanism. The HTTP-layer protocol that lets a browser holding a Kerberos ticket authenticate to a web app over the wire. RFC 4178. Spring Security has built-in support.
**kadmin / kadmin.local** — The KDC's administration tool. `kadmin` runs over the network with admin credentials; `kadmin.local` runs on the KDC host itself, bypassing the network protocol. Round 1 calls `kadmin.local` from a sidecar container.
**TGT (Ticket-Granting Ticket)** — The first ticket the KDC issues to a user after they prove their identity. Used to request further service tickets without re-entering credentials.
**TOTP** — Time-based One-Time Password. RFC 6238. The rotating six-digit code Google Authenticator and Authy show. A shared secret + the current Unix time bucketed into 30-second windows produces the code.
**OAuth2** — Authorization framework. Lets a user grant an app limited access to their account at another service. Concerns *delegation*, not *identity*.
**OIDC (OpenID Connect)** — An identity layer on top of OAuth2. Adds the `id_token` (a signed JWT with claims about the user). When we say "Google sign-in" we mean OIDC over Google's OAuth2.
**`hd` claim** — In a Google OIDC `id_token`, the user's hosted-domain (i.e. their Google Workspace). Optional, present only for Workspace accounts. We *record* it in `knoe.identity.provider_hd` for audit but do **not** gate access on it.
**Workload Identity** — GKE feature that binds a Kubernetes ServiceAccount to a Google Cloud IAM service account. Lets pods talk to GCP APIs (e.g. KMS for our master key) without long-lived JSON keyfiles.
**CNPG** — CloudNativePG. The Postgres operator we use for `knoe-db`. Runs in `knoe-cnpg-0`. See `CLAUDE.md` for cluster topology.
**knobject** — A platform-managed resource that an admin can hand out to a user — a Gitea repo, a Postgres role, an OpenBao policy. The name is a portmanteau of "knoe" + "object". Modeled by `knoe.knobject`; granted via `knoe.access_grant`.
**Round 1, Round 2, …** — Versioning convention for the auth roadmap. Each round is a self-contained, shippable increment. Round 1 stops where the platform can safely onboard contributing engineers; Round 2 introduces the OIDC provider; later rounds tighten the screws.