# Junie brief: end-to-end k3s reset/deploy that produces a working KNOE.LOCAL↔PROLE.ORG Kerberos trust

## Why

A full afternoon on 2026-05-10/11 went into wiring up the
PROLE.ORG ↔ KNOE.LOCAL cross-realm Kerberos trust on the prole k3s
cluster (myrddin/merlin/gandalf). The Samba side now works
(`infrastructure/playbooks/kerberos_trust_setup.yml`, commits
`5cece40`…`ad1eced`), but the in-cluster KDC pod was running for realm
**`PROLE.LOCAL`** — a pre-rebrand artifact — and could not decrypt
referral TGTs from Samba. Every "Server not found in Kerberos
database" / "Decrypt integrity check failed" / "PROCESS_TGS" symptom
we chased was downstream of that.

Root cause: the originally-deployed pod inherited
`PROLE_KDC_REALM=PROLE.LOCAL` from a stale `etc/init_kdc.sh` shell
default. The default was fixed in commit `4fe8647` today, and a set
of larger patches to `etc/init_kdc.sh` followed.

What's **not** done: running `install.sh --reset` end-to-end against
the k3s cluster to confirm the patches converge to a working trust
from a blank slate. That's this brief.

## What's already landed (do not re-do)

| Commit | Subject |
|---|---|
| `4fe8647` | `etc/init_kdc.sh:327` shell default `PROLE.LOCAL` → `KNOE.LOCAL` |
| `5cece40`…`227f490` | `infrastructure/playbooks/kerberos_trust_setup.yml` — full Samba-side trust account provisioning |
| `ad1eced` | `kerberos_trust_setup.yml` — `msDS-SupportedEncryptionTypes` 28 → 4 (RC4 only) |
| _(also pending in this branch)_ | `etc/init_kdc.sh` — both cross-realm krbtgts created in MIT with RC4, KDC volume switched from emptyDir to PVC, broken remote-kadmin block replaced with documentation pointing at the playbook |

The matching source fix also landed in `~/dev/knoe-db` (commit
`ff7546d`) so the GKE / canonical installer stays in sync. **This
brief runs against the prole k3s cluster from this repo (`~/dev/prole`)
— do NOT run it against the knoe-db repo.**

## What changes

Nothing more in source. Your job is to **run** the end-to-end
reset-and-deploy from `~/dev/prole` and verify it succeeds without
manual cluster surgery.

## Execution path

> **Sequencing constraint:** `install.sh --reset` for `--mode k3s` runs
> the `k3s_reset.yml` Ansible playbook, which **wipes and reinstalls
> k3s on all three k3s nodes (myrddin, merlin, gandalf)**. Schedule
> accordingly — anything currently running in that cluster goes away.
>
> Before starting, capture the current state so you can roll back if
> needed:
>
> ```bash
> kubectl --context=default -n knoe-system get pods,svc,pvc,configmap > /tmp/pre-reset.txt
> kubectl --context=default -n knoe-system get deploy authority-knoe-auth -o yaml > /tmp/pre-deploy.yaml || true
> ```

```bash
# Run from a control node with kubectl access to the prole k3s cluster
# (myrddin itself is fine — it's the k3s server)

cd ~/dev/prole
git pull   # pick up the trust-fix commits if you don't have them locally

# 1. RESET — wipes k3s on all three nodes
./install.sh --mode k3s --reset

# 2. DEPLOY — re-provisions services including the KDC.
#    Watch the init_kdc.sh output for:
#      "Applying Knoe KDC '...' in namespace '...' (realm KNOE.LOCAL)"
#      "Creating outbound trust principal krbtgt/PROLE.ORG@KNOE.LOCAL"
#      "Creating inbound trust principal krbtgt/KNOE.LOCAL@PROLE.ORG"
#      "Note: Samba-side trust account is provisioned out-of-band by ..."
./install.sh --mode k3s

# 3. Sync the Samba-side trust account to the (possibly new) cluster
#    Secret value.
ANSIBLE_VAULT_PASSWORD_FILE=$PWD/.vault_pass \
  ansible-playbook infrastructure/playbooks/kerberos_trust_setup.yml
```

## TDD acceptance criteria

All four must pass. Capture the output of each and paste into your
MR description.

### AC1 — KDC pod has realm `KNOE.LOCAL`

```bash
kubectl --context=default -n knoe-system exec deploy/auth -- \
  cat /etc/krb5.conf | grep default_realm
# EXPECT: default_realm = KNOE.LOCAL
```

### AC2 — Both cross-realm krbtgts exist in MIT

```bash
kubectl --context=default -n knoe-system exec deploy/auth -- \
  kadmin.local -q 'listprincs' | grep -E '^krbtgt/'
# EXPECT (at minimum):
#   krbtgt/KNOE.LOCAL@KNOE.LOCAL
#   krbtgt/KNOE.LOCAL@PROLE.ORG
#   krbtgt/PROLE.ORG@KNOE.LOCAL
```

### AC3 — KDC data PVC is bound

```bash
kubectl --context=default -n knoe-system get pvc knoe-kdc-data
# EXPECT: STATUS=Bound
```

If you see `Pending`, look at the StorageClass:

```bash
kubectl get sc
kubectl --context=default -n knoe-system describe pvc knoe-kdc-data
```

The default k3s StorageClass `local-path` should be sufficient. If
your cluster doesn't have a default SC, re-run install with
`PROLE_KDC_STORAGE_CLASS=local-path` in the environment.

### AC4 — End-to-end cross-realm ticket flow

```bash
# Pre-stage a test SPN in KNOE.LOCAL so we have something to kvno
# against beyond just the krbtgt pair (no real KNOE.LOCAL services
# are deployed yet that aren't already covered by AC2).
kubectl --context=default -n knoe-system exec deploy/auth -- \
  kadmin.local -q 'addprinc -randkey test/myrddin.prole.org@KNOE.LOCAL'

# From myrddin (which has a PROLE.ORG client config in /etc/krb5.conf
# and a working /etc/krb5.conf KNOE.LOCAL block courtesy of the
# kerberos_trust_setup.yml playbook):
kdestroy
kinit chrisfu@PROLE.ORG       # enter your AD password
kvno krbtgt/KNOE.LOCAL@PROLE.ORG       # cross-realm referral
kvno test/myrddin.prole.org@KNOE.LOCAL # full service ticket
klist
# EXPECT: both kvno calls succeed with no errors. klist shows three
#         entries:
#           chrisfu@PROLE.ORG                  (initial TGT)
#           krbtgt/KNOE.LOCAL@PROLE.ORG        (cross-realm TGT)
#           test/myrddin.prole.org@KNOE.LOCAL  (service ticket)

# Cleanup
kubectl --context=default -n knoe-system exec deploy/auth -- \
  kadmin.local -q 'delprinc -force test/myrddin.prole.org@KNOE.LOCAL'
```

## If any AC fails

Surface the failure with diagnostics rather than papering over it:

| AC | If it fails, capture |
|---|---|
| AC1 | `kubectl get deploy -A | grep -i auth` (in case the live pod is named something other than `authority-knoe-auth`); the pod's full env: `kubectl -n knoe-system get deploy authority-knoe-auth -o yaml | grep -A2 PROLE_KDC_REALM`; the pod logs from the init phase: `kubectl logs deploy/auth` |
| AC2 | The init script's stdout from the pod logs: `kubectl logs deploy/auth` |
| AC3 | `kubectl get sc` + `kubectl describe pvc knoe-kdc-data`; the deployment events: `kubectl describe deploy authority-knoe-auth` |
| AC4 | `KRB5_TRACE=/dev/stdout kvno test/myrddin.prole.org@KNOE.LOCAL 2>&1`; `kubectl logs deploy/auth --tail=100` |

**Do not** manually `kadmin.local addprinc` to paper over a missing
principal — that's exactly the trap of the previous attempt. If a
principal is missing, the bug is in `etc/init_kdc.sh` (or in its
environment) and we want to know.

Also be alert for any leftover pods named `auth-*` or
`authority-prole-auth-*` from the pre-rebrand era. The `--reset` path
should wipe them, but if `kubectl get pods -n knoe-system` shows
multiple deployments still alive, delete the stale ones explicitly:

```bash
kubectl -n knoe-system delete deploy auth authority-prole-auth --ignore-not-found
```

## Out of scope

- GKE (`--mode gke`) deployment — different realm (`KNOE.DEV`), different
  cluster, separate brief if needed.
- Image rebuild — every config the KDC pod consumes is generated by
  `etc/init_kdc.sh` and applied via `kubectl apply`. No image work
  required.
- The OIDC issuer (`auth.knoe.dev`) — that runs on GKE; this brief
  is about the in-cluster k3s KDC only.
- HA storage for the KDC PVC — `local-path` (k3s default
  StorageClass) is fine for now. A future task can move to longhorn
  / synology iSCSI.

## Commit shape

One commit covering this brief plus any `etc/init_kdc.sh` follow-ups
needed during execution. Title:

```
kdc: verify install.sh --reset converges to a working cross-realm trust
```

Body should embed the AC1-AC4 outputs as proof of acceptance.

## Reference: the cascade that led here

The hand-debugging session on 2026-05-10/11 walked these layers, in
order, before reaching the root cause. Preserved so the next person
who lands at "PROCESS_TGS error" doesn't repeat it:

1. "Cannot find KDC for realm" → workstation `/etc/krb5.conf` didn't
   know about KNOE.LOCAL. Fixed by editing the workstation conf via
   `infrastructure/playbooks/workstation_kerberos.yml`.
2. "Server not found" → kadmin.local writes were going to the wrong
   pod. Service `auth` routed to `authority-knoe-auth-...`, not the
   `auth-...` pod we'd been editing.
3. "Decrypt integrity check failed" → kvno mismatch between Samba's
   re-keyed `krbtgt_KNOE.LOCAL` (kvno=4) and MIT's freshly-created
   `krbtgt/KNOE.LOCAL@PROLE.ORG` (kvno=1).
4. "PROCESS_TGS" → MIT pod's realm was `PROLE.LOCAL`, not `KNOE.LOCAL`,
   so even matching kvnos failed because the realm-of-decryption
   didn't match the realm-of-issue.
5. `default_realm = PROLE.LOCAL` baked into the Deployment env →
   came from the prole `etc/init_kdc.sh:327` shell default never being
   updated post-rebrand. **THIS IS THE ROOT.**

All of (1)–(4) above are downstream symptoms of (5).