- New acme role: installs acme.sh, issues certs for myrddin/merlin/gandalf
via DNS-01 (name.com API), installs cert files to /etc/ssl/certs/prole/,
sets up renewal cron. rc=2 (already valid) treated as ok.
- New playbooks/acme.yml: runs acme role on myrddin (issues all 3 host certs)
- New playbooks/certmgr.yml: cert-manager and letsencrypt-prod ClusterIssuer
(HTTP-01/Traefik) for external endpoints (db/svc/api.prole.org)
- Add vault references for name.com API creds to group_vars/all/vars.yml
- Add prole_ssl_src_dir and prole_ssl_files to myrddin/merlin/gandalf host_vars
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Followup from commit dba8a2d's findings. ConfigMixin._save_knoe_cfg in
knoe/ui/screens/cfg.py reads Tk widget vars via `var.get()` and writes
the result to conf/<mode>.cfg. When a var is a MagicMock (interactive
./install.py run in a non-Tk context, partially-mocked widgets), the
save path serializes the mock's repr-string into the cfg, then the next
installer pass calls os.makedirs() on those values and produces
directories literally named `<MagicMock name='Canvas().tk.call().strip()'
id='4743999712'>/`.
The brief lays out a TDD approach for Junie:
1. Write failing test at tests/installer/test_cfg_save_refuses_mock_values.py
that passes MagicMock widget vars and asserts _save_knoe_cfg raises
TypeError naming the field.
2. Implement the minimal fix: a `_str_value(var, field=...)` helper in
cfg.py that validates widget reads and raises if non-str. Use it
in the .get()/.strip() chains across lines 103-155.
3. Verify the 750 existing installer tests still pass.
Brief includes file pointers (cfg.py:44 _save_knoe_cfg, line 102
globals_to_save assembly, line 277 cfg_path.write_text), the canonical
failing test stub, both fix-approach options (per-read validator vs
end-of-flow dict walk), and explicit commit-shape guidance.
Index updates:
docs/plans/junie/README.md — todo-1 row added under Active
docs/TODO.md §"In progress" — todo-1 promoted above Phase 3 (smaller
scope, easy to land first)
Naming convention: `todo-N-<slug>.md` for follow-up bugs, distinct from
the `NN-<slug>.md` pattern reserved for ranked queue items.
- Add gandalf.prole.org → 10.0.0.7 to prole_dns_records and
prole_internal_a_records in dns.yml (was missing, causing nslookup
to fail on myrddin)
- prole_ssl: stat-check each source file on the controller before
copying; emit a warning and skip rather than hard-failing when certs
have not yet been placed in ssl/prole/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Installs Tailscale on k3s agents via the Debian stable apt repo using
the modern signed-by keyring approach. Auth key stored in vault.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
init_gitea.sh requires GITEA_PV_NODE in k3s mode. PVs for Gitea storage
(gitea-local-d005) are pinned to gandalf.prole.org per nodeAffinity on
the existing gitea-d005-shared-storage PV.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cleanup_legacy_prole_kong() only checked for the prole-svc-kong deployment
and service, so the early-return guard fired after those were already deleted,
leaving svc-prole-ingress in place. On re-runs this Traefik ingress blocked
the duplicate-host check and prevented Kong from deploying.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k8s/knoe/iscsi-pvs.yaml had *.knoe.org nodeAffinity after the rename, but the
live k3s cluster nodes are still *.prole.org — causing immutable field errors on
Bound PVs during kubectl apply. Revert to *.prole.org to match live state.
conf/k3s.cfg KUBECTL_CONTEXT was renamed to knoe-service-cluster, but both the
system kubeconfig and etc/secrets/k3s.kubeconfig still use prole-service-cluster.
install.sh _check_gcp_tools() now skips gcloud check for k3s mode and any non-gke
config path (only GKE deploys need gcloud/kubectl pre-flight).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
conf/dev/knoe.cfg had a stale SERVICE_NAMESPACE=default written by a prior TUI session.
Also guard _service_namespace() so the value 'default' is never returned — it is never
a valid service namespace and signals a stale generated config.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Combobox width=30 chars extends to ~320-380px depending on font metrics,
colliding with Add at x+290 and Delete at x+350. Narrowed combobox to
width=22 and shifted buttons to x+260 and x+320 to give clear separation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
install_barman_plugin() fetches from GitHub on every run (up to 6×120 s
retries), stalling the installer worker thread even when the CNPG cluster
is already at 'Cluster in healthy state' with all pods 2/2 Ready.
Add a fast-path at the top of initialize(): check .status.phase for
'healthy' then verify all pods show 2/N ready — if both pass, return
immediately. Full pipeline (ensure_operator, install_barman_plugin,
_apply_manifest, _wait_cnpg_pods) is only entered when needed.
Also adds test_cnpg_initialize_skips_when_cluster_healthy to
test_deployment_mode_isolation.py to guard against regression.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
kube-prometheus-stack CRD re-application stalls on every installer run
even when the release is in deployed state with all pods Running. For k3d
this is purely idempotency noise — skip the upgrade and return immediately.
Broken state (failed status, unbound PVCs) still triggers a full purge+reinstall.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k3d clusters only have local-path provisioner; defaulting to synology-iscsi
caused all Supabase PVCs to get stuck Pending. Also add local-path to the
validation allowlist so it is not rejected as an unknown storage class.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The container's /bin/sh is dash, which doesn't support set -o pipefail.
The preceding [ -s /tmp/pg.keytab ] check already guarantees the file is
non-empty before this line runs, so pipefail was redundant.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
conf/service/prod.cfg: prole.org homelab overrides
- SERVICE_HOST_DOMAIN=prole.org, DB_UI_HOST=db.prole.org
- OIDC_ISSUER_URL=https://api.prole.org/auth
- GOOGLE_WORKSPACE_DOMAIN=prole.org (credentials in OpenBao bao/kv/knoe/oauth2/google-prole-org)
- kerberos_config: enabled, realm=PROLE.ORG, kdc=10.0.0.3 (myrddin.prole.org)
- PROLE_KDC_TRUST_REALM=PROLE.ORG, SERVICE_INGRESS_TLS_ENABLED=1
conf/k3d.cfg: update local dev paths to ~/dev/prole; enable kerberos for dev
env.sh: KNOE_HOME points to ~/dev/prole
application-prole.yml: Spring profile with prole.org Google Workspace client binding
These files are prole.org homelab specifics — do NOT merge upstream to knoe/main.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Keytab export (silent failure bug):
- Always delete+recreate the postgres SPN with fresh random key; old key gone
after EmptyDir wipe; kadmin.local -q exits 0 even on error so -norandkey
silently failed
- Clean /tmp/pg.keytab before ktadd; verify non-empty with [ -s ] before
proceeding; die loudly if keytab not written
- Fix base64 pipeline: set -o pipefail inside sh -c so base64 failure is not
masked by tr exit code
Next steps:
- Suppress myrddin.prole.org Samba trust step in k3d mode (no AD server)
- Remove Grafana auth.proxy reminder (configured by Helm values already)
Gitea admin token:
- Auto-generate via kubectl exec into running Gitea pod before falling back to
manual warning; persist as gitea-admin-token secret for future re-runs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs in the EmptyDir auto-recovery path:
1. get_kdc_pod returned items[0] without a phase filter, so a Terminating
pod from the previous rollout could be selected. Add
--field-selector=status.phase=Running to always get a live pod.
2. The rollout completes (pod Running) before the in-container entrypoint
finishes kdb5_util create (no readiness probe configured). The single
immediate kadmin.local check raced against DB init and lost. Replace
with a 120s poll loop (5s interval) that re-queries the pod each tick
so it handles both the timing race and any remaining pod-selection lag.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
init_kdc.sh initialize rolls out a new deployment, creating a new pod with a
new name. The retry pre-flight was still exec-ing into the old (terminated)
pod. Re-query get_kdc_pod() after re-init so the retry and all subsequent
kadmin calls use the new pod.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the KDC pod restarts (e.g. after k3d node restart to apply registry
mirror config), its EmptyDir volume is wiped. ensure_kdc_pod() finds the
running pod and returns it, bypassing the bootstrap path. The subsequent
pre-flight check then fails because kadmin.local cannot access the database.
Instead of dying with a manual-intervention message, auto-recover by calling
init_kdc.sh initialize (with PROLE_KDC_NAMESPACE/NAME/MASTER_PASSWORD set)
then retrying the pre-flight. Only die if the retry also fails.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
check_kerberos.sh defaulted PROLE_KDC_NAME=auth but init_kdc.sh deploys the
pod with label app=authority-knoe-auth (authority-gcp-auth for k8s). The pod
selector never matched, so wait_for_kdc_pod always returned empty and the test
failed with "KDC pod 'auth' not found" even after a successful rollout.
Add _default_kdc_name() using the same mode-aware logic as init_kdc.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k3d registry containers are addressed as k3d-NAME.localhost:PORT from
inside the cluster, but containerd resolves that as 127.0.0.1 (loopback)
which has nothing listening. The fix is to write /etc/rancher/k3s/registries.yaml
on every server/agent node to mirror k3d-NAME.localhost:PORT -> http://k3d-NAME:PORT
(Docker network DNS), then restart the node so containerd picks it up.
_patch_k3d_registry_mirrors() is called from _ensure_k3d_registry() after
the registry is confirmed running (idempotent — skips nodes already configured).
This makes knoe-authority and any other image pushed to localhost:5000
immediately pullable inside the cluster as k3d-knoe-registry.localhost:5000/...
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
check_kerberos.sh (already updated) sourced knoe_env.sh, knoe_secrets.sh,
knoe_string.sh, and knoe_yaml.sh but the files were still named prole_*.sh.
No other script referenced the old names, so this is a pure rename.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Port-forward setup was failing the installation because:
1. argocd and dashboard entries were always added to port-mapping even when
those services haven't been selected or installed yet
2. Grafana/Prometheus service names were hardcoded as kps-* but monitoring
deploys with release name "prometheus" → services are prometheus-grafana etc.
3. The grafana status poll after init_port_forwards.sh start would mark the
entire installation failed if grafana wasn't reachable
Changes:
- _build_required_port_forwards (env.py): add argocd_enabled, dashboard_enabled,
monitoring_release params; argocd/dashboard only included when enabled; grafana
and prometheus targets use f"svc/{monitoring_release}-grafana" etc.
- _sync_port_forward_mappings (actions.py): read argocd_enabled from installer
config; read MONITORING_RELEASE from Monitoring section; clean up argocd and
dashboard entries from [Port Forwards] when disabled
- run_init_scripts (services.py): port-forward step is non-fatal — failure logs
a warning and continues; blocking grafana poll removed
- 2 new regression tests (7e) covering the above invariants
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
services.py run_init_scripts() called init_kong.sh with -n env[NAMESPACE]
where NAMESPACE is the database namespace (e.g. knoe-db). The -n flag
sets COMMON_CORE_NAMESPACE which wins over SERVICE_NAMESPACE in
common_core_resolve_namespace, so Kong deployed into knoe-db.
Fix: resolve kong_ns from SERVICE_NAMESPACE, falling back to
_get_service_namespace() and finally knoe-system. Exclude default
as it is the Tk StringVar initial value, not a real service namespace.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When SERVICE_NAMESPACE was absent from the config file and OS env,
_get_script_env() fell back to env["NAMESPACE"] — the database namespace
(e.g. knoe-db). InitializationScriptsMilestone then called init_kong.sh
with SERVICE_NAMESPACE=knoe-db, causing Kong to deploy into the DB
namespace and collide with the existing knoe-system/svc-knoe-ingress.
Fix: default to "knoe-system", matching _service_namespace() in actions.py.
Configs that set SERVICE_NAMESPACE explicitly are unaffected.
Add tests/test_deployment_mode_isolation.py to guard against k3d/k3s/gke
mode-specific config bleeding into each other: SERVICE_NAMESPACE fallback,
monitoring storage class separation, Kong gitea host gating, and
_service_namespace() across all three deployment modes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three issues caused the purge-and-reinstall loop:
1. _purge_broken_monitoring deleted ALL PVCs including Bound ones; grafana's
working PVC was wiped on every retry. Now only delete PVCs in Pending state.
2. Broken-state detection keyed on Pending pods + unbound PVCs, which is true
during any still-converging install (including ones that timed out but
whose pods eventually came up). Gate on helm status=='failed' + unbound PVCs.
3. k3d install used --wait, which blocks on all kube-prometheus-stack components
(prometheus, alertmanager, node-exporter). They converge async after the
Prometheus operator starts; --wait always timed out. Drop --wait for k3d;
the status_common_services.sh check verifies readiness independently.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
kube-prometheus-stack pulls multiple large images on a fresh k3d install;
600s / 10m is not enough. Set --timeout 20m and process timeout 1320s
(20m + 2m buffer) for k3d, keep 10m for k8s/k3s where images are warm.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs kept prometheus-grafana-0 Pending for 21 days in k3d:
1. update() routed k3d through _values_yaml_k3s() which hardcodes
merlin-local-iscsi-* storage classes and merlin.prole.org node affinity
— neither exists in k3d, so grafana's PVC could never bind.
Add _values_yaml_k3d() that uses local-path (or MONITORING_STORAGE_CLASS
override) with no node affinity; dispatch k3d separately in update().
2. No broken-state recovery for non-k8s mode. Add _purge_broken_monitoring()
(helm uninstall + delete stuck PVCs) and call it from update() when
Pending pods and unbound PVCs are both detected in the monitoring namespace.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The shell path (init_registry.sh apply_registry) is the real k3d registry
entry point. It checked 'k3d registry list knoe-registry' which returned 0
because k3d-knoe-registry existed in 'created' state, so the create was
skipped and k3d-prole-registry (the port-5000 squatter) was never touched.
- init_registry.sh: before the knoe-registry ensure, detect k3d-prole-registry
via both k3d and docker inspect, then stop/rm the Docker container regardless
of whether k3d manages it; also detect knoe-registry in non-running state and
nuke+recreate it (port was unavailable on the previous attempt)
- k3d_registry.py: same cleanup logic for the Python fallback path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k3d-prole-registry was squatting on port 5000, leaving k3d-knoe-registry
in created state. Two bugs made this invisible:
- k3d_registry.py: substring match (name in stdout) matched k3d-knoe-registry
as already-exists without checking STATUS=running; add _k3d_registry_running()
requiring last column == running, detect and delete prole-registry before
creating knoe-registry, recreate if found in non-running state
- status_common_services.sh: grep -qx 'knoe-registry' (exact) never matched
k3d prefix 'k3d-knoe-registry'; fix with awk suffix match + STATUS==running
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k3s manifests still used prole-svc-kong after the rebrand; init_kong.sh
was waiting on `rollout status deployment/knoe-svc-kong` which never
existed, hanging the Common Services milestone.
- Rename kong-deployment.yaml and kong-service.yaml: prole-svc-kong →
knoe-svc-kong (labels, selector, configmap volume ref)
- Remove prole-svc-kong-configmap.yaml (static file replaced by
init_kong.sh dynamic ConfigMap generation; hardcoded namespace and
prole-branded routes were dead weight)
- init_kong.sh: add cleanup_legacy_prole_kong() called from action_update()
to remove stale prole-era resources before deploying knoe-svc-kong
- init_kong.sh: add _is_host_claimed_by_other_ingress() helper; set
include_gitea_host=0 for k3d mode (git is port-forward only, no public
hostname) — keeps the safety-net pre-check for k3s
- actions.py: extend legacy namespace dedupe to sweep prole-svc-kong from
both default and service_ns
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Junie's session targeted the prompt "stabilize ./install.py -c conf/k3d.cfg
using strict TDD" — broad installer-side work, not the k3d-mirror Phase 3
brief I had filed (which she didn't pick up; phase-3 brief stays open). All
750 installer tests pass post-change.
What Junie produced:
install.py (NEW) Top-level CLI entry point. Was
imagined by the prompt but didn't
exist; this commit makes it real.
knoe/deployment.py (NEW) `KnoeDeployment` orchestrator for
the k3s service-mode deploy pipeline.
Wraps Ansible kubeconfig fetch,
opentofu apply, init_*.sh post-apply
scripts, and (optionally) supabase/
deploy.sh.
knoe/ui/screens/cluster.py Dual-cluster GKE kubecontext UI: prod env
knoe/ui/screens/cfg.py now shows separate "App Cluster:" and
"DB Cluster:" dropdowns instead of a
single "Kubernetes Context:" combo.
New _app_kubectx_combo + _db_kubectx_combo
widgets; new app/db_cluster_kubecontext
tk.StringVars.
knoe/core/{actions,env,milestones}.py
knoe/core/ops/storage.py
knoe/config.py, knoe/knoe_conf.py Plumbing changes for the dual-cluster
kubecontext flow + storage-class topology
detection cleanup.
knoe/tools/cleanup_cnpg_storage.py (NEW) Stand-alone cleanup utility.
tools/dashboard.sh (NEW) Dashboard helper.
conf/knoe.cfg (NEW) Master cfg generated by knoe_conf.
conf/dev/ (NEW) Dev-mode cfg directory.
conf/port-mapping.cfg Port mapping tweaks for k3d.
tests/installer/* (8 files) New + extended tests for the dual-cluster
tests/test_database_options.py TUI, kubecontext save flow, storage ops,
topology detection, deploy helpers,
database-options screen.
Issues found in Junie's working state and fixed here:
1. install.py was a 11-line import shim with no shebang, no `chmod +x`,
no `if __name__ == '__main__'` block. `./install.py -c conf/k3d.cfg`
returned `Permission denied` and `python install.py` did nothing.
Added `#!/usr/bin/env python3`, `chmod +x`, and a __main__ block
that delegates to `knoe.ui.screens.main()`. `./install.py --help`
now prints the canonical argparse help.
2. knoe/deployment.py had FIVE `subprocess.run()` call sites with no
`timeout=` argument (`_run_script`, `_run_cmd`, the Ansible playbook
fetch, `tofu init`, `tofu apply`). A hung child process — typical
failure mode is a script waiting on stdin or a stalled network
call — would lock up the installer indefinitely. Added timeouts:
- Ansible kubeconfig fetch: 120s
- tofu init: 300s
- tofu apply, _run_script, _run_cmd: bounded by new module
constant `_MILESTONE_TIMEOUT` (default 1800s = 30 min, override
via `KNOE_MILESTONE_TIMEOUT_SECONDS` env var).
`subprocess.TimeoutExpired` is caught explicitly; on timeout the
run helpers return exit code 124 (conventional timeout code).
3. `conf/k3d.cfg` was corrupted with MagicMock string-reprs on disk:
KNOE_CONF = <MagicMock name='Canvas().tk.call().strip()' id='4743999712'>
argocd.node_selector = <MagicMock name='mock.StringVar().get().strip()' id='...'>
Likely path: Junie ran `./install.py -c conf/k3d.cfg` interactively
in a non-Tk environment (or with a partially-mocked widget set) and
the installer's "save current state" path wrote the mock-objects'
`__repr__` strings into the cfg file. This commit reverts the cfg
to its pre-Junie state. **Followup: harden the cfg save path
against non-string widget values** — track separately.
4. The corrupted cfg caused the installer to call `os.makedirs()` on
the mock-string values, producing 10 directories on disk literally
named `<MagicMock name='Canvas().tk.call().strip()' id='4733210304'>/`
etc., with 5–86 files of install artifacts inside each. Removed.
The "final step is timing out" the user reported was almost certainly
issue #2 above: install.py walked the milestone pipeline, hit one of
the unbounded subprocess.run calls, and the wrapped command (probably
supabase/deploy.sh, which Junie was reading for context when her
session timed out) hung. With the timeouts in place that path now
exits cleanly with rc=124 instead of locking up.
Verification:
- pytest tests/installer/ -q 750 passed in ~25s
- python3 -c "import knoe.deployment" imports clean
- ./install.py --help prints argparse help
- find . -maxdepth 1 -type d -name '<MagicMock*' | wc -l 0
- head -7 conf/k3d.cfg clean (no MagicMock)
Out of scope for this commit (followups):
- The cfg save-path that wrote mock-objects-as-strings (issue #3 root cause).
Reproducer: launch the installer in an env where Tk widget vars are
`unittest.mock.MagicMock` instances. The cfg save code should refuse to
serialize non-str values rather than calling `str()` on a MagicMock.
- The k3d-mirror Phase 3 brief (`docs/plans/junie/k3d-knoe-auth-pod-deploy.md`)
is still open — Junie picked a different prompt this round.
Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 (host loop) and the Phase 2 OIDC sandbox are shipped; this
brief queues Phase 3 of k3d-mirror-of-GKE: build the knoe-auth image,
k3d-import, run as a pod inside the cluster. Pre-merge smoke loop —
not the daily inner loop.
Deliverables (see brief for the full Definition of done):
- k8s/knoe/knoe-auth-deployment.yaml (NEW; sibling of GKE manifest)
- scripts/k3d-knoe-deploy.sh (NEW; build + import + apply + wait)
- Makefile k3d-knoe-{deploy,redeploy,undeploy} targets
- Phase 2 signing key flows from etc/secrets/knoe-auth-oidc-key.b64 into
a knoe-auth-oidc-signing-key K8s Secret in knoe-system, mirroring how
the GKE deploy reads it.
Index updates:
docs/plans/junie/README.md — k3d Phase 3 in Active row
docs/TODO.md §"In progress" — promotes the brief to top
docs/plans/k3d-gke-mirror.md — banner + §6 Phase 3 entry
flipped from "out of scope"
to "in flight"
Three independent lines of work landing in one commit because they were all
on disk together end-of-session and the cross-file edits (TODO, knoe-system,
junie/README) interleave cleanly:
1. **Queue #3 — image rename `knoe-authority` → `knoe-auth`** (Junie).
Closes drift R6. The Maven artifact has been `knoe-auth.jar` since
commit b355855; the deploy manifests now match.
authority/Dockerfile.app (NEW)
deploy/gcp/gke/knoe-auth-deployment.yaml (3 image tags renamed)
deploy/opentofu/k3s/manifests/knoe/knoe-auth-deployment.yaml (2 image tags renamed)
Makefile (docker-build-auth +
docker-push-auth + REGISTRY/KNOE_AUTH_VERSION
defaults)
docs/plans/junie/03-image-rename-knoe-authority-to-knoe-auth.md (brief, kept as design record)
2. **Phase 2 OIDC provider — GKE deploy** (Junie). Source landed via the
merge that brought claude/crazy-bose-fec256 back; the k3d sandbox
shipped earlier today (commit 93157b0). This commit completes the GKE
path: Kong route `/auth` → knoe-auth, default values added, signing
key wired via `knoe-auth-oidc` secret per the brief.
supabase/helm/knoe-supabase/templates/kong/config.yaml (Kong route /auth, gated
on .Values.knoeAuth.enabled)
supabase/helm/knoe-supabase/values.yaml (knoeAuth.enabled=false default;
studioIngress block — also
fixes the helm lint issue noted
in earlier commit 03bb731)
docs/plans/junie/phase2-oidc-gke-deploy.md (brief, kept as design record)
3. **k3d dev-user seed: `chrisfu` + `knoe_developer`** (Claude). Closes
"I want auth as chrisfu@knoey.com to my local cnpg database" for the
local dev loop. `etc/init_knoe_auth.sh --mode k3d` now runs
`seed_dev_users_k3d()` after the schema bootstrap, creating:
- `knoe_developer` group role with R/W on `knoe`+`public` (mirrors
the GKE production layout from docs/db-access.md; was hand-rolled
in production per the 2026-04-30 onboarding work, never baked into
postInitTemplateSQL).
- `chrisfu` LOGIN role with password `chrisfu-dev` (idempotent —
resets on every `make k3d-knoe-up`, so the rebuild loop is
deterministic). Granted into `knoe_developer`.
From the host with port-forward up:
PGPASSWORD=chrisfu-dev psql "postgresql://chrisfu@localhost:5432/knoe-db?sslmode=require"
Cross-cutting doc updates:
docs/TODO.md — Phase 2 GKE entry removed from "In progress"
(now "(none — all items shipped or paused)");
queue #3 + reality table R6 removed; pg_oauth
paused note flipped to "OIDC issuer now
deployed to GKE — pg_oauth can resume"; Done
section gets new entries for #3, Phase 2 GKE,
and chrisfu seed.
docs/knoe-system.md — Phase 2 GKE row → Shipped; pg_oauth → Ready;
knoe-auth-deployment.yaml note "knoe-auth:latest"
instead of "knoe-authority:latest"; "One-time
cluster setup" mentions the chrisfu seed.
docs/local-dev-knoe-auth.md — "Verify psql connectivity" rewritten to
connect as chrisfu (was `knoe`); new "Reset /
rebuild loop" section explaining idempotency.
docs/plans/junie/README.md — two new rows under Shipped: brief 03 and
phase2-oidc-gke-deploy.
Verification (the bits I ran locally):
- bash -n etc/init_knoe_auth.sh OK
- The seed SQL is idempotent (DO blocks with EXISTS checks +
ALTER ROLE on the password reset path)
- The Kong /auth route is gated on .Values.knoeAuth.enabled — default
false, no behavior change until someone flips it on per Junie's brief
Out of scope for this commit:
- Actually flipping knoeAuth.enabled=true on the live GKE chart and
rolling out — Junie's brief covers the runbook; needs the
knoe-auth-oidc K8s secret populated from 1Password first
- Rebuilding + pushing the new knoe-auth:latest image — `make
docker-push-auth` is wired, just hasn't been run yet
- Round 1.5 OpenBao transit-key encryption (still queued)
Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Completes the Phase 2 OIDC laptop dev path. Source for the OIDC surface
(discovery, authorize, token, userinfo, JWKS controllers + signing /
session services) landed on `main` via the merge that brought
claude/crazy-bose-fec256 back. This commit makes Phase 2 actually
exercisable in the k3d dev loop without any GKE deploy.
What's new:
authority/src/main/resources/application-k3d.yml
Spring profile activated by `-Dspring-boot.run.profiles=k3d`.
Enables OIDC (knoe.oidc.enabled=true), points the issuer at
`http://localhost:8080`, sets Kerberos realm to KNOE.LOCAL, and
aligns the datasource with the port-forwarded localhost:5432 DB.
etc/gen_oidc_signing_key.sh (executable)
Idempotent RS256 PKCS#8 keypair generator. Outputs:
etc/secrets/knoe-auth-oidc-key.pem (PEM)
etc/secrets/knoe-auth-oidc-key.b64 (single-line base64 of DER —
directly consumable as KNOE_AUTH_OIDC_SIGNING_KEY by
OidcTokenService.init())
`etc/secrets/` is already gitignored. Set FORCE=1 to rotate.
What's wired:
scripts/k3d-knoe-up.sh
New §7 calls etc/gen_oidc_signing_key.sh after schema seed. Keypair
persists across `make k3d-knoe-down && make k3d-knoe-up` cycles.
scripts/k3d-knoe-pf.sh
Output now includes the KNOE_AUTH_OIDC_SIGNING_KEY export line, the
full `mvn spring-boot:run` invocation with -Dspring-boot.run.profiles=k3d,
and the three OIDC endpoints to curl-test.
What's documented:
docs/local-dev-knoe-auth.md
"Daily loop" Terminal B: now exports KNOE_AUTH_OIDC_SIGNING_KEY,
runs with `-Dspring-boot.run.profiles=k3d`, and the verify section
includes /jwks.json. IntelliJ run config: adds Active Profiles: k3d
and a note about pasting the b64 directly (no shell expansion in
the env-var field).
docs/knoe-system.md
Phase 2 status row split: "k3d setup" → Shipped, "GKE deploy" →
Pending. The "Open work items" Phase 2 entry rewritten to flag
that the GKE deploy is the remaining thread (gated on queue #3 for
the image rebuild as `knoe-auth:latest`).
docs/TODO.md
Promoted "Phase 2 OIDC provider — GKE deploy" into §In progress
(replacing the empty "(none)" placeholder). Done section updated
with two entries: the k3d Phase 1 dev loop (Junie's c3...) and
this Phase 2 OIDC k3d sandbox.
End-to-end loop the engineer can run:
make k3d-knoe-up # one-time, ~5 min
make k3d-knoe-pf & # port-forwards
export KRB5_CONFIG=$PWD/etc/krb5.local.conf
export KNOE_AUTH_OIDC_SIGNING_KEY=$(cat etc/secrets/knoe-auth-oidc-key.b64)
mvn -pl authority spring-boot:run \
-Dspring-boot.run.jvmArguments="-Djava.security.krb5.conf=$PWD/etc/krb5.local.conf" \
-Dspring-boot.run.profiles=k3d
# then:
curl -s http://localhost:8080/.well-known/openid-configuration | jq .issuer
# → "http://localhost:8080"
curl -s http://localhost:8080/jwks.json | jq '.keys[0].kty'
# → "RSA"
Verified locally: keypair generator round-trips through openssl pkey -inform DER
(produces valid 2048-bit RSA keys); idempotent (existing key kept by default,
FORCE=1 rotates); bash -n clean on all 5 touched scripts.
Out of scope (TODO §In progress captures it):
- GKE deploy of Phase 2 (image rebuild + K8s Secret + deployment env vars)
- SPNEGO E2E from host browsers (k3d-mirror Phase 2)
- knoe-auth-as-pod in k3d (k3d-mirror Phase 3)
- OidcCodeService DB persistence (separate track)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings up the smallest k3d-resident stack that lets a host-side knoe-auth
(run via `mvn spring-boot:run` or IntelliJ) iterate against real Postgres
+ Kerberos. Closes Phase 1 of the k3d-gke-mirror plan (docs/plans/k3d-gke-mirror.md).
Scope:
- k8s/knoe/knoe-kdc-{configmap,deployment,service,pvc,init-job}.yaml
NEW; standalone KDC, realm KNOE.LOCAL (distinct from KNOE.DEV).
- etc/init_knoe_auth.sh: --mode k3d flag added; swaps realm + skips
GCP-specific steps. GKE behavior unchanged when flag absent.
- Makefile: k3d-knoe-up, k3d-knoe-pf, k3d-knoe-down (delegate to
scripts/k3d-knoe-{up,pf,down}.sh).
- scripts/k3d-knoe-{up,pf,down,smoke}.sh NEW; up = full bring-up,
pf = three port-forwards (5432/88/464) + JDBC URL + ^C cleanup,
down = teardown, smoke = sanity check.
- etc/krb5.local.conf NEW; checked-in libdefaults+realms config
pointing at localhost:88. udp_preference_limit=1 to dodge
kubectl port-forward UDP flakiness on macOS.
- docs/local-dev-knoe-auth.md NEW; one-time setup + daily loop +
IntelliJ run config.
- docs/knoe-system.md NEW; unified reference for the knoe-auth
service (GKE deployment + k3d dev loop + schema overview +
source map + open work items).
Verified per the brief's Definition of done: fresh-clone laptop can
`make k3d-knoe-up` + `make k3d-knoe-pf` + `mvn -pl authority spring-boot:run`
and hit /health, /.well-known/openid-configuration in <8 minutes.
Out of scope (parent plan docs/plans/k3d-gke-mirror.md §6):
- SPNEGO from host browsers (Phase 2)
- knoe-auth-as-pod / image build/load (Phase 3)
- Supabase stack on k3d (Phase 4)
- OidcCodeService DB persistence (separate track)
docs/plans/junie/README.md — k3d brief moved from Active to Shipped.
docs/TODO.md — In-progress now empty; Phase 2 pg_oauth notes that the
local dev loop is in place so it can resume.
Closes Phase 1; Phase 2+ briefs filed as needed.
Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The next development thread is knoe-auth Phase 2 OIDC iteration, which
just landed on main but has no fast inner-loop. Instead of bouncing
edits through the GKE deploy chain, we want a laptop-resident model
where knoe-auth runs from the IDE / mvn against real Postgres + KDC
brought up in k3d.
Two new docs:
docs/plans/k3d-gke-mirror.md
Architectural plan. Top-to-bottom rationale for what the k3d
model should mirror, what we deliberately skip (supabase, kong,
oauth2-proxy, Studio — all unnecessary for knoe-auth's surfaces),
and the phasing. Captures the three scope decisions taken
2026-05-02 with the user:
- knoe-auth runs on the laptop, not as a pod (faster inner loop)
- stack scope is CNPG + KDC only
- single-replica CNPG, realm KNOE.LOCAL (distinct from KNOE.DEV)
Out-of-scope for Phase 1 listed explicitly so reviewer
expectations match scope: SPNEGO E2E, image-build-and-load,
supabase, oauth2-proxy, OidcCodeService DB persistence.
docs/plans/junie/k3d-knoe-auth-dev-loop.md
Phase 1 brief. Concrete deliverables:
- k8s/knoe/knoe-kdc-{deployment,service,configmap,pvc,init-job}.yaml
- etc/init_knoe_auth.sh --mode k3d flag (or sibling script)
- Makefile: k3d-knoe-{up,pf,down}
- etc/krb5.local.conf (with udp_preference_limit=1 for
kubectl port-forward UDP flakiness on macOS)
- docs/local-dev-knoe-auth.md (engineer-facing)
- scripts/k3d-knoe-smoke.sh
Definition of done: a fresh-clone laptop can `make k3d-knoe-up`
+ `make k3d-knoe-pf` + `mvn -pl authority spring-boot:run` and
hit /health, /.well-known/openid-configuration in <8 minutes.
Index updates:
docs/plans/README.md — k3d-gke-mirror.md row added
docs/plans/junie/README.md — split into Active / Shipped;
k3d brief listed under Active;
the 5 shipped-2026-05-02 briefs
moved to Shipped with commit refs.
docs/TODO.md In-progress — k3d brief now top of "In progress";
the previously in-flight "Phase 2
pg_oauth in install/deploy.sh"
item demoted to "paused" (resume
after the local dev loop lands).
No code changes; all docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirroring the 2026-04-29 DB-cluster removal. Live state showed garage
in an unconfigured limbo (layout v0, no role assigned, 21d of "Ring
not yet ready" warnings); nothing outside garage's own state
referenced its S3 endpoint anymore.
Removed from knoe-system namespace on knoe-dev-0:
- statefulset/garage (1/1 pod, 21d uptime)
- service/garage (ClusterIP; no garage-s3-ilb on app cluster)
- configmap/garage-config
- secret/garage-secrets
- pvc/data-garage-0 (29 Gi, garage-hdd)
- pv/pvc-656e1936-... (was Bound, reclaim=Retain)
- storageclass/garage-hdd
Underlying GCE pd-standard disk (us-west3-c/pvc-656e1936-...) reclaimed
via `gcloud compute disks delete` since reclaim=Retain leaves the disk
behind after the PV is gone.
Source manifests intact:
- k8s/knoe/garage-*.yaml (k3d, min)
- deploy/opentofu/k3s/manifests/knoe/garage-*.yaml (k3s)
- etc/init_garage_store.sh + knoe/core/ops/k8s_garage_store.py
- k8s/knoe/garage-statefulset-gcp.yaml (GKE variant)
A future ./install.sh against GKE could re-deploy garage if that's
what the deploy mode wires; whether it SHOULD wire garage given GCS
now handles backups is a separate cleanup decision.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Queue #5 Phase B orphan cleanup landed:
App cluster (knoe-dev-0):
- gitlab/gitlab-migrations-58a3e27-97-23-n72j8 (Failed/Evicted,
11d stale)
- pv/pvc-42e8545b-… (10Gi, monitoring/storage-kps-grafana-0)
- pv/pvc-86422179-… (30Gi, monitoring/prometheus-…-prometheus-0)
- pv/pvc-e2c29f7a-… (5Gi, monitoring/alertmanager-…-alertmanager-0)
DB cluster (knoe-dev-cnpg-0):
- kube-system/prole-supabase-dirprep Job (FailureTarget, 22d stale)
Post-state: 0 Released PVs on app cluster; gitlab namespace has only
the Completed migration pod; DB-cluster kube-system has no prole-*
artifacts. Live-cluster ops only; no source changes.
Queue #4 reframed: tried the planned `garage repair --yes blocks`,
but found garage in an unconfigured state — layout v0, no role
assigned, "Ring not yet ready" warnings continuous since whenever the
layout was lost (likely during a reconcile after the 2026-04-29 PVC
resize). Cross-cluster sweep shows nothing outside garage references
its S3 endpoint anymore (CNPG backups moved to GCS, no other workload
points at 10.180.15.239:3900). Garage on app-cluster is now a dead
pod holding an 80 GiB PVC. The original block-repair task is moot.
Reframed as a decommission-or-restore decision; details captured in
the ranked queue entry. Awaiting direction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a 2026-05-02 Done entry covering the cluster-side work that paired
with c3fae73 (source) and 03bb731 (chart cleanups): PodMonitor migration
applied with verified zero-data-loss handoff; supabase chart upgraded
revision 1→3; helm field-manager conflict on Kong configmap resolved;
node label drift on knoe-dev-0 closed (3 nodes labelled
knoe.dev/node-role=general alongside legacy prole.org/node-role); DB
cluster nodes still pending same treatment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two follow-ups to commit c3fae73 (queue #15) surfaced when applying
the DASHBOARD removal to live `supabase` release on knoe-dev-0:
1. **Delete `templates/test/kong.yaml`.** This was a Helm Test Job that
validated the Studio dashboard's basic-auth gate. The basic-auth
plugin was retired in commit 25f1b2e (oauth2-proxy fronting); the
consumer + credentials in commit c3fae73. The test has nothing
left to test — it referenced `supabase.secret.dashboard` template
(gone) and `.Values.secret.dashboard.{secretRef,secretRefKey}`
(gone), causing `helm template`/`upgrade` to fail. Removing the
file is the right move; if dashboard auth ever comes back as
something else, write a fresh test for that.
2. **Add `secret.oauth2proxy` + clarify `deployment.oauth2proxy`
defaults in values.yaml.** The chart has
`templates/studio/oauth2proxy-{deployment,service}.yaml` and
`templates/secrets/oauth2proxy.yaml` (Junie's Phase 2 OIDC adds)
guarded by `if .Values.deployment.oauth2proxy.enabled`. The guard
short-circuits when the parent map exists, but if the map itself
is undefined helm fails with `nil pointer evaluating interface{}.enabled`.
Added a `secret.oauth2proxy` block (empty defaults) so the secrets
template guard can evaluate cleanly. `deployment.oauth2proxy.enabled`
was already `false` in values.yaml; added a comment explaining the
chart oauth2-proxy templates are vestigial — the deployed
oauth2-proxy lives in `deploy/gcp/gke/oauth2-proxy-deployment.yaml`,
in FRONT of supabase-kong (commit 25f1b2e), not as a Kong upstream.
Live state after this commit:
- `helm upgrade supabase ./supabase/helm/knoe-supabase
--reset-then-reuse-values` succeeds.
- revision 3 deployed; configmap field manager = `helm` (sole owner).
- `kubectl get cm supabase-kong -o yaml | grep DASHBOARD` returns 0
matches.
- All Kong env vars on the new pod are clean (no DASHBOARD_*).
- External smoke tests:
/onboard.html → 200
/rest/v1/ no auth → 401
/ (Studio gate) → 302 to Google
/healthz → 302
svc.knoe.dev/grafana/login → 200
- cnpg-grafana panels still populate (PodMonitor migration verified
earlier in same session: 3 postgres exporter targets up; cluster
label injection working).
Side-finding handled live: nodes lacked the new
`knoe.dev/node-role=general` label that Junie's chart defaults to in
nodeAffinity. Added the label to all 3 knoe-dev-0 nodes alongside the
legacy `prole.org/node-role=general`. The label rename has been
tracked in CLAUDE.md as a follow-up; this is partial closure for the
app cluster (DB cluster nodes still need the same treatment when next
relevant).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three Junie briefs landed in one commit:
#07 — Wire cnpg-backup-sa into CNPG cluster spec (drift R8)
deploy/gcp/gke/knoe-db.yaml: add spec.serviceAccountName: cnpg-backup-sa
(requires CNPG v1.29+, which is the live operator version).
etc/init_cnpg_gke.sh: operator install URL now uses CNPG_OPERATOR_VERSION
variable (default 1.29.0); new §11 patches knoe-db and
knoe-db-barman-cloud RoleBindings to add cnpg-backup-sa as a subject
if not already present — matching the 2026-04-29 live stabilization.
#13 — Migrate off deprecated enablePodMonitor + podMonitorRelabelings
Both deprecated fields removed from deploy/gcp/gke/knoe-db.yaml
spec.monitoring. New deploy/gcp/gke/knoe-db-podmonitor.yaml carries
the PodMonitor with the cluster relabeling rule (cnpg.io/cluster pod
label → cluster label; required for all 85 CNPG Grafana panels).
Apply alongside knoe-db.yaml on next cluster patch.
#15 — Remove dead DASHBOARD consumer + basicauth_credentials
supabase/helm/knoe-supabase:
- wrapper.sh: drop DASHBOARD_USERNAME / DASHBOARD_PASSWORD envsubst lines
- config.yaml: drop DASHBOARD consumer + basicauth_credentials block
- kong/deployment.yaml: drop both DASHBOARD env-var secret refs
- values.yaml: rename secret.dashboard → secret.openai (apiKey only;
username/password dropped — no enforcer since commit 25f1b2e)
- secrets/dashboard.yaml + _helpers.tpl: renamed to openai /
supabase.secret.openai
- studio/deployment.yaml: reads from secret.openai.apiKey
- ci/example.yaml: updated to secret.openai.apiKey
helm template confirms knoe-supabase-openai secret referenced; no
DASHBOARD output.
docs/TODO.md: queue items #7, #13, #15 + drift row R8 archived to Done.
Co-authored-by: Junie <junie@jetbrains.com>
Defects A, B, C from docs/plans/junie/06-patch-garage-script-fixes.md:
A. DB_CLUSTER default was knoe-cnpg-0 (stale rebrand artifact);
corrected to knoe-dev-cnpg-0 to match conf/gke.cfg and the
cluster table in CLAUDE.md. Header comment + overrides block
also updated.
B. Phase 1 deletion loop missed service/garage-s3-ilb (the
LoadBalancer the ObjectStore endpoint historically pointed at).
Now included alongside service/garage.
C. Phase 2 was applying knoe-db-backup-gcs.yaml whole, including a
legacy ScheduledBackup using method:barmanObjectStore (being
removed in CNPG v1.30). Manifest split handled in prior commit
(34a25dd); script's Phase 2 is now safe to re-run.
bash -n clean; CONFIRM=false dry-run prints knoe-dev-cnpg-0 correctly.
docs/TODO.md: queue items #2 and #6 + drift rows R5 and R9 archived to Done.
Closes queue item #6 in docs/TODO.md (drift R9).
Co-authored-by: Junie <junie@jetbrains.com>
deploy/gcp/gke/knoe-db-backup-gcs.yaml previously contained both the
cnpg-backup-sa ServiceAccount (wanted) and a ScheduledBackup using
method: barmanObjectStore (unwanted — legacy in-tree method being
removed in CNPG v1.30, and the live cluster already has a plugin-method
ScheduledBackup that would be duplicated on re-apply).
Verified no other consumers of knoe-db-daily in the repo (grep clean).
File is now SA-only, matching its filename's intent.
Part 1/2 of queue item #6 fix (docs/plans/junie/06-patch-garage-script-fixes.md).
Co-authored-by: Junie <junie@jetbrains.com>
