When SERVICE_NAMESPACE was absent from the config file and OS env,
_get_script_env() fell back to env["NAMESPACE"] — the database namespace
(e.g. knoe-db). InitializationScriptsMilestone then called init_kong.sh
with SERVICE_NAMESPACE=knoe-db, causing Kong to deploy into the DB
namespace and collide with the existing knoe-system/svc-knoe-ingress.
Fix: default to "knoe-system", matching _service_namespace() in actions.py.
Configs that set SERVICE_NAMESPACE explicitly are unaffected.
Add tests/test_deployment_mode_isolation.py to guard against k3d/k3s/gke
mode-specific config bleeding into each other: SERVICE_NAMESPACE fallback,
monitoring storage class separation, Kong gitea host gating, and
_service_namespace() across all three deployment modes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three issues caused the purge-and-reinstall loop:
1. _purge_broken_monitoring deleted ALL PVCs including Bound ones; grafana's
working PVC was wiped on every retry. Now only delete PVCs in Pending state.
2. Broken-state detection keyed on Pending pods + unbound PVCs, which is true
during any still-converging install (including ones that timed out but
whose pods eventually came up). Gate on helm status=='failed' + unbound PVCs.
3. k3d install used --wait, which blocks on all kube-prometheus-stack components
(prometheus, alertmanager, node-exporter). They converge async after the
Prometheus operator starts; --wait always timed out. Drop --wait for k3d;
the status_common_services.sh check verifies readiness independently.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
kube-prometheus-stack pulls multiple large images on a fresh k3d install;
600s / 10m is not enough. Set --timeout 20m and process timeout 1320s
(20m + 2m buffer) for k3d, keep 10m for k8s/k3s where images are warm.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs kept prometheus-grafana-0 Pending for 21 days in k3d:
1. update() routed k3d through _values_yaml_k3s() which hardcodes
merlin-local-iscsi-* storage classes and merlin.prole.org node affinity
— neither exists in k3d, so grafana's PVC could never bind.
Add _values_yaml_k3d() that uses local-path (or MONITORING_STORAGE_CLASS
override) with no node affinity; dispatch k3d separately in update().
2. No broken-state recovery for non-k8s mode. Add _purge_broken_monitoring()
(helm uninstall + delete stuck PVCs) and call it from update() when
Pending pods and unbound PVCs are both detected in the monitoring namespace.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The shell path (init_registry.sh apply_registry) is the real k3d registry
entry point. It checked 'k3d registry list knoe-registry' which returned 0
because k3d-knoe-registry existed in 'created' state, so the create was
skipped and k3d-prole-registry (the port-5000 squatter) was never touched.
- init_registry.sh: before the knoe-registry ensure, detect k3d-prole-registry
via both k3d and docker inspect, then stop/rm the Docker container regardless
of whether k3d manages it; also detect knoe-registry in non-running state and
nuke+recreate it (port was unavailable on the previous attempt)
- k3d_registry.py: same cleanup logic for the Python fallback path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k3d-prole-registry was squatting on port 5000, leaving k3d-knoe-registry
in created state. Two bugs made this invisible:
- k3d_registry.py: substring match (name in stdout) matched k3d-knoe-registry
as already-exists without checking STATUS=running; add _k3d_registry_running()
requiring last column == running, detect and delete prole-registry before
creating knoe-registry, recreate if found in non-running state
- status_common_services.sh: grep -qx 'knoe-registry' (exact) never matched
k3d prefix 'k3d-knoe-registry'; fix with awk suffix match + STATUS==running
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k3s manifests still used prole-svc-kong after the rebrand; init_kong.sh
was waiting on `rollout status deployment/knoe-svc-kong` which never
existed, hanging the Common Services milestone.
- Rename kong-deployment.yaml and kong-service.yaml: prole-svc-kong →
knoe-svc-kong (labels, selector, configmap volume ref)
- Remove prole-svc-kong-configmap.yaml (static file replaced by
init_kong.sh dynamic ConfigMap generation; hardcoded namespace and
prole-branded routes were dead weight)
- init_kong.sh: add cleanup_legacy_prole_kong() called from action_update()
to remove stale prole-era resources before deploying knoe-svc-kong
- init_kong.sh: add _is_host_claimed_by_other_ingress() helper; set
include_gitea_host=0 for k3d mode (git is port-forward only, no public
hostname) — keeps the safety-net pre-check for k3s
- actions.py: extend legacy namespace dedupe to sweep prole-svc-kong from
both default and service_ns
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Junie's session targeted the prompt "stabilize ./install.py -c conf/k3d.cfg
using strict TDD" — broad installer-side work, not the k3d-mirror Phase 3
brief I had filed (which she didn't pick up; phase-3 brief stays open). All
750 installer tests pass post-change.
What Junie produced:
install.py (NEW) Top-level CLI entry point. Was
imagined by the prompt but didn't
exist; this commit makes it real.
knoe/deployment.py (NEW) `KnoeDeployment` orchestrator for
the k3s service-mode deploy pipeline.
Wraps Ansible kubeconfig fetch,
opentofu apply, init_*.sh post-apply
scripts, and (optionally) supabase/
deploy.sh.
knoe/ui/screens/cluster.py Dual-cluster GKE kubecontext UI: prod env
knoe/ui/screens/cfg.py now shows separate "App Cluster:" and
"DB Cluster:" dropdowns instead of a
single "Kubernetes Context:" combo.
New _app_kubectx_combo + _db_kubectx_combo
widgets; new app/db_cluster_kubecontext
tk.StringVars.
knoe/core/{actions,env,milestones}.py
knoe/core/ops/storage.py
knoe/config.py, knoe/knoe_conf.py Plumbing changes for the dual-cluster
kubecontext flow + storage-class topology
detection cleanup.
knoe/tools/cleanup_cnpg_storage.py (NEW) Stand-alone cleanup utility.
tools/dashboard.sh (NEW) Dashboard helper.
conf/knoe.cfg (NEW) Master cfg generated by knoe_conf.
conf/dev/ (NEW) Dev-mode cfg directory.
conf/port-mapping.cfg Port mapping tweaks for k3d.
tests/installer/* (8 files) New + extended tests for the dual-cluster
tests/test_database_options.py TUI, kubecontext save flow, storage ops,
topology detection, deploy helpers,
database-options screen.
Issues found in Junie's working state and fixed here:
1. install.py was a 11-line import shim with no shebang, no `chmod +x`,
no `if __name__ == '__main__'` block. `./install.py -c conf/k3d.cfg`
returned `Permission denied` and `python install.py` did nothing.
Added `#!/usr/bin/env python3`, `chmod +x`, and a __main__ block
that delegates to `knoe.ui.screens.main()`. `./install.py --help`
now prints the canonical argparse help.
2. knoe/deployment.py had FIVE `subprocess.run()` call sites with no
`timeout=` argument (`_run_script`, `_run_cmd`, the Ansible playbook
fetch, `tofu init`, `tofu apply`). A hung child process — typical
failure mode is a script waiting on stdin or a stalled network
call — would lock up the installer indefinitely. Added timeouts:
- Ansible kubeconfig fetch: 120s
- tofu init: 300s
- tofu apply, _run_script, _run_cmd: bounded by new module
constant `_MILESTONE_TIMEOUT` (default 1800s = 30 min, override
via `KNOE_MILESTONE_TIMEOUT_SECONDS` env var).
`subprocess.TimeoutExpired` is caught explicitly; on timeout the
run helpers return exit code 124 (conventional timeout code).
3. `conf/k3d.cfg` was corrupted with MagicMock string-reprs on disk:
KNOE_CONF = <MagicMock name='Canvas().tk.call().strip()' id='4743999712'>
argocd.node_selector = <MagicMock name='mock.StringVar().get().strip()' id='...'>
Likely path: Junie ran `./install.py -c conf/k3d.cfg` interactively
in a non-Tk environment (or with a partially-mocked widget set) and
the installer's "save current state" path wrote the mock-objects'
`__repr__` strings into the cfg file. This commit reverts the cfg
to its pre-Junie state. **Followup: harden the cfg save path
against non-string widget values** — track separately.
4. The corrupted cfg caused the installer to call `os.makedirs()` on
the mock-string values, producing 10 directories on disk literally
named `<MagicMock name='Canvas().tk.call().strip()' id='4733210304'>/`
etc., with 5–86 files of install artifacts inside each. Removed.
The "final step is timing out" the user reported was almost certainly
issue #2 above: install.py walked the milestone pipeline, hit one of
the unbounded subprocess.run calls, and the wrapped command (probably
supabase/deploy.sh, which Junie was reading for context when her
session timed out) hung. With the timeouts in place that path now
exits cleanly with rc=124 instead of locking up.
Verification:
- pytest tests/installer/ -q 750 passed in ~25s
- python3 -c "import knoe.deployment" imports clean
- ./install.py --help prints argparse help
- find . -maxdepth 1 -type d -name '<MagicMock*' | wc -l 0
- head -7 conf/k3d.cfg clean (no MagicMock)
Out of scope for this commit (followups):
- The cfg save-path that wrote mock-objects-as-strings (issue #3 root cause).
Reproducer: launch the installer in an env where Tk widget vars are
`unittest.mock.MagicMock` instances. The cfg save code should refuse to
serialize non-str values rather than calling `str()` on a MagicMock.
- The k3d-mirror Phase 3 brief (`docs/plans/junie/k3d-knoe-auth-pod-deploy.md`)
is still open — Junie picked a different prompt this round.
Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 (host loop) and the Phase 2 OIDC sandbox are shipped; this
brief queues Phase 3 of k3d-mirror-of-GKE: build the knoe-auth image,
k3d-import, run as a pod inside the cluster. Pre-merge smoke loop —
not the daily inner loop.
Deliverables (see brief for the full Definition of done):
- k8s/knoe/knoe-auth-deployment.yaml (NEW; sibling of GKE manifest)
- scripts/k3d-knoe-deploy.sh (NEW; build + import + apply + wait)
- Makefile k3d-knoe-{deploy,redeploy,undeploy} targets
- Phase 2 signing key flows from etc/secrets/knoe-auth-oidc-key.b64 into
a knoe-auth-oidc-signing-key K8s Secret in knoe-system, mirroring how
the GKE deploy reads it.
Index updates:
docs/plans/junie/README.md — k3d Phase 3 in Active row
docs/TODO.md §"In progress" — promotes the brief to top
docs/plans/k3d-gke-mirror.md — banner + §6 Phase 3 entry
flipped from "out of scope"
to "in flight"
Three independent lines of work landing in one commit because they were all
on disk together end-of-session and the cross-file edits (TODO, knoe-system,
junie/README) interleave cleanly:
1. **Queue #3 — image rename `knoe-authority` → `knoe-auth`** (Junie).
Closes drift R6. The Maven artifact has been `knoe-auth.jar` since
commit b355855; the deploy manifests now match.
authority/Dockerfile.app (NEW)
deploy/gcp/gke/knoe-auth-deployment.yaml (3 image tags renamed)
deploy/opentofu/k3s/manifests/knoe/knoe-auth-deployment.yaml (2 image tags renamed)
Makefile (docker-build-auth +
docker-push-auth + REGISTRY/KNOE_AUTH_VERSION
defaults)
docs/plans/junie/03-image-rename-knoe-authority-to-knoe-auth.md (brief, kept as design record)
2. **Phase 2 OIDC provider — GKE deploy** (Junie). Source landed via the
merge that brought claude/crazy-bose-fec256 back; the k3d sandbox
shipped earlier today (commit 93157b0). This commit completes the GKE
path: Kong route `/auth` → knoe-auth, default values added, signing
key wired via `knoe-auth-oidc` secret per the brief.
supabase/helm/knoe-supabase/templates/kong/config.yaml (Kong route /auth, gated
on .Values.knoeAuth.enabled)
supabase/helm/knoe-supabase/values.yaml (knoeAuth.enabled=false default;
studioIngress block — also
fixes the helm lint issue noted
in earlier commit 03bb731)
docs/plans/junie/phase2-oidc-gke-deploy.md (brief, kept as design record)
3. **k3d dev-user seed: `chrisfu` + `knoe_developer`** (Claude). Closes
"I want auth as chrisfu@knoey.com to my local cnpg database" for the
local dev loop. `etc/init_knoe_auth.sh --mode k3d` now runs
`seed_dev_users_k3d()` after the schema bootstrap, creating:
- `knoe_developer` group role with R/W on `knoe`+`public` (mirrors
the GKE production layout from docs/db-access.md; was hand-rolled
in production per the 2026-04-30 onboarding work, never baked into
postInitTemplateSQL).
- `chrisfu` LOGIN role with password `chrisfu-dev` (idempotent —
resets on every `make k3d-knoe-up`, so the rebuild loop is
deterministic). Granted into `knoe_developer`.
From the host with port-forward up:
PGPASSWORD=chrisfu-dev psql "postgresql://chrisfu@localhost:5432/knoe-db?sslmode=require"
Cross-cutting doc updates:
docs/TODO.md — Phase 2 GKE entry removed from "In progress"
(now "(none — all items shipped or paused)");
queue #3 + reality table R6 removed; pg_oauth
paused note flipped to "OIDC issuer now
deployed to GKE — pg_oauth can resume"; Done
section gets new entries for #3, Phase 2 GKE,
and chrisfu seed.
docs/knoe-system.md — Phase 2 GKE row → Shipped; pg_oauth → Ready;
knoe-auth-deployment.yaml note "knoe-auth:latest"
instead of "knoe-authority:latest"; "One-time
cluster setup" mentions the chrisfu seed.
docs/local-dev-knoe-auth.md — "Verify psql connectivity" rewritten to
connect as chrisfu (was `knoe`); new "Reset /
rebuild loop" section explaining idempotency.
docs/plans/junie/README.md — two new rows under Shipped: brief 03 and
phase2-oidc-gke-deploy.
Verification (the bits I ran locally):
- bash -n etc/init_knoe_auth.sh OK
- The seed SQL is idempotent (DO blocks with EXISTS checks +
ALTER ROLE on the password reset path)
- The Kong /auth route is gated on .Values.knoeAuth.enabled — default
false, no behavior change until someone flips it on per Junie's brief
Out of scope for this commit:
- Actually flipping knoeAuth.enabled=true on the live GKE chart and
rolling out — Junie's brief covers the runbook; needs the
knoe-auth-oidc K8s secret populated from 1Password first
- Rebuilding + pushing the new knoe-auth:latest image — `make
docker-push-auth` is wired, just hasn't been run yet
- Round 1.5 OpenBao transit-key encryption (still queued)
Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Completes the Phase 2 OIDC laptop dev path. Source for the OIDC surface
(discovery, authorize, token, userinfo, JWKS controllers + signing /
session services) landed on `main` via the merge that brought
claude/crazy-bose-fec256 back. This commit makes Phase 2 actually
exercisable in the k3d dev loop without any GKE deploy.
What's new:
authority/src/main/resources/application-k3d.yml
Spring profile activated by `-Dspring-boot.run.profiles=k3d`.
Enables OIDC (knoe.oidc.enabled=true), points the issuer at
`http://localhost:8080`, sets Kerberos realm to KNOE.LOCAL, and
aligns the datasource with the port-forwarded localhost:5432 DB.
etc/gen_oidc_signing_key.sh (executable)
Idempotent RS256 PKCS#8 keypair generator. Outputs:
etc/secrets/knoe-auth-oidc-key.pem (PEM)
etc/secrets/knoe-auth-oidc-key.b64 (single-line base64 of DER —
directly consumable as KNOE_AUTH_OIDC_SIGNING_KEY by
OidcTokenService.init())
`etc/secrets/` is already gitignored. Set FORCE=1 to rotate.
What's wired:
scripts/k3d-knoe-up.sh
New §7 calls etc/gen_oidc_signing_key.sh after schema seed. Keypair
persists across `make k3d-knoe-down && make k3d-knoe-up` cycles.
scripts/k3d-knoe-pf.sh
Output now includes the KNOE_AUTH_OIDC_SIGNING_KEY export line, the
full `mvn spring-boot:run` invocation with -Dspring-boot.run.profiles=k3d,
and the three OIDC endpoints to curl-test.
What's documented:
docs/local-dev-knoe-auth.md
"Daily loop" Terminal B: now exports KNOE_AUTH_OIDC_SIGNING_KEY,
runs with `-Dspring-boot.run.profiles=k3d`, and the verify section
includes /jwks.json. IntelliJ run config: adds Active Profiles: k3d
and a note about pasting the b64 directly (no shell expansion in
the env-var field).
docs/knoe-system.md
Phase 2 status row split: "k3d setup" → Shipped, "GKE deploy" →
Pending. The "Open work items" Phase 2 entry rewritten to flag
that the GKE deploy is the remaining thread (gated on queue #3 for
the image rebuild as `knoe-auth:latest`).
docs/TODO.md
Promoted "Phase 2 OIDC provider — GKE deploy" into §In progress
(replacing the empty "(none)" placeholder). Done section updated
with two entries: the k3d Phase 1 dev loop (Junie's c3...) and
this Phase 2 OIDC k3d sandbox.
End-to-end loop the engineer can run:
make k3d-knoe-up # one-time, ~5 min
make k3d-knoe-pf & # port-forwards
export KRB5_CONFIG=$PWD/etc/krb5.local.conf
export KNOE_AUTH_OIDC_SIGNING_KEY=$(cat etc/secrets/knoe-auth-oidc-key.b64)
mvn -pl authority spring-boot:run \
-Dspring-boot.run.jvmArguments="-Djava.security.krb5.conf=$PWD/etc/krb5.local.conf" \
-Dspring-boot.run.profiles=k3d
# then:
curl -s http://localhost:8080/.well-known/openid-configuration | jq .issuer
# → "http://localhost:8080"
curl -s http://localhost:8080/jwks.json | jq '.keys[0].kty'
# → "RSA"
Verified locally: keypair generator round-trips through openssl pkey -inform DER
(produces valid 2048-bit RSA keys); idempotent (existing key kept by default,
FORCE=1 rotates); bash -n clean on all 5 touched scripts.
Out of scope (TODO §In progress captures it):
- GKE deploy of Phase 2 (image rebuild + K8s Secret + deployment env vars)
- SPNEGO E2E from host browsers (k3d-mirror Phase 2)
- knoe-auth-as-pod in k3d (k3d-mirror Phase 3)
- OidcCodeService DB persistence (separate track)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings up the smallest k3d-resident stack that lets a host-side knoe-auth
(run via `mvn spring-boot:run` or IntelliJ) iterate against real Postgres
+ Kerberos. Closes Phase 1 of the k3d-gke-mirror plan (docs/plans/k3d-gke-mirror.md).
Scope:
- k8s/knoe/knoe-kdc-{configmap,deployment,service,pvc,init-job}.yaml
NEW; standalone KDC, realm KNOE.LOCAL (distinct from KNOE.DEV).
- etc/init_knoe_auth.sh: --mode k3d flag added; swaps realm + skips
GCP-specific steps. GKE behavior unchanged when flag absent.
- Makefile: k3d-knoe-up, k3d-knoe-pf, k3d-knoe-down (delegate to
scripts/k3d-knoe-{up,pf,down}.sh).
- scripts/k3d-knoe-{up,pf,down,smoke}.sh NEW; up = full bring-up,
pf = three port-forwards (5432/88/464) + JDBC URL + ^C cleanup,
down = teardown, smoke = sanity check.
- etc/krb5.local.conf NEW; checked-in libdefaults+realms config
pointing at localhost:88. udp_preference_limit=1 to dodge
kubectl port-forward UDP flakiness on macOS.
- docs/local-dev-knoe-auth.md NEW; one-time setup + daily loop +
IntelliJ run config.
- docs/knoe-system.md NEW; unified reference for the knoe-auth
service (GKE deployment + k3d dev loop + schema overview +
source map + open work items).
Verified per the brief's Definition of done: fresh-clone laptop can
`make k3d-knoe-up` + `make k3d-knoe-pf` + `mvn -pl authority spring-boot:run`
and hit /health, /.well-known/openid-configuration in <8 minutes.
Out of scope (parent plan docs/plans/k3d-gke-mirror.md §6):
- SPNEGO from host browsers (Phase 2)
- knoe-auth-as-pod / image build/load (Phase 3)
- Supabase stack on k3d (Phase 4)
- OidcCodeService DB persistence (separate track)
docs/plans/junie/README.md — k3d brief moved from Active to Shipped.
docs/TODO.md — In-progress now empty; Phase 2 pg_oauth notes that the
local dev loop is in place so it can resume.
Closes Phase 1; Phase 2+ briefs filed as needed.
Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The next development thread is knoe-auth Phase 2 OIDC iteration, which
just landed on main but has no fast inner-loop. Instead of bouncing
edits through the GKE deploy chain, we want a laptop-resident model
where knoe-auth runs from the IDE / mvn against real Postgres + KDC
brought up in k3d.
Two new docs:
docs/plans/k3d-gke-mirror.md
Architectural plan. Top-to-bottom rationale for what the k3d
model should mirror, what we deliberately skip (supabase, kong,
oauth2-proxy, Studio — all unnecessary for knoe-auth's surfaces),
and the phasing. Captures the three scope decisions taken
2026-05-02 with the user:
- knoe-auth runs on the laptop, not as a pod (faster inner loop)
- stack scope is CNPG + KDC only
- single-replica CNPG, realm KNOE.LOCAL (distinct from KNOE.DEV)
Out-of-scope for Phase 1 listed explicitly so reviewer
expectations match scope: SPNEGO E2E, image-build-and-load,
supabase, oauth2-proxy, OidcCodeService DB persistence.
docs/plans/junie/k3d-knoe-auth-dev-loop.md
Phase 1 brief. Concrete deliverables:
- k8s/knoe/knoe-kdc-{deployment,service,configmap,pvc,init-job}.yaml
- etc/init_knoe_auth.sh --mode k3d flag (or sibling script)
- Makefile: k3d-knoe-{up,pf,down}
- etc/krb5.local.conf (with udp_preference_limit=1 for
kubectl port-forward UDP flakiness on macOS)
- docs/local-dev-knoe-auth.md (engineer-facing)
- scripts/k3d-knoe-smoke.sh
Definition of done: a fresh-clone laptop can `make k3d-knoe-up`
+ `make k3d-knoe-pf` + `mvn -pl authority spring-boot:run` and
hit /health, /.well-known/openid-configuration in <8 minutes.
Index updates:
docs/plans/README.md — k3d-gke-mirror.md row added
docs/plans/junie/README.md — split into Active / Shipped;
k3d brief listed under Active;
the 5 shipped-2026-05-02 briefs
moved to Shipped with commit refs.
docs/TODO.md In-progress — k3d brief now top of "In progress";
the previously in-flight "Phase 2
pg_oauth in install/deploy.sh"
item demoted to "paused" (resume
after the local dev loop lands).
No code changes; all docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirroring the 2026-04-29 DB-cluster removal. Live state showed garage
in an unconfigured limbo (layout v0, no role assigned, 21d of "Ring
not yet ready" warnings); nothing outside garage's own state
referenced its S3 endpoint anymore.
Removed from knoe-system namespace on knoe-dev-0:
- statefulset/garage (1/1 pod, 21d uptime)
- service/garage (ClusterIP; no garage-s3-ilb on app cluster)
- configmap/garage-config
- secret/garage-secrets
- pvc/data-garage-0 (29 Gi, garage-hdd)
- pv/pvc-656e1936-... (was Bound, reclaim=Retain)
- storageclass/garage-hdd
Underlying GCE pd-standard disk (us-west3-c/pvc-656e1936-...) reclaimed
via `gcloud compute disks delete` since reclaim=Retain leaves the disk
behind after the PV is gone.
Source manifests intact:
- k8s/knoe/garage-*.yaml (k3d, min)
- deploy/opentofu/k3s/manifests/knoe/garage-*.yaml (k3s)
- etc/init_garage_store.sh + knoe/core/ops/k8s_garage_store.py
- k8s/knoe/garage-statefulset-gcp.yaml (GKE variant)
A future ./install.sh against GKE could re-deploy garage if that's
what the deploy mode wires; whether it SHOULD wire garage given GCS
now handles backups is a separate cleanup decision.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Queue #5 Phase B orphan cleanup landed:
App cluster (knoe-dev-0):
- gitlab/gitlab-migrations-58a3e27-97-23-n72j8 (Failed/Evicted,
11d stale)
- pv/pvc-42e8545b-… (10Gi, monitoring/storage-kps-grafana-0)
- pv/pvc-86422179-… (30Gi, monitoring/prometheus-…-prometheus-0)
- pv/pvc-e2c29f7a-… (5Gi, monitoring/alertmanager-…-alertmanager-0)
DB cluster (knoe-dev-cnpg-0):
- kube-system/prole-supabase-dirprep Job (FailureTarget, 22d stale)
Post-state: 0 Released PVs on app cluster; gitlab namespace has only
the Completed migration pod; DB-cluster kube-system has no prole-*
artifacts. Live-cluster ops only; no source changes.
Queue #4 reframed: tried the planned `garage repair --yes blocks`,
but found garage in an unconfigured state — layout v0, no role
assigned, "Ring not yet ready" warnings continuous since whenever the
layout was lost (likely during a reconcile after the 2026-04-29 PVC
resize). Cross-cluster sweep shows nothing outside garage references
its S3 endpoint anymore (CNPG backups moved to GCS, no other workload
points at 10.180.15.239:3900). Garage on app-cluster is now a dead
pod holding an 80 GiB PVC. The original block-repair task is moot.
Reframed as a decommission-or-restore decision; details captured in
the ranked queue entry. Awaiting direction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a 2026-05-02 Done entry covering the cluster-side work that paired
with c3fae73 (source) and 03bb731 (chart cleanups): PodMonitor migration
applied with verified zero-data-loss handoff; supabase chart upgraded
revision 1→3; helm field-manager conflict on Kong configmap resolved;
node label drift on knoe-dev-0 closed (3 nodes labelled
knoe.dev/node-role=general alongside legacy prole.org/node-role); DB
cluster nodes still pending same treatment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two follow-ups to commit c3fae73 (queue #15) surfaced when applying
the DASHBOARD removal to live `supabase` release on knoe-dev-0:
1. **Delete `templates/test/kong.yaml`.** This was a Helm Test Job that
validated the Studio dashboard's basic-auth gate. The basic-auth
plugin was retired in commit 25f1b2e (oauth2-proxy fronting); the
consumer + credentials in commit c3fae73. The test has nothing
left to test — it referenced `supabase.secret.dashboard` template
(gone) and `.Values.secret.dashboard.{secretRef,secretRefKey}`
(gone), causing `helm template`/`upgrade` to fail. Removing the
file is the right move; if dashboard auth ever comes back as
something else, write a fresh test for that.
2. **Add `secret.oauth2proxy` + clarify `deployment.oauth2proxy`
defaults in values.yaml.** The chart has
`templates/studio/oauth2proxy-{deployment,service}.yaml` and
`templates/secrets/oauth2proxy.yaml` (Junie's Phase 2 OIDC adds)
guarded by `if .Values.deployment.oauth2proxy.enabled`. The guard
short-circuits when the parent map exists, but if the map itself
is undefined helm fails with `nil pointer evaluating interface{}.enabled`.
Added a `secret.oauth2proxy` block (empty defaults) so the secrets
template guard can evaluate cleanly. `deployment.oauth2proxy.enabled`
was already `false` in values.yaml; added a comment explaining the
chart oauth2-proxy templates are vestigial — the deployed
oauth2-proxy lives in `deploy/gcp/gke/oauth2-proxy-deployment.yaml`,
in FRONT of supabase-kong (commit 25f1b2e), not as a Kong upstream.
Live state after this commit:
- `helm upgrade supabase ./supabase/helm/knoe-supabase
--reset-then-reuse-values` succeeds.
- revision 3 deployed; configmap field manager = `helm` (sole owner).
- `kubectl get cm supabase-kong -o yaml | grep DASHBOARD` returns 0
matches.
- All Kong env vars on the new pod are clean (no DASHBOARD_*).
- External smoke tests:
/onboard.html → 200
/rest/v1/ no auth → 401
/ (Studio gate) → 302 to Google
/healthz → 302
svc.knoe.dev/grafana/login → 200
- cnpg-grafana panels still populate (PodMonitor migration verified
earlier in same session: 3 postgres exporter targets up; cluster
label injection working).
Side-finding handled live: nodes lacked the new
`knoe.dev/node-role=general` label that Junie's chart defaults to in
nodeAffinity. Added the label to all 3 knoe-dev-0 nodes alongside the
legacy `prole.org/node-role=general`. The label rename has been
tracked in CLAUDE.md as a follow-up; this is partial closure for the
app cluster (DB cluster nodes still need the same treatment when next
relevant).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three Junie briefs landed in one commit:
#07 — Wire cnpg-backup-sa into CNPG cluster spec (drift R8)
deploy/gcp/gke/knoe-db.yaml: add spec.serviceAccountName: cnpg-backup-sa
(requires CNPG v1.29+, which is the live operator version).
etc/init_cnpg_gke.sh: operator install URL now uses CNPG_OPERATOR_VERSION
variable (default 1.29.0); new §11 patches knoe-db and
knoe-db-barman-cloud RoleBindings to add cnpg-backup-sa as a subject
if not already present — matching the 2026-04-29 live stabilization.
#13 — Migrate off deprecated enablePodMonitor + podMonitorRelabelings
Both deprecated fields removed from deploy/gcp/gke/knoe-db.yaml
spec.monitoring. New deploy/gcp/gke/knoe-db-podmonitor.yaml carries
the PodMonitor with the cluster relabeling rule (cnpg.io/cluster pod
label → cluster label; required for all 85 CNPG Grafana panels).
Apply alongside knoe-db.yaml on next cluster patch.
#15 — Remove dead DASHBOARD consumer + basicauth_credentials
supabase/helm/knoe-supabase:
- wrapper.sh: drop DASHBOARD_USERNAME / DASHBOARD_PASSWORD envsubst lines
- config.yaml: drop DASHBOARD consumer + basicauth_credentials block
- kong/deployment.yaml: drop both DASHBOARD env-var secret refs
- values.yaml: rename secret.dashboard → secret.openai (apiKey only;
username/password dropped — no enforcer since commit 25f1b2e)
- secrets/dashboard.yaml + _helpers.tpl: renamed to openai /
supabase.secret.openai
- studio/deployment.yaml: reads from secret.openai.apiKey
- ci/example.yaml: updated to secret.openai.apiKey
helm template confirms knoe-supabase-openai secret referenced; no
DASHBOARD output.
docs/TODO.md: queue items #7, #13, #15 + drift row R8 archived to Done.
Co-authored-by: Junie <junie@jetbrains.com>
Defects A, B, C from docs/plans/junie/06-patch-garage-script-fixes.md:
A. DB_CLUSTER default was knoe-cnpg-0 (stale rebrand artifact);
corrected to knoe-dev-cnpg-0 to match conf/gke.cfg and the
cluster table in CLAUDE.md. Header comment + overrides block
also updated.
B. Phase 1 deletion loop missed service/garage-s3-ilb (the
LoadBalancer the ObjectStore endpoint historically pointed at).
Now included alongside service/garage.
C. Phase 2 was applying knoe-db-backup-gcs.yaml whole, including a
legacy ScheduledBackup using method:barmanObjectStore (being
removed in CNPG v1.30). Manifest split handled in prior commit
(34a25dd); script's Phase 2 is now safe to re-run.
bash -n clean; CONFIRM=false dry-run prints knoe-dev-cnpg-0 correctly.
docs/TODO.md: queue items #2 and #6 + drift rows R5 and R9 archived to Done.
Closes queue item #6 in docs/TODO.md (drift R9).
Co-authored-by: Junie <junie@jetbrains.com>
deploy/gcp/gke/knoe-db-backup-gcs.yaml previously contained both the
cnpg-backup-sa ServiceAccount (wanted) and a ScheduledBackup using
method: barmanObjectStore (unwanted — legacy in-tree method being
removed in CNPG v1.30, and the live cluster already has a plugin-method
ScheduledBackup that would be duplicated on re-apply).
Verified no other consumers of knoe-db-daily in the repo (grep clean).
File is now SA-only, matching its filename's intent.
Part 1/2 of queue item #6 fix (docs/plans/junie/06-patch-garage-script-fixes.md).
Co-authored-by: Junie <junie@jetbrains.com>
Mirrors commit b355855 (auth-side rename). The kustomization at
deploy/opentofu/k3s/manifests/knoe/kustomization.yaml has been
referencing knoe-* names since the rebrand; kustomize was broken
on the k3s deploy mode until this lands.
Five git mv:
prole-configmap.yaml -> knoe-configmap.yaml
prole-deployment.yaml -> knoe-deployment.yaml
prole-service.yaml -> knoe-service.yaml
prole-kdc-configmap.yaml -> knoe-kdc-configmap.yaml
prole-kdc-secrets.example.yaml -> knoe-kdc-secrets.example.yaml
Plus content sweep of knoe-kdc-configmap.yaml: replaced stale
PROLE.LOCAL/PROLE.ORG realm refs and PROLE_KDC_* env vars with
KNOE.DEV/KNOE_KDC_* to match the canonical GKE configmap.
Closes queue item #2 in docs/TODO.md (drift R5).
Co-authored-by: Junie <junie@jetbrains.com>
Bringing the long-running session-feature branch back into main in one
deliberate sweep. The branch carried the cluster work that's been live for
weeks (cross-cluster CNPG metrics, Grafana w/ Google OAuth, supabase
oauth2-proxy, cluster recovery, pg.0.knoe.dev + per-engineer onboarding,
GCS-backed CNPG backups via Workload Identity, the env-contamination
guard, the Junie brief queue, the cnpg-grafana CSRF + memory-request
fixes from today), while main accumulated Junie's parallel knoe-auth
Phase 2 OIDC work (full provider surface: discovery, authorize, token,
userinfo, JWKS, RS256 signing, code exchange, session services).
Key decision: the two branches did COMPETING rebrands off the same
starting point (5ba9b63, 2026-04-27):
- claude branch (commit b355855, earlier): org.prole.authority.* →
dev.knoe.auth.*
(artifact renamed to
knoe-auth.jar)
- main (commit 9daa94b, recent): org.prole.authority.* →
dev.knoe.authority.*
(kept "authority" artifact name)
dev.knoe.auth wins: cluster runs from this name, the Maven artifact is
already knoe-auth.jar, and the broader rename is the documented
namespace direction (per ~/.claude/projects/-Users-chrisfu-dev-knoe-db/
memory/MEMORY.md). All of main's recent Phase 2 OIDC content was ported
from authority/src/.../dev/knoe/authority/ into
authority/src/.../dev/knoe/auth/ with package declarations rewritten.
== File-level resolution summary ==
Textual conflicts (4):
authority/pom.xml
- Took our artifactId="auth"
- Took our branch's removal of spring-security-kerberos-client
(verified: Junie's Phase 2 OIDC code does not import it; the dep
was already-dead config)
docs/pipeline-phases.md
- Took our branch's "Phase 1 not started" status. Main had a
misplaced "✅ Complete" with a knoe-auth-Phase-1 commit ref
in the autobuild Phase 1 section — different domain.
docs/plans/knoe-auth-round-1.md
- Took our branch's dev.knoe.auth file table (vs main's
dev.knoe.authority listing). Pure rename mismatch.
supabase/helm/knoe-supabase/templates/kong/config.yaml
- Took our branch's onboard route + plain dashboard wiring.
Main had an oauth2proxy.enabled toggle that put oauth2-proxy as
a Kong upstream — but the deployed architecture (commit 25f1b2e)
has oauth2-proxy in FRONT of Kong, not behind. Main's wrapper
reflected an architecture that was never deployed.
- Took our branch's removal of basic-auth from dashboard route
(queue #15 brief still tracks the matching values.yaml /
kong/deployment.yaml cleanup).
Java tree reconciliation (44 file-pairs):
20 dual-path source files + 2 dual-path tests
Body-identical between main's authority/ and our branch's auth/
after stripping package decls — main's commit 9daa94b was a pure
rebrand. Took our branch's auth/ version for all 22.
8 main-only source files (Phase 2 OIDC), ported into auth/:
web/JwksController.java
web/OidcAuthorizeController.java
web/OidcDiscoveryController.java
web/OidcTokenController.java
web/OidcUserInfoController.java
session/OidcCodeService.java
session/OidcTokenService.java
session/SessionService.java
12 main-only test files, ported into auth/:
HealthControllerTest.java
enroll/EnrollValueTypesTest.java
enroll/EnrollmentControllerTest.java
enroll/TotpServiceTest.java
kerberos/KadminClientTest.java
kerberos/KerberosSpnegoResultTest.java
web/LoginControllerTest.java
admin/AdminControllerTest.java
user/PrincipalNormalizerTest.java
regression/IdentityRegressionTest.java
session/OidcCodeServiceTest.java
session/SessionServiceTest.java
Port mechanics: read main:authority/...<file> via git show, then sed
rewrite `package dev.knoe.authority` → `package dev.knoe.auth` and
`import dev.knoe.authority` → `import dev.knoe.auth`. Body content
unchanged.
authority/src/main/java/dev/knoe/authority/ — DELETED (duplicate)
authority/src/test/java/dev/knoe/authority/ — DELETED (duplicate)
== Verification ==
- grep -rln '<<<<<<<' across .java/.md/.yaml/.yml/.sh/.xml/.tpl: clean
- find authority/src -path '*/dev/knoe/authority*': empty (subtree gone)
- grep 'package dev.knoe.authority' across repo: clean
- bash -n install.sh deploy.sh etc/preflight_kubecontext.sh: clean
- git ls-files -u | wc -l: 0 unmerged paths
- helm lint supabase/helm/knoe-supabase: pre-existing failure on
studioIngress.enabled undefined in values.yaml (introduced by Junie
on main; unrelated to this merge — flagging as follow-up).
== Followups (carried into TODO ranked queue or noted here) ==
- helm lint failure: studioIngress block in values.yaml is missing
enable flag; templates/studio/{ingress,oauth2proxy-deployment,
oauth2proxy-service}.yaml all reference studioIngress.enabled with
no default. Pre-existing on main; not introduced by this merge.
- The five Junie briefs filed on this branch are now reachable from
main at docs/plans/junie/{02,06,07,13,15}-*.md. Junie can pick them
up in any order.
- knoe-auth Phase 2 OIDC source (now at dev.knoe.auth.*) is not yet
deployed to the cluster. Deployment is its own task.
- The branch claude/crazy-bose-fec256 stays in place (worktree at
.claude/worktrees/crazy-bose-fec256 may have ongoing context for
Claude Code sessions). Safe to delete once next session starts
cleanly from main.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Filed in response to the 2026-04-28 14:00 UTC backup outage. An
`install.sh --mode k3d` run with the shell pointed at GKE silently
overwrote the GKE cluster's GCS-backed ObjectStore + ScheduledBackup
with k3d-mode defaults; Garage filled up and CNPG backups failed for
hours before the next manual check. The class of bug is "config says
target cluster A, shell context says target cluster B, installer
proceeds against B without warning."
New shared bash helper at etc/preflight_kubecontext.sh with two
functions:
- verify_kubecontext_matches_config <cfg-path>
Strict gate. Reads [Global] APP_CLUSTER_KUBECONTEXT from the
config and exits 1 if `kubectl config current-context` differs.
Skipped silently when the config has no baked APP_CLUSTER_KUBECONTEXT
(e.g. fresh k3d.cfg) or when there's no live current-context.
- print_kubecontext_notice
Informational. Prints what's about to be inherited so the user
can abort before the TUI launches if it looks wrong. Never fails.
Wiring:
- deploy.sh sources the helper and calls the strict gate against
${PROLE_DEPLOY_CFG:-conf/gke.cfg} before invoking Python.
Unattended path -> hard refusal on mismatch.
- install.sh sources the helper and calls the informational notice
(gated on not-`--min`) right after entering the local-checkout
branch. The TUI is interactive, so the strict mode-aware gate is
a follow-up once the welcome screen records a mode in
state.inputs.
Bypass for deliberate cross-cluster maintenance:
KNOE_SKIP_KUBECONTEXT_GUARD=true ./deploy.sh
End-to-end verified:
- deploy.sh with current=cnpg-0, gke.cfg=app-0 -> exit 1, clear msg
- deploy.sh with KNOE_SKIP_...=true -> bypasses, prints
"skipping check"
- install.sh --min -> notice skipped
- install.sh (no flag) and install.sh --silent -> notice printed
Doc updates:
- CLAUDE.md §"Env-contamination warning" rewritten to describe the
live guard (was a forward-looking TODO).
- CLAUDE.md drift table row R4 removed; "Closed 2026-05-01" line added.
- docs/TODO.md queue item #1 archived to Done; R4 dropped from the
reality-vs-intent table. Queue numbering retained (no #1 placeholder)
so the docs/plans/junie/<NN>-...md filenames still match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five self-contained work briefs in a new docs/plans/junie/ subdirectory,
each tagged against a numbered item in docs/TODO.md so Junie can take
them independently in any order.
02-k3s-prole-rename.md -> queue #2 (drift R5)
06-patch-garage-script-fixes.md -> queue #6 (drift R9)
07-init-cnpg-gke-sa-wiring.md -> queue #7 (drift R8)
13-podmonitor-manual-management.md-> queue #13
15-remove-dead-dashboard-consumer.md-> queue #15
Each brief follows the same shape: Why -> What changes (concrete file
paths + line numbers + before/after) -> Verification -> Out of scope ->
Commit shape -> Definition of done. The intent is that Junie reads cold
(no shared chat history) and lands the change without escalating
questions.
Also adds:
- docs/plans/junie/README.md describing the convention.
- Row in docs/plans/README.md so newcomers find the subdirectory.
- Brief reference + "Assigned to Junie" tag on each of the five queue
items in docs/TODO.md.
Existing in-progress assignment to Junie (Phase 2 pg_oauth) is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Implement Google OIDC support in Authority module via GoogleOAuthService
- Update AuthProperties and application.yml with OIDC configuration
- Add oidc-setup.md documentation for GKE/Google Cloud setup
- Update etc/init_knoe_auth.sh to handle OIDC secrets and path-B configuration
- Configure knoe-auth-deployment.yaml and gke.cfg for production auth
Co-authored-by: Junie <junie@jetbrains.com>
pytest_all.xml updated by IntelliJ to bind SDK name
'Python 3.14 (knoe-db)' and add PYTHONUNBUFFERED=1.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds authority module to root POM, updates Spring Boot and
dependency versions in authority/pom.xml, application.yml
updated for KNOE.DEV realm defaults.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Java implementation of the knoe-auth identity backbone (Round 1):
authority/.../admin/ user admin REST endpoints (list, deactivate,
reset-totp, role management)
authority/.../enroll/ invite-OTP web enrollment flow — accepts invite
token, creates Kerberos principal via kadmin,
seeds TOTP secret, returns keytab
authority/.../provisioning/ contributor provisioning service — orchestrates
invite generation, principal lifecycle, role grants
authority/.../kerberos/
KadminClient.java typed wrapper around the kadmin(1) subprocess;
create/delete/get-keytab/change-password
GKE manifests:
deploy/gcp/gke/knoe-auth-deployment.yaml two-container Pod (knoe-auth HTTP
+ kdc sidecar) with keytab-bootstrap
initContainer
deploy/gcp/gke/knoe-kdc-configmap.yaml krb5.conf, kdc.conf, kadm5.acl and
entrypoint for the embedded KDC
deploy/gcp/gke/knoe-kdc-secrets.yaml placeholder template — real values
created by init_knoe_auth.sh via 1Password
See docs/plans/knoe-auth-round-1.md for full architectural narrative.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds docs/plans/ as the canonical engineering reference for completed and
in-flight initiatives. Written for jr/mid engineers who have not seen the
repo before — each plan starts with strategic context and links to existing
code before asking for changes.
README.md index, audience, and status conventions
deployment-modes.md four-mode installer (min/k3d/k3s/gke), welcome-screen
mode selector, min-mode fast-path. Status: shipped.
knoe-auth-round-1.md Kerberos KNOE.DEV realm, invite-OTP enrollment,
Google corroboration, TOTP 2FA. Status: operational.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Delete tracked knoe-db/.prole_user_password (dead secret, rotated
in ansible since the original commit)
- Remove postgres-password.txt and build/ copy from working tree
- Add explicit ignore rules for *_password, postgres-password*,
.prole_user_password, *-secret variants
Add docs/plans/customer-deploy-resync.md with the upstream/downstream
migration plan and update CLAUDE.md to reflect prole's role as a
customer deploy of knoe-db.
