Provisions the docker engine on [build_hosts] (gandalf) for the native arm64 daily
knoe-db image build → registry.knoe.dev, run via a remote buildx builder. gandalf is
also a k3s_agent, so the role holds iptables FORWARD=ACCEPT to protect k3s pod
networking (docker's containerd coexists with k3s's on separate sockets). Verified
idempotent: --check against gandalf = ok=5, changed=0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Captured for the knoe-db convergence (Phase 2): configure_kubernetes_auth() (OpenBao
K8s auth → scoped knoe-jobs tokens vs root), the Kong knoe-secret route (secure-dropbox
bridge /secret/* → knoe-jobs:8081), and the prole.org canary-deploy runbook. None of
this is in canonical knoe-db — it is prole-staging-specific, not stale.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The AD DC's smb.conf set no `interfaces` / `bind interfaces only`, so Samba
bound to — and samba_dnsupdate registered into DNS — every interface on the
host. On myrddin that meant the Docker bridge (172.17.0.1) and k3s/flannel
CNI addresses (10.42.0.0, 10.42.0.1) were published as A records for both the
prole.org apex and `myrddin`, alongside the real 10.0.0.3. Clients then
round-robined onto unroutable addresses, producing the long-standing
"resolves, then doesn't" internal DNS flakiness.
Confine Samba to loopback + the LAN service IP:
interfaces = lo 10.0.0.3
bind interfaces only = yes
This scopes both service binding and DNS self-registration to the real
address, so the junk records stop being (re)created on restart.
- smb.conf.j2: emit the two directives, gated on bind-interfaces-only being
enabled AND a non-loopback IP being present (empty -> directives omitted,
never binds loopback-only by accident).
- defaults: samba_ad_dc_lan_ip ("" by default), samba_ad_dc_bind_interfaces_only
(true), samba_ad_dc_interfaces (lo + lan_ip), all documented.
- tasks: assert samba_ad_dc_lan_ip is non-empty before deploying smb.conf
when bind-interfaces-only is on, so a missing value fails fast instead of
rendering the DC unreachable.
- group_vars/ad_dc: set samba_ad_dc_lan_ip=10.0.0.3 (myrddin's LAN address).
Deploying notifies the existing Restart samba-ad-dc handler. Pre-existing junk
records must be deleted once by hand; they will not be re-registered after the
restart. Template rendering verified for both the set and empty-IP cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`Create reverse DNS zones if missing` looped over `samba_reverse_zones`, a fact
built by a separate `set_fact` task that carried no tags. Under a tag-filtered
run (e.g. --tags samba_reverse_dns) that set_fact was skipped, so the variable
was undefined and the play failed:
TASK [samba_reverse_dns : Create reverse DNS zones if missing]
'samba_reverse_zones' is undefined
A set_fact must carry the same tags as every task that consumes it. Rather than
re-add tags (which breaks again under any other tag combination), compute the
zone list inline in the loop and drop the now-dead set_fact. The role is now
correct under any tag selection. lan_reverse_zone and k3s_reverse_zones are
defined in group_vars/all/vars.yml, so they are always available.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Forward A/CNAME and reverse PTR records reported changed=0 / ok but never
resolved by short name on the Samba AD DC (myrddin.prole.org). Root cause was
regex escaping inside YAML folded block scalars (">-"): unlike double-quoted
scalars, block scalars do NOT process backslash escapes, so a pattern written
as '\\.' reached Jinja as a literal backslash + any-char and never matched.
Consequences:
- regex_replace() never stripped the zone suffix, so records were created
under their full FQDN (e.g. a record literally named "sg2428lp.prole.org"
inside zone "prole.org"). `samba-tool dns query ... prole.org sg2428lp A`
then returns WERR_DNS_ERROR_NAME_DOES_NOT_EXIST.
- regex_findall() of existing values always returned [], so stale records
were never pruned and idempotency only survived via RECORD_ALREADY_EXISTS.
The leading/trailing whitespace from the folded scalar was a red herring: the
command module's shlex tokenization collapses it, so a single-token name
survived intact. The escaping was the actual defect.
Fix:
- Convert the templated set_fact expressions from ">-" block scalars to
double-quoted single-line scalars (one consistent escaping convention,
matching the already-working samba_target_zone line; also removes the
stray whitespace).
- Add an assertion that the computed record name is a clean, non-empty,
whitespace-free token.
- Add a post-apply verification re-query + assert that the expected
value(s) are actually present, turning a silent no-op into a hard failure.
Applies to roles/samba_dns (ensure_a.yml, ensure_cname.yml) and
roles/samba_reverse_dns (ensure_ptr.yml).
Verified end-to-end through ansible-playbook against a stateful fake
samba-tool: correct short names (sg2428lp, git, @, registry), stale-value
removal, internal.prole.org zone routing, idempotent re-runs (changed=0),
and the verify-assert failing loudly when an add silently does not persist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
vault_winrm_password_morgoth and vault_winrm_password_fairyland stored for
Ansible WinRM auth to Windows GPU workstations.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pi-hole 6 uses pihole-FTL --config dns.hosts as the authoritative local DNS
store. dnsmasq address= directives still work for new domains (e.g. mcp.0.knoe.dev)
but Pi-hole may serve stale upstream cache over them for previously-resolved
domains (e.g. auth.0.knoe.dev was cached from old DNS before the fix).
dns.hosts entries always win, idempotent approach: strip old *.0.knoe.dev entries,
append new ones, update via pihole-FTL --config + pihole reloaddns.
auth.0.knoe.dev: 34.120.221.5 (wrong) → 34.36.71.141 (knoe-auth GCE LB) ✓
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds dnsmasq address= directives for all active knoe.dev GKE endpoints
so LAN clients (morgana, Pi nodes) resolve them without hitting external DNS.
Records added:
mcp.0.knoe.dev → 8.232.93.14 (knoe-mcp GCE LB)
auth.0.knoe.dev → 34.36.71.141 (knoe-auth GCE LB)
api.0.knoe.dev → 34.120.221.5 (supabase-kong GCE LB)
db.0.knoe.dev → 34.120.221.5 (supabase-kong GCE LB)
app.0.knoe.dev → 34.120.221.5 (supabase-kong GCE LB)
auth.0.knoe.dev was previously resolving to 34.120.221.5 (wrong — supabase LB)
causing the knoe-auth managed cert to be stuck Provisioning for 20 days.
Correct IP is 34.36.71.141. Also update auth.0.knoe.dev in external DNS
(name.com) to unblock the GCP managed cert.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
INSERT ... RETURNING needs RETURN QUERY in PL/pgSQL RETURNS TABLE functions.
tenant_id column is smallint in knoe.tenants; cast to integer to match
the function's declared return type.
Reproduced on pg.prole.org at 2026-05-30 during Phase 2 canary deploy.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Replace the knoe ASCII art banner with a Prole. banner linking to all
four live services (svc/db/git/api.prole.org). Add # Prole. as the
top-level header with a one-paragraph description of the homelab
deployment and its relationship to upstream knoe-db. Demote the Knoe
platform overview to ## Knoe platform.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add infrastructure/logs/ to .gitignore so ansible run logs
(infrastructure/logs/ansible/*.log) are never accidentally committed
- Track docs/plans/junie/ekosystem-uuid-cnpg-wire.md — Junie brief for
wiring the ekosystem UUID schema to the CNPG cluster (queue item #13)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add KNOE_AUTH_BASE_URL=https://api.prole.org/auth to knoe-auth
deployment; LoginController.java defaulted to api.0.knoe.dev causing
redirect_uri mismatch on every Google login attempt
- Update oauth2-proxy-deployment-prole.yaml to reflect live config:
--provider=oidc --oidc-issuer-url=https://api.prole.org/auth (was
stale --provider=google; live deployment already switched to OIDC)
- Fix redirect URIs in knoe-auth-google-prole-secret.example.yaml:
remove spurious double /auth/auth/ prefix (Kong strip_path=false so
Spring @RequestMapping(/auth) gives /auth/callback/google not
/auth/auth/callback/google)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three-part fix to make browser and curl SPNEGO auth work end-to-end:
1. spnego-proxy: issue WWW-Authenticate: Negotiate challenge when no
Authorization header is present so Chrome (with AuthServerAllowlist)
and curl --negotiate automatically present Kerberos tokens. Previously
the proxy only validated tokens if the client proactively sent them.
Pass-through preserved for non-Negotiate schemes (Basic/token) so
git CLI users with PATs continue to work via Gitea own auth.
2. gitea_spnego_keytab.yml: new Ansible playbook that provisions the
gitea-http AD account (AES-only, msDS-SupportedEncryptionTypes=24),
registers SPN HTTP/git.prole.org, resets the password to derive fresh
AES keys, exports the domain keytab, and rekeys it to principal name
HTTP/git.prole.org@PROLE.ORG that gokrb5 needs for keytab lookup.
Key lesson: samba-tool exportkeytab --principal=HTTP/... returns empty;
must export full domain keytab and rekey in Python.
3. init_gitea.sh: add setup_gitea_spnego() calling the Ansible playbook
in k3s mode as part of the standard deploy flow, with inline notes
on every non-obvious constraint discovered during this work.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ansible_become_ask_pass in group_vars does not reliably trigger an
interactive sudo prompt for ansible_connection=local — Ansible cannot
prompt per-host mid-play. The fix is a wrapper script that passes
-K / --ask-become-pass before any play execution.
- infrastructure/bin/install_workstation.sh: auto-detects FQDN, adds
--ask-become-pass, passes extra args through. Run from repo root:
./infrastructure/bin/install_workstation.sh
make workstation
- Makefile: add 'workstation' target pointing to the script
- group_vars/workstations.yml: drop ansible_become_ask_pass (does not
work), keep ansible_pipelining=false (needed for SSH workstations
with sudo passwords — pipelining replaces stdin, breaking sudo -S)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Personal Macs require a sudo password; the global become_ask_pass=False
and pipelining=True combination silently swallows the prompt and fails.
- ansible_become_ask_pass: true → Ansible prompts for sudo password
- ansible_pipelining: false → sudo can read stdin for the password;
pipelining replaces stdin with a pipe
which sudo treats as non-interactive and
refuses to ask for a password at all.
Applies to [workstations] group (mac + linux). Has no effect on
[workstations_windows] (WinRM auth is separate from become).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both ansible.cfg files lacked vault_password_file, causing
'Attempting to decrypt but no vault secrets found' when running any
playbook that auto-loads the encrypted group_vars/all/ files.
- ansible.cfg (root): vault_password_file = .vault_pass
- infrastructure/ansible.cfg: vault_password_file = ../.vault_pass
.vault_pass is gitignored. Each machine needs its own copy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- KerberosSpnegoService: replace Subject.doAs with Subject.callAs (JDK 18+)
so Subject.current() is set correctly for JGSS credential lookup in JDK 21.
Pass null GSSCredential to createContext so the mechanism auto-selects
rather than failing with "No credential found for 1.3.6.1.5.5.2".
- knoe-auth-deployment: fix PROLE_KERBEROS_KEYTAB_PATH to /etc/knoe-auth/http.keytab
(matches the knoe-auth-http-keytab secret volume mount).
- knoe-kdc-configmap: full rewrite from stale KNOE.DEV to KNOE.LOCAL;
add PROLE.ORG realm (Samba AD at 10.0.0.3) and [capaths] for future
cross-realm trust. Drop arcfour-hmac/des3 from permitted_enctypes —
RC4 hard-removed in JDK 21 and service account now AES-only
(msDS-SupportedEncryptionTypes=24). Fix hardcoded secret values in
entrypoint.sh to use ${PROLE_KDC_MASTER_PASSWORD} variable references.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add @RequestMapping(/auth) to all OIDC and login controllers so paths
match the Kong route (strip_path: false keeps /auth in the forwarded path).
Changes:
- authority/web: add @RequestMapping(/auth) to LoginController,
OidcAuthorizeController, OidcTokenController, OidcUserInfoController
- authority/resources: add V1__oidc_authorization_codes Flyway migration
- application-prole.yml: add OIDC clientId/clientSecret env bindings
- knoe-auth-deployment.yaml: add Flyway baseline env vars and
KNOE_AUTH_OIDC_CLIENT_ID/SECRET from knoe-auth-oidc-client secret
Result: db.prole.org login flow redirects correctly through
https://api.prole.org/auth/login
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Copy pg-knoe-auth/ wholesale from upstream/knoe-db/20260523 (Task 1 of
docs/plans/junie/upstream-knoe-db-20260523-integration.md).
- Extension: PG18 OAUTHBEARER JWT validator using libcurl + OpenSSL RS256.
- knoe-db/Dockerfile: add libcurl4-openssl-dev to dev deps; COPY src/ and
build with make USE_PGXS=1 install after tds_fdw.
- NOT enabled in the default database build (absent from 20_create_extensions.sh).
To enable: CREATE EXTENSION pg_knoe_auth; (requires pg_hba.conf oauth_issuer).
Closes Task 1 of upstream-knoe-db-20260523-integration.md.
Replace all <MagicMock ...> values with empty strings or sensible defaults:
- BooleanVar mocks → false
- StringVar mocks for known fields (cluster_name, db_host_port, db_namespace,
db_username, cluster_env) → correct defaults
- All other StringVar mocks → empty string
Files are now safe to parse and commit. Protected going forward by
_validate_cfg_values() added in ef20c8a.
Add common_core_default_config_path() which resolves the correct knoe.cfg
path (k3d/k3s/gke) from KNOE_MODE/DEPLOYMENT_MODE/CLUSTER_ENV env vars,
falling back to the first cfg file found under conf/. Replaces the
hardcoded knoe.cfg reference in the config-loading comment.
Co-authored-by: Junie <junie@jetbrains.com>
docs/completed/ — new directory; 9 shipped Junie briefs moved from
docs/plans/junie/ (02-k3s-prole-rename, 03-image-rename, 06-patch-garage,
07-init-cnpg-gke-sa-wiring, 13-podmonitor, 15-remove-dead-dashboard,
k3d-knoe-auth-dev-loop, phase2-oidc-gke-deploy, todo-1-cfg-save-path-bug).
docs/completed/README.md summarises all shipped work with dates/refs.
docs/plans/junie/README.md — updated to 2026-05-23; active/pending tables
reflect current state; shipped section now points to docs/completed/.
conf/service/knoe.cfg — new unattended deploy config for the service/k3s
environment (used by ./deploy.sh).
Co-authored-by: Junie <junie@jetbrains.com>
infrastructure/playbooks/pihole_flush_dns.yml — Ansible playbook to
flush Pi-hole DNS cache across inventory hosts.
infrastructure/logs/ — log directory for Ansible run output.
Co-authored-by: Junie <junie@jetbrains.com>
knoe/knoe_conf.py: normalize_environment() now recognises the gke_ prefix
(e.g. gke_plenary-truck-485623-p7_us-west3_knoe-dev-0) as the 'prod'
environment, matching real GKE kubecontext naming.
status.py: add _current_kubecontext() and _cfg_path_from_kubecontext()
so status.py auto-selects the correct cfg file based on the active
kubectl context without manual KNOE_CONF overrides.
Co-authored-by: Junie <junie@jetbrains.com>
Add _validate_cfg_values() to knoe/ui/screens/cfg.py — called before
_filter_cfg_values_for_persistence so MagicMock (or any non-str) widget
values raise TypeError instead of being silently str()-coerced into
conf/<mode>.cfg.
Also re-raise TypeError/ValueError from _save_knoe_cfg so the error
escapes the outer broad except-Exception handler.
New test: tests/installer/test_cfg_save_refuses_mock_values.py
- test_save_knoe_cfg_refuses_non_string_widget_values: MagicMock vars → TypeError
- test_save_knoe_cfg_real_strings_produce_clean_cfg: real _Var stubs → clean cfg
Fixes TODO-1 / tracked in docs/completed/todo-1-cfg-save-path-bug.md.
conf/k3d.cfg and conf/k3s.cfg still contain stale MagicMock values from
before this fix and must be regenerated before committing.
Co-authored-by: Junie <junie@jetbrains.com>
vault_samba_dns_admin_pass was stale — samba-tool commands were failing
with NT_STATUS_LOGON_FAILURE. Updated to match current AD Administrator
credential so the samba_dns role can manage records end-to-end.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All samba-tool dns commands were using -P (machine account Kerberos) which
silently failed with no error — tasks reported changed=0 but records were
never written. Switch to -U Administrator --password={{ samba_dns_admin_pass }}
which uses the vault-protected admin credentials that were already defined
but never wired up.
Also fix regex patterns in record parsers: samba-tool output uses `A: IP`
and `PTR: fqdn` format, not `A IP` / `PTR fqdn` (space-separated), so
updated regex_findall patterns to match `TYPE:\s+value`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- knoe-db.yaml: switch to CNPG-managed TLS cert with serverAltDNSNames
(pg.prole.org + knoe-db-rw cluster service) — removes static serverTLSSecret/serverCASecret
- dns.yml: add pg.prole.org A record to prole_k3s_dns_records (10.0.0.3, 10.0.0.6)
for Ansible-managed split-horizon DNS via Samba AD DC
- k3s.cfg: align KNOE_HOME paths to ~/dev/prole, add PROLE_KDC_* vars, remove
hardcoded KUBECTL_CONTEXT (kubeconfig current-context is authoritative)
- prod.cfg: add PROLE_KDC_STORAGE_CLASS = prole-iscsi
- onepassword.py: skip vault check gracefully when no 1Password session active (non-TTY)
- CLAUDE.md: document production postgres connection string and DNS/CA cert ops
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mermaid 10.7.0 chokes on bare <placeholder> in message text, parsing
it as HTML. Swap to square brackets in the cross-realm §4.3 sequence
diagram and the Gitea SPNEGO §5 diagram:
<svc-host>, <SPNEGO> (§4.3)
<blob> (§5)
Matches the same fix landed in knoe-db's docs/knoe-auth.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Companion to the canonical architecture doc at
~/dev/knoe-db/docs/knoe-auth.md. Documents what's specific to the
prole.org homelab deployment:
- Two upstream identity layers (Samba AD on myrddin for PROLE.ORG
Kerberos, Google Workspace knoey.com for OIDC)
- Three deploy targets (k3s on myrddin/merlin/gandalf, k3d for
engineer dev loop, GKE for production API endpoint)
- Hostname map (myrddin/morgana/zinfandel/db.prole.org/
git.prole.org/api.knoe.dev/db.knoe.dev/etc.)
- Cross-realm Kerberos trust setup (KNOE.LOCAL ↔ PROLE.ORG) via
infrastructure/playbooks/kerberos_trust_setup.yml — RC4 pin
explanation, kvno sync notes
- Gitea SPNEGO at git.prole.org with Mermaid flow diagram
- Supabase Studio status (db.prole.org Google OAuth working;
db.knoe.dev knoe-auth OIDC blocked on pg_oauth image)
- Engineer onboarding checklist (prole-specific steps)
- Known fragility / current incidents table
Mermaid sequence diagram covers the cross-realm flow end-to-end
(workstation → Samba KDC → MIT KDC → service ticket), distinct from
the canonical doc's generic version.
References the matching Junie briefs in knoe-db for the open
fragility items (pg_oauth-image-install, init-user-schemas-heredoc-fix,
deploy-pipeline-restore).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Five fixes Junie surfaced while running the kdc-trust-reset-repeatable
Junie brief end-to-end (companion to commit 6f99f95). All hit during
the unattended `install.sh --mode k3s --reset` pipeline.
- knoe/core/milestones.py (KerberosMilestone):
For k3s and k3d modes, deploy the KDC pod via `init_kdc.sh start`
before running init_kerberos.sh. init_kerberos.sh only chains into
init_kdc.sh when PROLE_KDC_STANDALONE=1; without this hook the
cluster came up with no KDC pod and the cross-realm trust principals
had nowhere to land.
- knoe/milestone.py (Milestone._get_script_env):
Clear KUBECTL_CONTEXT in addition to KUBECONTEXT so stale entries
from a different machine's cfg don't override the kubeconfig's
own current-context.
- etc/knoe_cfg.sh (_knoe_read_cfg):
Skip KUBECTL_CONTEXT / KUBE_CONTEXT_NAME / KUBECONTEXT entries when
reading cfg in k3s mode. Same theme: kubeconfig current-context is
authoritative.
- etc/init_1password.sh + knoe/core/onepassword.py:
When running non-interactively (no TTY on stdin) and no `op`
session exists, skip rather than hang on `op signin`. Lets the
unattended pipeline proceed for k3s/k3d where in-cluster secrets
are managed separately from 1Password.
Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fix _deployment_mode_hint() to correctly map cluster_env=service → k3s
mode. Previously 'service' was not handled in the normalized_env checks,
causing fallthrough to build.deploy_env='Dev' → mode='dev' → k3d
dependency required. On a k3s node (myrddin/merlin/gandalf) k3d is not
installed and the DependenciesMilestone fatally aborted the install.
Also fix get_required_dependency_ids(): k3s mode does not require k3d
(k3s is provisioned on the cluster nodes by Ansible, not by the
installer binary).
Fixes: install.sh --mode k3s --reset failing with
'Dependencies unresolved after install attempts. Required: k3d'
Co-authored-by: Junie <junie@jetbrains.com>
Mirrors the knoe-db commit `ff7546d` patches into the prole copy of
`etc/init_kdc.sh` so a re-run of `install.sh --mode k3s --reset` from
this repo produces a working cross-realm trust without manual cluster
surgery. The k3s cluster is provisioned from this repo, so the source
fix must live here (knoe-db remains canonical for GKE).
Changes to etc/init_kdc.sh:
1. Create BOTH cross-realm krbtgts in MIT, not just the outbound one.
The inbound `krbtgt/<REALM>@<TRUST_REALM>` (issued by Samba,
decrypted here) was missing entirely; without it, MIT cannot
decrypt inbound TGTs and the trust never carries traffic.
2. Pin both cross-realm krbtgts to RC4 (`arcfour-hmac:normal`). AES
keys depend on salt, and Samba's `<remote_realm>+UPN` salt does
not match MIT's `<local_realm>+<principal-no-realm>`; RC4 has no
salt so both sides converge from the password alone. Matches the
already-pinned Samba side (commit `ad1eced`).
3. Replace the broken "remote kadmin to Samba" reciprocal-trust block
with a documented no-op pointing at
`infrastructure/playbooks/kerberos_trust_setup.yml`. Samba AD
does not accept additions over MIT's kadmin protocol; the block
always failed with "Missing parameters in krb5.conf required for
kadmin client".
4. Switch the KDC data volume from emptyDir to a PVC
(claimName `knoe-kdc-data`, parameterized by
`$PROLE_KDC_STORAGE_SIZE` and `$PROLE_KDC_STORAGE_CLASS`).
State now survives pod restarts.
Adds Junie brief `docs/plans/junie/kdc-trust-reset-repeatable.md`
with four TDD acceptance criteria for an end-to-end --reset run on
the prole k3s cluster.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The matching knoe-db change (commit `ff7546d`) re-keys the MIT-side
cross-realm krbtgts with `-e arcfour-hmac:normal` to dodge the
AES-salt mismatch between Samba and MIT. For the trust to actually
carry traffic, the Samba side must also offer only RC4 on the
trust account — otherwise the client picks AES, the two sides
derive different keys from the same password (different salt
conventions), and TGT decryption fails.
Change `msDS-SupportedEncryptionTypes` from 28 (RC4+AES128+AES256)
to 4 (RC4 only). RC4 has no salt, so keys derive from the password
alone and both sides converge.
Updated docstring + summary debug print to reflect the new value
and the why.
Tracked alongside the Junie brief at
~/dev/knoe-db/docs/plans/junie/kdc-trust-reset-repeatable.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The post-rebrand realm is KNOE.LOCAL (matches knoe-db/etc/init_kdc.sh
already). The prole copy of init_kdc.sh was never updated, so the k3s
KDC pods on myrddin/merlin/gandalf got initialized with PROLE.LOCAL
realm baked into /etc/krb5.conf and /etc/krb5kdc/kdc.conf.
When chrisfu later wired up the Samba <-> in-cluster KDC cross-realm
trust playbook (kerberos_trust_setup.yml, targeting KNOE.LOCAL), the
trust was bidirectionally correct on the Samba side but the in-cluster
KDC could not decrypt the inbound krbtgt/KNOE.LOCAL@PROLE.ORG TGT
because its own realm was PROLE.LOCAL — different krbtgt key, every
TGS_REQ failed with "Decrypt integrity check failed" (manifested as
"Server not found in Kerberos database" in the client trace).
This commit fixes the source. To pick it up in-cluster, the active
KDC pod must be redeployed (its DB sits on an emptyDir volume so a
restart re-initializes against this default).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
klist shows the Kerberos principal (slash form
krbtgt/KNOE.LOCAL@PROLE.ORG), not the sAMAccountName underscore form
(krbtgt_KNOE.LOCAL). Update the summary debug task accordingly and
split the kvno hint into two stages (the cross-realm TGT first, then
a service ticket) so the user can isolate which step fails when
chasing salt/enctype issues.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The apt install task was positioned after the user-create step, so a
failure in create (or anywhere upstream) meant ldb-tools never got
installed. Move the apt task to the very top of the play, before the
cluster-secret lookups, so it always runs first on every invocation.
Drop the duplicate apt task that previously sat just before the
ldbmodify locator.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The _trust_exists probe was still grepping samba-tool user list for
"krbtgt/KNOE.LOCAL" (slash form), but the actual sAMAccountName is
"krbtgt_KNOE.LOCAL" (underscore form, since slash is invalid in
sAMAccountName). _trust_exists was therefore always false, the create
step retried against the existing account on every re-run, and Samba
rejected it as a name collision.
Match against samba_account_name directly so re-runs route through
the setpassword branch instead.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
myrddin doesn't have ldbmodify installed — it's in the ldb-tools
Debian package, separate from samba-tool. Add an apt task ahead of
the locator so the package is in place before we try to use it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ldbmodify is part of the ldb-tools Debian package but is not in
root's default PATH under sudo on the Samba DC, so the user-attribute
modify step failed with "ldbmodify: command not found".
Add a locator task that probes the conventional install paths
(/usr/bin, /usr/sbin, /usr/local/bin, /opt/samba/bin) and falls back
to a depth-limited find under /usr and /opt. The downstream
ldbmodify invocation now uses the absolute path resolved here.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
samba-tool user create rejected the cluster-generated trust_shared_password
with "0000052D: Constraint violation - check_password_restrictions: the
password does not meet the complexity criteria". The trust password is
high-entropy random bytes from init_kdc.sh and may not satisfy AD's
"3 of 4 character categories" rule.
Wrap the create + setpassword steps in a block that:
1. reads the current `samba-tool domain passwordsettings show` state
2. flips complexity off if it was on
3. does the user write
4. always: restores complexity to its prior on/off state
The restore is in an `always:` clause so a failure inside the write
block does not leave the domain policy weakened.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
samba-tool user create rejected the sAMAccountName "krbtgt/KNOE.LOCAL"
with "samldb: sAMAccountName contains invalid '/' character". AD/Samba
disallow "/" in sAMAccountName even though Kerberos principal names
require it (krbtgt/REMOTE@LOCAL).
Switch storage name to "krbtgt_KNOE.LOCAL" and add an ldbmodify pass
that sets the canonical Kerberos identity on the same object:
userPrincipalName = krbtgt/KNOE.LOCAL@PROLE.ORG
servicePrincipalName = krbtgt/KNOE.LOCAL
msDS-SupportedEncryptionTypes = 28 (RC4+AES128+AES256)
Samba KDC resolves principals by UPN/SPN, so a TGS-REQ for
krbtgt/KNOE.LOCAL@PROLE.ORG will hit this account.
Note: key-salt parity with the MIT side is NOT guaranteed yet. Samba's
default salt for AES keys is REALM+UPN; MIT's default for cross-realm
krbtgt is REALM+"krbtgt"+REMOTE. If kvno fails with "decrypt integrity
check failed" we'll add a keytab-export/import step in a follow-up
rather than try to coerce Samba's salt at creation time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
In --check mode the user-create step is skipped (it's mutating), so
the subsequent `samba-tool user show krbtgt/KNOE.LOCAL` probe fails
with rc=1 on a first dry-run because the user doesn't yet exist. Add
the same guarded failed_when used on the trust validate/show tasks
earlier: only fail when NOT (check_mode AND trust didn't pre-exist).
Real runs still fail hard on rc!=0.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
samba-tool domain trust create --type=external only works against
another writeable AD-style DC; against an MIT KDC it fails with
"Failed to find a writeable DC for domain 'KNOE.LOCAL'". Our peer at
svc/auth in knoe-system is a plain Kerberos KDC, so we cannot use that
path.
Replace the trust create/validate/show steps with the supported
Samba-AD <-> MIT-KDC approach: create the inter-realm TGT principal as
an ordinary user account in Samba whose sAMAccountName is
"krbtgt/KNOE.LOCAL", with its password synced to the cluster Secret
knoe-system/knoe-kdc-secrets/trust_shared_password (which the MIT side
already keys against). Steps:
1. samba-tool user list (idempotency probe)
2. samba-tool user create krbtgt/KNOE.LOCAL ... (when missing)
3. samba-tool user setpassword krbtgt/KNOE.LOCAL (when present, to
re-sync after a cluster Secret rotation)
4. samba-tool user setexpiry --noexpiry
5. ldbmodify msDS-SupportedEncryptionTypes=28 (RC4+AES128+AES256
to match what init_kdc.sh sets on the MIT side)
6. samba-tool user show (smoke probe)
7. debug task prints the manual kvno smoke-test command
Cluster-secret lookups and krb5.conf [realms]/[domain_realm] edits are
unchanged - those parts were correct.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ansible --check mode auto-skips command/shell tasks unless they
explicitly opt in via check_mode:false. The two cluster-state lookups
(svc/auth ClusterIP and knoe-kdc-secrets/trust_shared_password) were
silently skipped during dry-run, leaving trust_kdc_ip empty and tripping
the assertion immediately.
Add check_mode:false to the five read-only tasks so a --check run can
still resolve cluster state and report what would change:
- Resolve trust_kdc_ip from cluster
- Resolve trust_shared_password from knoe-kdc-secrets
- samba-tool domain trust list (idempotency probe)
- samba-tool domain trust validate
- samba-tool domain trust show (smoke print)
The mutating create step (samba-tool domain trust create) keeps the
default behavior - skipped in check mode. validate/show gain a guarded
failed_when so a check-mode dry-run on a host where the trust does not
yet exist does not fail (the create was skipped, so a non-zero rc is
expected). Real runs still fail hard on rc!=0.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The cross-realm trust playbook was written for the pre-rebrand realm name
PROLE.LOCAL, which no longer exists in the in-cluster KDC — the canonical
realm is now KNOE.LOCAL. The Samba AD trust was therefore never actually
established, leaving chrisfu@KNOE.LOCAL→chrisfu@PROLE.ORG service ticket
flows blocked (and blocking pg_oauth / db.prole.org Kerberos work).
Changes:
- Realm: PROLE.LOCAL → KNOE.LOCAL across all task names, vars, and the
krb5.conf [realms] / [domain_realm] blocks added on myrddin.
- samba_admin_password now resolves from the Ansible vault var
vault_samba_dns_admin_pass (group_vars/ad_dc/vault.yml) by default,
with SAMBA_ADMIN_PASSWORD env and -e overrides preserved for CI.
- trust_shared_password auto-resolves from the in-cluster Secret
knoe-system/knoe-kdc-secrets (key trust_shared_password) when not
passed explicitly — same Secret init_kdc.sh writes.
- Added [domain_realm] mapping for *.knoe.local → KNOE.LOCAL so Samba
can resolve in-cluster service principals.
- Added a final `samba-tool domain trust show` smoke step so a successful
run prints the visible trust state for log review.
Run:
ANSIBLE_VAULT_PASSWORD_FILE=$PWD/.vault_pass ansible-playbook infrastructure/playbooks/kerberos_trust_setup.yml
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
win_regedit writes HKLM keys without privilege escalation when
ansible_user is already a local/domain Administrator. become:true
with --ask-become-pass was overriding the playbook's runas method.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A single play with become:true uses sudo by default, which Ansible's
Windows exec wrapper rejects immediately at gather_facts. Split into:
- Play 1: hosts workstations,!workstations_windows — macOS/Linux, sudo
- Play 2: hosts workstations_windows — Windows, become_method:runas
Also set morgana to ansible_connection=local (running Ansible from
morgana itself, so SSH self-lookup fails DNS resolution).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add morgana and zinfandel (macOS) and morgoth + fairyland (Windows)
to the workstations inventory.
Extend workstation_kerberos.yml to handle Windows targets:
- Chrome and Edge AuthServerAllowlist set via win_regedit under
HKLM\SOFTWARE\Policies\{Google\Chrome,Microsoft\Edge}
- krb5.conf and plist tasks skip on Windows (uses built-in SSPI/AD)
- Windows hosts in [workstations_windows] group with WinRM/NTLM
transport; switch to kerberos transport after domain join
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea's AdminEditUser PATCH with source_id:0 resets the password field
as a side effect when applied to a SPNEGO-registered user (the auth
record re-initialisation clears the local password). Setting the
password before the promotion PATCH meant it was immediately wiped.
Move the 1Password recovery password step to run after the admin
promotion PATCH so the final Gitea state matches 1Password.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea's AdminEditUser endpoint requires email alongside login_name
and source_id. Without it the server returns 200 but silently
ignores the password field — causing login failures despite the
script reporting success.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 1Password password-setting block was nested inside path 2 of
promote_gitea_admin(). When path 1 found a cached gitea-admin-token
k8s secret, path 2 was skipped entirely and the recovery password
was never set in Gitea, leaving the user unable to log in.
Move the password-setting step to run unconditionally after all four
token paths complete. Uses _op_ensure_auth() so it degrades gracefully
in CI/headless environments where 1Password is unavailable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Problem: 'defaults write' is not an onboarding path, it's a support
incident waiting to happen. Browser SPNEGO must be deployed via managed
policy at machine provisioning time, not per-user.
Changes:
etc/init_knoe_users.sh
- _op_ensure_auth(): tries OP_SERVICE_ACCOUNT_TOKEN env, k8s secret
'op-service-account-token', existing interactive session, and op-run
inheritance — in that order. Gives actionable instructions when none
work, including the correct 'op run --' invocation documented in k3s.cfg.
- gitea_ensure_password(): calls _op_ensure_auth() at entry; eliminates
the silent 'op not authenticated' failure path.
- Script header: documents 'op run -- bash etc/init_knoe_users.sh' as
the intended invocation for admin's laptop.
- 'Next steps' output: replaces 'defaults write' with reference to
workstation_kerberos.yml Ansible playbook.
infrastructure/playbooks/workstation_kerberos.yml (new)
- Deploys /etc/krb5.conf (PROLE.ORG realm, myrddin KDC) to all managed
endpoints.
- Deploys Chrome + Edge managed policy (AuthServerAllowlist) on macOS
and Linux — no per-user browser configuration ever required.
- Idempotent; run during laptop provisioning or re-run at any time.
- Targets 'workstations' Ansible group.
infrastructure/inventory/hosts.ini
- Adds [workstations] group with example entries and onboarding notes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SPNEGO auto-registration assigns a UUID@localhost placeholder email.
promote_gitea_admin() now detects the UUID pattern and replaces it with
the user's real configured email (KNOE_ADMIN_EMAIL / GITEA_ADMIN_EMAIL).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1Password credentials are optional (primary auth is Kerberos SPNEGO).
When op is not signed in on a headless server, show a WARN with a
helpful hint instead of an ERROR that suggests a hard failure.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
log() writes to stdout, so log calls inside functions that return
values via printf/stdout contaminate the captured output.
gitea_helm_admin_token and gitea_ensure_password now redirect all
log() calls to stderr with >&2, keeping stdout clean for the
returned token/password string.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea admin edit-user API requires email in PATCH body.
Fix HTTP status parser (tr -d vs broken cut -d_).
Add logging to user-existence check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea 1.22 requires explicit scopes in the token creation POST body.
Without them the API returns: "access token must have a scope".
Now requests: read/write:admin, read/write:user, read/write:issue,
read/write:repository — sufficient for admin user management.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a Deployment has been updated but the new pod keeps crashing before
configure-gitea runs (e.g. after a failed Helm upgrade), the running pod
still carries the old RS's password — which differs from the current
Deployment spec.
gitea_helm_admin_token() now walks: running-pod-RS → current-Deployment-spec,
trying each password until one produces a valid API token.
Also adds HTTP status + response body to gitea_api_token() failures so the
cause is visible without exposing credentials.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Gitea Helm chart (gitea-12.5.3) stores GITEA_ADMIN_USERNAME and
GITEA_ADMIN_PASSWORD as plain values in the configure-gitea init container
spec — not in a k8s Secret with key 'admin-password'. The previous code
looked for a non-existent secretKeyRef and returned empty, causing the
REST API bootstrap path to fail.
Now reads credentials via:
kubectl get deployment gitea ... env[?(@.name=="GITEA_ADMIN_PASSWORD")].value
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Gitea Helm chart names the admin credentials secret after the release
name alone, not '{release}-gitea'. For release name 'gitea' the secret is
just 'gitea'.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
deploy.sh: resolve_knoe_db_namespace() now checks DATABASE_NAMESPACE from
config (k3s.cfg: knoe-db) before probing namespaces. Previously it fell back
to 'default' because only 'knoe' was probed, baking a wrong DB hostname into
app.ini. The running server was unaffected (uses GITEA__database__HOST env var)
but the gitea admin CLI, which reads app.ini directly, could not connect.
init_knoe_users.sh: add gitea_helm_admin_token() which reads the Helm
bootstrap admin password from the gitea-gitea k8s secret and exchanges it for
a REST API token — entirely avoiding the broken CLI. Add gitea_api_set_password()
which uses PATCH /api/v1/admin/users/{user} to set the password via the running
web server (which has the correct DB connection) instead of kubectl exec.
promote_gitea_admin() now prefers the REST-API-only path:
1. existing token (env / k8s secret)
2. Helm admin k8s secret → REST token; set KNOE_ADMIN_PRINCIPAL password via API
3. 1Password → CLI set-password → basic-auth token (fallback)
4. kubectl exec generate-access-token (last resort)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
knoe_cfg.sh selects the config file based on KNOE_MODE at source time.
Without pre-scanning argv, KNOE_MODE is empty and k3d.cfg wins the
fallback loop even when --mode k3s is passed. This caused k3s.cfg values
(e.g. KNOE_ADMIN_PRINCIPAL=chrisfu) to be silently ignored, defaulting
to 'admin' and causing gitea_set_password to fail against a non-existent user.
Also surface gitea_set_password errors (without logging the password) so
failures are diagnosable rather than silent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the predictable temp-password pattern with a secure credential flow:
- gitea_ensure_password(): creates a 1Password item with --generate-password
(1P generator, 32-char letters+digits) if one does not exist; retrieves it
on re-runs (idempotent). Password never appears in logs or env vars.
- gitea_set_password(): sets the Gitea account password via kubectl exec
(gitea admin CLI, no prior Gitea auth required).
- gitea_api_token(): exchanges basic-auth credentials for a REST API token,
deleting any stale token of the same name first. Token only; password
cleared from memory immediately after use.
- promote_gitea_admin(): tries credential sources in order:
1. GITEA_ADMIN_TOKEN env / gitea-admin-token k8s secret
2. 1Password (generates if missing) → set password → REST token
3. kubectl exec generate-access-token fallback (Helm bootstrap admin)
Tokens persisted in k8s secret (revocable); passwords only in 1Password.
The Gitea password is a vault credential for emergency recovery only.
Normal auth is always Kerberos SPNEGO — users never need to type a password.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ron has no Google Workspace prole.org account. Email is now NULL in the DB
(NULLIF('')). The provision_user log line now always shows the Kerberos
principal explicitly and labels the email field separately so it is clear
which identity type is in use.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
In k3s mode, LOCAL_REGISTRY_INTERNAL set to a k3d value in the shell environment
leaked into init_kdc.sh subprocesses, causing pods to use the wrong image address
while the push went to myrddin.prole.org:5000.
Add _resolve_kdc_registry() helper that derives correct registry values from
KNOE_IMAGE_REGISTRY and PROLE_K3S_SERVER. Both init_kdc.sh call sites now pass
explicit overrides to prevent any inherited k3d value from leaking through.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 'die' call with a multiline string containing '|' (for base64 pipeline
in the help text) was parsed by bash as a pipe operator. Replace with
individual err() calls followed by explicit exit 1.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Password resolution order for PROLE_KDC_MASTER_PASSWORD and KNOE_GUEST_PASSWORD:
1. Env var (existing)
2. knoe-kdc-secrets k8s Secret (existing)
3. prole-kdc-secrets k8s Secret (new — used by live prole deployment)
4. 1Password via op read (new — OP_KDC_MASTER_PASSWORD_REF / OP_KDC_GUEST_PASSWORD_REF)
5. PROLE_LOCAL_ADMIN_PASSWORD (existing)
6. die with actionable message
Add try_op_read() helper: wraps op CLI, returns 1 if op not available or ref empty.
conf/k3s.cfg: add KNOE_ADMIN_PRINCIPAL=chrisfu, OP_KDC_MASTER_PASSWORD_REF,
OP_KDC_GUEST_PASSWORD_REF placeholders (empty — user fills in op:// URIs).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The k3s-generated kubeconfig on the cluster server node uses 'default'
as its context name. The existing guard rejected this unconditionally,
causing init_knoe_users.sh to always fail when run from myrddin.
Tighten the guard: only reject 'default' when KNOE_MODE is k3d (where
it would indicate the k3d context was not found). In k3s/k8s mode,
'default' is legitimate and the script proceeds normally.
Usage on myrddin: bash etc/init_knoe_users.sh --mode k3s initialize
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add ensure_gitea_spnego_keytab(): provisions HTTP/git.prole.org@PROLE.ORG
service principal, keytab, and gitea-krb5-keytab Secret automatically when
samba-tool is available (i.e. running on myrddin); otherwise prints manual
steps. Restarts gitea-spnego-proxy after keytab is created.
- Add GITEA_SPNEGO_HOST, GITEA_KRB5_AD_REALM, GITEA_KRB5_AD_USER config vars.
- Fix promote_gitea_admin: exec with 'su git' (gitea refuses to run as root);
use KNOE_ADMIN_PRINCIPAL instead of hardcoded 'admin'; fall back to
'gitea_admin' (Helm chart default) if the primary user has no token yet.
- Rename initialize() step numbers (9=ArgoCD, 10=SPNEGO keytab, 11=Gitea, 12=GitLab).
- Fix PROLE.LOCAL → KNOE.LOCAL in: default realm, SQL schema default, comments,
next-steps message, trust principal comments.
- show_status: add gitea-krb5-keytab check alongside pg keytab check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea Helm chart v10.x changed defaults:
image.registry: docker.gitea.com (was docker.io)
image.rootless: true (was false)
Our values set repository/tag but not registry/rootless, so the new
defaults produced docker.gitea.com/gitea/gitea:1.22.3-rootless which
can't be pulled from gandalf. Pin registry: docker.io and rootless: false
to keep using the standard Docker Hub image.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Helm upgrade was failing for two reasons:
1. persistence.storageClass was unset (rendered null), conflicting with the
existing PVC's immutable gitea-local-d005 storageClassName.
2. valkey-cluster (and redis-cluster) were not explicitly disabled, causing
the latest gitea chart to spin up a valkey StatefulSet on each upgrade.
Add GITEA_STORAGE_CLASS env-var override for non-k3s modes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gokrb5 returns 401 silently on invalid tickets. Wrap ResponseWriter to
capture status code and log failures with remote addr, host, path, and
first 20 chars of the Negotiate token for easier debugging.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Go proxy exposes /_healthz so use httpGet probes (tcpSocket was only a
connectivity check, not a health check). Also reduce memory request/limit
to 32Mi/64Mi — the Go binary is lean and in-cluster load is minimal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds the ScheduledBackup CR to source control. Fixes schedule from
5-field to 6-field format (0 0 3 * * *) — CNPG requires seconds as
first field. The broken schedule caused the reconcile loop that
generated 746 stale Backup CRs. Deleted 7 orphaned GitLab Garage
buckets (~8.5 GB) and resumed daily backups.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path does not exist on this k3s cluster. The Prometheus Operator
was stuck in a reconcile error loop, blocking all PodMonitor updates.
Prometheus: merlin-local-iscsi-prometheus (30Gi pre-existing PV)
Alertmanager: merlin-local-iscsi-alertmanager (5Gi pre-existing PV)
Grafana: remains local-path to match immutable StatefulSet VolumeClaimTemplate;
actual PVC is already bound to merlin-local-iscsi-grafana PV.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
knoe-auth /auth/verify not yet implemented — auth_request returns 500,
blocking all Grafana access. Simplify nginx to plain passthrough; strip
inbound X-WEBAUTH-USER to prevent header forgery. Disable Grafana
auth.proxy until knoe-auth is ready. Google auth.google still active.
authority-prole-auth deployment had no Service. Create knoe-auth ClusterIP
service selecting app=authority-prole-auth on port 8080 — consistent with
Kong's knoe-auth route URL. Update nginx configmap to use the same FQDN.
Without this, kubectl apply without -n flag lands in default namespace
instead of monitoring, leaving the live configmap with the old
knoe-auth hostname.
Traefik ingress (svc-knoe-ingress) routes db.prole.org to
service/oauth2-proxy:80 in knoe-system. oauth2-proxy itself runs in
supabase namespace, so this ExternalName alias bridges the gap.
- kong-configmap: sync to live content; fix grafana upstream to route through
knoe-grafana-proxy (nginx auth proxy) instead of kps-grafana (stale service
name from old helm release); add db.prole.org → oauth2-proxy route; fix
knoe-auth FQDN to authority-prole-auth
- grafana-proxy-configmap: fix upstream to prometheus-grafana (helm release
name is 'prometheus', not 'kps'); fix auth_request FQDN to
authority-prole-auth.knoe-system; fix login redirect to api.prole.org
- grafana-proxy-{deployment,service}: add namespace: monitoring
openssl rand -base64 32 produces a 44-char string; oauth2-proxy treats it
as 44 raw bytes and rejects it. -base64 24 encodes 24 bytes → 32 base64
chars with no padding, which satisfies the 32-byte AES requirement.
Reads svc-prole-org (Grafana) and db-prole-org (oauth2-proxy) API_CREDENTIAL
items from the Personal 1Password vault and writes them into etc/secrets/ so
init_grafana_oauth_prole.sh and init_oauth2_proxy_prole.sh can consume them.
Generates and saves the oauth2-proxy cookie secret (32-byte base64) back to
the db-prole-org 1Password item as a cookie_secret field on first run;
idempotent thereafter.
Usage:
./etc/fetch_prole_secrets.sh
./etc/init_grafana_oauth_prole.sh
./etc/init_oauth2_proxy_prole.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- monitoring/kps-values-k3s.yaml: Grafana helm values for k3s homelab with
dual auth (auth.proxy Kerberos + auth.google for prole.org Workspace)
- grafana-proxy-configmap.yaml: nginx passthrough for /grafana/login/google
and /grafana/login to allow Google OAuth flow without knoe-auth redirect
- grafana-google-oidc-secret-prole.example.yaml: Secret template for Grafana
Google OAuth client (svc.prole.org, Internal consent, prole.org Workspace)
- oauth2-proxy-google-oidc-secret-prole.example.yaml: Secret template for
oauth2-proxy gating db.prole.org Studio
- oauth2-proxy-deployment-prole.yaml: k3s oauth2-proxy deployment for
db.prole.org (prole.org domain, no BackendConfig)
- init_grafana_oauth_prole.sh: Bootstrap script for Grafana OAuth secret
- init_oauth2_proxy_prole.sh: Bootstrap script for Studio oauth2-proxy
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
selectattr with 'match' regex was returning empty sequence on this Ansible
version. Since each install entry has exactly two items (fullchain.cer +
key), rejectattr the fullchain to get the key — no regex needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The dns_namecom.sh plugin uses Namecom_Username and Namecom_Token (not
NAMECOM_*). Fix the environment dict, the SAVED_ keys written to
account.conf, and add a cleanup task to remove the stale all-caps entries
already present on hosts from previous runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Re-adds no_log: true removed in 5e4e17e and switches the issue task from
ansible.builtin.shell with inline VAR=value to ansible.builtin.command with
an environment: dict. This prevents credentials appearing in Ansible failure
output and avoids the inline env var being clobbered when acme.sh sources
account.conf internally before the dns_namecom plugin runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ansible command module + become does not reliably pass environment:
vars through sudo env_reset. Use ansible.builtin.shell with POSIX
inline assignment (VAR=val cmd) so credentials are set inside the
sudo-spawned shell subprocess, never touching the sudo env barrier.
account.conf writing retained for cron renewal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_readaccountconf_mutable checks SAVED_NAMECOM_Username first; if that
key lookup exits 0 with empty output it short-circuits and never reads
the bare NAMECOM_Username key. Write the SAVED_ prefix that acme.sh
uses internally when persisting credentials.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sudo env_reset strips the environment: key through become. Write
NAMECOM_Username and NAMECOM_Token to acme.sh's account.conf directly,
which dns_namecom reads as a fallback via _readaccountconf_mutable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New acme role: installs acme.sh, issues certs for myrddin/merlin/gandalf
via DNS-01 (name.com API), installs cert files to /etc/ssl/certs/prole/,
sets up renewal cron. rc=2 (already valid) treated as ok.
- New playbooks/acme.yml: runs acme role on myrddin (issues all 3 host certs)
- New playbooks/certmgr.yml: cert-manager and letsencrypt-prod ClusterIssuer
(HTTP-01/Traefik) for external endpoints (db/svc/api.prole.org)
- Add vault references for name.com API creds to group_vars/all/vars.yml
- Add prole_ssl_src_dir and prole_ssl_files to myrddin/merlin/gandalf host_vars
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Followup from commit dba8a2d's findings. ConfigMixin._save_knoe_cfg in
knoe/ui/screens/cfg.py reads Tk widget vars via `var.get()` and writes
the result to conf/<mode>.cfg. When a var is a MagicMock (interactive
./install.py run in a non-Tk context, partially-mocked widgets), the
save path serializes the mock's repr-string into the cfg, then the next
installer pass calls os.makedirs() on those values and produces
directories literally named `<MagicMock name='Canvas().tk.call().strip()'
id='4743999712'>/`.
The brief lays out a TDD approach for Junie:
1. Write failing test at tests/installer/test_cfg_save_refuses_mock_values.py
that passes MagicMock widget vars and asserts _save_knoe_cfg raises
TypeError naming the field.
2. Implement the minimal fix: a `_str_value(var, field=...)` helper in
cfg.py that validates widget reads and raises if non-str. Use it
in the .get()/.strip() chains across lines 103-155.
3. Verify the 750 existing installer tests still pass.
Brief includes file pointers (cfg.py:44 _save_knoe_cfg, line 102
globals_to_save assembly, line 277 cfg_path.write_text), the canonical
failing test stub, both fix-approach options (per-read validator vs
end-of-flow dict walk), and explicit commit-shape guidance.
Index updates:
docs/plans/junie/README.md — todo-1 row added under Active
docs/TODO.md §"In progress" — todo-1 promoted above Phase 3 (smaller
scope, easy to land first)
Naming convention: `todo-N-<slug>.md` for follow-up bugs, distinct from
the `NN-<slug>.md` pattern reserved for ranked queue items.
- Add gandalf.prole.org → 10.0.0.7 to prole_dns_records and
prole_internal_a_records in dns.yml (was missing, causing nslookup
to fail on myrddin)
- prole_ssl: stat-check each source file on the controller before
copying; emit a warning and skip rather than hard-failing when certs
have not yet been placed in ssl/prole/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Installs Tailscale on k3s agents via the Debian stable apt repo using
the modern signed-by keyring approach. Auth key stored in vault.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
init_gitea.sh requires GITEA_PV_NODE in k3s mode. PVs for Gitea storage
(gitea-local-d005) are pinned to gandalf.prole.org per nodeAffinity on
the existing gitea-d005-shared-storage PV.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cleanup_legacy_prole_kong() only checked for the prole-svc-kong deployment
and service, so the early-return guard fired after those were already deleted,
leaving svc-prole-ingress in place. On re-runs this Traefik ingress blocked
the duplicate-host check and prevented Kong from deploying.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k8s/knoe/iscsi-pvs.yaml had *.knoe.org nodeAffinity after the rename, but the
live k3s cluster nodes are still *.prole.org — causing immutable field errors on
Bound PVs during kubectl apply. Revert to *.prole.org to match live state.
conf/k3s.cfg KUBECTL_CONTEXT was renamed to knoe-service-cluster, but both the
system kubeconfig and etc/secrets/k3s.kubeconfig still use prole-service-cluster.
install.sh _check_gcp_tools() now skips gcloud check for k3s mode and any non-gke
config path (only GKE deploys need gcloud/kubectl pre-flight).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
conf/dev/knoe.cfg had a stale SERVICE_NAMESPACE=default written by a prior TUI session.
Also guard _service_namespace() so the value 'default' is never returned — it is never
a valid service namespace and signals a stale generated config.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Combobox width=30 chars extends to ~320-380px depending on font metrics,
colliding with Add at x+290 and Delete at x+350. Narrowed combobox to
width=22 and shifted buttons to x+260 and x+320 to give clear separation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
install_barman_plugin() fetches from GitHub on every run (up to 6×120 s
retries), stalling the installer worker thread even when the CNPG cluster
is already at 'Cluster in healthy state' with all pods 2/2 Ready.
Add a fast-path at the top of initialize(): check .status.phase for
'healthy' then verify all pods show 2/N ready — if both pass, return
immediately. Full pipeline (ensure_operator, install_barman_plugin,
_apply_manifest, _wait_cnpg_pods) is only entered when needed.
Also adds test_cnpg_initialize_skips_when_cluster_healthy to
test_deployment_mode_isolation.py to guard against regression.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
kube-prometheus-stack CRD re-application stalls on every installer run
even when the release is in deployed state with all pods Running. For k3d
this is purely idempotency noise — skip the upgrade and return immediately.
Broken state (failed status, unbound PVCs) still triggers a full purge+reinstall.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k3d clusters only have local-path provisioner; defaulting to synology-iscsi
caused all Supabase PVCs to get stuck Pending. Also add local-path to the
validation allowlist so it is not rejected as an unknown storage class.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The container's /bin/sh is dash, which doesn't support set -o pipefail.
The preceding [ -s /tmp/pg.keytab ] check already guarantees the file is
non-empty before this line runs, so pipefail was redundant.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
conf/service/prod.cfg: prole.org homelab overrides
- SERVICE_HOST_DOMAIN=prole.org, DB_UI_HOST=db.prole.org
- OIDC_ISSUER_URL=https://api.prole.org/auth
- GOOGLE_WORKSPACE_DOMAIN=prole.org (credentials in OpenBao bao/kv/knoe/oauth2/google-prole-org)
- kerberos_config: enabled, realm=PROLE.ORG, kdc=10.0.0.3 (myrddin.prole.org)
- PROLE_KDC_TRUST_REALM=PROLE.ORG, SERVICE_INGRESS_TLS_ENABLED=1
conf/k3d.cfg: update local dev paths to ~/dev/prole; enable kerberos for dev
env.sh: KNOE_HOME points to ~/dev/prole
application-prole.yml: Spring profile with prole.org Google Workspace client binding
These files are prole.org homelab specifics — do NOT merge upstream to knoe/main.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Keytab export (silent failure bug):
- Always delete+recreate the postgres SPN with fresh random key; old key gone
after EmptyDir wipe; kadmin.local -q exits 0 even on error so -norandkey
silently failed
- Clean /tmp/pg.keytab before ktadd; verify non-empty with [ -s ] before
proceeding; die loudly if keytab not written
- Fix base64 pipeline: set -o pipefail inside sh -c so base64 failure is not
masked by tr exit code
Next steps:
- Suppress myrddin.prole.org Samba trust step in k3d mode (no AD server)
- Remove Grafana auth.proxy reminder (configured by Helm values already)
Gitea admin token:
- Auto-generate via kubectl exec into running Gitea pod before falling back to
manual warning; persist as gitea-admin-token secret for future re-runs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs in the EmptyDir auto-recovery path:
1. get_kdc_pod returned items[0] without a phase filter, so a Terminating
pod from the previous rollout could be selected. Add
--field-selector=status.phase=Running to always get a live pod.
2. The rollout completes (pod Running) before the in-container entrypoint
finishes kdb5_util create (no readiness probe configured). The single
immediate kadmin.local check raced against DB init and lost. Replace
with a 120s poll loop (5s interval) that re-queries the pod each tick
so it handles both the timing race and any remaining pod-selection lag.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
init_kdc.sh initialize rolls out a new deployment, creating a new pod with a
new name. The retry pre-flight was still exec-ing into the old (terminated)
pod. Re-query get_kdc_pod() after re-init so the retry and all subsequent
kadmin calls use the new pod.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the KDC pod restarts (e.g. after k3d node restart to apply registry
mirror config), its EmptyDir volume is wiped. ensure_kdc_pod() finds the
running pod and returns it, bypassing the bootstrap path. The subsequent
pre-flight check then fails because kadmin.local cannot access the database.
Instead of dying with a manual-intervention message, auto-recover by calling
init_kdc.sh initialize (with PROLE_KDC_NAMESPACE/NAME/MASTER_PASSWORD set)
then retrying the pre-flight. Only die if the retry also fails.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
check_kerberos.sh defaulted PROLE_KDC_NAME=auth but init_kdc.sh deploys the
pod with label app=authority-knoe-auth (authority-gcp-auth for k8s). The pod
selector never matched, so wait_for_kdc_pod always returned empty and the test
failed with "KDC pod 'auth' not found" even after a successful rollout.
Add _default_kdc_name() using the same mode-aware logic as init_kdc.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k3d registry containers are addressed as k3d-NAME.localhost:PORT from
inside the cluster, but containerd resolves that as 127.0.0.1 (loopback)
which has nothing listening. The fix is to write /etc/rancher/k3s/registries.yaml
on every server/agent node to mirror k3d-NAME.localhost:PORT -> http://k3d-NAME:PORT
(Docker network DNS), then restart the node so containerd picks it up.
_patch_k3d_registry_mirrors() is called from _ensure_k3d_registry() after
the registry is confirmed running (idempotent — skips nodes already configured).
This makes knoe-authority and any other image pushed to localhost:5000
immediately pullable inside the cluster as k3d-knoe-registry.localhost:5000/...
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
check_kerberos.sh (already updated) sourced knoe_env.sh, knoe_secrets.sh,
knoe_string.sh, and knoe_yaml.sh but the files were still named prole_*.sh.
No other script referenced the old names, so this is a pure rename.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Port-forward setup was failing the installation because:
1. argocd and dashboard entries were always added to port-mapping even when
those services haven't been selected or installed yet
2. Grafana/Prometheus service names were hardcoded as kps-* but monitoring
deploys with release name "prometheus" → services are prometheus-grafana etc.
3. The grafana status poll after init_port_forwards.sh start would mark the
entire installation failed if grafana wasn't reachable
Changes:
- _build_required_port_forwards (env.py): add argocd_enabled, dashboard_enabled,
monitoring_release params; argocd/dashboard only included when enabled; grafana
and prometheus targets use f"svc/{monitoring_release}-grafana" etc.
- _sync_port_forward_mappings (actions.py): read argocd_enabled from installer
config; read MONITORING_RELEASE from Monitoring section; clean up argocd and
dashboard entries from [Port Forwards] when disabled
- run_init_scripts (services.py): port-forward step is non-fatal — failure logs
a warning and continues; blocking grafana poll removed
- 2 new regression tests (7e) covering the above invariants
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
services.py run_init_scripts() called init_kong.sh with -n env[NAMESPACE]
where NAMESPACE is the database namespace (e.g. knoe-db). The -n flag
sets COMMON_CORE_NAMESPACE which wins over SERVICE_NAMESPACE in
common_core_resolve_namespace, so Kong deployed into knoe-db.
Fix: resolve kong_ns from SERVICE_NAMESPACE, falling back to
_get_service_namespace() and finally knoe-system. Exclude default
as it is the Tk StringVar initial value, not a real service namespace.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When SERVICE_NAMESPACE was absent from the config file and OS env,
_get_script_env() fell back to env["NAMESPACE"] — the database namespace
(e.g. knoe-db). InitializationScriptsMilestone then called init_kong.sh
with SERVICE_NAMESPACE=knoe-db, causing Kong to deploy into the DB
namespace and collide with the existing knoe-system/svc-knoe-ingress.
Fix: default to "knoe-system", matching _service_namespace() in actions.py.
Configs that set SERVICE_NAMESPACE explicitly are unaffected.
Add tests/test_deployment_mode_isolation.py to guard against k3d/k3s/gke
mode-specific config bleeding into each other: SERVICE_NAMESPACE fallback,
monitoring storage class separation, Kong gitea host gating, and
_service_namespace() across all three deployment modes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three issues caused the purge-and-reinstall loop:
1. _purge_broken_monitoring deleted ALL PVCs including Bound ones; grafana's
working PVC was wiped on every retry. Now only delete PVCs in Pending state.
2. Broken-state detection keyed on Pending pods + unbound PVCs, which is true
during any still-converging install (including ones that timed out but
whose pods eventually came up). Gate on helm status=='failed' + unbound PVCs.
3. k3d install used --wait, which blocks on all kube-prometheus-stack components
(prometheus, alertmanager, node-exporter). They converge async after the
Prometheus operator starts; --wait always timed out. Drop --wait for k3d;
the status_common_services.sh check verifies readiness independently.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
kube-prometheus-stack pulls multiple large images on a fresh k3d install;
600s / 10m is not enough. Set --timeout 20m and process timeout 1320s
(20m + 2m buffer) for k3d, keep 10m for k8s/k3s where images are warm.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs kept prometheus-grafana-0 Pending for 21 days in k3d:
1. update() routed k3d through _values_yaml_k3s() which hardcodes
merlin-local-iscsi-* storage classes and merlin.prole.org node affinity
— neither exists in k3d, so grafana's PVC could never bind.
Add _values_yaml_k3d() that uses local-path (or MONITORING_STORAGE_CLASS
override) with no node affinity; dispatch k3d separately in update().
2. No broken-state recovery for non-k8s mode. Add _purge_broken_monitoring()
(helm uninstall + delete stuck PVCs) and call it from update() when
Pending pods and unbound PVCs are both detected in the monitoring namespace.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The shell path (init_registry.sh apply_registry) is the real k3d registry
entry point. It checked 'k3d registry list knoe-registry' which returned 0
because k3d-knoe-registry existed in 'created' state, so the create was
skipped and k3d-prole-registry (the port-5000 squatter) was never touched.
- init_registry.sh: before the knoe-registry ensure, detect k3d-prole-registry
via both k3d and docker inspect, then stop/rm the Docker container regardless
of whether k3d manages it; also detect knoe-registry in non-running state and
nuke+recreate it (port was unavailable on the previous attempt)
- k3d_registry.py: same cleanup logic for the Python fallback path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k3d-prole-registry was squatting on port 5000, leaving k3d-knoe-registry
in created state. Two bugs made this invisible:
- k3d_registry.py: substring match (name in stdout) matched k3d-knoe-registry
as already-exists without checking STATUS=running; add _k3d_registry_running()
requiring last column == running, detect and delete prole-registry before
creating knoe-registry, recreate if found in non-running state
- status_common_services.sh: grep -qx 'knoe-registry' (exact) never matched
k3d prefix 'k3d-knoe-registry'; fix with awk suffix match + STATUS==running
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k3s manifests still used prole-svc-kong after the rebrand; init_kong.sh
was waiting on `rollout status deployment/knoe-svc-kong` which never
existed, hanging the Common Services milestone.
- Rename kong-deployment.yaml and kong-service.yaml: prole-svc-kong →
knoe-svc-kong (labels, selector, configmap volume ref)
- Remove prole-svc-kong-configmap.yaml (static file replaced by
init_kong.sh dynamic ConfigMap generation; hardcoded namespace and
prole-branded routes were dead weight)
- init_kong.sh: add cleanup_legacy_prole_kong() called from action_update()
to remove stale prole-era resources before deploying knoe-svc-kong
- init_kong.sh: add _is_host_claimed_by_other_ingress() helper; set
include_gitea_host=0 for k3d mode (git is port-forward only, no public
hostname) — keeps the safety-net pre-check for k3s
- actions.py: extend legacy namespace dedupe to sweep prole-svc-kong from
both default and service_ns
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Junie's session targeted the prompt "stabilize ./install.py -c conf/k3d.cfg
using strict TDD" — broad installer-side work, not the k3d-mirror Phase 3
brief I had filed (which she didn't pick up; phase-3 brief stays open). All
750 installer tests pass post-change.
What Junie produced:
install.py (NEW) Top-level CLI entry point. Was
imagined by the prompt but didn't
exist; this commit makes it real.
knoe/deployment.py (NEW) `KnoeDeployment` orchestrator for
the k3s service-mode deploy pipeline.
Wraps Ansible kubeconfig fetch,
opentofu apply, init_*.sh post-apply
scripts, and (optionally) supabase/
deploy.sh.
knoe/ui/screens/cluster.py Dual-cluster GKE kubecontext UI: prod env
knoe/ui/screens/cfg.py now shows separate "App Cluster:" and
"DB Cluster:" dropdowns instead of a
single "Kubernetes Context:" combo.
New _app_kubectx_combo + _db_kubectx_combo
widgets; new app/db_cluster_kubecontext
tk.StringVars.
knoe/core/{actions,env,milestones}.py
knoe/core/ops/storage.py
knoe/config.py, knoe/knoe_conf.py Plumbing changes for the dual-cluster
kubecontext flow + storage-class topology
detection cleanup.
knoe/tools/cleanup_cnpg_storage.py (NEW) Stand-alone cleanup utility.
tools/dashboard.sh (NEW) Dashboard helper.
conf/knoe.cfg (NEW) Master cfg generated by knoe_conf.
conf/dev/ (NEW) Dev-mode cfg directory.
conf/port-mapping.cfg Port mapping tweaks for k3d.
tests/installer/* (8 files) New + extended tests for the dual-cluster
tests/test_database_options.py TUI, kubecontext save flow, storage ops,
topology detection, deploy helpers,
database-options screen.
Issues found in Junie's working state and fixed here:
1. install.py was a 11-line import shim with no shebang, no `chmod +x`,
no `if __name__ == '__main__'` block. `./install.py -c conf/k3d.cfg`
returned `Permission denied` and `python install.py` did nothing.
Added `#!/usr/bin/env python3`, `chmod +x`, and a __main__ block
that delegates to `knoe.ui.screens.main()`. `./install.py --help`
now prints the canonical argparse help.
2. knoe/deployment.py had FIVE `subprocess.run()` call sites with no
`timeout=` argument (`_run_script`, `_run_cmd`, the Ansible playbook
fetch, `tofu init`, `tofu apply`). A hung child process — typical
failure mode is a script waiting on stdin or a stalled network
call — would lock up the installer indefinitely. Added timeouts:
- Ansible kubeconfig fetch: 120s
- tofu init: 300s
- tofu apply, _run_script, _run_cmd: bounded by new module
constant `_MILESTONE_TIMEOUT` (default 1800s = 30 min, override
via `KNOE_MILESTONE_TIMEOUT_SECONDS` env var).
`subprocess.TimeoutExpired` is caught explicitly; on timeout the
run helpers return exit code 124 (conventional timeout code).
3. `conf/k3d.cfg` was corrupted with MagicMock string-reprs on disk:
KNOE_CONF = <MagicMock name='Canvas().tk.call().strip()' id='4743999712'>
argocd.node_selector = <MagicMock name='mock.StringVar().get().strip()' id='...'>
Likely path: Junie ran `./install.py -c conf/k3d.cfg` interactively
in a non-Tk environment (or with a partially-mocked widget set) and
the installer's "save current state" path wrote the mock-objects'
`__repr__` strings into the cfg file. This commit reverts the cfg
to its pre-Junie state. **Followup: harden the cfg save path
against non-string widget values** — track separately.
4. The corrupted cfg caused the installer to call `os.makedirs()` on
the mock-string values, producing 10 directories on disk literally
named `<MagicMock name='Canvas().tk.call().strip()' id='4733210304'>/`
etc., with 5–86 files of install artifacts inside each. Removed.
The "final step is timing out" the user reported was almost certainly
issue #2 above: install.py walked the milestone pipeline, hit one of
the unbounded subprocess.run calls, and the wrapped command (probably
supabase/deploy.sh, which Junie was reading for context when her
session timed out) hung. With the timeouts in place that path now
exits cleanly with rc=124 instead of locking up.
Verification:
- pytest tests/installer/ -q 750 passed in ~25s
- python3 -c "import knoe.deployment" imports clean
- ./install.py --help prints argparse help
- find . -maxdepth 1 -type d -name '<MagicMock*' | wc -l 0
- head -7 conf/k3d.cfg clean (no MagicMock)
Out of scope for this commit (followups):
- The cfg save-path that wrote mock-objects-as-strings (issue #3 root cause).
Reproducer: launch the installer in an env where Tk widget vars are
`unittest.mock.MagicMock` instances. The cfg save code should refuse to
serialize non-str values rather than calling `str()` on a MagicMock.
- The k3d-mirror Phase 3 brief (`docs/plans/junie/k3d-knoe-auth-pod-deploy.md`)
is still open — Junie picked a different prompt this round.
Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 (host loop) and the Phase 2 OIDC sandbox are shipped; this
brief queues Phase 3 of k3d-mirror-of-GKE: build the knoe-auth image,
k3d-import, run as a pod inside the cluster. Pre-merge smoke loop —
not the daily inner loop.
Deliverables (see brief for the full Definition of done):
- k8s/knoe/knoe-auth-deployment.yaml (NEW; sibling of GKE manifest)
- scripts/k3d-knoe-deploy.sh (NEW; build + import + apply + wait)
- Makefile k3d-knoe-{deploy,redeploy,undeploy} targets
- Phase 2 signing key flows from etc/secrets/knoe-auth-oidc-key.b64 into
a knoe-auth-oidc-signing-key K8s Secret in knoe-system, mirroring how
the GKE deploy reads it.
Index updates:
docs/plans/junie/README.md — k3d Phase 3 in Active row
docs/TODO.md §"In progress" — promotes the brief to top
docs/plans/k3d-gke-mirror.md — banner + §6 Phase 3 entry
flipped from "out of scope"
to "in flight"
Three independent lines of work landing in one commit because they were all
on disk together end-of-session and the cross-file edits (TODO, knoe-system,
junie/README) interleave cleanly:
1. **Queue #3 — image rename `knoe-authority` → `knoe-auth`** (Junie).
Closes drift R6. The Maven artifact has been `knoe-auth.jar` since
commit b355855; the deploy manifests now match.
authority/Dockerfile.app (NEW)
deploy/gcp/gke/knoe-auth-deployment.yaml (3 image tags renamed)
deploy/opentofu/k3s/manifests/knoe/knoe-auth-deployment.yaml (2 image tags renamed)
Makefile (docker-build-auth +
docker-push-auth + REGISTRY/KNOE_AUTH_VERSION
defaults)
docs/plans/junie/03-image-rename-knoe-authority-to-knoe-auth.md (brief, kept as design record)
2. **Phase 2 OIDC provider — GKE deploy** (Junie). Source landed via the
merge that brought claude/crazy-bose-fec256 back; the k3d sandbox
shipped earlier today (commit 93157b0). This commit completes the GKE
path: Kong route `/auth` → knoe-auth, default values added, signing
key wired via `knoe-auth-oidc` secret per the brief.
supabase/helm/knoe-supabase/templates/kong/config.yaml (Kong route /auth, gated
on .Values.knoeAuth.enabled)
supabase/helm/knoe-supabase/values.yaml (knoeAuth.enabled=false default;
studioIngress block — also
fixes the helm lint issue noted
in earlier commit 03bb731)
docs/plans/junie/phase2-oidc-gke-deploy.md (brief, kept as design record)
3. **k3d dev-user seed: `chrisfu` + `knoe_developer`** (Claude). Closes
"I want auth as chrisfu@knoey.com to my local cnpg database" for the
local dev loop. `etc/init_knoe_auth.sh --mode k3d` now runs
`seed_dev_users_k3d()` after the schema bootstrap, creating:
- `knoe_developer` group role with R/W on `knoe`+`public` (mirrors
the GKE production layout from docs/db-access.md; was hand-rolled
in production per the 2026-04-30 onboarding work, never baked into
postInitTemplateSQL).
- `chrisfu` LOGIN role with password `chrisfu-dev` (idempotent —
resets on every `make k3d-knoe-up`, so the rebuild loop is
deterministic). Granted into `knoe_developer`.
From the host with port-forward up:
PGPASSWORD=chrisfu-dev psql "postgresql://chrisfu@localhost:5432/knoe-db?sslmode=require"
Cross-cutting doc updates:
docs/TODO.md — Phase 2 GKE entry removed from "In progress"
(now "(none — all items shipped or paused)");
queue #3 + reality table R6 removed; pg_oauth
paused note flipped to "OIDC issuer now
deployed to GKE — pg_oauth can resume"; Done
section gets new entries for #3, Phase 2 GKE,
and chrisfu seed.
docs/knoe-system.md — Phase 2 GKE row → Shipped; pg_oauth → Ready;
knoe-auth-deployment.yaml note "knoe-auth:latest"
instead of "knoe-authority:latest"; "One-time
cluster setup" mentions the chrisfu seed.
docs/local-dev-knoe-auth.md — "Verify psql connectivity" rewritten to
connect as chrisfu (was `knoe`); new "Reset /
rebuild loop" section explaining idempotency.
docs/plans/junie/README.md — two new rows under Shipped: brief 03 and
phase2-oidc-gke-deploy.
Verification (the bits I ran locally):
- bash -n etc/init_knoe_auth.sh OK
- The seed SQL is idempotent (DO blocks with EXISTS checks +
ALTER ROLE on the password reset path)
- The Kong /auth route is gated on .Values.knoeAuth.enabled — default
false, no behavior change until someone flips it on per Junie's brief
Out of scope for this commit:
- Actually flipping knoeAuth.enabled=true on the live GKE chart and
rolling out — Junie's brief covers the runbook; needs the
knoe-auth-oidc K8s secret populated from 1Password first
- Rebuilding + pushing the new knoe-auth:latest image — `make
docker-push-auth` is wired, just hasn't been run yet
- Round 1.5 OpenBao transit-key encryption (still queued)
Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Completes the Phase 2 OIDC laptop dev path. Source for the OIDC surface
(discovery, authorize, token, userinfo, JWKS controllers + signing /
session services) landed on `main` via the merge that brought
claude/crazy-bose-fec256 back. This commit makes Phase 2 actually
exercisable in the k3d dev loop without any GKE deploy.
What's new:
authority/src/main/resources/application-k3d.yml
Spring profile activated by `-Dspring-boot.run.profiles=k3d`.
Enables OIDC (knoe.oidc.enabled=true), points the issuer at
`http://localhost:8080`, sets Kerberos realm to KNOE.LOCAL, and
aligns the datasource with the port-forwarded localhost:5432 DB.
etc/gen_oidc_signing_key.sh (executable)
Idempotent RS256 PKCS#8 keypair generator. Outputs:
etc/secrets/knoe-auth-oidc-key.pem (PEM)
etc/secrets/knoe-auth-oidc-key.b64 (single-line base64 of DER —
directly consumable as KNOE_AUTH_OIDC_SIGNING_KEY by
OidcTokenService.init())
`etc/secrets/` is already gitignored. Set FORCE=1 to rotate.
What's wired:
scripts/k3d-knoe-up.sh
New §7 calls etc/gen_oidc_signing_key.sh after schema seed. Keypair
persists across `make k3d-knoe-down && make k3d-knoe-up` cycles.
scripts/k3d-knoe-pf.sh
Output now includes the KNOE_AUTH_OIDC_SIGNING_KEY export line, the
full `mvn spring-boot:run` invocation with -Dspring-boot.run.profiles=k3d,
and the three OIDC endpoints to curl-test.
What's documented:
docs/local-dev-knoe-auth.md
"Daily loop" Terminal B: now exports KNOE_AUTH_OIDC_SIGNING_KEY,
runs with `-Dspring-boot.run.profiles=k3d`, and the verify section
includes /jwks.json. IntelliJ run config: adds Active Profiles: k3d
and a note about pasting the b64 directly (no shell expansion in
the env-var field).
docs/knoe-system.md
Phase 2 status row split: "k3d setup" → Shipped, "GKE deploy" →
Pending. The "Open work items" Phase 2 entry rewritten to flag
that the GKE deploy is the remaining thread (gated on queue #3 for
the image rebuild as `knoe-auth:latest`).
docs/TODO.md
Promoted "Phase 2 OIDC provider — GKE deploy" into §In progress
(replacing the empty "(none)" placeholder). Done section updated
with two entries: the k3d Phase 1 dev loop (Junie's c3...) and
this Phase 2 OIDC k3d sandbox.
End-to-end loop the engineer can run:
make k3d-knoe-up # one-time, ~5 min
make k3d-knoe-pf & # port-forwards
export KRB5_CONFIG=$PWD/etc/krb5.local.conf
export KNOE_AUTH_OIDC_SIGNING_KEY=$(cat etc/secrets/knoe-auth-oidc-key.b64)
mvn -pl authority spring-boot:run \
-Dspring-boot.run.jvmArguments="-Djava.security.krb5.conf=$PWD/etc/krb5.local.conf" \
-Dspring-boot.run.profiles=k3d
# then:
curl -s http://localhost:8080/.well-known/openid-configuration | jq .issuer
# → "http://localhost:8080"
curl -s http://localhost:8080/jwks.json | jq '.keys[0].kty'
# → "RSA"
Verified locally: keypair generator round-trips through openssl pkey -inform DER
(produces valid 2048-bit RSA keys); idempotent (existing key kept by default,
FORCE=1 rotates); bash -n clean on all 5 touched scripts.
Out of scope (TODO §In progress captures it):
- GKE deploy of Phase 2 (image rebuild + K8s Secret + deployment env vars)
- SPNEGO E2E from host browsers (k3d-mirror Phase 2)
- knoe-auth-as-pod in k3d (k3d-mirror Phase 3)
- OidcCodeService DB persistence (separate track)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings up the smallest k3d-resident stack that lets a host-side knoe-auth
(run via `mvn spring-boot:run` or IntelliJ) iterate against real Postgres
+ Kerberos. Closes Phase 1 of the k3d-gke-mirror plan (docs/plans/k3d-gke-mirror.md).
Scope:
- k8s/knoe/knoe-kdc-{configmap,deployment,service,pvc,init-job}.yaml
NEW; standalone KDC, realm KNOE.LOCAL (distinct from KNOE.DEV).
- etc/init_knoe_auth.sh: --mode k3d flag added; swaps realm + skips
GCP-specific steps. GKE behavior unchanged when flag absent.
- Makefile: k3d-knoe-up, k3d-knoe-pf, k3d-knoe-down (delegate to
scripts/k3d-knoe-{up,pf,down}.sh).
- scripts/k3d-knoe-{up,pf,down,smoke}.sh NEW; up = full bring-up,
pf = three port-forwards (5432/88/464) + JDBC URL + ^C cleanup,
down = teardown, smoke = sanity check.
- etc/krb5.local.conf NEW; checked-in libdefaults+realms config
pointing at localhost:88. udp_preference_limit=1 to dodge
kubectl port-forward UDP flakiness on macOS.
- docs/local-dev-knoe-auth.md NEW; one-time setup + daily loop +
IntelliJ run config.
- docs/knoe-system.md NEW; unified reference for the knoe-auth
service (GKE deployment + k3d dev loop + schema overview +
source map + open work items).
Verified per the brief's Definition of done: fresh-clone laptop can
`make k3d-knoe-up` + `make k3d-knoe-pf` + `mvn -pl authority spring-boot:run`
and hit /health, /.well-known/openid-configuration in <8 minutes.
Out of scope (parent plan docs/plans/k3d-gke-mirror.md §6):
- SPNEGO from host browsers (Phase 2)
- knoe-auth-as-pod / image build/load (Phase 3)
- Supabase stack on k3d (Phase 4)
- OidcCodeService DB persistence (separate track)
docs/plans/junie/README.md — k3d brief moved from Active to Shipped.
docs/TODO.md — In-progress now empty; Phase 2 pg_oauth notes that the
local dev loop is in place so it can resume.
Closes Phase 1; Phase 2+ briefs filed as needed.
Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The next development thread is knoe-auth Phase 2 OIDC iteration, which
just landed on main but has no fast inner-loop. Instead of bouncing
edits through the GKE deploy chain, we want a laptop-resident model
where knoe-auth runs from the IDE / mvn against real Postgres + KDC
brought up in k3d.
Two new docs:
docs/plans/k3d-gke-mirror.md
Architectural plan. Top-to-bottom rationale for what the k3d
model should mirror, what we deliberately skip (supabase, kong,
oauth2-proxy, Studio — all unnecessary for knoe-auth's surfaces),
and the phasing. Captures the three scope decisions taken
2026-05-02 with the user:
- knoe-auth runs on the laptop, not as a pod (faster inner loop)
- stack scope is CNPG + KDC only
- single-replica CNPG, realm KNOE.LOCAL (distinct from KNOE.DEV)
Out-of-scope for Phase 1 listed explicitly so reviewer
expectations match scope: SPNEGO E2E, image-build-and-load,
supabase, oauth2-proxy, OidcCodeService DB persistence.
docs/plans/junie/k3d-knoe-auth-dev-loop.md
Phase 1 brief. Concrete deliverables:
- k8s/knoe/knoe-kdc-{deployment,service,configmap,pvc,init-job}.yaml
- etc/init_knoe_auth.sh --mode k3d flag (or sibling script)
- Makefile: k3d-knoe-{up,pf,down}
- etc/krb5.local.conf (with udp_preference_limit=1 for
kubectl port-forward UDP flakiness on macOS)
- docs/local-dev-knoe-auth.md (engineer-facing)
- scripts/k3d-knoe-smoke.sh
Definition of done: a fresh-clone laptop can `make k3d-knoe-up`
+ `make k3d-knoe-pf` + `mvn -pl authority spring-boot:run` and
hit /health, /.well-known/openid-configuration in <8 minutes.
Index updates:
docs/plans/README.md — k3d-gke-mirror.md row added
docs/plans/junie/README.md — split into Active / Shipped;
k3d brief listed under Active;
the 5 shipped-2026-05-02 briefs
moved to Shipped with commit refs.
docs/TODO.md In-progress — k3d brief now top of "In progress";
the previously in-flight "Phase 2
pg_oauth in install/deploy.sh"
item demoted to "paused" (resume
after the local dev loop lands).
No code changes; all docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirroring the 2026-04-29 DB-cluster removal. Live state showed garage
in an unconfigured limbo (layout v0, no role assigned, 21d of "Ring
not yet ready" warnings); nothing outside garage's own state
referenced its S3 endpoint anymore.
Removed from knoe-system namespace on knoe-dev-0:
- statefulset/garage (1/1 pod, 21d uptime)
- service/garage (ClusterIP; no garage-s3-ilb on app cluster)
- configmap/garage-config
- secret/garage-secrets
- pvc/data-garage-0 (29 Gi, garage-hdd)
- pv/pvc-656e1936-... (was Bound, reclaim=Retain)
- storageclass/garage-hdd
Underlying GCE pd-standard disk (us-west3-c/pvc-656e1936-...) reclaimed
via `gcloud compute disks delete` since reclaim=Retain leaves the disk
behind after the PV is gone.
Source manifests intact:
- k8s/knoe/garage-*.yaml (k3d, min)
- deploy/opentofu/k3s/manifests/knoe/garage-*.yaml (k3s)
- etc/init_garage_store.sh + knoe/core/ops/k8s_garage_store.py
- k8s/knoe/garage-statefulset-gcp.yaml (GKE variant)
A future ./install.sh against GKE could re-deploy garage if that's
what the deploy mode wires; whether it SHOULD wire garage given GCS
now handles backups is a separate cleanup decision.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Queue #5 Phase B orphan cleanup landed:
App cluster (knoe-dev-0):
- gitlab/gitlab-migrations-58a3e27-97-23-n72j8 (Failed/Evicted,
11d stale)
- pv/pvc-42e8545b-… (10Gi, monitoring/storage-kps-grafana-0)
- pv/pvc-86422179-… (30Gi, monitoring/prometheus-…-prometheus-0)
- pv/pvc-e2c29f7a-… (5Gi, monitoring/alertmanager-…-alertmanager-0)
DB cluster (knoe-dev-cnpg-0):
- kube-system/prole-supabase-dirprep Job (FailureTarget, 22d stale)
Post-state: 0 Released PVs on app cluster; gitlab namespace has only
the Completed migration pod; DB-cluster kube-system has no prole-*
artifacts. Live-cluster ops only; no source changes.
Queue #4 reframed: tried the planned `garage repair --yes blocks`,
but found garage in an unconfigured state — layout v0, no role
assigned, "Ring not yet ready" warnings continuous since whenever the
layout was lost (likely during a reconcile after the 2026-04-29 PVC
resize). Cross-cluster sweep shows nothing outside garage references
its S3 endpoint anymore (CNPG backups moved to GCS, no other workload
points at 10.180.15.239:3900). Garage on app-cluster is now a dead
pod holding an 80 GiB PVC. The original block-repair task is moot.
Reframed as a decommission-or-restore decision; details captured in
the ranked queue entry. Awaiting direction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a 2026-05-02 Done entry covering the cluster-side work that paired
with c3fae73 (source) and 03bb731 (chart cleanups): PodMonitor migration
applied with verified zero-data-loss handoff; supabase chart upgraded
revision 1→3; helm field-manager conflict on Kong configmap resolved;
node label drift on knoe-dev-0 closed (3 nodes labelled
knoe.dev/node-role=general alongside legacy prole.org/node-role); DB
cluster nodes still pending same treatment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two follow-ups to commit c3fae73 (queue #15) surfaced when applying
the DASHBOARD removal to live `supabase` release on knoe-dev-0:
1. **Delete `templates/test/kong.yaml`.** This was a Helm Test Job that
validated the Studio dashboard's basic-auth gate. The basic-auth
plugin was retired in commit 25f1b2e (oauth2-proxy fronting); the
consumer + credentials in commit c3fae73. The test has nothing
left to test — it referenced `supabase.secret.dashboard` template
(gone) and `.Values.secret.dashboard.{secretRef,secretRefKey}`
(gone), causing `helm template`/`upgrade` to fail. Removing the
file is the right move; if dashboard auth ever comes back as
something else, write a fresh test for that.
2. **Add `secret.oauth2proxy` + clarify `deployment.oauth2proxy`
defaults in values.yaml.** The chart has
`templates/studio/oauth2proxy-{deployment,service}.yaml` and
`templates/secrets/oauth2proxy.yaml` (Junie's Phase 2 OIDC adds)
guarded by `if .Values.deployment.oauth2proxy.enabled`. The guard
short-circuits when the parent map exists, but if the map itself
is undefined helm fails with `nil pointer evaluating interface{}.enabled`.
Added a `secret.oauth2proxy` block (empty defaults) so the secrets
template guard can evaluate cleanly. `deployment.oauth2proxy.enabled`
was already `false` in values.yaml; added a comment explaining the
chart oauth2-proxy templates are vestigial — the deployed
oauth2-proxy lives in `deploy/gcp/gke/oauth2-proxy-deployment.yaml`,
in FRONT of supabase-kong (commit 25f1b2e), not as a Kong upstream.
Live state after this commit:
- `helm upgrade supabase ./supabase/helm/knoe-supabase
--reset-then-reuse-values` succeeds.
- revision 3 deployed; configmap field manager = `helm` (sole owner).
- `kubectl get cm supabase-kong -o yaml | grep DASHBOARD` returns 0
matches.
- All Kong env vars on the new pod are clean (no DASHBOARD_*).
- External smoke tests:
/onboard.html → 200
/rest/v1/ no auth → 401
/ (Studio gate) → 302 to Google
/healthz → 302
svc.knoe.dev/grafana/login → 200
- cnpg-grafana panels still populate (PodMonitor migration verified
earlier in same session: 3 postgres exporter targets up; cluster
label injection working).
Side-finding handled live: nodes lacked the new
`knoe.dev/node-role=general` label that Junie's chart defaults to in
nodeAffinity. Added the label to all 3 knoe-dev-0 nodes alongside the
legacy `prole.org/node-role=general`. The label rename has been
tracked in CLAUDE.md as a follow-up; this is partial closure for the
app cluster (DB cluster nodes still need the same treatment when next
relevant).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three Junie briefs landed in one commit:
#07 — Wire cnpg-backup-sa into CNPG cluster spec (drift R8)
deploy/gcp/gke/knoe-db.yaml: add spec.serviceAccountName: cnpg-backup-sa
(requires CNPG v1.29+, which is the live operator version).
etc/init_cnpg_gke.sh: operator install URL now uses CNPG_OPERATOR_VERSION
variable (default 1.29.0); new §11 patches knoe-db and
knoe-db-barman-cloud RoleBindings to add cnpg-backup-sa as a subject
if not already present — matching the 2026-04-29 live stabilization.
#13 — Migrate off deprecated enablePodMonitor + podMonitorRelabelings
Both deprecated fields removed from deploy/gcp/gke/knoe-db.yaml
spec.monitoring. New deploy/gcp/gke/knoe-db-podmonitor.yaml carries
the PodMonitor with the cluster relabeling rule (cnpg.io/cluster pod
label → cluster label; required for all 85 CNPG Grafana panels).
Apply alongside knoe-db.yaml on next cluster patch.
#15 — Remove dead DASHBOARD consumer + basicauth_credentials
supabase/helm/knoe-supabase:
- wrapper.sh: drop DASHBOARD_USERNAME / DASHBOARD_PASSWORD envsubst lines
- config.yaml: drop DASHBOARD consumer + basicauth_credentials block
- kong/deployment.yaml: drop both DASHBOARD env-var secret refs
- values.yaml: rename secret.dashboard → secret.openai (apiKey only;
username/password dropped — no enforcer since commit 25f1b2e)
- secrets/dashboard.yaml + _helpers.tpl: renamed to openai /
supabase.secret.openai
- studio/deployment.yaml: reads from secret.openai.apiKey
- ci/example.yaml: updated to secret.openai.apiKey
helm template confirms knoe-supabase-openai secret referenced; no
DASHBOARD output.
docs/TODO.md: queue items #7, #13, #15 + drift row R8 archived to Done.
Co-authored-by: Junie <junie@jetbrains.com>
Defects A, B, C from docs/plans/junie/06-patch-garage-script-fixes.md:
A. DB_CLUSTER default was knoe-cnpg-0 (stale rebrand artifact);
corrected to knoe-dev-cnpg-0 to match conf/gke.cfg and the
cluster table in CLAUDE.md. Header comment + overrides block
also updated.
B. Phase 1 deletion loop missed service/garage-s3-ilb (the
LoadBalancer the ObjectStore endpoint historically pointed at).
Now included alongside service/garage.
C. Phase 2 was applying knoe-db-backup-gcs.yaml whole, including a
legacy ScheduledBackup using method:barmanObjectStore (being
removed in CNPG v1.30). Manifest split handled in prior commit
(34a25dd); script's Phase 2 is now safe to re-run.
bash -n clean; CONFIRM=false dry-run prints knoe-dev-cnpg-0 correctly.
docs/TODO.md: queue items #2 and #6 + drift rows R5 and R9 archived to Done.
Closes queue item #6 in docs/TODO.md (drift R9).
Co-authored-by: Junie <junie@jetbrains.com>
deploy/gcp/gke/knoe-db-backup-gcs.yaml previously contained both the
cnpg-backup-sa ServiceAccount (wanted) and a ScheduledBackup using
method: barmanObjectStore (unwanted — legacy in-tree method being
removed in CNPG v1.30, and the live cluster already has a plugin-method
ScheduledBackup that would be duplicated on re-apply).
Verified no other consumers of knoe-db-daily in the repo (grep clean).
File is now SA-only, matching its filename's intent.
Part 1/2 of queue item #6 fix (docs/plans/junie/06-patch-garage-script-fixes.md).
Co-authored-by: Junie <junie@jetbrains.com>
Mirrors commit b355855 (auth-side rename). The kustomization at
deploy/opentofu/k3s/manifests/knoe/kustomization.yaml has been
referencing knoe-* names since the rebrand; kustomize was broken
on the k3s deploy mode until this lands.
Five git mv:
prole-configmap.yaml -> knoe-configmap.yaml
prole-deployment.yaml -> knoe-deployment.yaml
prole-service.yaml -> knoe-service.yaml
prole-kdc-configmap.yaml -> knoe-kdc-configmap.yaml
prole-kdc-secrets.example.yaml -> knoe-kdc-secrets.example.yaml
Plus content sweep of knoe-kdc-configmap.yaml: replaced stale
PROLE.LOCAL/PROLE.ORG realm refs and PROLE_KDC_* env vars with
KNOE.DEV/KNOE_KDC_* to match the canonical GKE configmap.
Closes queue item #2 in docs/TODO.md (drift R5).
Co-authored-by: Junie <junie@jetbrains.com>
Bringing the long-running session-feature branch back into main in one
deliberate sweep. The branch carried the cluster work that's been live for
weeks (cross-cluster CNPG metrics, Grafana w/ Google OAuth, supabase
oauth2-proxy, cluster recovery, pg.0.knoe.dev + per-engineer onboarding,
GCS-backed CNPG backups via Workload Identity, the env-contamination
guard, the Junie brief queue, the cnpg-grafana CSRF + memory-request
fixes from today), while main accumulated Junie's parallel knoe-auth
Phase 2 OIDC work (full provider surface: discovery, authorize, token,
userinfo, JWKS, RS256 signing, code exchange, session services).
Key decision: the two branches did COMPETING rebrands off the same
starting point (5ba9b63, 2026-04-27):
- claude branch (commit b355855, earlier): org.prole.authority.* →
dev.knoe.auth.*
(artifact renamed to
knoe-auth.jar)
- main (commit 9daa94b, recent): org.prole.authority.* →
dev.knoe.authority.*
(kept "authority" artifact name)
dev.knoe.auth wins: cluster runs from this name, the Maven artifact is
already knoe-auth.jar, and the broader rename is the documented
namespace direction (per ~/.claude/projects/-Users-chrisfu-dev-knoe-db/
memory/MEMORY.md). All of main's recent Phase 2 OIDC content was ported
from authority/src/.../dev/knoe/authority/ into
authority/src/.../dev/knoe/auth/ with package declarations rewritten.
== File-level resolution summary ==
Textual conflicts (4):
authority/pom.xml
- Took our artifactId="auth"
- Took our branch's removal of spring-security-kerberos-client
(verified: Junie's Phase 2 OIDC code does not import it; the dep
was already-dead config)
docs/pipeline-phases.md
- Took our branch's "Phase 1 not started" status. Main had a
misplaced "✅ Complete" with a knoe-auth-Phase-1 commit ref
in the autobuild Phase 1 section — different domain.
docs/plans/knoe-auth-round-1.md
- Took our branch's dev.knoe.auth file table (vs main's
dev.knoe.authority listing). Pure rename mismatch.
supabase/helm/knoe-supabase/templates/kong/config.yaml
- Took our branch's onboard route + plain dashboard wiring.
Main had an oauth2proxy.enabled toggle that put oauth2-proxy as
a Kong upstream — but the deployed architecture (commit 25f1b2e)
has oauth2-proxy in FRONT of Kong, not behind. Main's wrapper
reflected an architecture that was never deployed.
- Took our branch's removal of basic-auth from dashboard route
(queue #15 brief still tracks the matching values.yaml /
kong/deployment.yaml cleanup).
Java tree reconciliation (44 file-pairs):
20 dual-path source files + 2 dual-path tests
Body-identical between main's authority/ and our branch's auth/
after stripping package decls — main's commit 9daa94b was a pure
rebrand. Took our branch's auth/ version for all 22.
8 main-only source files (Phase 2 OIDC), ported into auth/:
web/JwksController.java
web/OidcAuthorizeController.java
web/OidcDiscoveryController.java
web/OidcTokenController.java
web/OidcUserInfoController.java
session/OidcCodeService.java
session/OidcTokenService.java
session/SessionService.java
12 main-only test files, ported into auth/:
HealthControllerTest.java
enroll/EnrollValueTypesTest.java
enroll/EnrollmentControllerTest.java
enroll/TotpServiceTest.java
kerberos/KadminClientTest.java
kerberos/KerberosSpnegoResultTest.java
web/LoginControllerTest.java
admin/AdminControllerTest.java
user/PrincipalNormalizerTest.java
regression/IdentityRegressionTest.java
session/OidcCodeServiceTest.java
session/SessionServiceTest.java
Port mechanics: read main:authority/...<file> via git show, then sed
rewrite `package dev.knoe.authority` → `package dev.knoe.auth` and
`import dev.knoe.authority` → `import dev.knoe.auth`. Body content
unchanged.
authority/src/main/java/dev/knoe/authority/ — DELETED (duplicate)
authority/src/test/java/dev/knoe/authority/ — DELETED (duplicate)
== Verification ==
- grep -rln '<<<<<<<' across .java/.md/.yaml/.yml/.sh/.xml/.tpl: clean
- find authority/src -path '*/dev/knoe/authority*': empty (subtree gone)
- grep 'package dev.knoe.authority' across repo: clean
- bash -n install.sh deploy.sh etc/preflight_kubecontext.sh: clean
- git ls-files -u | wc -l: 0 unmerged paths
- helm lint supabase/helm/knoe-supabase: pre-existing failure on
studioIngress.enabled undefined in values.yaml (introduced by Junie
on main; unrelated to this merge — flagging as follow-up).
== Followups (carried into TODO ranked queue or noted here) ==
- helm lint failure: studioIngress block in values.yaml is missing
enable flag; templates/studio/{ingress,oauth2proxy-deployment,
oauth2proxy-service}.yaml all reference studioIngress.enabled with
no default. Pre-existing on main; not introduced by this merge.
- The five Junie briefs filed on this branch are now reachable from
main at docs/plans/junie/{02,06,07,13,15}-*.md. Junie can pick them
up in any order.
- knoe-auth Phase 2 OIDC source (now at dev.knoe.auth.*) is not yet
deployed to the cluster. Deployment is its own task.
- The branch claude/crazy-bose-fec256 stays in place (worktree at
.claude/worktrees/crazy-bose-fec256 may have ongoing context for
Claude Code sessions). Safe to delete once next session starts
cleanly from main.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Filed in response to the 2026-04-28 14:00 UTC backup outage. An
`install.sh --mode k3d` run with the shell pointed at GKE silently
overwrote the GKE cluster's GCS-backed ObjectStore + ScheduledBackup
with k3d-mode defaults; Garage filled up and CNPG backups failed for
hours before the next manual check. The class of bug is "config says
target cluster A, shell context says target cluster B, installer
proceeds against B without warning."
New shared bash helper at etc/preflight_kubecontext.sh with two
functions:
- verify_kubecontext_matches_config <cfg-path>
Strict gate. Reads [Global] APP_CLUSTER_KUBECONTEXT from the
config and exits 1 if `kubectl config current-context` differs.
Skipped silently when the config has no baked APP_CLUSTER_KUBECONTEXT
(e.g. fresh k3d.cfg) or when there's no live current-context.
- print_kubecontext_notice
Informational. Prints what's about to be inherited so the user
can abort before the TUI launches if it looks wrong. Never fails.
Wiring:
- deploy.sh sources the helper and calls the strict gate against
${PROLE_DEPLOY_CFG:-conf/gke.cfg} before invoking Python.
Unattended path -> hard refusal on mismatch.
- install.sh sources the helper and calls the informational notice
(gated on not-`--min`) right after entering the local-checkout
branch. The TUI is interactive, so the strict mode-aware gate is
a follow-up once the welcome screen records a mode in
state.inputs.
Bypass for deliberate cross-cluster maintenance:
KNOE_SKIP_KUBECONTEXT_GUARD=true ./deploy.sh
End-to-end verified:
- deploy.sh with current=cnpg-0, gke.cfg=app-0 -> exit 1, clear msg
- deploy.sh with KNOE_SKIP_...=true -> bypasses, prints
"skipping check"
- install.sh --min -> notice skipped
- install.sh (no flag) and install.sh --silent -> notice printed
Doc updates:
- CLAUDE.md §"Env-contamination warning" rewritten to describe the
live guard (was a forward-looking TODO).
- CLAUDE.md drift table row R4 removed; "Closed 2026-05-01" line added.
- docs/TODO.md queue item #1 archived to Done; R4 dropped from the
reality-vs-intent table. Queue numbering retained (no #1 placeholder)
so the docs/plans/junie/<NN>-...md filenames still match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five self-contained work briefs in a new docs/plans/junie/ subdirectory,
each tagged against a numbered item in docs/TODO.md so Junie can take
them independently in any order.
02-k3s-prole-rename.md -> queue #2 (drift R5)
06-patch-garage-script-fixes.md -> queue #6 (drift R9)
07-init-cnpg-gke-sa-wiring.md -> queue #7 (drift R8)
13-podmonitor-manual-management.md-> queue #13
15-remove-dead-dashboard-consumer.md-> queue #15
Each brief follows the same shape: Why -> What changes (concrete file
paths + line numbers + before/after) -> Verification -> Out of scope ->
Commit shape -> Definition of done. The intent is that Junie reads cold
(no shared chat history) and lands the change without escalating
questions.
Also adds:
- docs/plans/junie/README.md describing the convention.
- Row in docs/plans/README.md so newcomers find the subdirectory.
- Brief reference + "Assigned to Junie" tag on each of the five queue
items in docs/TODO.md.
Existing in-progress assignment to Junie (Phase 2 pg_oauth) is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Implement Google OIDC support in Authority module via GoogleOAuthService
- Update AuthProperties and application.yml with OIDC configuration
- Add oidc-setup.md documentation for GKE/Google Cloud setup
- Update etc/init_knoe_auth.sh to handle OIDC secrets and path-B configuration
- Configure knoe-auth-deployment.yaml and gke.cfg for production auth
Co-authored-by: Junie <junie@jetbrains.com>
pytest_all.xml updated by IntelliJ to bind SDK name
'Python 3.14 (knoe-db)' and add PYTHONUNBUFFERED=1.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds authority module to root POM, updates Spring Boot and
dependency versions in authority/pom.xml, application.yml
updated for KNOE.DEV realm defaults.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Java implementation of the knoe-auth identity backbone (Round 1):
authority/.../admin/ user admin REST endpoints (list, deactivate,
reset-totp, role management)
authority/.../enroll/ invite-OTP web enrollment flow — accepts invite
token, creates Kerberos principal via kadmin,
seeds TOTP secret, returns keytab
authority/.../provisioning/ contributor provisioning service — orchestrates
invite generation, principal lifecycle, role grants
authority/.../kerberos/
KadminClient.java typed wrapper around the kadmin(1) subprocess;
create/delete/get-keytab/change-password
GKE manifests:
deploy/gcp/gke/knoe-auth-deployment.yaml two-container Pod (knoe-auth HTTP
+ kdc sidecar) with keytab-bootstrap
initContainer
deploy/gcp/gke/knoe-kdc-configmap.yaml krb5.conf, kdc.conf, kadm5.acl and
entrypoint for the embedded KDC
deploy/gcp/gke/knoe-kdc-secrets.yaml placeholder template — real values
created by init_knoe_auth.sh via 1Password
See docs/plans/knoe-auth-round-1.md for full architectural narrative.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds docs/plans/ as the canonical engineering reference for completed and
in-flight initiatives. Written for jr/mid engineers who have not seen the
repo before — each plan starts with strategic context and links to existing
code before asking for changes.
README.md index, audience, and status conventions
deployment-modes.md four-mode installer (min/k3d/k3s/gke), welcome-screen
mode selector, min-mode fast-path. Status: shipped.
knoe-auth-round-1.md Kerberos KNOE.DEV realm, invite-OTP enrollment,
Google corroboration, TOTP 2FA. Status: operational.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
| `knoe-cnpg-0` | `gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0` | DB cluster — CNPG/PostgreSQL only |
| `knoe-dev-cnpg-0` | `gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0` | DB cluster — CNPG/PostgreSQL only |
**Critical:** Garage must NEVER be deployed to `knoe-cnpg-0`. SSD quota (300 GB) is fully consumed by CNPG — all non-CNPG PVCs must use `standard` storage class (HDD), not `standard-rwo`/`premium-rwo`.
**Critical:** SSD quota (300 GB) is fully consumed by CNPG — all non-CNPG PVCs must use `standard` storage class (HDD), not `standard-rwo`/`premium-rwo`.**Garage runs ONLY on `knoe-dev-0`** (removed from DB cluster on 2026-04-29).
### Deployment environments / modes
### Deployment environments / modes
| `cluster_env` | `KNOE_MODE` | Target |
| `cluster_env` | `KNOE_MODE` | Target |
@ -110,8 +110,13 @@ Missing `init_cluster.app_cluster_kubecontext` in config causes Garage to deploy
## CNPG / backup specifics
## CNPG / backup specifics
- CNPG backups go to **GCS** (not Garage): `gs://knoe-0-backups/` and `gs://knoe-0-wal/`
- CNPG backups go to **GCS**: `gs://knoe-0-backups/` (single bucket; `knoe-db/base/` and `knoe-db/wals/` prefixes). `gs://knoe-0-wal/` exists but is unused.
- GCP SA: `cnpg-backup@plenary-truck-485623-p7.iam.gserviceaccount.com` (`storage.objectAdmin` + `storage.legacyBucketReader` on the bucket)
- K8s SA: cluster pods run as **`cnpg-backup-sa`** in `knoe-db-0`, set via `cluster.spec.serviceAccountName: cnpg-backup-sa`. The SA has the `iam.gke.io/gcp-service-account` annotation. RoleBindings `knoe-db` and `knoe-db-barman-cloud` include `cnpg-backup-sa` as a subject so the pod has the same RBAC the auto-generated SA would have had.
- ObjectStore manifest: `k8s/knoe/knoe-db-barman-objectstore-gcs.yaml` — includes `googleCredentials.gkeEnvironment: true`
- Setup: `etc/init_cnpg_gke.sh` and `etc/init_cnpg_backup.sh`
- Setup: `etc/init_cnpg_gke.sh` and `etc/init_cnpg_backup.sh`
> **Performance note:** Pod `memory: 512Mi` makes `barman-cloud-backup` runs 30–90 min for the 9 GB DB. Bump pending in [`docs/TODO.md`](docs/TODO.md).
This working tree (`prole`) is being converged into a **customer-deploy branch** of the upstream platform repo `knoe-db` (remote: `git@git-ssh.knoe.dev:knoe.dev/knoe-db.git`, configured here as the `knoe` remote). Platform code — `authority/`, `knoe/`, `etc/init_*.sh`, `deploy/gcp/gke/*`, the test pipeline — is not maintained in this repo; its source of truth is `knoe-db/main`. Customer-specific divergence (config, branding, on-prem `prole.org` manifests) is what stays on the customer branch.
This is `knoe-db` (remote: `git@git-ssh.knoe.dev:knoe.dev/knoe-db.git`, configured as both `origin` and `knoe`). It's the platform's source of truth — `authority/`, `knoe/`, `etc/init_*.sh`, `deploy/gcp/gke/*`, the test pipeline, all live here.
**Before changing anything in this tree, check whether the change belongs upstream.** If it's platform code, land it on `knoe-db/main` and rebase the customer branch.
Customer deploys are intended to live as **branches** in this repo (e.g. a future `customer/prole.org`), not as separate forks. As of this writing, no customer branch is active — the prole→knoe rebrand has merged into `main` and there is no separate `~/dev/prole` working tree under development. See [`docs/plans/customer-deploy-resync.md`](docs/plans/customer-deploy-resync.md) for the original plan, currently dormant.
## Master TODO
Single source of truth for unfinished work, including the reality-vs-intent gaps flagged below: **[`docs/TODO.md`](docs/TODO.md)**.
## Active plans
## Active plans
- [`docs/plans/customer-deploy-resync.md`](docs/plans/customer-deploy-resync.md) — The active plan for converging `prole` onto `knoe-db/main` as a customer-deploy branch. Inventory + cherry-pick + rebase cadence. **Read this first** if you are about to touch anything in this tree.
- [`docs/plans/README.md`](docs/plans/README.md) — Index and conventions for this directory.
- [`docs/plans/README.md`](docs/plans/README.md) — Index and conventions for this directory.
For platform-level plans (knoe-auth, deployment modes, etc.), read [`knoe-db/docs/plans/`](../knoe-db/docs/plans/) (path relative to your `~/dev/` root). Those documents are the architectural reference for the shipped platform features and are versioned with the upstream code, not here.
- [`docs/pipeline-phases.md`](docs/pipeline-phases.md) — Autobuild & test pipeline phase reference. Phase 0 ✅, Phase 1 next (first task: f-string fix at `knoe/core/ops/cloudnative_pg.py:1372`).
- [`docs/plans/customer-deploy-resync.md`](docs/plans/customer-deploy-resync.md) — The original plan to converge `~/dev/prole` onto `knoe-db/main` as a customer-deploy branch. **Dormant** (rebrand is now in main, no separate prole tree active).
---
---
## Dual-cluster GKE architecture
## Dual-cluster GKE architecture
This project uses **two separate GKE Standard clusters** in `us-west3`, both currently provisioned with `e2-standard-2`× 3 nodes (2 vCPU / 8 GB each, ~7.1 GB allocatable). Verify with `gcloud container clusters list` — `knoe/core/actions.py` still carries an `e2-small` default for the app cluster but the live cluster is `e2-standard-2`.
This project uses **two separate GKE Standard clusters** in `us-west3`, both currently provisioned with `e2-standard-2`× 3 nodes (2 vCPU / 8 GB each, ~7.1 GB allocatable). Verify with `gcloud container clusters list`.
| `knoe-cnpg-0` | `gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0` | DB cluster — CNPG/PostgreSQL only (note: `conf/gke.cfg` currently deploys this as `knoe-dev-cnpg-0`; the code default remains `knoe-cnpg-0`) |
| `knoe-dev-cnpg-0` | `gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0` | DB cluster — CNPG/PostgreSQL only |
**Storage quota:** the project has `SSD_TOTAL_GB = 300 GB` in `us-west3`, **fully consumed by CNPG**. All non-CNPG PVCs must use `standard` (pd-standard / HDD) — not `standard-rwo` / `premium-rwo`, which are SSD-backed and will fail to provision with a quota error.`GITLAB_GITALY_STORAGE_CLASS = standard` is set in `conf/gke.cfg` accordingly.
**Storage quota:** the project has `SSD_TOTAL_GB = 300 GB` in `us-west3`, **fully consumed by CNPG**. All non-CNPG PVCs must use `standard` (pd-standard / HDD) — not `standard-rwo` / `premium-rwo`, which are SSD-backed and will fail to provision with a quota error. `GITLAB_GITALY_STORAGE_CLASS = standard` is set in `conf/gke.cfg` accordingly.
| Kong API gateway | `knoe-dev-0` | `knoe-system` |
| Kong API gateway | `knoe-dev-0` | `knoe-system` |
| Monitoring | `knoe-dev-0` | `monitoring` |
| Monitoring | `knoe-dev-0` | `monitoring` |
**Garage runs ONLY in `knoe-dev-0`.** Do NOT deploy Garage to `knoe-cnpg-0`.
**Garage runs ONLY in `knoe-dev-0`.** The DB cluster (`knoe-dev-cnpg-0`) has none — was removed 2026-04-29. Do NOT redeploy Garage to the DB cluster (use GCS for backups there).
### CNPG backups → GCS (not Garage)
### CNPG backups → GCS
Backups use **GCS with Workload Identity** (not Garage):
Backups use **GCS with Workload Identity**:
- Data bucket: `gs://knoe-0-backups/`
- Data + WAL: `gs://knoe-0-backups/` (single bucket; `knoe-db/base/` and `knoe-db/wals/` prefixes). `gs://knoe-0-wal/` exists but is unused.
- WAL bucket: `gs://knoe-0-wal/`
- GCP SA: `cnpg-backup@plenary-truck-485623-p7.iam.gserviceaccount.com` (roles on bucket: `storage.objectAdmin`, `storage.legacyBucketReader`)
- K8s SA: cluster pods run as **`cnpg-backup-sa`** in `knoe-db-0` (set via `cluster.spec.serviceAccountName`, requires CNPG ≥ v1.29.0). The SA is annotated with `iam.gke.io/gcp-service-account=cnpg-backup@…`. Two `RoleBinding` subjects (`knoe-db` and `knoe-db-barman-cloud`) include `cnpg-backup-sa` so the pod has the same RBAC the auto-generated SA would have had.
- K8s SA: `cnpg-backup-sa` in `knoe-db-0` (annotated with WI)
- ObjectStore manifest: [`k8s/knoe/knoe-db-barman-objectstore-gcs.yaml`](k8s/knoe/knoe-db-barman-objectstore-gcs.yaml) — includes `googleCredentials.gkeEnvironment: true` (required by plugin-barman-cloud v0.12.0).
**Why these matter:** `Milestone._get_script_env()` (in `knoe/milestone.py`) reads these to set `KUBECONTEXT=app_ctx` for common services and `DB_CLUSTER_KUBECONTEXT=db_ctx` for CNPG ops. Without them, all kubectl calls use the ambient context, which may be the DB cluster.
**Why these matter:** `Milestone._get_script_env()` (in `knoe/milestone.py`) reads these to set `KUBECONTEXT=app_ctx` for common services and `DB_CLUSTER_KUBECONTEXT=db_ctx` for CNPG ops. Without them, all kubectl calls use the ambient context, which may be the DB cluster.
Missing `init_cluster.app_cluster_kubecontext` → `_cluster_kubecontext("app")` returns `""` → installer falls back to `Global.KUBECONTEXT` for **both** app and db environments → **Garage deploys to knoe-cnpg-0** (wrong).
Missing `init_cluster.app_cluster_kubecontext` → `_cluster_kubecontext("app")` returns `""` → installer falls back to `Global.KUBECONTEXT` for **both** app and db environments → **Garage deploys to knoe-dev-cnpg-0** (wrong).
- CPU quota: 16 vCPUs (all regions) — 2 clusters × 3 ×`e2-standard-2` = 12 vCPUs used
- CPU quota: 16 vCPUs (all regions) — 2 clusters × 3 ×`e2-standard-2` = 12 vCPUs used
- SSD quota: `SSD_TOTAL_GB = 300 GB` — fully consumed by CNPG; all other PVCs must use `standard` (pd-standard / HDD)
- SSD quota: `SSD_TOTAL_GB = 300 GB` — fully consumed by CNPG; all other PVCs must use `standard` (pd-standard / HDD)
---
## Reality TODOs / Drift Log
Quick reference. Each entry links to the master index where context, owner, and rank live.
| # | Drift | Where described above | Where tracked |
|---|---|---|---|
| _(none currently)_ | | | |
**Closed in 2026-04-29 stabilization session:** Garage on DB cluster removed; cluster pods migrated to `cnpg-backup-sa` via CNPG v1.29.0 `spec.serviceAccountName`; both operators restarted clean.
**Closed 2026-05-01:** R4 — installer env-contamination guard now live in `deploy.sh` (strict) + `install.sh` (informational notice). Helper at [`etc/preflight_kubecontext.sh`](etc/preflight_kubecontext.sh).
---
## k3s CNPG database (production)
The prole.org k3s CNPG cluster is this project's **production** PostgreSQL database.
| Internal service | `knoe-db-rw.knoe-db.svc.cluster.local:5432` |
| kubectl context | `prole-service-cluster` |
| Namespace | `knoe-db` |
| CA cert | `~/.knoe/knoe-db-ca.crt` |
| sslmode | `verify-full` |
**DNS**: `pg.prole.org` resolves internally via split-horizon DNS on `myrddin.prole.org` (Samba AD DC) to the k3s ServiceLB node IPs (`10.0.0.3`, `10.0.0.6`). External DNS resolves to the public IP — do not access from outside the LAN without a VPN.
**Prole** is the reference homelab deployment of the **[Knoe](https://git.knoe.dev)** platform — running on a three-node k3s ARM64 cluster with Samba Active Directory, Kerberos SSO, and Google Workspace login across all services.
This repo is the `prole.org` fork of [knoe-db](https://git.knoe.dev/knoe-dev/knoe-db), tracking upstream and carrying the cluster-specific manifests, identity configuration, and operational tooling for `prole.org`.
---
## Knoe platform
**Knoe makes it practical to run a Supabase-style platform across air-gapped, edge, and cloud environments.**
**Knoe makes it practical to run a Supabase-style platform across air-gapped, edge, and cloud environments.**
@ -297,16 +315,229 @@ Detailed setup docs, cluster procedures, and host-specific notes should live in
---
---
## prole.org deployment
This repository is the **prole.org fork** of the upstream knoe-db platform. It runs a production k3s cluster on a three-node ARM64 LAN (myrddin / gandalf / merlin) with Samba AD providing Kerberos identity for single sign-on across all platform services.
### Live services
| Service | URL | Auth |
|---|---|---|
| Supabase Studio | `https://db.prole.org` | SPNEGO → OIDC → Studio |
sudo samba-tool user setpassword knoe-auth --newpassword="$(openssl rand -base64 32 | tr -d '=/+' | head -c 40)"
# Export full domain keytab; rekey to HTTP/api.prole.org@PROLE.ORG in Python
# (samba-tool exportkeytab --principal=HTTP/... returns empty — see note below)
```
**Note:** `samba-tool domain exportkeytab --principal=HTTP/hostname` always produces an empty keytab on Samba 4 — the filter only matches UPN format, not SPN format. The correct procedure is to export the full domain keytab and rekey the AES entries to the SPN principal name using the Python script embedded in `infrastructure/playbooks/gitea_spnego_keytab.yml` (same logic applies for all HTTP service principals).
Working tree is clean. Browser SPNEGO SSO is live on `db.prole.org` and `git.prole.org`.
---
## Upstream fork — knoe-db
This repository is an independent fork of the upstream knoe-db project:
```
git@git.knoe.dev:knoe-dev/knoe-db.git
```
Upstream changes are pulled into a dated review branch (`upstream/knoe-db/YYYYMMDD`) using the sync script, inspected, and selectively merged into `main`.
```bash
# Preview what the script will do
./scripts/sync_upstream_knoe_db.sh --dry-run
# Run the sync (requires a clean working tree)
./scripts/sync_upstream_knoe_db.sh
```
See [`docs/upstream-knoe-db-sync.md`](docs/upstream-knoe-db-sync.md) for the full review-and-merge procedure.
### Fork-specific files
Files added or substantially modified in the prole.org fork (not present or not relevant upstream):
| Path | Purpose |
|---|---|
| `infrastructure/` | Ansible roles, playbooks, and inventory for the prole.org cluster |
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
