local-path does not exist on this k3s cluster. The Prometheus Operator
was stuck in a reconcile error loop, blocking all PodMonitor updates.
Prometheus: merlin-local-iscsi-prometheus (30Gi pre-existing PV)
Alertmanager: merlin-local-iscsi-alertmanager (5Gi pre-existing PV)
Grafana: remains local-path to match immutable StatefulSet VolumeClaimTemplate;
actual PVC is already bound to merlin-local-iscsi-grafana PV.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
knoe-auth /auth/verify not yet implemented — auth_request returns 500,
blocking all Grafana access. Simplify nginx to plain passthrough; strip
inbound X-WEBAUTH-USER to prevent header forgery. Disable Grafana
auth.proxy until knoe-auth is ready. Google auth.google still active.
authority-prole-auth deployment had no Service. Create knoe-auth ClusterIP
service selecting app=authority-prole-auth on port 8080 — consistent with
Kong's knoe-auth route URL. Update nginx configmap to use the same FQDN.
Without this, kubectl apply without -n flag lands in default namespace
instead of monitoring, leaving the live configmap with the old
knoe-auth hostname.
Traefik ingress (svc-knoe-ingress) routes db.prole.org to
service/oauth2-proxy:80 in knoe-system. oauth2-proxy itself runs in
supabase namespace, so this ExternalName alias bridges the gap.
- kong-configmap: sync to live content; fix grafana upstream to route through
knoe-grafana-proxy (nginx auth proxy) instead of kps-grafana (stale service
name from old helm release); add db.prole.org → oauth2-proxy route; fix
knoe-auth FQDN to authority-prole-auth
- grafana-proxy-configmap: fix upstream to prometheus-grafana (helm release
name is 'prometheus', not 'kps'); fix auth_request FQDN to
authority-prole-auth.knoe-system; fix login redirect to api.prole.org
- grafana-proxy-{deployment,service}: add namespace: monitoring
openssl rand -base64 32 produces a 44-char string; oauth2-proxy treats it
as 44 raw bytes and rejects it. -base64 24 encodes 24 bytes → 32 base64
chars with no padding, which satisfies the 32-byte AES requirement.
Reads svc-prole-org (Grafana) and db-prole-org (oauth2-proxy) API_CREDENTIAL
items from the Personal 1Password vault and writes them into etc/secrets/ so
init_grafana_oauth_prole.sh and init_oauth2_proxy_prole.sh can consume them.
Generates and saves the oauth2-proxy cookie secret (32-byte base64) back to
the db-prole-org 1Password item as a cookie_secret field on first run;
idempotent thereafter.
Usage:
./etc/fetch_prole_secrets.sh
./etc/init_grafana_oauth_prole.sh
./etc/init_oauth2_proxy_prole.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- monitoring/kps-values-k3s.yaml: Grafana helm values for k3s homelab with
dual auth (auth.proxy Kerberos + auth.google for prole.org Workspace)
- grafana-proxy-configmap.yaml: nginx passthrough for /grafana/login/google
and /grafana/login to allow Google OAuth flow without knoe-auth redirect
- grafana-google-oidc-secret-prole.example.yaml: Secret template for Grafana
Google OAuth client (svc.prole.org, Internal consent, prole.org Workspace)
- oauth2-proxy-google-oidc-secret-prole.example.yaml: Secret template for
oauth2-proxy gating db.prole.org Studio
- oauth2-proxy-deployment-prole.yaml: k3s oauth2-proxy deployment for
db.prole.org (prole.org domain, no BackendConfig)
- init_grafana_oauth_prole.sh: Bootstrap script for Grafana OAuth secret
- init_oauth2_proxy_prole.sh: Bootstrap script for Studio oauth2-proxy
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
selectattr with 'match' regex was returning empty sequence on this Ansible
version. Since each install entry has exactly two items (fullchain.cer +
key), rejectattr the fullchain to get the key — no regex needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The dns_namecom.sh plugin uses Namecom_Username and Namecom_Token (not
NAMECOM_*). Fix the environment dict, the SAVED_ keys written to
account.conf, and add a cleanup task to remove the stale all-caps entries
already present on hosts from previous runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Re-adds no_log: true removed in 5e4e17e and switches the issue task from
ansible.builtin.shell with inline VAR=value to ansible.builtin.command with
an environment: dict. This prevents credentials appearing in Ansible failure
output and avoids the inline env var being clobbered when acme.sh sources
account.conf internally before the dns_namecom plugin runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ansible command module + become does not reliably pass environment:
vars through sudo env_reset. Use ansible.builtin.shell with POSIX
inline assignment (VAR=val cmd) so credentials are set inside the
sudo-spawned shell subprocess, never touching the sudo env barrier.
account.conf writing retained for cron renewal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_readaccountconf_mutable checks SAVED_NAMECOM_Username first; if that
key lookup exits 0 with empty output it short-circuits and never reads
the bare NAMECOM_Username key. Write the SAVED_ prefix that acme.sh
uses internally when persisting credentials.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sudo env_reset strips the environment: key through become. Write
NAMECOM_Username and NAMECOM_Token to acme.sh's account.conf directly,
which dns_namecom reads as a fallback via _readaccountconf_mutable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New acme role: installs acme.sh, issues certs for myrddin/merlin/gandalf
via DNS-01 (name.com API), installs cert files to /etc/ssl/certs/prole/,
sets up renewal cron. rc=2 (already valid) treated as ok.
- New playbooks/acme.yml: runs acme role on myrddin (issues all 3 host certs)
- New playbooks/certmgr.yml: cert-manager and letsencrypt-prod ClusterIssuer
(HTTP-01/Traefik) for external endpoints (db/svc/api.prole.org)
- Add vault references for name.com API creds to group_vars/all/vars.yml
- Add prole_ssl_src_dir and prole_ssl_files to myrddin/merlin/gandalf host_vars
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Followup from commit dba8a2d's findings. ConfigMixin._save_knoe_cfg in
knoe/ui/screens/cfg.py reads Tk widget vars via `var.get()` and writes
the result to conf/<mode>.cfg. When a var is a MagicMock (interactive
./install.py run in a non-Tk context, partially-mocked widgets), the
save path serializes the mock's repr-string into the cfg, then the next
installer pass calls os.makedirs() on those values and produces
directories literally named `<MagicMock name='Canvas().tk.call().strip()'
id='4743999712'>/`.
The brief lays out a TDD approach for Junie:
1. Write failing test at tests/installer/test_cfg_save_refuses_mock_values.py
that passes MagicMock widget vars and asserts _save_knoe_cfg raises
TypeError naming the field.
2. Implement the minimal fix: a `_str_value(var, field=...)` helper in
cfg.py that validates widget reads and raises if non-str. Use it
in the .get()/.strip() chains across lines 103-155.
3. Verify the 750 existing installer tests still pass.
Brief includes file pointers (cfg.py:44 _save_knoe_cfg, line 102
globals_to_save assembly, line 277 cfg_path.write_text), the canonical
failing test stub, both fix-approach options (per-read validator vs
end-of-flow dict walk), and explicit commit-shape guidance.
Index updates:
docs/plans/junie/README.md — todo-1 row added under Active
docs/TODO.md §"In progress" — todo-1 promoted above Phase 3 (smaller
scope, easy to land first)
Naming convention: `todo-N-<slug>.md` for follow-up bugs, distinct from
the `NN-<slug>.md` pattern reserved for ranked queue items.
- Add gandalf.prole.org → 10.0.0.7 to prole_dns_records and
prole_internal_a_records in dns.yml (was missing, causing nslookup
to fail on myrddin)
- prole_ssl: stat-check each source file on the controller before
copying; emit a warning and skip rather than hard-failing when certs
have not yet been placed in ssl/prole/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Installs Tailscale on k3s agents via the Debian stable apt repo using
the modern signed-by keyring approach. Auth key stored in vault.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
init_gitea.sh requires GITEA_PV_NODE in k3s mode. PVs for Gitea storage
(gitea-local-d005) are pinned to gandalf.prole.org per nodeAffinity on
the existing gitea-d005-shared-storage PV.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cleanup_legacy_prole_kong() only checked for the prole-svc-kong deployment
and service, so the early-return guard fired after those were already deleted,
leaving svc-prole-ingress in place. On re-runs this Traefik ingress blocked
the duplicate-host check and prevented Kong from deploying.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k8s/knoe/iscsi-pvs.yaml had *.knoe.org nodeAffinity after the rename, but the
live k3s cluster nodes are still *.prole.org — causing immutable field errors on
Bound PVs during kubectl apply. Revert to *.prole.org to match live state.
conf/k3s.cfg KUBECTL_CONTEXT was renamed to knoe-service-cluster, but both the
system kubeconfig and etc/secrets/k3s.kubeconfig still use prole-service-cluster.
install.sh _check_gcp_tools() now skips gcloud check for k3s mode and any non-gke
config path (only GKE deploys need gcloud/kubectl pre-flight).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
conf/dev/knoe.cfg had a stale SERVICE_NAMESPACE=default written by a prior TUI session.
Also guard _service_namespace() so the value 'default' is never returned — it is never
a valid service namespace and signals a stale generated config.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Combobox width=30 chars extends to ~320-380px depending on font metrics,
colliding with Add at x+290 and Delete at x+350. Narrowed combobox to
width=22 and shifted buttons to x+260 and x+320 to give clear separation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
install_barman_plugin() fetches from GitHub on every run (up to 6×120 s
retries), stalling the installer worker thread even when the CNPG cluster
is already at 'Cluster in healthy state' with all pods 2/2 Ready.
Add a fast-path at the top of initialize(): check .status.phase for
'healthy' then verify all pods show 2/N ready — if both pass, return
immediately. Full pipeline (ensure_operator, install_barman_plugin,
_apply_manifest, _wait_cnpg_pods) is only entered when needed.
Also adds test_cnpg_initialize_skips_when_cluster_healthy to
test_deployment_mode_isolation.py to guard against regression.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
kube-prometheus-stack CRD re-application stalls on every installer run
even when the release is in deployed state with all pods Running. For k3d
this is purely idempotency noise — skip the upgrade and return immediately.
Broken state (failed status, unbound PVCs) still triggers a full purge+reinstall.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k3d clusters only have local-path provisioner; defaulting to synology-iscsi
caused all Supabase PVCs to get stuck Pending. Also add local-path to the
validation allowlist so it is not rejected as an unknown storage class.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The container's /bin/sh is dash, which doesn't support set -o pipefail.
The preceding [ -s /tmp/pg.keytab ] check already guarantees the file is
non-empty before this line runs, so pipefail was redundant.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
conf/service/prod.cfg: prole.org homelab overrides
- SERVICE_HOST_DOMAIN=prole.org, DB_UI_HOST=db.prole.org
- OIDC_ISSUER_URL=https://api.prole.org/auth
- GOOGLE_WORKSPACE_DOMAIN=prole.org (credentials in OpenBao bao/kv/knoe/oauth2/google-prole-org)
- kerberos_config: enabled, realm=PROLE.ORG, kdc=10.0.0.3 (myrddin.prole.org)
- PROLE_KDC_TRUST_REALM=PROLE.ORG, SERVICE_INGRESS_TLS_ENABLED=1
conf/k3d.cfg: update local dev paths to ~/dev/prole; enable kerberos for dev
env.sh: KNOE_HOME points to ~/dev/prole
application-prole.yml: Spring profile with prole.org Google Workspace client binding
These files are prole.org homelab specifics — do NOT merge upstream to knoe/main.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Keytab export (silent failure bug):
- Always delete+recreate the postgres SPN with fresh random key; old key gone
after EmptyDir wipe; kadmin.local -q exits 0 even on error so -norandkey
silently failed
- Clean /tmp/pg.keytab before ktadd; verify non-empty with [ -s ] before
proceeding; die loudly if keytab not written
- Fix base64 pipeline: set -o pipefail inside sh -c so base64 failure is not
masked by tr exit code
Next steps:
- Suppress myrddin.prole.org Samba trust step in k3d mode (no AD server)
- Remove Grafana auth.proxy reminder (configured by Helm values already)
Gitea admin token:
- Auto-generate via kubectl exec into running Gitea pod before falling back to
manual warning; persist as gitea-admin-token secret for future re-runs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs in the EmptyDir auto-recovery path:
1. get_kdc_pod returned items[0] without a phase filter, so a Terminating
pod from the previous rollout could be selected. Add
--field-selector=status.phase=Running to always get a live pod.
2. The rollout completes (pod Running) before the in-container entrypoint
finishes kdb5_util create (no readiness probe configured). The single
immediate kadmin.local check raced against DB init and lost. Replace
with a 120s poll loop (5s interval) that re-queries the pod each tick
so it handles both the timing race and any remaining pod-selection lag.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
init_kdc.sh initialize rolls out a new deployment, creating a new pod with a
new name. The retry pre-flight was still exec-ing into the old (terminated)
pod. Re-query get_kdc_pod() after re-init so the retry and all subsequent
kadmin calls use the new pod.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the KDC pod restarts (e.g. after k3d node restart to apply registry
mirror config), its EmptyDir volume is wiped. ensure_kdc_pod() finds the
running pod and returns it, bypassing the bootstrap path. The subsequent
pre-flight check then fails because kadmin.local cannot access the database.
Instead of dying with a manual-intervention message, auto-recover by calling
init_kdc.sh initialize (with PROLE_KDC_NAMESPACE/NAME/MASTER_PASSWORD set)
then retrying the pre-flight. Only die if the retry also fails.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
check_kerberos.sh defaulted PROLE_KDC_NAME=auth but init_kdc.sh deploys the
pod with label app=authority-knoe-auth (authority-gcp-auth for k8s). The pod
selector never matched, so wait_for_kdc_pod always returned empty and the test
failed with "KDC pod 'auth' not found" even after a successful rollout.
Add _default_kdc_name() using the same mode-aware logic as init_kdc.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k3d registry containers are addressed as k3d-NAME.localhost:PORT from
inside the cluster, but containerd resolves that as 127.0.0.1 (loopback)
which has nothing listening. The fix is to write /etc/rancher/k3s/registries.yaml
on every server/agent node to mirror k3d-NAME.localhost:PORT -> http://k3d-NAME:PORT
(Docker network DNS), then restart the node so containerd picks it up.
_patch_k3d_registry_mirrors() is called from _ensure_k3d_registry() after
the registry is confirmed running (idempotent — skips nodes already configured).
This makes knoe-authority and any other image pushed to localhost:5000
immediately pullable inside the cluster as k3d-knoe-registry.localhost:5000/...
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
check_kerberos.sh (already updated) sourced knoe_env.sh, knoe_secrets.sh,
knoe_string.sh, and knoe_yaml.sh but the files were still named prole_*.sh.
No other script referenced the old names, so this is a pure rename.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Port-forward setup was failing the installation because:
1. argocd and dashboard entries were always added to port-mapping even when
those services haven't been selected or installed yet
2. Grafana/Prometheus service names were hardcoded as kps-* but monitoring
deploys with release name "prometheus" → services are prometheus-grafana etc.
3. The grafana status poll after init_port_forwards.sh start would mark the
entire installation failed if grafana wasn't reachable
Changes:
- _build_required_port_forwards (env.py): add argocd_enabled, dashboard_enabled,
monitoring_release params; argocd/dashboard only included when enabled; grafana
and prometheus targets use f"svc/{monitoring_release}-grafana" etc.
- _sync_port_forward_mappings (actions.py): read argocd_enabled from installer
config; read MONITORING_RELEASE from Monitoring section; clean up argocd and
dashboard entries from [Port Forwards] when disabled
- run_init_scripts (services.py): port-forward step is non-fatal — failure logs
a warning and continues; blocking grafana poll removed
- 2 new regression tests (7e) covering the above invariants
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
services.py run_init_scripts() called init_kong.sh with -n env[NAMESPACE]
where NAMESPACE is the database namespace (e.g. knoe-db). The -n flag
sets COMMON_CORE_NAMESPACE which wins over SERVICE_NAMESPACE in
common_core_resolve_namespace, so Kong deployed into knoe-db.
Fix: resolve kong_ns from SERVICE_NAMESPACE, falling back to
_get_service_namespace() and finally knoe-system. Exclude default
as it is the Tk StringVar initial value, not a real service namespace.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
