Password resolution order for PROLE_KDC_MASTER_PASSWORD and KNOE_GUEST_PASSWORD:
1. Env var (existing)
2. knoe-kdc-secrets k8s Secret (existing)
3. prole-kdc-secrets k8s Secret (new — used by live prole deployment)
4. 1Password via op read (new — OP_KDC_MASTER_PASSWORD_REF / OP_KDC_GUEST_PASSWORD_REF)
5. PROLE_LOCAL_ADMIN_PASSWORD (existing)
6. die with actionable message
Add try_op_read() helper: wraps op CLI, returns 1 if op not available or ref empty.
conf/k3s.cfg: add KNOE_ADMIN_PRINCIPAL=chrisfu, OP_KDC_MASTER_PASSWORD_REF,
OP_KDC_GUEST_PASSWORD_REF placeholders (empty — user fills in op:// URIs).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The k3s-generated kubeconfig on the cluster server node uses 'default'
as its context name. The existing guard rejected this unconditionally,
causing init_knoe_users.sh to always fail when run from myrddin.
Tighten the guard: only reject 'default' when KNOE_MODE is k3d (where
it would indicate the k3d context was not found). In k3s/k8s mode,
'default' is legitimate and the script proceeds normally.
Usage on myrddin: bash etc/init_knoe_users.sh --mode k3s initialize
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add ensure_gitea_spnego_keytab(): provisions HTTP/git.prole.org@PROLE.ORG
service principal, keytab, and gitea-krb5-keytab Secret automatically when
samba-tool is available (i.e. running on myrddin); otherwise prints manual
steps. Restarts gitea-spnego-proxy after keytab is created.
- Add GITEA_SPNEGO_HOST, GITEA_KRB5_AD_REALM, GITEA_KRB5_AD_USER config vars.
- Fix promote_gitea_admin: exec with 'su git' (gitea refuses to run as root);
use KNOE_ADMIN_PRINCIPAL instead of hardcoded 'admin'; fall back to
'gitea_admin' (Helm chart default) if the primary user has no token yet.
- Rename initialize() step numbers (9=ArgoCD, 10=SPNEGO keytab, 11=Gitea, 12=GitLab).
- Fix PROLE.LOCAL → KNOE.LOCAL in: default realm, SQL schema default, comments,
next-steps message, trust principal comments.
- show_status: add gitea-krb5-keytab check alongside pg keytab check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea Helm chart v10.x changed defaults:
image.registry: docker.gitea.com (was docker.io)
image.rootless: true (was false)
Our values set repository/tag but not registry/rootless, so the new
defaults produced docker.gitea.com/gitea/gitea:1.22.3-rootless which
can't be pulled from gandalf. Pin registry: docker.io and rootless: false
to keep using the standard Docker Hub image.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Helm upgrade was failing for two reasons:
1. persistence.storageClass was unset (rendered null), conflicting with the
existing PVC's immutable gitea-local-d005 storageClassName.
2. valkey-cluster (and redis-cluster) were not explicitly disabled, causing
the latest gitea chart to spin up a valkey StatefulSet on each upgrade.
Add GITEA_STORAGE_CLASS env-var override for non-k3s modes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gokrb5 returns 401 silently on invalid tickets. Wrap ResponseWriter to
capture status code and log failures with remote addr, host, path, and
first 20 chars of the Negotiate token for easier debugging.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Go proxy exposes /_healthz so use httpGet probes (tcpSocket was only a
connectivity check, not a health check). Also reduce memory request/limit
to 32Mi/64Mi — the Go binary is lean and in-cluster load is minimal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds the ScheduledBackup CR to source control. Fixes schedule from
5-field to 6-field format (0 0 3 * * *) — CNPG requires seconds as
first field. The broken schedule caused the reconcile loop that
generated 746 stale Backup CRs. Deleted 7 orphaned GitLab Garage
buckets (~8.5 GB) and resumed daily backups.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path does not exist on this k3s cluster. The Prometheus Operator
was stuck in a reconcile error loop, blocking all PodMonitor updates.
Prometheus: merlin-local-iscsi-prometheus (30Gi pre-existing PV)
Alertmanager: merlin-local-iscsi-alertmanager (5Gi pre-existing PV)
Grafana: remains local-path to match immutable StatefulSet VolumeClaimTemplate;
actual PVC is already bound to merlin-local-iscsi-grafana PV.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
knoe-auth /auth/verify not yet implemented — auth_request returns 500,
blocking all Grafana access. Simplify nginx to plain passthrough; strip
inbound X-WEBAUTH-USER to prevent header forgery. Disable Grafana
auth.proxy until knoe-auth is ready. Google auth.google still active.
authority-prole-auth deployment had no Service. Create knoe-auth ClusterIP
service selecting app=authority-prole-auth on port 8080 — consistent with
Kong's knoe-auth route URL. Update nginx configmap to use the same FQDN.
Without this, kubectl apply without -n flag lands in default namespace
instead of monitoring, leaving the live configmap with the old
knoe-auth hostname.
Traefik ingress (svc-knoe-ingress) routes db.prole.org to
service/oauth2-proxy:80 in knoe-system. oauth2-proxy itself runs in
supabase namespace, so this ExternalName alias bridges the gap.
- kong-configmap: sync to live content; fix grafana upstream to route through
knoe-grafana-proxy (nginx auth proxy) instead of kps-grafana (stale service
name from old helm release); add db.prole.org → oauth2-proxy route; fix
knoe-auth FQDN to authority-prole-auth
- grafana-proxy-configmap: fix upstream to prometheus-grafana (helm release
name is 'prometheus', not 'kps'); fix auth_request FQDN to
authority-prole-auth.knoe-system; fix login redirect to api.prole.org
- grafana-proxy-{deployment,service}: add namespace: monitoring
openssl rand -base64 32 produces a 44-char string; oauth2-proxy treats it
as 44 raw bytes and rejects it. -base64 24 encodes 24 bytes → 32 base64
chars with no padding, which satisfies the 32-byte AES requirement.
Reads svc-prole-org (Grafana) and db-prole-org (oauth2-proxy) API_CREDENTIAL
items from the Personal 1Password vault and writes them into etc/secrets/ so
init_grafana_oauth_prole.sh and init_oauth2_proxy_prole.sh can consume them.
Generates and saves the oauth2-proxy cookie secret (32-byte base64) back to
the db-prole-org 1Password item as a cookie_secret field on first run;
idempotent thereafter.
Usage:
./etc/fetch_prole_secrets.sh
./etc/init_grafana_oauth_prole.sh
./etc/init_oauth2_proxy_prole.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- monitoring/kps-values-k3s.yaml: Grafana helm values for k3s homelab with
dual auth (auth.proxy Kerberos + auth.google for prole.org Workspace)
- grafana-proxy-configmap.yaml: nginx passthrough for /grafana/login/google
and /grafana/login to allow Google OAuth flow without knoe-auth redirect
- grafana-google-oidc-secret-prole.example.yaml: Secret template for Grafana
Google OAuth client (svc.prole.org, Internal consent, prole.org Workspace)
- oauth2-proxy-google-oidc-secret-prole.example.yaml: Secret template for
oauth2-proxy gating db.prole.org Studio
- oauth2-proxy-deployment-prole.yaml: k3s oauth2-proxy deployment for
db.prole.org (prole.org domain, no BackendConfig)
- init_grafana_oauth_prole.sh: Bootstrap script for Grafana OAuth secret
- init_oauth2_proxy_prole.sh: Bootstrap script for Studio oauth2-proxy
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
selectattr with 'match' regex was returning empty sequence on this Ansible
version. Since each install entry has exactly two items (fullchain.cer +
key), rejectattr the fullchain to get the key — no regex needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The dns_namecom.sh plugin uses Namecom_Username and Namecom_Token (not
NAMECOM_*). Fix the environment dict, the SAVED_ keys written to
account.conf, and add a cleanup task to remove the stale all-caps entries
already present on hosts from previous runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Re-adds no_log: true removed in 5e4e17e and switches the issue task from
ansible.builtin.shell with inline VAR=value to ansible.builtin.command with
an environment: dict. This prevents credentials appearing in Ansible failure
output and avoids the inline env var being clobbered when acme.sh sources
account.conf internally before the dns_namecom plugin runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ansible command module + become does not reliably pass environment:
vars through sudo env_reset. Use ansible.builtin.shell with POSIX
inline assignment (VAR=val cmd) so credentials are set inside the
sudo-spawned shell subprocess, never touching the sudo env barrier.
account.conf writing retained for cron renewal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_readaccountconf_mutable checks SAVED_NAMECOM_Username first; if that
key lookup exits 0 with empty output it short-circuits and never reads
the bare NAMECOM_Username key. Write the SAVED_ prefix that acme.sh
uses internally when persisting credentials.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sudo env_reset strips the environment: key through become. Write
NAMECOM_Username and NAMECOM_Token to acme.sh's account.conf directly,
which dns_namecom reads as a fallback via _readaccountconf_mutable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New acme role: installs acme.sh, issues certs for myrddin/merlin/gandalf
via DNS-01 (name.com API), installs cert files to /etc/ssl/certs/prole/,
sets up renewal cron. rc=2 (already valid) treated as ok.
- New playbooks/acme.yml: runs acme role on myrddin (issues all 3 host certs)
- New playbooks/certmgr.yml: cert-manager and letsencrypt-prod ClusterIssuer
(HTTP-01/Traefik) for external endpoints (db/svc/api.prole.org)
- Add vault references for name.com API creds to group_vars/all/vars.yml
- Add prole_ssl_src_dir and prole_ssl_files to myrddin/merlin/gandalf host_vars
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Followup from commit dba8a2d's findings. ConfigMixin._save_knoe_cfg in
knoe/ui/screens/cfg.py reads Tk widget vars via `var.get()` and writes
the result to conf/<mode>.cfg. When a var is a MagicMock (interactive
./install.py run in a non-Tk context, partially-mocked widgets), the
save path serializes the mock's repr-string into the cfg, then the next
installer pass calls os.makedirs() on those values and produces
directories literally named `<MagicMock name='Canvas().tk.call().strip()'
id='4743999712'>/`.
The brief lays out a TDD approach for Junie:
1. Write failing test at tests/installer/test_cfg_save_refuses_mock_values.py
that passes MagicMock widget vars and asserts _save_knoe_cfg raises
TypeError naming the field.
2. Implement the minimal fix: a `_str_value(var, field=...)` helper in
cfg.py that validates widget reads and raises if non-str. Use it
in the .get()/.strip() chains across lines 103-155.
3. Verify the 750 existing installer tests still pass.
Brief includes file pointers (cfg.py:44 _save_knoe_cfg, line 102
globals_to_save assembly, line 277 cfg_path.write_text), the canonical
failing test stub, both fix-approach options (per-read validator vs
end-of-flow dict walk), and explicit commit-shape guidance.
Index updates:
docs/plans/junie/README.md — todo-1 row added under Active
docs/TODO.md §"In progress" — todo-1 promoted above Phase 3 (smaller
scope, easy to land first)
Naming convention: `todo-N-<slug>.md` for follow-up bugs, distinct from
the `NN-<slug>.md` pattern reserved for ranked queue items.
- Add gandalf.prole.org → 10.0.0.7 to prole_dns_records and
prole_internal_a_records in dns.yml (was missing, causing nslookup
to fail on myrddin)
- prole_ssl: stat-check each source file on the controller before
copying; emit a warning and skip rather than hard-failing when certs
have not yet been placed in ssl/prole/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Installs Tailscale on k3s agents via the Debian stable apt repo using
the modern signed-by keyring approach. Auth key stored in vault.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
init_gitea.sh requires GITEA_PV_NODE in k3s mode. PVs for Gitea storage
(gitea-local-d005) are pinned to gandalf.prole.org per nodeAffinity on
the existing gitea-d005-shared-storage PV.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cleanup_legacy_prole_kong() only checked for the prole-svc-kong deployment
and service, so the early-return guard fired after those were already deleted,
leaving svc-prole-ingress in place. On re-runs this Traefik ingress blocked
the duplicate-host check and prevented Kong from deploying.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k8s/knoe/iscsi-pvs.yaml had *.knoe.org nodeAffinity after the rename, but the
live k3s cluster nodes are still *.prole.org — causing immutable field errors on
Bound PVs during kubectl apply. Revert to *.prole.org to match live state.
conf/k3s.cfg KUBECTL_CONTEXT was renamed to knoe-service-cluster, but both the
system kubeconfig and etc/secrets/k3s.kubeconfig still use prole-service-cluster.
install.sh _check_gcp_tools() now skips gcloud check for k3s mode and any non-gke
config path (only GKE deploys need gcloud/kubectl pre-flight).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
conf/dev/knoe.cfg had a stale SERVICE_NAMESPACE=default written by a prior TUI session.
Also guard _service_namespace() so the value 'default' is never returned — it is never
a valid service namespace and signals a stale generated config.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Combobox width=30 chars extends to ~320-380px depending on font metrics,
colliding with Add at x+290 and Delete at x+350. Narrowed combobox to
width=22 and shifted buttons to x+260 and x+320 to give clear separation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
install_barman_plugin() fetches from GitHub on every run (up to 6×120 s
retries), stalling the installer worker thread even when the CNPG cluster
is already at 'Cluster in healthy state' with all pods 2/2 Ready.
Add a fast-path at the top of initialize(): check .status.phase for
'healthy' then verify all pods show 2/N ready — if both pass, return
immediately. Full pipeline (ensure_operator, install_barman_plugin,
_apply_manifest, _wait_cnpg_pods) is only entered when needed.
Also adds test_cnpg_initialize_skips_when_cluster_healthy to
test_deployment_mode_isolation.py to guard against regression.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
kube-prometheus-stack CRD re-application stalls on every installer run
even when the release is in deployed state with all pods Running. For k3d
this is purely idempotency noise — skip the upgrade and return immediately.
Broken state (failed status, unbound PVCs) still triggers a full purge+reinstall.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k3d clusters only have local-path provisioner; defaulting to synology-iscsi
caused all Supabase PVCs to get stuck Pending. Also add local-path to the
validation allowlist so it is not rejected as an unknown storage class.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
