The Gitea Helm chart names the admin credentials secret after the release
name alone, not '{release}-gitea'. For release name 'gitea' the secret is
just 'gitea'.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
deploy.sh: resolve_knoe_db_namespace() now checks DATABASE_NAMESPACE from
config (k3s.cfg: knoe-db) before probing namespaces. Previously it fell back
to 'default' because only 'knoe' was probed, baking a wrong DB hostname into
app.ini. The running server was unaffected (uses GITEA__database__HOST env var)
but the gitea admin CLI, which reads app.ini directly, could not connect.
init_knoe_users.sh: add gitea_helm_admin_token() which reads the Helm
bootstrap admin password from the gitea-gitea k8s secret and exchanges it for
a REST API token — entirely avoiding the broken CLI. Add gitea_api_set_password()
which uses PATCH /api/v1/admin/users/{user} to set the password via the running
web server (which has the correct DB connection) instead of kubectl exec.
promote_gitea_admin() now prefers the REST-API-only path:
1. existing token (env / k8s secret)
2. Helm admin k8s secret → REST token; set KNOE_ADMIN_PRINCIPAL password via API
3. 1Password → CLI set-password → basic-auth token (fallback)
4. kubectl exec generate-access-token (last resort)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
knoe_cfg.sh selects the config file based on KNOE_MODE at source time.
Without pre-scanning argv, KNOE_MODE is empty and k3d.cfg wins the
fallback loop even when --mode k3s is passed. This caused k3s.cfg values
(e.g. KNOE_ADMIN_PRINCIPAL=chrisfu) to be silently ignored, defaulting
to 'admin' and causing gitea_set_password to fail against a non-existent user.
Also surface gitea_set_password errors (without logging the password) so
failures are diagnosable rather than silent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the predictable temp-password pattern with a secure credential flow:
- gitea_ensure_password(): creates a 1Password item with --generate-password
(1P generator, 32-char letters+digits) if one does not exist; retrieves it
on re-runs (idempotent). Password never appears in logs or env vars.
- gitea_set_password(): sets the Gitea account password via kubectl exec
(gitea admin CLI, no prior Gitea auth required).
- gitea_api_token(): exchanges basic-auth credentials for a REST API token,
deleting any stale token of the same name first. Token only; password
cleared from memory immediately after use.
- promote_gitea_admin(): tries credential sources in order:
1. GITEA_ADMIN_TOKEN env / gitea-admin-token k8s secret
2. 1Password (generates if missing) → set password → REST token
3. kubectl exec generate-access-token fallback (Helm bootstrap admin)
Tokens persisted in k8s secret (revocable); passwords only in 1Password.
The Gitea password is a vault credential for emergency recovery only.
Normal auth is always Kerberos SPNEGO — users never need to type a password.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ron has no Google Workspace prole.org account. Email is now NULL in the DB
(NULLIF('')). The provision_user log line now always shows the Kerberos
principal explicitly and labels the email field separately so it is clear
which identity type is in use.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
In k3s mode, LOCAL_REGISTRY_INTERNAL set to a k3d value in the shell environment
leaked into init_kdc.sh subprocesses, causing pods to use the wrong image address
while the push went to myrddin.prole.org:5000.
Add _resolve_kdc_registry() helper that derives correct registry values from
KNOE_IMAGE_REGISTRY and PROLE_K3S_SERVER. Both init_kdc.sh call sites now pass
explicit overrides to prevent any inherited k3d value from leaking through.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 'die' call with a multiline string containing '|' (for base64 pipeline
in the help text) was parsed by bash as a pipe operator. Replace with
individual err() calls followed by explicit exit 1.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Password resolution order for PROLE_KDC_MASTER_PASSWORD and KNOE_GUEST_PASSWORD:
1. Env var (existing)
2. knoe-kdc-secrets k8s Secret (existing)
3. prole-kdc-secrets k8s Secret (new — used by live prole deployment)
4. 1Password via op read (new — OP_KDC_MASTER_PASSWORD_REF / OP_KDC_GUEST_PASSWORD_REF)
5. PROLE_LOCAL_ADMIN_PASSWORD (existing)
6. die with actionable message
Add try_op_read() helper: wraps op CLI, returns 1 if op not available or ref empty.
conf/k3s.cfg: add KNOE_ADMIN_PRINCIPAL=chrisfu, OP_KDC_MASTER_PASSWORD_REF,
OP_KDC_GUEST_PASSWORD_REF placeholders (empty — user fills in op:// URIs).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The k3s-generated kubeconfig on the cluster server node uses 'default'
as its context name. The existing guard rejected this unconditionally,
causing init_knoe_users.sh to always fail when run from myrddin.
Tighten the guard: only reject 'default' when KNOE_MODE is k3d (where
it would indicate the k3d context was not found). In k3s/k8s mode,
'default' is legitimate and the script proceeds normally.
Usage on myrddin: bash etc/init_knoe_users.sh --mode k3s initialize
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add ensure_gitea_spnego_keytab(): provisions HTTP/git.prole.org@PROLE.ORG
service principal, keytab, and gitea-krb5-keytab Secret automatically when
samba-tool is available (i.e. running on myrddin); otherwise prints manual
steps. Restarts gitea-spnego-proxy after keytab is created.
- Add GITEA_SPNEGO_HOST, GITEA_KRB5_AD_REALM, GITEA_KRB5_AD_USER config vars.
- Fix promote_gitea_admin: exec with 'su git' (gitea refuses to run as root);
use KNOE_ADMIN_PRINCIPAL instead of hardcoded 'admin'; fall back to
'gitea_admin' (Helm chart default) if the primary user has no token yet.
- Rename initialize() step numbers (9=ArgoCD, 10=SPNEGO keytab, 11=Gitea, 12=GitLab).
- Fix PROLE.LOCAL → KNOE.LOCAL in: default realm, SQL schema default, comments,
next-steps message, trust principal comments.
- show_status: add gitea-krb5-keytab check alongside pg keytab check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea Helm chart v10.x changed defaults:
image.registry: docker.gitea.com (was docker.io)
image.rootless: true (was false)
Our values set repository/tag but not registry/rootless, so the new
defaults produced docker.gitea.com/gitea/gitea:1.22.3-rootless which
can't be pulled from gandalf. Pin registry: docker.io and rootless: false
to keep using the standard Docker Hub image.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Helm upgrade was failing for two reasons:
1. persistence.storageClass was unset (rendered null), conflicting with the
existing PVC's immutable gitea-local-d005 storageClassName.
2. valkey-cluster (and redis-cluster) were not explicitly disabled, causing
the latest gitea chart to spin up a valkey StatefulSet on each upgrade.
Add GITEA_STORAGE_CLASS env-var override for non-k3s modes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gokrb5 returns 401 silently on invalid tickets. Wrap ResponseWriter to
capture status code and log failures with remote addr, host, path, and
first 20 chars of the Negotiate token for easier debugging.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Go proxy exposes /_healthz so use httpGet probes (tcpSocket was only a
connectivity check, not a health check). Also reduce memory request/limit
to 32Mi/64Mi — the Go binary is lean and in-cluster load is minimal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds the ScheduledBackup CR to source control. Fixes schedule from
5-field to 6-field format (0 0 3 * * *) — CNPG requires seconds as
first field. The broken schedule caused the reconcile loop that
generated 746 stale Backup CRs. Deleted 7 orphaned GitLab Garage
buckets (~8.5 GB) and resumed daily backups.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path does not exist on this k3s cluster. The Prometheus Operator
was stuck in a reconcile error loop, blocking all PodMonitor updates.
Prometheus: merlin-local-iscsi-prometheus (30Gi pre-existing PV)
Alertmanager: merlin-local-iscsi-alertmanager (5Gi pre-existing PV)
Grafana: remains local-path to match immutable StatefulSet VolumeClaimTemplate;
actual PVC is already bound to merlin-local-iscsi-grafana PV.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
knoe-auth /auth/verify not yet implemented — auth_request returns 500,
blocking all Grafana access. Simplify nginx to plain passthrough; strip
inbound X-WEBAUTH-USER to prevent header forgery. Disable Grafana
auth.proxy until knoe-auth is ready. Google auth.google still active.
authority-prole-auth deployment had no Service. Create knoe-auth ClusterIP
service selecting app=authority-prole-auth on port 8080 — consistent with
Kong's knoe-auth route URL. Update nginx configmap to use the same FQDN.
Without this, kubectl apply without -n flag lands in default namespace
instead of monitoring, leaving the live configmap with the old
knoe-auth hostname.
Traefik ingress (svc-knoe-ingress) routes db.prole.org to
service/oauth2-proxy:80 in knoe-system. oauth2-proxy itself runs in
supabase namespace, so this ExternalName alias bridges the gap.
- kong-configmap: sync to live content; fix grafana upstream to route through
knoe-grafana-proxy (nginx auth proxy) instead of kps-grafana (stale service
name from old helm release); add db.prole.org → oauth2-proxy route; fix
knoe-auth FQDN to authority-prole-auth
- grafana-proxy-configmap: fix upstream to prometheus-grafana (helm release
name is 'prometheus', not 'kps'); fix auth_request FQDN to
authority-prole-auth.knoe-system; fix login redirect to api.prole.org
- grafana-proxy-{deployment,service}: add namespace: monitoring
openssl rand -base64 32 produces a 44-char string; oauth2-proxy treats it
as 44 raw bytes and rejects it. -base64 24 encodes 24 bytes → 32 base64
chars with no padding, which satisfies the 32-byte AES requirement.
Reads svc-prole-org (Grafana) and db-prole-org (oauth2-proxy) API_CREDENTIAL
items from the Personal 1Password vault and writes them into etc/secrets/ so
init_grafana_oauth_prole.sh and init_oauth2_proxy_prole.sh can consume them.
Generates and saves the oauth2-proxy cookie secret (32-byte base64) back to
the db-prole-org 1Password item as a cookie_secret field on first run;
idempotent thereafter.
Usage:
./etc/fetch_prole_secrets.sh
./etc/init_grafana_oauth_prole.sh
./etc/init_oauth2_proxy_prole.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- monitoring/kps-values-k3s.yaml: Grafana helm values for k3s homelab with
dual auth (auth.proxy Kerberos + auth.google for prole.org Workspace)
- grafana-proxy-configmap.yaml: nginx passthrough for /grafana/login/google
and /grafana/login to allow Google OAuth flow without knoe-auth redirect
- grafana-google-oidc-secret-prole.example.yaml: Secret template for Grafana
Google OAuth client (svc.prole.org, Internal consent, prole.org Workspace)
- oauth2-proxy-google-oidc-secret-prole.example.yaml: Secret template for
oauth2-proxy gating db.prole.org Studio
- oauth2-proxy-deployment-prole.yaml: k3s oauth2-proxy deployment for
db.prole.org (prole.org domain, no BackendConfig)
- init_grafana_oauth_prole.sh: Bootstrap script for Grafana OAuth secret
- init_oauth2_proxy_prole.sh: Bootstrap script for Studio oauth2-proxy
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
selectattr with 'match' regex was returning empty sequence on this Ansible
version. Since each install entry has exactly two items (fullchain.cer +
key), rejectattr the fullchain to get the key — no regex needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The dns_namecom.sh plugin uses Namecom_Username and Namecom_Token (not
NAMECOM_*). Fix the environment dict, the SAVED_ keys written to
account.conf, and add a cleanup task to remove the stale all-caps entries
already present on hosts from previous runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Re-adds no_log: true removed in 5e4e17e and switches the issue task from
ansible.builtin.shell with inline VAR=value to ansible.builtin.command with
an environment: dict. This prevents credentials appearing in Ansible failure
output and avoids the inline env var being clobbered when acme.sh sources
account.conf internally before the dns_namecom plugin runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ansible command module + become does not reliably pass environment:
vars through sudo env_reset. Use ansible.builtin.shell with POSIX
inline assignment (VAR=val cmd) so credentials are set inside the
sudo-spawned shell subprocess, never touching the sudo env barrier.
account.conf writing retained for cron renewal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_readaccountconf_mutable checks SAVED_NAMECOM_Username first; if that
key lookup exits 0 with empty output it short-circuits and never reads
the bare NAMECOM_Username key. Write the SAVED_ prefix that acme.sh
uses internally when persisting credentials.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sudo env_reset strips the environment: key through become. Write
NAMECOM_Username and NAMECOM_Token to acme.sh's account.conf directly,
which dns_namecom reads as a fallback via _readaccountconf_mutable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New acme role: installs acme.sh, issues certs for myrddin/merlin/gandalf
via DNS-01 (name.com API), installs cert files to /etc/ssl/certs/prole/,
sets up renewal cron. rc=2 (already valid) treated as ok.
- New playbooks/acme.yml: runs acme role on myrddin (issues all 3 host certs)
- New playbooks/certmgr.yml: cert-manager and letsencrypt-prod ClusterIssuer
(HTTP-01/Traefik) for external endpoints (db/svc/api.prole.org)
- Add vault references for name.com API creds to group_vars/all/vars.yml
- Add prole_ssl_src_dir and prole_ssl_files to myrddin/merlin/gandalf host_vars
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Followup from commit dba8a2d's findings. ConfigMixin._save_knoe_cfg in
knoe/ui/screens/cfg.py reads Tk widget vars via `var.get()` and writes
the result to conf/<mode>.cfg. When a var is a MagicMock (interactive
./install.py run in a non-Tk context, partially-mocked widgets), the
save path serializes the mock's repr-string into the cfg, then the next
installer pass calls os.makedirs() on those values and produces
directories literally named `<MagicMock name='Canvas().tk.call().strip()'
id='4743999712'>/`.
The brief lays out a TDD approach for Junie:
1. Write failing test at tests/installer/test_cfg_save_refuses_mock_values.py
that passes MagicMock widget vars and asserts _save_knoe_cfg raises
TypeError naming the field.
2. Implement the minimal fix: a `_str_value(var, field=...)` helper in
cfg.py that validates widget reads and raises if non-str. Use it
in the .get()/.strip() chains across lines 103-155.
3. Verify the 750 existing installer tests still pass.
Brief includes file pointers (cfg.py:44 _save_knoe_cfg, line 102
globals_to_save assembly, line 277 cfg_path.write_text), the canonical
failing test stub, both fix-approach options (per-read validator vs
end-of-flow dict walk), and explicit commit-shape guidance.
Index updates:
docs/plans/junie/README.md — todo-1 row added under Active
docs/TODO.md §"In progress" — todo-1 promoted above Phase 3 (smaller
scope, easy to land first)
Naming convention: `todo-N-<slug>.md` for follow-up bugs, distinct from
the `NN-<slug>.md` pattern reserved for ranked queue items.
- Add gandalf.prole.org → 10.0.0.7 to prole_dns_records and
prole_internal_a_records in dns.yml (was missing, causing nslookup
to fail on myrddin)
- prole_ssl: stat-check each source file on the controller before
copying; emit a warning and skip rather than hard-failing when certs
have not yet been placed in ssl/prole/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Installs Tailscale on k3s agents via the Debian stable apt repo using
the modern signed-by keyring approach. Auth key stored in vault.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
init_gitea.sh requires GITEA_PV_NODE in k3s mode. PVs for Gitea storage
(gitea-local-d005) are pinned to gandalf.prole.org per nodeAffinity on
the existing gitea-d005-shared-storage PV.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cleanup_legacy_prole_kong() only checked for the prole-svc-kong deployment
and service, so the early-return guard fired after those were already deleted,
leaving svc-prole-ingress in place. On re-runs this Traefik ingress blocked
the duplicate-host check and prevented Kong from deploying.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
k8s/knoe/iscsi-pvs.yaml had *.knoe.org nodeAffinity after the rename, but the
live k3s cluster nodes are still *.prole.org — causing immutable field errors on
Bound PVs during kubectl apply. Revert to *.prole.org to match live state.
conf/k3s.cfg KUBECTL_CONTEXT was renamed to knoe-service-cluster, but both the
system kubeconfig and etc/secrets/k3s.kubeconfig still use prole-service-cluster.
install.sh _check_gcp_tools() now skips gcloud check for k3s mode and any non-gke
config path (only GKE deploys need gcloud/kubectl pre-flight).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
conf/dev/knoe.cfg had a stale SERVICE_NAMESPACE=default written by a prior TUI session.
Also guard _service_namespace() so the value 'default' is never returned — it is never
a valid service namespace and signals a stale generated config.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
