The cross-realm trust playbook was written for the pre-rebrand realm name
PROLE.LOCAL, which no longer exists in the in-cluster KDC — the canonical
realm is now KNOE.LOCAL. The Samba AD trust was therefore never actually
established, leaving chrisfu@KNOE.LOCAL→chrisfu@PROLE.ORG service ticket
flows blocked (and blocking pg_oauth / db.prole.org Kerberos work).
Changes:
- Realm: PROLE.LOCAL → KNOE.LOCAL across all task names, vars, and the
krb5.conf [realms] / [domain_realm] blocks added on myrddin.
- samba_admin_password now resolves from the Ansible vault var
vault_samba_dns_admin_pass (group_vars/ad_dc/vault.yml) by default,
with SAMBA_ADMIN_PASSWORD env and -e overrides preserved for CI.
- trust_shared_password auto-resolves from the in-cluster Secret
knoe-system/knoe-kdc-secrets (key trust_shared_password) when not
passed explicitly — same Secret init_kdc.sh writes.
- Added [domain_realm] mapping for *.knoe.local → KNOE.LOCAL so Samba
can resolve in-cluster service principals.
- Added a final `samba-tool domain trust show` smoke step so a successful
run prints the visible trust state for log review.
Run:
ANSIBLE_VAULT_PASSWORD_FILE=$PWD/.vault_pass ansible-playbook infrastructure/playbooks/kerberos_trust_setup.yml
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
win_regedit writes HKLM keys without privilege escalation when
ansible_user is already a local/domain Administrator. become:true
with --ask-become-pass was overriding the playbook's runas method.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A single play with become:true uses sudo by default, which Ansible's
Windows exec wrapper rejects immediately at gather_facts. Split into:
- Play 1: hosts workstations,!workstations_windows — macOS/Linux, sudo
- Play 2: hosts workstations_windows — Windows, become_method:runas
Also set morgana to ansible_connection=local (running Ansible from
morgana itself, so SSH self-lookup fails DNS resolution).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add morgana and zinfandel (macOS) and morgoth + fairyland (Windows)
to the workstations inventory.
Extend workstation_kerberos.yml to handle Windows targets:
- Chrome and Edge AuthServerAllowlist set via win_regedit under
HKLM\SOFTWARE\Policies\{Google\Chrome,Microsoft\Edge}
- krb5.conf and plist tasks skip on Windows (uses built-in SSPI/AD)
- Windows hosts in [workstations_windows] group with WinRM/NTLM
transport; switch to kerberos transport after domain join
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea's AdminEditUser PATCH with source_id:0 resets the password field
as a side effect when applied to a SPNEGO-registered user (the auth
record re-initialisation clears the local password). Setting the
password before the promotion PATCH meant it was immediately wiped.
Move the 1Password recovery password step to run after the admin
promotion PATCH so the final Gitea state matches 1Password.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea's AdminEditUser endpoint requires email alongside login_name
and source_id. Without it the server returns 200 but silently
ignores the password field — causing login failures despite the
script reporting success.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 1Password password-setting block was nested inside path 2 of
promote_gitea_admin(). When path 1 found a cached gitea-admin-token
k8s secret, path 2 was skipped entirely and the recovery password
was never set in Gitea, leaving the user unable to log in.
Move the password-setting step to run unconditionally after all four
token paths complete. Uses _op_ensure_auth() so it degrades gracefully
in CI/headless environments where 1Password is unavailable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Problem: 'defaults write' is not an onboarding path, it's a support
incident waiting to happen. Browser SPNEGO must be deployed via managed
policy at machine provisioning time, not per-user.
Changes:
etc/init_knoe_users.sh
- _op_ensure_auth(): tries OP_SERVICE_ACCOUNT_TOKEN env, k8s secret
'op-service-account-token', existing interactive session, and op-run
inheritance — in that order. Gives actionable instructions when none
work, including the correct 'op run --' invocation documented in k3s.cfg.
- gitea_ensure_password(): calls _op_ensure_auth() at entry; eliminates
the silent 'op not authenticated' failure path.
- Script header: documents 'op run -- bash etc/init_knoe_users.sh' as
the intended invocation for admin's laptop.
- 'Next steps' output: replaces 'defaults write' with reference to
workstation_kerberos.yml Ansible playbook.
infrastructure/playbooks/workstation_kerberos.yml (new)
- Deploys /etc/krb5.conf (PROLE.ORG realm, myrddin KDC) to all managed
endpoints.
- Deploys Chrome + Edge managed policy (AuthServerAllowlist) on macOS
and Linux — no per-user browser configuration ever required.
- Idempotent; run during laptop provisioning or re-run at any time.
- Targets 'workstations' Ansible group.
infrastructure/inventory/hosts.ini
- Adds [workstations] group with example entries and onboarding notes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SPNEGO auto-registration assigns a UUID@localhost placeholder email.
promote_gitea_admin() now detects the UUID pattern and replaces it with
the user's real configured email (KNOE_ADMIN_EMAIL / GITEA_ADMIN_EMAIL).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1Password credentials are optional (primary auth is Kerberos SPNEGO).
When op is not signed in on a headless server, show a WARN with a
helpful hint instead of an ERROR that suggests a hard failure.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
log() writes to stdout, so log calls inside functions that return
values via printf/stdout contaminate the captured output.
gitea_helm_admin_token and gitea_ensure_password now redirect all
log() calls to stderr with >&2, keeping stdout clean for the
returned token/password string.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea admin edit-user API requires email in PATCH body.
Fix HTTP status parser (tr -d vs broken cut -d_).
Add logging to user-existence check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea 1.22 requires explicit scopes in the token creation POST body.
Without them the API returns: "access token must have a scope".
Now requests: read/write:admin, read/write:user, read/write:issue,
read/write:repository — sufficient for admin user management.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a Deployment has been updated but the new pod keeps crashing before
configure-gitea runs (e.g. after a failed Helm upgrade), the running pod
still carries the old RS's password — which differs from the current
Deployment spec.
gitea_helm_admin_token() now walks: running-pod-RS → current-Deployment-spec,
trying each password until one produces a valid API token.
Also adds HTTP status + response body to gitea_api_token() failures so the
cause is visible without exposing credentials.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Gitea Helm chart (gitea-12.5.3) stores GITEA_ADMIN_USERNAME and
GITEA_ADMIN_PASSWORD as plain values in the configure-gitea init container
spec — not in a k8s Secret with key 'admin-password'. The previous code
looked for a non-existent secretKeyRef and returned empty, causing the
REST API bootstrap path to fail.
Now reads credentials via:
kubectl get deployment gitea ... env[?(@.name=="GITEA_ADMIN_PASSWORD")].value
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Gitea Helm chart names the admin credentials secret after the release
name alone, not '{release}-gitea'. For release name 'gitea' the secret is
just 'gitea'.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
deploy.sh: resolve_knoe_db_namespace() now checks DATABASE_NAMESPACE from
config (k3s.cfg: knoe-db) before probing namespaces. Previously it fell back
to 'default' because only 'knoe' was probed, baking a wrong DB hostname into
app.ini. The running server was unaffected (uses GITEA__database__HOST env var)
but the gitea admin CLI, which reads app.ini directly, could not connect.
init_knoe_users.sh: add gitea_helm_admin_token() which reads the Helm
bootstrap admin password from the gitea-gitea k8s secret and exchanges it for
a REST API token — entirely avoiding the broken CLI. Add gitea_api_set_password()
which uses PATCH /api/v1/admin/users/{user} to set the password via the running
web server (which has the correct DB connection) instead of kubectl exec.
promote_gitea_admin() now prefers the REST-API-only path:
1. existing token (env / k8s secret)
2. Helm admin k8s secret → REST token; set KNOE_ADMIN_PRINCIPAL password via API
3. 1Password → CLI set-password → basic-auth token (fallback)
4. kubectl exec generate-access-token (last resort)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
knoe_cfg.sh selects the config file based on KNOE_MODE at source time.
Without pre-scanning argv, KNOE_MODE is empty and k3d.cfg wins the
fallback loop even when --mode k3s is passed. This caused k3s.cfg values
(e.g. KNOE_ADMIN_PRINCIPAL=chrisfu) to be silently ignored, defaulting
to 'admin' and causing gitea_set_password to fail against a non-existent user.
Also surface gitea_set_password errors (without logging the password) so
failures are diagnosable rather than silent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the predictable temp-password pattern with a secure credential flow:
- gitea_ensure_password(): creates a 1Password item with --generate-password
(1P generator, 32-char letters+digits) if one does not exist; retrieves it
on re-runs (idempotent). Password never appears in logs or env vars.
- gitea_set_password(): sets the Gitea account password via kubectl exec
(gitea admin CLI, no prior Gitea auth required).
- gitea_api_token(): exchanges basic-auth credentials for a REST API token,
deleting any stale token of the same name first. Token only; password
cleared from memory immediately after use.
- promote_gitea_admin(): tries credential sources in order:
1. GITEA_ADMIN_TOKEN env / gitea-admin-token k8s secret
2. 1Password (generates if missing) → set password → REST token
3. kubectl exec generate-access-token fallback (Helm bootstrap admin)
Tokens persisted in k8s secret (revocable); passwords only in 1Password.
The Gitea password is a vault credential for emergency recovery only.
Normal auth is always Kerberos SPNEGO — users never need to type a password.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ron has no Google Workspace prole.org account. Email is now NULL in the DB
(NULLIF('')). The provision_user log line now always shows the Kerberos
principal explicitly and labels the email field separately so it is clear
which identity type is in use.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
In k3s mode, LOCAL_REGISTRY_INTERNAL set to a k3d value in the shell environment
leaked into init_kdc.sh subprocesses, causing pods to use the wrong image address
while the push went to myrddin.prole.org:5000.
Add _resolve_kdc_registry() helper that derives correct registry values from
KNOE_IMAGE_REGISTRY and PROLE_K3S_SERVER. Both init_kdc.sh call sites now pass
explicit overrides to prevent any inherited k3d value from leaking through.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 'die' call with a multiline string containing '|' (for base64 pipeline
in the help text) was parsed by bash as a pipe operator. Replace with
individual err() calls followed by explicit exit 1.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Password resolution order for PROLE_KDC_MASTER_PASSWORD and KNOE_GUEST_PASSWORD:
1. Env var (existing)
2. knoe-kdc-secrets k8s Secret (existing)
3. prole-kdc-secrets k8s Secret (new — used by live prole deployment)
4. 1Password via op read (new — OP_KDC_MASTER_PASSWORD_REF / OP_KDC_GUEST_PASSWORD_REF)
5. PROLE_LOCAL_ADMIN_PASSWORD (existing)
6. die with actionable message
Add try_op_read() helper: wraps op CLI, returns 1 if op not available or ref empty.
conf/k3s.cfg: add KNOE_ADMIN_PRINCIPAL=chrisfu, OP_KDC_MASTER_PASSWORD_REF,
OP_KDC_GUEST_PASSWORD_REF placeholders (empty — user fills in op:// URIs).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The k3s-generated kubeconfig on the cluster server node uses 'default'
as its context name. The existing guard rejected this unconditionally,
causing init_knoe_users.sh to always fail when run from myrddin.
Tighten the guard: only reject 'default' when KNOE_MODE is k3d (where
it would indicate the k3d context was not found). In k3s/k8s mode,
'default' is legitimate and the script proceeds normally.
Usage on myrddin: bash etc/init_knoe_users.sh --mode k3s initialize
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add ensure_gitea_spnego_keytab(): provisions HTTP/git.prole.org@PROLE.ORG
service principal, keytab, and gitea-krb5-keytab Secret automatically when
samba-tool is available (i.e. running on myrddin); otherwise prints manual
steps. Restarts gitea-spnego-proxy after keytab is created.
- Add GITEA_SPNEGO_HOST, GITEA_KRB5_AD_REALM, GITEA_KRB5_AD_USER config vars.
- Fix promote_gitea_admin: exec with 'su git' (gitea refuses to run as root);
use KNOE_ADMIN_PRINCIPAL instead of hardcoded 'admin'; fall back to
'gitea_admin' (Helm chart default) if the primary user has no token yet.
- Rename initialize() step numbers (9=ArgoCD, 10=SPNEGO keytab, 11=Gitea, 12=GitLab).
- Fix PROLE.LOCAL → KNOE.LOCAL in: default realm, SQL schema default, comments,
next-steps message, trust principal comments.
- show_status: add gitea-krb5-keytab check alongside pg keytab check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitea Helm chart v10.x changed defaults:
image.registry: docker.gitea.com (was docker.io)
image.rootless: true (was false)
Our values set repository/tag but not registry/rootless, so the new
defaults produced docker.gitea.com/gitea/gitea:1.22.3-rootless which
can't be pulled from gandalf. Pin registry: docker.io and rootless: false
to keep using the standard Docker Hub image.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Helm upgrade was failing for two reasons:
1. persistence.storageClass was unset (rendered null), conflicting with the
existing PVC's immutable gitea-local-d005 storageClassName.
2. valkey-cluster (and redis-cluster) were not explicitly disabled, causing
the latest gitea chart to spin up a valkey StatefulSet on each upgrade.
Add GITEA_STORAGE_CLASS env-var override for non-k3s modes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gokrb5 returns 401 silently on invalid tickets. Wrap ResponseWriter to
capture status code and log failures with remote addr, host, path, and
first 20 chars of the Negotiate token for easier debugging.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Go proxy exposes /_healthz so use httpGet probes (tcpSocket was only a
connectivity check, not a health check). Also reduce memory request/limit
to 32Mi/64Mi — the Go binary is lean and in-cluster load is minimal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds the ScheduledBackup CR to source control. Fixes schedule from
5-field to 6-field format (0 0 3 * * *) — CNPG requires seconds as
first field. The broken schedule caused the reconcile loop that
generated 746 stale Backup CRs. Deleted 7 orphaned GitLab Garage
buckets (~8.5 GB) and resumed daily backups.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path does not exist on this k3s cluster. The Prometheus Operator
was stuck in a reconcile error loop, blocking all PodMonitor updates.
Prometheus: merlin-local-iscsi-prometheus (30Gi pre-existing PV)
Alertmanager: merlin-local-iscsi-alertmanager (5Gi pre-existing PV)
Grafana: remains local-path to match immutable StatefulSet VolumeClaimTemplate;
actual PVC is already bound to merlin-local-iscsi-grafana PV.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
knoe-auth /auth/verify not yet implemented — auth_request returns 500,
blocking all Grafana access. Simplify nginx to plain passthrough; strip
inbound X-WEBAUTH-USER to prevent header forgery. Disable Grafana
auth.proxy until knoe-auth is ready. Google auth.google still active.
authority-prole-auth deployment had no Service. Create knoe-auth ClusterIP
service selecting app=authority-prole-auth on port 8080 — consistent with
Kong's knoe-auth route URL. Update nginx configmap to use the same FQDN.
Without this, kubectl apply without -n flag lands in default namespace
instead of monitoring, leaving the live configmap with the old
knoe-auth hostname.
Traefik ingress (svc-knoe-ingress) routes db.prole.org to
service/oauth2-proxy:80 in knoe-system. oauth2-proxy itself runs in
supabase namespace, so this ExternalName alias bridges the gap.
- kong-configmap: sync to live content; fix grafana upstream to route through
knoe-grafana-proxy (nginx auth proxy) instead of kps-grafana (stale service
name from old helm release); add db.prole.org → oauth2-proxy route; fix
knoe-auth FQDN to authority-prole-auth
- grafana-proxy-configmap: fix upstream to prometheus-grafana (helm release
name is 'prometheus', not 'kps'); fix auth_request FQDN to
authority-prole-auth.knoe-system; fix login redirect to api.prole.org
- grafana-proxy-{deployment,service}: add namespace: monitoring
openssl rand -base64 32 produces a 44-char string; oauth2-proxy treats it
as 44 raw bytes and rejects it. -base64 24 encodes 24 bytes → 32 base64
chars with no padding, which satisfies the 32-byte AES requirement.
Reads svc-prole-org (Grafana) and db-prole-org (oauth2-proxy) API_CREDENTIAL
items from the Personal 1Password vault and writes them into etc/secrets/ so
init_grafana_oauth_prole.sh and init_oauth2_proxy_prole.sh can consume them.
Generates and saves the oauth2-proxy cookie secret (32-byte base64) back to
the db-prole-org 1Password item as a cookie_secret field on first run;
idempotent thereafter.
Usage:
./etc/fetch_prole_secrets.sh
./etc/init_grafana_oauth_prole.sh
./etc/init_oauth2_proxy_prole.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- monitoring/kps-values-k3s.yaml: Grafana helm values for k3s homelab with
dual auth (auth.proxy Kerberos + auth.google for prole.org Workspace)
- grafana-proxy-configmap.yaml: nginx passthrough for /grafana/login/google
and /grafana/login to allow Google OAuth flow without knoe-auth redirect
- grafana-google-oidc-secret-prole.example.yaml: Secret template for Grafana
Google OAuth client (svc.prole.org, Internal consent, prole.org Workspace)
- oauth2-proxy-google-oidc-secret-prole.example.yaml: Secret template for
oauth2-proxy gating db.prole.org Studio
- oauth2-proxy-deployment-prole.yaml: k3s oauth2-proxy deployment for
db.prole.org (prole.org domain, no BackendConfig)
- init_grafana_oauth_prole.sh: Bootstrap script for Grafana OAuth secret
- init_oauth2_proxy_prole.sh: Bootstrap script for Studio oauth2-proxy
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
selectattr with 'match' regex was returning empty sequence on this Ansible
version. Since each install entry has exactly two items (fullchain.cer +
key), rejectattr the fullchain to get the key — no regex needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
