Three-part fix to make browser and curl SPNEGO auth work end-to-end:
1. spnego-proxy: issue WWW-Authenticate: Negotiate challenge when no
Authorization header is present so Chrome (with AuthServerAllowlist)
and curl --negotiate automatically present Kerberos tokens. Previously
the proxy only validated tokens if the client proactively sent them.
Pass-through preserved for non-Negotiate schemes (Basic/token) so
git CLI users with PATs continue to work via Gitea own auth.
2. gitea_spnego_keytab.yml: new Ansible playbook that provisions the
gitea-http AD account (AES-only, msDS-SupportedEncryptionTypes=24),
registers SPN HTTP/git.prole.org, resets the password to derive fresh
AES keys, exports the domain keytab, and rekeys it to principal name
HTTP/git.prole.org@PROLE.ORG that gokrb5 needs for keytab lookup.
Key lesson: samba-tool exportkeytab --principal=HTTP/... returns empty;
must export full domain keytab and rekey in Python.
3. init_gitea.sh: add setup_gitea_spnego() calling the Ansible playbook
in k3s mode as part of the standard deploy flow, with inline notes
on every non-obvious constraint discovered during this work.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
infrastructure/playbooks/pihole_flush_dns.yml — Ansible playbook to
flush Pi-hole DNS cache across inventory hosts.
infrastructure/logs/ — log directory for Ansible run output.
Co-authored-by: Junie <junie@jetbrains.com>
The matching knoe-db change (commit `ff7546d`) re-keys the MIT-side
cross-realm krbtgts with `-e arcfour-hmac:normal` to dodge the
AES-salt mismatch between Samba and MIT. For the trust to actually
carry traffic, the Samba side must also offer only RC4 on the
trust account — otherwise the client picks AES, the two sides
derive different keys from the same password (different salt
conventions), and TGT decryption fails.
Change `msDS-SupportedEncryptionTypes` from 28 (RC4+AES128+AES256)
to 4 (RC4 only). RC4 has no salt, so keys derive from the password
alone and both sides converge.
Updated docstring + summary debug print to reflect the new value
and the why.
Tracked alongside the Junie brief at
~/dev/knoe-db/docs/plans/junie/kdc-trust-reset-repeatable.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
klist shows the Kerberos principal (slash form
krbtgt/KNOE.LOCAL@PROLE.ORG), not the sAMAccountName underscore form
(krbtgt_KNOE.LOCAL). Update the summary debug task accordingly and
split the kvno hint into two stages (the cross-realm TGT first, then
a service ticket) so the user can isolate which step fails when
chasing salt/enctype issues.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The apt install task was positioned after the user-create step, so a
failure in create (or anywhere upstream) meant ldb-tools never got
installed. Move the apt task to the very top of the play, before the
cluster-secret lookups, so it always runs first on every invocation.
Drop the duplicate apt task that previously sat just before the
ldbmodify locator.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The _trust_exists probe was still grepping samba-tool user list for
"krbtgt/KNOE.LOCAL" (slash form), but the actual sAMAccountName is
"krbtgt_KNOE.LOCAL" (underscore form, since slash is invalid in
sAMAccountName). _trust_exists was therefore always false, the create
step retried against the existing account on every re-run, and Samba
rejected it as a name collision.
Match against samba_account_name directly so re-runs route through
the setpassword branch instead.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
myrddin doesn't have ldbmodify installed — it's in the ldb-tools
Debian package, separate from samba-tool. Add an apt task ahead of
the locator so the package is in place before we try to use it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ldbmodify is part of the ldb-tools Debian package but is not in
root's default PATH under sudo on the Samba DC, so the user-attribute
modify step failed with "ldbmodify: command not found".
Add a locator task that probes the conventional install paths
(/usr/bin, /usr/sbin, /usr/local/bin, /opt/samba/bin) and falls back
to a depth-limited find under /usr and /opt. The downstream
ldbmodify invocation now uses the absolute path resolved here.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
samba-tool user create rejected the cluster-generated trust_shared_password
with "0000052D: Constraint violation - check_password_restrictions: the
password does not meet the complexity criteria". The trust password is
high-entropy random bytes from init_kdc.sh and may not satisfy AD's
"3 of 4 character categories" rule.
Wrap the create + setpassword steps in a block that:
1. reads the current `samba-tool domain passwordsettings show` state
2. flips complexity off if it was on
3. does the user write
4. always: restores complexity to its prior on/off state
The restore is in an `always:` clause so a failure inside the write
block does not leave the domain policy weakened.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
samba-tool user create rejected the sAMAccountName "krbtgt/KNOE.LOCAL"
with "samldb: sAMAccountName contains invalid '/' character". AD/Samba
disallow "/" in sAMAccountName even though Kerberos principal names
require it (krbtgt/REMOTE@LOCAL).
Switch storage name to "krbtgt_KNOE.LOCAL" and add an ldbmodify pass
that sets the canonical Kerberos identity on the same object:
userPrincipalName = krbtgt/KNOE.LOCAL@PROLE.ORG
servicePrincipalName = krbtgt/KNOE.LOCAL
msDS-SupportedEncryptionTypes = 28 (RC4+AES128+AES256)
Samba KDC resolves principals by UPN/SPN, so a TGS-REQ for
krbtgt/KNOE.LOCAL@PROLE.ORG will hit this account.
Note: key-salt parity with the MIT side is NOT guaranteed yet. Samba's
default salt for AES keys is REALM+UPN; MIT's default for cross-realm
krbtgt is REALM+"krbtgt"+REMOTE. If kvno fails with "decrypt integrity
check failed" we'll add a keytab-export/import step in a follow-up
rather than try to coerce Samba's salt at creation time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
In --check mode the user-create step is skipped (it's mutating), so
the subsequent `samba-tool user show krbtgt/KNOE.LOCAL` probe fails
with rc=1 on a first dry-run because the user doesn't yet exist. Add
the same guarded failed_when used on the trust validate/show tasks
earlier: only fail when NOT (check_mode AND trust didn't pre-exist).
Real runs still fail hard on rc!=0.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
samba-tool domain trust create --type=external only works against
another writeable AD-style DC; against an MIT KDC it fails with
"Failed to find a writeable DC for domain 'KNOE.LOCAL'". Our peer at
svc/auth in knoe-system is a plain Kerberos KDC, so we cannot use that
path.
Replace the trust create/validate/show steps with the supported
Samba-AD <-> MIT-KDC approach: create the inter-realm TGT principal as
an ordinary user account in Samba whose sAMAccountName is
"krbtgt/KNOE.LOCAL", with its password synced to the cluster Secret
knoe-system/knoe-kdc-secrets/trust_shared_password (which the MIT side
already keys against). Steps:
1. samba-tool user list (idempotency probe)
2. samba-tool user create krbtgt/KNOE.LOCAL ... (when missing)
3. samba-tool user setpassword krbtgt/KNOE.LOCAL (when present, to
re-sync after a cluster Secret rotation)
4. samba-tool user setexpiry --noexpiry
5. ldbmodify msDS-SupportedEncryptionTypes=28 (RC4+AES128+AES256
to match what init_kdc.sh sets on the MIT side)
6. samba-tool user show (smoke probe)
7. debug task prints the manual kvno smoke-test command
Cluster-secret lookups and krb5.conf [realms]/[domain_realm] edits are
unchanged - those parts were correct.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ansible --check mode auto-skips command/shell tasks unless they
explicitly opt in via check_mode:false. The two cluster-state lookups
(svc/auth ClusterIP and knoe-kdc-secrets/trust_shared_password) were
silently skipped during dry-run, leaving trust_kdc_ip empty and tripping
the assertion immediately.
Add check_mode:false to the five read-only tasks so a --check run can
still resolve cluster state and report what would change:
- Resolve trust_kdc_ip from cluster
- Resolve trust_shared_password from knoe-kdc-secrets
- samba-tool domain trust list (idempotency probe)
- samba-tool domain trust validate
- samba-tool domain trust show (smoke print)
The mutating create step (samba-tool domain trust create) keeps the
default behavior - skipped in check mode. validate/show gain a guarded
failed_when so a check-mode dry-run on a host where the trust does not
yet exist does not fail (the create was skipped, so a non-zero rc is
expected). Real runs still fail hard on rc!=0.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The cross-realm trust playbook was written for the pre-rebrand realm name
PROLE.LOCAL, which no longer exists in the in-cluster KDC — the canonical
realm is now KNOE.LOCAL. The Samba AD trust was therefore never actually
established, leaving chrisfu@KNOE.LOCAL→chrisfu@PROLE.ORG service ticket
flows blocked (and blocking pg_oauth / db.prole.org Kerberos work).
Changes:
- Realm: PROLE.LOCAL → KNOE.LOCAL across all task names, vars, and the
krb5.conf [realms] / [domain_realm] blocks added on myrddin.
- samba_admin_password now resolves from the Ansible vault var
vault_samba_dns_admin_pass (group_vars/ad_dc/vault.yml) by default,
with SAMBA_ADMIN_PASSWORD env and -e overrides preserved for CI.
- trust_shared_password auto-resolves from the in-cluster Secret
knoe-system/knoe-kdc-secrets (key trust_shared_password) when not
passed explicitly — same Secret init_kdc.sh writes.
- Added [domain_realm] mapping for *.knoe.local → KNOE.LOCAL so Samba
can resolve in-cluster service principals.
- Added a final `samba-tool domain trust show` smoke step so a successful
run prints the visible trust state for log review.
Run:
ANSIBLE_VAULT_PASSWORD_FILE=$PWD/.vault_pass ansible-playbook infrastructure/playbooks/kerberos_trust_setup.yml
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
win_regedit writes HKLM keys without privilege escalation when
ansible_user is already a local/domain Administrator. become:true
with --ask-become-pass was overriding the playbook's runas method.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A single play with become:true uses sudo by default, which Ansible's
Windows exec wrapper rejects immediately at gather_facts. Split into:
- Play 1: hosts workstations,!workstations_windows — macOS/Linux, sudo
- Play 2: hosts workstations_windows — Windows, become_method:runas
Also set morgana to ansible_connection=local (running Ansible from
morgana itself, so SSH self-lookup fails DNS resolution).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add morgana and zinfandel (macOS) and morgoth + fairyland (Windows)
to the workstations inventory.
Extend workstation_kerberos.yml to handle Windows targets:
- Chrome and Edge AuthServerAllowlist set via win_regedit under
HKLM\SOFTWARE\Policies\{Google\Chrome,Microsoft\Edge}
- krb5.conf and plist tasks skip on Windows (uses built-in SSPI/AD)
- Windows hosts in [workstations_windows] group with WinRM/NTLM
transport; switch to kerberos transport after domain join
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Problem: 'defaults write' is not an onboarding path, it's a support
incident waiting to happen. Browser SPNEGO must be deployed via managed
policy at machine provisioning time, not per-user.
Changes:
etc/init_knoe_users.sh
- _op_ensure_auth(): tries OP_SERVICE_ACCOUNT_TOKEN env, k8s secret
'op-service-account-token', existing interactive session, and op-run
inheritance — in that order. Gives actionable instructions when none
work, including the correct 'op run --' invocation documented in k3s.cfg.
- gitea_ensure_password(): calls _op_ensure_auth() at entry; eliminates
the silent 'op not authenticated' failure path.
- Script header: documents 'op run -- bash etc/init_knoe_users.sh' as
the intended invocation for admin's laptop.
- 'Next steps' output: replaces 'defaults write' with reference to
workstation_kerberos.yml Ansible playbook.
infrastructure/playbooks/workstation_kerberos.yml (new)
- Deploys /etc/krb5.conf (PROLE.ORG realm, myrddin KDC) to all managed
endpoints.
- Deploys Chrome + Edge managed policy (AuthServerAllowlist) on macOS
and Linux — no per-user browser configuration ever required.
- Idempotent; run during laptop provisioning or re-run at any time.
- Targets 'workstations' Ansible group.
infrastructure/inventory/hosts.ini
- Adds [workstations] group with example entries and onboarding notes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New acme role: installs acme.sh, issues certs for myrddin/merlin/gandalf
via DNS-01 (name.com API), installs cert files to /etc/ssl/certs/prole/,
sets up renewal cron. rc=2 (already valid) treated as ok.
- New playbooks/acme.yml: runs acme role on myrddin (issues all 3 host certs)
- New playbooks/certmgr.yml: cert-manager and letsencrypt-prod ClusterIssuer
(HTTP-01/Traefik) for external endpoints (db/svc/api.prole.org)
- Add vault references for name.com API creds to group_vars/all/vars.yml
- Add prole_ssl_src_dir and prole_ssl_files to myrddin/merlin/gandalf host_vars
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Installs Tailscale on k3s agents via the Debian stable apt repo using
the modern signed-by keyring approach. Auth key stored in vault.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
knoe-db deploys cluster-native (mode k3d for local dev, k8s for the
shared company cluster, k3s still supported for individual homelab
collaborators via conf/k3s.cfg). The ansible-managed distributed cluster
of prole.org machines is no longer the target topology, and the Ansible
inventory in this tree targets only the original author's specific hosts
(merlin/gandalf/myrddin/pi/raspberry/retropie.prole.org). It is neither
portable nor useful for the new deployment shape.
Removed:
- infrastructure/inventory/ (hosts.ini + 7 *.prole.org host_vars)
- infrastructure/playbooks/ (11 playbooks, all hardcoded to *.prole.org)
- infrastructure/roles/ (20 roles: pihole, samba AD DC, mariadb,
netplan, iscsi, prole_ssl, etc. -- all
tied to the bare-metal prole.org topology)
- infrastructure/setup/ (cr_ansible_user.sh, cr_samba_family_users.sh,
filesystem.txt -- prole.org bootstrap)
- infrastructure/deployments/svc-check-helm/
(legacy; etc/init_kong.sh already tears
down svc-check resources on every run)
- infrastructure/ansible.cfg (references deleted inventory)
- ansible.cfg, ansible.sh, ansible_min.cfg, ansible_recovery.log (root;
orphaned without inventory)
- vault_backup/ (ansible-vault encrypted prole.org secrets
backup: ad_dc_vault.yml, all_prole_vault.yml,
iscsi_vault.yml)
- tools/ (dashboard.sh, ghcr_docker_login.sh,
k3s-cluster-rename.sh, manage-node.sh,
export_mariadb_synology.sh,
prepare_mariadb_usb.sh, and a dumped
synology.prole.org mariadb .sql file --
all prole.org-era deployment helpers)
Kept: scripts/ (cluster ops), bin/ (prole-env.sh, prole-kpf.sh dev
shims), conf/k3s.cfg / conf/k3d.cfg (still supported modes; any
*.prole.org defaults in those files are legitimate mode fallbacks for
collaborators who keep running their own k3s/k3d setups). Test
fixtures referencing *.prole.org hostnames are left as-is (regression
data).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summarize CLAUDE.md updates:
- Document both live GKE clusters as e2-standard-2 x3 in us-west3 and note stale app default in code.
- Call out DB cluster naming mismatch between conf (knoe-dev-cnpg-0) and code default (knoe-cnpg-0).
- Add SSD quota guidance: 300GB fully consumed by CNPG, so non-CNPG PVCs must use standard/pd-standard.
- Refresh reset script reference and quota section details to match current runtime state.
Co-authored-by: Junie <junie@jetbrains.com>
- Rename host mount namespace from /prole/d00x to /synology/d00x
- Make LUN ownership explicit per host in inventory (myrddin=d001; merlin=d002,d004; pi=d003)
- Add safe migration cleanup for legacy /prole/d00x mounts and stale /etc/fstab entries
- Update k3s/ArgoCD/OpenTofu manifests and PV node affinities to match new mount paths
- Improve iSCSI role check-mode behavior and add quiesce/detach flow in ansible.sh
Update inventory/roles to keep K3s state on local storage and prevent iSCSI from managing /var/lib/rancher.
Also add a single-host k3s install playbook, a systemd override template, and docs describing the installer ↔ Ansible boundary and slow-storage knobs.
Co-authored-by: Junie <junie@jetbrains.com>
- Add diagnostics/reset/retry logic for stuck monitoring Helm installs\n- Add optional cluster-side node/lease deletion during k3s reset via delegated control-plane\n- Bump prole-db image tag/version to 18-123\n- Update monitoring init unit test for reset/diagnostics path
- k3s_sync: choose any server as sync source fallback; refresh kubeconfig and validate nodes\n- fetch_kubeconfig: save to prole-k3s.kubeconfig and etc/secrets/k3s.kubeconfig\n- k3s sync: sync server URL into config.yaml; restore TLS bundle only on servers\n- cert-manager/ACME: wait for CNI/webhook readiness; retry ClusterIssuer/Certificate apply\n- inventory: tune myrddin control-plane timing args; add render test\n- misc: update port mappings and prole-db manifests/version
Co-authored-by: Junie <junie@jetbrains.com>
- Add ssl_hosts group and prole_ssl role to deploy registry cert/key to /etc/ssl/certs/prole
- Configure k8s registry deployment to serve HTTPS using mounted host certs
- Switch k3s/containerd registry mirror config to https:// and prefer HTTPS checks with HTTP fallback
Co-authored-by: Junie <junie@jetbrains.com>
- Added tasks and playbooks for K3s datastore export/import using MariaDB Tools role, with associated tests and defaults.
- Enhanced iSCSI role to support mkfs-once logic and safer re-initialization of block storage.
- Migrated iSCSI-backed Rancher data from retropie to merlin.prole.org.
- Updated k3s roles/playbooks to relax Rancher storage preflight checks, supporting PARTUUID-based mounts.
- Adjusted Samba AD NetBIOS name derivation to use uppercase short hostname by default.
- Incremented prole DB version to 104, updated generated prole.cfg, inventory, and recovery manifest templates.
ArgoCD IO load distribution: Updated k8s/argocd/install.yaml to use hostPath volumes for application-controller (/prole/d001), redis (/prole/d002), and repo-server (/prole/d003). Updated Ansible configure task to create hostPath directories on myrddin.prole.org.
K3s configuration and testing: Added test playbooks for kubeconfig rewriting and argument validation. Updated K3s install, configure, and image import tasks. Configured required mounts and host-specific variables for myrddin.prole.org.
Miscellaneous: Updated conf/prole.cfg with K3S_TOKEN. Bumped prole-db version and updated recovery manifest templates. Minor UI adjustments in cluster screen.
Co-authored-by: Junie <junie@jetbrains.com>
Repairs and improvements:
- iSCSI: Added cleanup tasks to remove stale mounts and fstab entries. Improved robustness of iSCSI target management and added 'iscsi_absent_mounts' support.
- K3s:
- Updated service start logic to accept 'activating' state, preventing premature failure during slow startups.
- Improved service stop logic to safely handle missing or not-found services.
- Ensured 'prole-installer' ServiceAccount and ClusterRoleBinding exist for K8s administration.
- Added leader election and etcd tuning arguments (forgiving leases) to config.yaml.j2.
- Removed deprecated 'prole-port-forwards' systemd service.
- Installer & Scripts:
- Updated legacy_tk.py to support K3s mode, secret resolution for passwords, and better environment management (including ~/.prole/env.sh for service mode).
- Updated init_ansible.sh to support PROLE_VAULT_PASS_FILE and ANSIBLE_VAULT_PASSWORD_FILE.
- Improved directory and kubeconfig path resolution in prole_cfg.sh to support fallback to ~/.prole.
- Enhanced Grafana password resolution in init_monitoring.sh.
- Added automatic application of iSCSI StorageClass and PersistentVolumes in init_openbao.sh.
- General: Switched conf/prole.cfg to k3s deployment mode and updated vault_k3s.yml token.
New Ansible Tasks and Playbooks:
- infrastructure/playbooks/iscsi_cleanup.yml: Automates logout and removal of stale iSCSI node records.
- infrastructure/playbooks/prole_logs_migrate.yml: Orchestrates /prole/logs migration to iSCSI storage.
- infrastructure/playbooks/tmp_bao_dir.yml: Ensures host-level storage directories for OpenBao.
- infrastructure/playbooks/tmp_mount.yml: Utility to verify and enforce host-level mounts.
- infrastructure/playbooks/k3s_sync.yml: Added tasks to start K3s after sync and update local kubeconfig on the controller.
- Added 'Unmount stale iSCSI mounts' and 'Remove stale iSCSI fstab entries' to the iscsi role.
- Added 'Ensure prole-installer service account exists' to the k3s role.
- Moved files from 'prole/' subdirectory to root level or appropriate subdirectories (tests, authority, infrastructure) to flatten the project structure.
- Updated 'install.py' and initialization scripts in 'etc/' to reflect the new directory layout.
- Added 'etc/repair_pipeline.sh' for automated pipeline repairs.
- Updated configuration files including 'conf/prole.cfg' and 'env.sh'.
- Integrated ArgoCD manifests in 'k8s/argocd/'.
- Updated 'prole-app' environment and properties.
- Moved and updated test scripts for better organization and reliability.
- Added 'tests/silent_install_test.sh' for automated installation testing.
- Installer: Updated k3s deployment logic and configuration generation.
- k3s Role: Implemented token drift protection to verify Vault secrets against live node tokens.
- DNS: Enhanced samba_reverse_dns role to support multiple reverse zones.
- Service Init: Updated initialization scripts and status reporting.
- Infrastructure: Added prole management role and k3s diagnostic playbook.
- Configuration: Updated prole.cfg and added vaulted group variables.
- Introduced cgroup kernel parameter management for `k3s_hosts` via Ansible playbook.
- Added Kubernetes PersistentVolumes for iSCSI storage with specific node affinity.
- Defined `prole-iscsi` StorageClass for volume provisioning.
- Added `nodeSelector` for multiple Kubernetes resources to ensure scheduling on `myrddin.prole.org`.
- Modified `storage` requests and set `storageClassName` in `garage-statefulset.yaml`.
- Enhanced `install.py` for dynamic environment configuration and kubeconfig handling.
- Improved cgroup management tasks in Ansible with conflict resolution and parameter updates.
- Simplified vault token update process in playbooks and updated encryption checks.
