Provisions the docker engine on [build_hosts] (gandalf) for the native arm64 daily
knoe-db image build → registry.knoe.dev, run via a remote buildx builder. gandalf is
also a k3s_agent, so the role holds iptables FORWARD=ACCEPT to protect k3s pod
networking (docker's containerd coexists with k3s's on separate sockets). Verified
idempotent: --check against gandalf = ok=5, changed=0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The AD DC's smb.conf set no `interfaces` / `bind interfaces only`, so Samba
bound to — and samba_dnsupdate registered into DNS — every interface on the
host. On myrddin that meant the Docker bridge (172.17.0.1) and k3s/flannel
CNI addresses (10.42.0.0, 10.42.0.1) were published as A records for both the
prole.org apex and `myrddin`, alongside the real 10.0.0.3. Clients then
round-robined onto unroutable addresses, producing the long-standing
"resolves, then doesn't" internal DNS flakiness.
Confine Samba to loopback + the LAN service IP:
interfaces = lo 10.0.0.3
bind interfaces only = yes
This scopes both service binding and DNS self-registration to the real
address, so the junk records stop being (re)created on restart.
- smb.conf.j2: emit the two directives, gated on bind-interfaces-only being
enabled AND a non-loopback IP being present (empty -> directives omitted,
never binds loopback-only by accident).
- defaults: samba_ad_dc_lan_ip ("" by default), samba_ad_dc_bind_interfaces_only
(true), samba_ad_dc_interfaces (lo + lan_ip), all documented.
- tasks: assert samba_ad_dc_lan_ip is non-empty before deploying smb.conf
when bind-interfaces-only is on, so a missing value fails fast instead of
rendering the DC unreachable.
- group_vars/ad_dc: set samba_ad_dc_lan_ip=10.0.0.3 (myrddin's LAN address).
Deploying notifies the existing Restart samba-ad-dc handler. Pre-existing junk
records must be deleted once by hand; they will not be re-registered after the
restart. Template rendering verified for both the set and empty-IP cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`Create reverse DNS zones if missing` looped over `samba_reverse_zones`, a fact
built by a separate `set_fact` task that carried no tags. Under a tag-filtered
run (e.g. --tags samba_reverse_dns) that set_fact was skipped, so the variable
was undefined and the play failed:
TASK [samba_reverse_dns : Create reverse DNS zones if missing]
'samba_reverse_zones' is undefined
A set_fact must carry the same tags as every task that consumes it. Rather than
re-add tags (which breaks again under any other tag combination), compute the
zone list inline in the loop and drop the now-dead set_fact. The role is now
correct under any tag selection. lan_reverse_zone and k3s_reverse_zones are
defined in group_vars/all/vars.yml, so they are always available.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Forward A/CNAME and reverse PTR records reported changed=0 / ok but never
resolved by short name on the Samba AD DC (myrddin.prole.org). Root cause was
regex escaping inside YAML folded block scalars (">-"): unlike double-quoted
scalars, block scalars do NOT process backslash escapes, so a pattern written
as '\\.' reached Jinja as a literal backslash + any-char and never matched.
Consequences:
- regex_replace() never stripped the zone suffix, so records were created
under their full FQDN (e.g. a record literally named "sg2428lp.prole.org"
inside zone "prole.org"). `samba-tool dns query ... prole.org sg2428lp A`
then returns WERR_DNS_ERROR_NAME_DOES_NOT_EXIST.
- regex_findall() of existing values always returned [], so stale records
were never pruned and idempotency only survived via RECORD_ALREADY_EXISTS.
The leading/trailing whitespace from the folded scalar was a red herring: the
command module's shlex tokenization collapses it, so a single-token name
survived intact. The escaping was the actual defect.
Fix:
- Convert the templated set_fact expressions from ">-" block scalars to
double-quoted single-line scalars (one consistent escaping convention,
matching the already-working samba_target_zone line; also removes the
stray whitespace).
- Add an assertion that the computed record name is a clean, non-empty,
whitespace-free token.
- Add a post-apply verification re-query + assert that the expected
value(s) are actually present, turning a silent no-op into a hard failure.
Applies to roles/samba_dns (ensure_a.yml, ensure_cname.yml) and
roles/samba_reverse_dns (ensure_ptr.yml).
Verified end-to-end through ansible-playbook against a stateful fake
samba-tool: correct short names (sg2428lp, git, @, registry), stale-value
removal, internal.prole.org zone routing, idempotent re-runs (changed=0),
and the verify-assert failing loudly when an add silently does not persist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pi-hole 6 uses pihole-FTL --config dns.hosts as the authoritative local DNS
store. dnsmasq address= directives still work for new domains (e.g. mcp.0.knoe.dev)
but Pi-hole may serve stale upstream cache over them for previously-resolved
domains (e.g. auth.0.knoe.dev was cached from old DNS before the fix).
dns.hosts entries always win, idempotent approach: strip old *.0.knoe.dev entries,
append new ones, update via pihole-FTL --config + pihole reloaddns.
auth.0.knoe.dev: 34.120.221.5 (wrong) → 34.36.71.141 (knoe-auth GCE LB) ✓
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds dnsmasq address= directives for all active knoe.dev GKE endpoints
so LAN clients (morgana, Pi nodes) resolve them without hitting external DNS.
Records added:
mcp.0.knoe.dev → 8.232.93.14 (knoe-mcp GCE LB)
auth.0.knoe.dev → 34.36.71.141 (knoe-auth GCE LB)
api.0.knoe.dev → 34.120.221.5 (supabase-kong GCE LB)
db.0.knoe.dev → 34.120.221.5 (supabase-kong GCE LB)
app.0.knoe.dev → 34.120.221.5 (supabase-kong GCE LB)
auth.0.knoe.dev was previously resolving to 34.120.221.5 (wrong — supabase LB)
causing the knoe-auth managed cert to be stuck Provisioning for 20 days.
Correct IP is 34.36.71.141. Also update auth.0.knoe.dev in external DNS
(name.com) to unblock the GCP managed cert.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All samba-tool dns commands were using -P (machine account Kerberos) which
silently failed with no error — tasks reported changed=0 but records were
never written. Switch to -U Administrator --password={{ samba_dns_admin_pass }}
which uses the vault-protected admin credentials that were already defined
but never wired up.
Also fix regex patterns in record parsers: samba-tool output uses `A: IP`
and `PTR: fqdn` format, not `A IP` / `PTR fqdn` (space-separated), so
updated regex_findall patterns to match `TYPE:\s+value`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
selectattr with 'match' regex was returning empty sequence on this Ansible
version. Since each install entry has exactly two items (fullchain.cer +
key), rejectattr the fullchain to get the key — no regex needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The dns_namecom.sh plugin uses Namecom_Username and Namecom_Token (not
NAMECOM_*). Fix the environment dict, the SAVED_ keys written to
account.conf, and add a cleanup task to remove the stale all-caps entries
already present on hosts from previous runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Re-adds no_log: true removed in 5e4e17e and switches the issue task from
ansible.builtin.shell with inline VAR=value to ansible.builtin.command with
an environment: dict. This prevents credentials appearing in Ansible failure
output and avoids the inline env var being clobbered when acme.sh sources
account.conf internally before the dns_namecom plugin runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ansible command module + become does not reliably pass environment:
vars through sudo env_reset. Use ansible.builtin.shell with POSIX
inline assignment (VAR=val cmd) so credentials are set inside the
sudo-spawned shell subprocess, never touching the sudo env barrier.
account.conf writing retained for cron renewal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_readaccountconf_mutable checks SAVED_NAMECOM_Username first; if that
key lookup exits 0 with empty output it short-circuits and never reads
the bare NAMECOM_Username key. Write the SAVED_ prefix that acme.sh
uses internally when persisting credentials.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sudo env_reset strips the environment: key through become. Write
NAMECOM_Username and NAMECOM_Token to acme.sh's account.conf directly,
which dns_namecom reads as a fallback via _readaccountconf_mutable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New acme role: installs acme.sh, issues certs for myrddin/merlin/gandalf
via DNS-01 (name.com API), installs cert files to /etc/ssl/certs/prole/,
sets up renewal cron. rc=2 (already valid) treated as ok.
- New playbooks/acme.yml: runs acme role on myrddin (issues all 3 host certs)
- New playbooks/certmgr.yml: cert-manager and letsencrypt-prod ClusterIssuer
(HTTP-01/Traefik) for external endpoints (db/svc/api.prole.org)
- Add vault references for name.com API creds to group_vars/all/vars.yml
- Add prole_ssl_src_dir and prole_ssl_files to myrddin/merlin/gandalf host_vars
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add gandalf.prole.org → 10.0.0.7 to prole_dns_records and
prole_internal_a_records in dns.yml (was missing, causing nslookup
to fail on myrddin)
- prole_ssl: stat-check each source file on the controller before
copying; emit a warning and skip rather than hard-failing when certs
have not yet been placed in ssl/prole/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Installs Tailscale on k3s agents via the Debian stable apt repo using
the modern signed-by keyring approach. Auth key stored in vault.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
knoe-db deploys cluster-native (mode k3d for local dev, k8s for the
shared company cluster, k3s still supported for individual homelab
collaborators via conf/k3s.cfg). The ansible-managed distributed cluster
of prole.org machines is no longer the target topology, and the Ansible
inventory in this tree targets only the original author's specific hosts
(merlin/gandalf/myrddin/pi/raspberry/retropie.prole.org). It is neither
portable nor useful for the new deployment shape.
Removed:
- infrastructure/inventory/ (hosts.ini + 7 *.prole.org host_vars)
- infrastructure/playbooks/ (11 playbooks, all hardcoded to *.prole.org)
- infrastructure/roles/ (20 roles: pihole, samba AD DC, mariadb,
netplan, iscsi, prole_ssl, etc. -- all
tied to the bare-metal prole.org topology)
- infrastructure/setup/ (cr_ansible_user.sh, cr_samba_family_users.sh,
filesystem.txt -- prole.org bootstrap)
- infrastructure/deployments/svc-check-helm/
(legacy; etc/init_kong.sh already tears
down svc-check resources on every run)
- infrastructure/ansible.cfg (references deleted inventory)
- ansible.cfg, ansible.sh, ansible_min.cfg, ansible_recovery.log (root;
orphaned without inventory)
- vault_backup/ (ansible-vault encrypted prole.org secrets
backup: ad_dc_vault.yml, all_prole_vault.yml,
iscsi_vault.yml)
- tools/ (dashboard.sh, ghcr_docker_login.sh,
k3s-cluster-rename.sh, manage-node.sh,
export_mariadb_synology.sh,
prepare_mariadb_usb.sh, and a dumped
synology.prole.org mariadb .sql file --
all prole.org-era deployment helpers)
Kept: scripts/ (cluster ops), bin/ (prole-env.sh, prole-kpf.sh dev
shims), conf/k3s.cfg / conf/k3d.cfg (still supported modes; any
*.prole.org defaults in those files are legitimate mode fallbacks for
collaborators who keep running their own k3s/k3d setups). Test
fixtures referencing *.prole.org hostnames are left as-is (regression
data).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summarize CLAUDE.md updates:
- Document both live GKE clusters as e2-standard-2 x3 in us-west3 and note stale app default in code.
- Call out DB cluster naming mismatch between conf (knoe-dev-cnpg-0) and code default (knoe-cnpg-0).
- Add SSD quota guidance: 300GB fully consumed by CNPG, so non-CNPG PVCs must use standard/pd-standard.
- Refresh reset script reference and quota section details to match current runtime state.
Co-authored-by: Junie <junie@jetbrains.com>
Rename env config files from conf/*/prole.cfg to conf/k3d.cfg, conf/k3s.cfg, and conf/gke.cfg. Update shell/Python loaders and etc/deploy scripts to resolve named configs cleanly while keeping legacy fallback behavior. Align k3s Ansible tasks, docs, and regression coverage with the new configuration layout.
Co-authored-by: Junie <junie@jetbrains.com>
Update conf/service/prole.cfg to persist the k3s kube context as knoe.dev.prole.org and replace host-specific absolute paths with $HOME-based paths for portability.
Include all pending project changes: local_user role package and screenrc provisioning updates, plus tools/k3s-cluster-rename.sh and tests/etc/test_k3s_cluster_rename.sh.
Co-authored-by: Junie <junie@jetbrains.com>
- Rename host mount namespace from /prole/d00x to /synology/d00x
- Make LUN ownership explicit per host in inventory (myrddin=d001; merlin=d002,d004; pi=d003)
- Add safe migration cleanup for legacy /prole/d00x mounts and stale /etc/fstab entries
- Update k3s/ArgoCD/OpenTofu manifests and PV node affinities to match new mount paths
- Improve iSCSI role check-mode behavior and add quiesce/detach flow in ansible.sh
- Persist and export supabase_hostname for canonical external Supabase entrypoint (db.prole.org)
- Render Supabase Helm ingress host and public URLs (API_EXTERNAL_URL/GOTRUE_SITE_URL/SUPABASE_PUBLIC_URL) from config
- Align tracked Traefik ingress manifests to db.prole.org
- Generalize service hostname/TLS wiring in k3s automation and refresh Kong/monitoring behavior
- Make optional workload policy checks deterministic when kube context is absent; add render tests
Prefer infra-managed manifests for bootstrap; reconcile CNPG instances based on Ready+schedulable labeled db nodes; update manifests to use node-role affinity + anti-affinity; add policy/tests and config touch-ups (incl. prole.cfg).
Update inventory/roles to keep K3s state on local storage and prevent iSCSI from managing /var/lib/rancher.
Also add a single-host k3s install playbook, a systemd override template, and docs describing the installer ↔ Ansible boundary and slow-storage knobs.
Co-authored-by: Junie <junie@jetbrains.com>
- Add diagnostics/reset/retry logic for stuck monitoring Helm installs\n- Add optional cluster-side node/lease deletion during k3s reset via delegated control-plane\n- Bump prole-db image tag/version to 18-123\n- Update monitoring init unit test for reset/diagnostics path
- k3s_sync: choose any server as sync source fallback; refresh kubeconfig and validate nodes\n- fetch_kubeconfig: save to prole-k3s.kubeconfig and etc/secrets/k3s.kubeconfig\n- k3s sync: sync server URL into config.yaml; restore TLS bundle only on servers\n- cert-manager/ACME: wait for CNI/webhook readiness; retry ClusterIssuer/Certificate apply\n- inventory: tune myrddin control-plane timing args; add render test\n- misc: update port mappings and prole-db manifests/version
Co-authored-by: Junie <junie@jetbrains.com>
- Introduce lib/shell helpers and keep etc/* scripts thin via compatibility shims
- Move Kerberos validation to scripts/validation/check_kerberos.sh and update callers
- Add deterministic shellspec unit tests under tests/shellspec/ and wire Maven to run them
- Add minimal Spring Boot authority module with startup + /health endpoint and Maven wiring
- Document the new layout in docs/layout.md
Co-authored-by: Junie <junie@jetbrains.com>
- Add `ansible.builtin.stat` task to verify `prole.cfg` availability before loading defaults.
- Prevent errors when `prole.cfg` is missing by skipping `lookup` operations.
- Update documentation to note `/external` may already be mounted before Ansible runs.
- Avoid redundant ownership and permissions enforcement in `mariadb_primary` role.
- Ensure dashboard service doesn't create an ordering cycle with `multi-user.target`.
- Verify service safely attaches to a virtual terminal (VT) and passes VT to Xorg.
- Ensure no `su` usage by default in the service.
- Validate script behavior for passing VT and `-keeptty` options to Xorg.
chore(dashboard): update systemd unit to avoid VT conflicts and improve logging
- Remove `After` and `Wants` directives for `multi-user.target` to prevent conflicts.
- Add `TTYPath`, `StandardInput`, and related options for VT attachment.
- Ensure `ExecStopPost` restores getty service for the dashboard VT.
fix(dashboard): improve Xorg resolution and clean up script logic
- Dynamically determine Xorg executable location in `dashboard.sh`.
- Pass VT, display, and `-keeptty` to Xorg to improve script reliability.
fix(mariadb): correct exFAT mount options for external storage
- Avoid `chown` failures by setting root ownership with group access for `mysql`.
- Update external mount options to `uid=0,gid=mysql,umask=0002`.
cleanup(mariadb): remove unnecessary owner/group permissions from migration marker task
- Enable mounting borrowed USB disk (exFAT) persistently at `/external`.
- Migrate MariaDB datadir to `/external/mariadb` with bind mount at `/srv/mariadb/mariadb`.
- Add migration marker and rollback support to prevent accidental remigration.
- Include Ansible tasks for external storage setup and migration validation.
- Add ssl_hosts group and prole_ssl role to deploy registry cert/key to /etc/ssl/certs/prole
- Configure k8s registry deployment to serve HTTPS using mounted host certs
- Switch k3s/containerd registry mirror config to https:// and prefer HTTPS checks with HTTP fallback
Co-authored-by: Junie <junie@jetbrains.com>
- Added tasks and playbooks for K3s datastore export/import using MariaDB Tools role, with associated tests and defaults.
- Enhanced iSCSI role to support mkfs-once logic and safer re-initialization of block storage.
- Migrated iSCSI-backed Rancher data from retropie to merlin.prole.org.
- Updated k3s roles/playbooks to relax Rancher storage preflight checks, supporting PARTUUID-based mounts.
- Adjusted Samba AD NetBIOS name derivation to use uppercase short hostname by default.
- Incremented prole DB version to 104, updated generated prole.cfg, inventory, and recovery manifest templates.
ArgoCD IO load distribution: Updated k8s/argocd/install.yaml to use hostPath volumes for application-controller (/prole/d001), redis (/prole/d002), and repo-server (/prole/d003). Updated Ansible configure task to create hostPath directories on myrddin.prole.org.
K3s configuration and testing: Added test playbooks for kubeconfig rewriting and argument validation. Updated K3s install, configure, and image import tasks. Configured required mounts and host-specific variables for myrddin.prole.org.
Miscellaneous: Updated conf/prole.cfg with K3S_TOKEN. Bumped prole-db version and updated recovery manifest templates. Minor UI adjustments in cluster screen.
Co-authored-by: Junie <junie@jetbrains.com>
* CNPG: Implement version pinning, automatic manifest URL derivation, and optional operator upgrades.
* CNPG: Add installation and management of the kubectl-cnpg plugin with architecture detection.
* k3s: Update etc/init_cloudnative_pg.sh to support configurable CNPG operator versions.
* Ansible: Enhance cgroups and k3s roles with improved check mode support and more robust variable defaults.
* Ansible: Improve kernel command-line path selection logic in the cgroups role.
Co-authored-by: Junie <junie@jetbrains.com>
