Provisions the docker engine on [build_hosts] (gandalf) for the native arm64 daily
knoe-db image build → registry.knoe.dev, run via a remote buildx builder. gandalf is
also a k3s_agent, so the role holds iptables FORWARD=ACCEPT to protect k3s pod
networking (docker's containerd coexists with k3s's on separate sockets). Verified
idempotent: --check against gandalf = ok=5, changed=0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Captured for the knoe-db convergence (Phase 2): configure_kubernetes_auth() (OpenBao
K8s auth → scoped knoe-jobs tokens vs root), the Kong knoe-secret route (secure-dropbox
bridge /secret/* → knoe-jobs:8081), and the prole.org canary-deploy runbook. None of
this is in canonical knoe-db — it is prole-staging-specific, not stale.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The AD DC's smb.conf set no `interfaces` / `bind interfaces only`, so Samba
bound to — and samba_dnsupdate registered into DNS — every interface on the
host. On myrddin that meant the Docker bridge (172.17.0.1) and k3s/flannel
CNI addresses (10.42.0.0, 10.42.0.1) were published as A records for both the
prole.org apex and `myrddin`, alongside the real 10.0.0.3. Clients then
round-robined onto unroutable addresses, producing the long-standing
"resolves, then doesn't" internal DNS flakiness.
Confine Samba to loopback + the LAN service IP:
interfaces = lo 10.0.0.3
bind interfaces only = yes
This scopes both service binding and DNS self-registration to the real
address, so the junk records stop being (re)created on restart.
- smb.conf.j2: emit the two directives, gated on bind-interfaces-only being
enabled AND a non-loopback IP being present (empty -> directives omitted,
never binds loopback-only by accident).
- defaults: samba_ad_dc_lan_ip ("" by default), samba_ad_dc_bind_interfaces_only
(true), samba_ad_dc_interfaces (lo + lan_ip), all documented.
- tasks: assert samba_ad_dc_lan_ip is non-empty before deploying smb.conf
when bind-interfaces-only is on, so a missing value fails fast instead of
rendering the DC unreachable.
- group_vars/ad_dc: set samba_ad_dc_lan_ip=10.0.0.3 (myrddin's LAN address).
Deploying notifies the existing Restart samba-ad-dc handler. Pre-existing junk
records must be deleted once by hand; they will not be re-registered after the
restart. Template rendering verified for both the set and empty-IP cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`Create reverse DNS zones if missing` looped over `samba_reverse_zones`, a fact
built by a separate `set_fact` task that carried no tags. Under a tag-filtered
run (e.g. --tags samba_reverse_dns) that set_fact was skipped, so the variable
was undefined and the play failed:
TASK [samba_reverse_dns : Create reverse DNS zones if missing]
'samba_reverse_zones' is undefined
A set_fact must carry the same tags as every task that consumes it. Rather than
re-add tags (which breaks again under any other tag combination), compute the
zone list inline in the loop and drop the now-dead set_fact. The role is now
correct under any tag selection. lan_reverse_zone and k3s_reverse_zones are
defined in group_vars/all/vars.yml, so they are always available.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Forward A/CNAME and reverse PTR records reported changed=0 / ok but never
resolved by short name on the Samba AD DC (myrddin.prole.org). Root cause was
regex escaping inside YAML folded block scalars (">-"): unlike double-quoted
scalars, block scalars do NOT process backslash escapes, so a pattern written
as '\\.' reached Jinja as a literal backslash + any-char and never matched.
Consequences:
- regex_replace() never stripped the zone suffix, so records were created
under their full FQDN (e.g. a record literally named "sg2428lp.prole.org"
inside zone "prole.org"). `samba-tool dns query ... prole.org sg2428lp A`
then returns WERR_DNS_ERROR_NAME_DOES_NOT_EXIST.
- regex_findall() of existing values always returned [], so stale records
were never pruned and idempotency only survived via RECORD_ALREADY_EXISTS.
The leading/trailing whitespace from the folded scalar was a red herring: the
command module's shlex tokenization collapses it, so a single-token name
survived intact. The escaping was the actual defect.
Fix:
- Convert the templated set_fact expressions from ">-" block scalars to
double-quoted single-line scalars (one consistent escaping convention,
matching the already-working samba_target_zone line; also removes the
stray whitespace).
- Add an assertion that the computed record name is a clean, non-empty,
whitespace-free token.
- Add a post-apply verification re-query + assert that the expected
value(s) are actually present, turning a silent no-op into a hard failure.
Applies to roles/samba_dns (ensure_a.yml, ensure_cname.yml) and
roles/samba_reverse_dns (ensure_ptr.yml).
Verified end-to-end through ansible-playbook against a stateful fake
samba-tool: correct short names (sg2428lp, git, @, registry), stale-value
removal, internal.prole.org zone routing, idempotent re-runs (changed=0),
and the verify-assert failing loudly when an add silently does not persist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
vault_winrm_password_morgoth and vault_winrm_password_fairyland stored for
Ansible WinRM auth to Windows GPU workstations.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pi-hole 6 uses pihole-FTL --config dns.hosts as the authoritative local DNS
store. dnsmasq address= directives still work for new domains (e.g. mcp.0.knoe.dev)
but Pi-hole may serve stale upstream cache over them for previously-resolved
domains (e.g. auth.0.knoe.dev was cached from old DNS before the fix).
dns.hosts entries always win, idempotent approach: strip old *.0.knoe.dev entries,
append new ones, update via pihole-FTL --config + pihole reloaddns.
auth.0.knoe.dev: 34.120.221.5 (wrong) → 34.36.71.141 (knoe-auth GCE LB) ✓
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds dnsmasq address= directives for all active knoe.dev GKE endpoints
so LAN clients (morgana, Pi nodes) resolve them without hitting external DNS.
Records added:
mcp.0.knoe.dev → 8.232.93.14 (knoe-mcp GCE LB)
auth.0.knoe.dev → 34.36.71.141 (knoe-auth GCE LB)
api.0.knoe.dev → 34.120.221.5 (supabase-kong GCE LB)
db.0.knoe.dev → 34.120.221.5 (supabase-kong GCE LB)
app.0.knoe.dev → 34.120.221.5 (supabase-kong GCE LB)
auth.0.knoe.dev was previously resolving to 34.120.221.5 (wrong — supabase LB)
causing the knoe-auth managed cert to be stuck Provisioning for 20 days.
Correct IP is 34.36.71.141. Also update auth.0.knoe.dev in external DNS
(name.com) to unblock the GCP managed cert.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
INSERT ... RETURNING needs RETURN QUERY in PL/pgSQL RETURNS TABLE functions.
tenant_id column is smallint in knoe.tenants; cast to integer to match
the function's declared return type.
Reproduced on pg.prole.org at 2026-05-30 during Phase 2 canary deploy.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Replace the knoe ASCII art banner with a Prole. banner linking to all
four live services (svc/db/git/api.prole.org). Add # Prole. as the
top-level header with a one-paragraph description of the homelab
deployment and its relationship to upstream knoe-db. Demote the Knoe
platform overview to ## Knoe platform.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add infrastructure/logs/ to .gitignore so ansible run logs
(infrastructure/logs/ansible/*.log) are never accidentally committed
- Track docs/plans/junie/ekosystem-uuid-cnpg-wire.md — Junie brief for
wiring the ekosystem UUID schema to the CNPG cluster (queue item #13)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add KNOE_AUTH_BASE_URL=https://api.prole.org/auth to knoe-auth
deployment; LoginController.java defaulted to api.0.knoe.dev causing
redirect_uri mismatch on every Google login attempt
- Update oauth2-proxy-deployment-prole.yaml to reflect live config:
--provider=oidc --oidc-issuer-url=https://api.prole.org/auth (was
stale --provider=google; live deployment already switched to OIDC)
- Fix redirect URIs in knoe-auth-google-prole-secret.example.yaml:
remove spurious double /auth/auth/ prefix (Kong strip_path=false so
Spring @RequestMapping(/auth) gives /auth/callback/google not
/auth/auth/callback/google)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three-part fix to make browser and curl SPNEGO auth work end-to-end:
1. spnego-proxy: issue WWW-Authenticate: Negotiate challenge when no
Authorization header is present so Chrome (with AuthServerAllowlist)
and curl --negotiate automatically present Kerberos tokens. Previously
the proxy only validated tokens if the client proactively sent them.
Pass-through preserved for non-Negotiate schemes (Basic/token) so
git CLI users with PATs continue to work via Gitea own auth.
2. gitea_spnego_keytab.yml: new Ansible playbook that provisions the
gitea-http AD account (AES-only, msDS-SupportedEncryptionTypes=24),
registers SPN HTTP/git.prole.org, resets the password to derive fresh
AES keys, exports the domain keytab, and rekeys it to principal name
HTTP/git.prole.org@PROLE.ORG that gokrb5 needs for keytab lookup.
Key lesson: samba-tool exportkeytab --principal=HTTP/... returns empty;
must export full domain keytab and rekey in Python.
3. init_gitea.sh: add setup_gitea_spnego() calling the Ansible playbook
in k3s mode as part of the standard deploy flow, with inline notes
on every non-obvious constraint discovered during this work.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ansible_become_ask_pass in group_vars does not reliably trigger an
interactive sudo prompt for ansible_connection=local — Ansible cannot
prompt per-host mid-play. The fix is a wrapper script that passes
-K / --ask-become-pass before any play execution.
- infrastructure/bin/install_workstation.sh: auto-detects FQDN, adds
--ask-become-pass, passes extra args through. Run from repo root:
./infrastructure/bin/install_workstation.sh
make workstation
- Makefile: add 'workstation' target pointing to the script
- group_vars/workstations.yml: drop ansible_become_ask_pass (does not
work), keep ansible_pipelining=false (needed for SSH workstations
with sudo passwords — pipelining replaces stdin, breaking sudo -S)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Personal Macs require a sudo password; the global become_ask_pass=False
and pipelining=True combination silently swallows the prompt and fails.
- ansible_become_ask_pass: true → Ansible prompts for sudo password
- ansible_pipelining: false → sudo can read stdin for the password;
pipelining replaces stdin with a pipe
which sudo treats as non-interactive and
refuses to ask for a password at all.
Applies to [workstations] group (mac + linux). Has no effect on
[workstations_windows] (WinRM auth is separate from become).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both ansible.cfg files lacked vault_password_file, causing
'Attempting to decrypt but no vault secrets found' when running any
playbook that auto-loads the encrypted group_vars/all/ files.
- ansible.cfg (root): vault_password_file = .vault_pass
- infrastructure/ansible.cfg: vault_password_file = ../.vault_pass
.vault_pass is gitignored. Each machine needs its own copy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- KerberosSpnegoService: replace Subject.doAs with Subject.callAs (JDK 18+)
so Subject.current() is set correctly for JGSS credential lookup in JDK 21.
Pass null GSSCredential to createContext so the mechanism auto-selects
rather than failing with "No credential found for 1.3.6.1.5.5.2".
- knoe-auth-deployment: fix PROLE_KERBEROS_KEYTAB_PATH to /etc/knoe-auth/http.keytab
(matches the knoe-auth-http-keytab secret volume mount).
- knoe-kdc-configmap: full rewrite from stale KNOE.DEV to KNOE.LOCAL;
add PROLE.ORG realm (Samba AD at 10.0.0.3) and [capaths] for future
cross-realm trust. Drop arcfour-hmac/des3 from permitted_enctypes —
RC4 hard-removed in JDK 21 and service account now AES-only
(msDS-SupportedEncryptionTypes=24). Fix hardcoded secret values in
entrypoint.sh to use ${PROLE_KDC_MASTER_PASSWORD} variable references.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add @RequestMapping(/auth) to all OIDC and login controllers so paths
match the Kong route (strip_path: false keeps /auth in the forwarded path).
Changes:
- authority/web: add @RequestMapping(/auth) to LoginController,
OidcAuthorizeController, OidcTokenController, OidcUserInfoController
- authority/resources: add V1__oidc_authorization_codes Flyway migration
- application-prole.yml: add OIDC clientId/clientSecret env bindings
- knoe-auth-deployment.yaml: add Flyway baseline env vars and
KNOE_AUTH_OIDC_CLIENT_ID/SECRET from knoe-auth-oidc-client secret
Result: db.prole.org login flow redirects correctly through
https://api.prole.org/auth/login
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Copy pg-knoe-auth/ wholesale from upstream/knoe-db/20260523 (Task 1 of
docs/plans/junie/upstream-knoe-db-20260523-integration.md).
- Extension: PG18 OAUTHBEARER JWT validator using libcurl + OpenSSL RS256.
- knoe-db/Dockerfile: add libcurl4-openssl-dev to dev deps; COPY src/ and
build with make USE_PGXS=1 install after tds_fdw.
- NOT enabled in the default database build (absent from 20_create_extensions.sh).
To enable: CREATE EXTENSION pg_knoe_auth; (requires pg_hba.conf oauth_issuer).
Closes Task 1 of upstream-knoe-db-20260523-integration.md.
Replace all <MagicMock ...> values with empty strings or sensible defaults:
- BooleanVar mocks → false
- StringVar mocks for known fields (cluster_name, db_host_port, db_namespace,
db_username, cluster_env) → correct defaults
- All other StringVar mocks → empty string
Files are now safe to parse and commit. Protected going forward by
_validate_cfg_values() added in ef20c8a.
Add common_core_default_config_path() which resolves the correct knoe.cfg
path (k3d/k3s/gke) from KNOE_MODE/DEPLOYMENT_MODE/CLUSTER_ENV env vars,
falling back to the first cfg file found under conf/. Replaces the
hardcoded knoe.cfg reference in the config-loading comment.
Co-authored-by: Junie <junie@jetbrains.com>
docs/completed/ — new directory; 9 shipped Junie briefs moved from
docs/plans/junie/ (02-k3s-prole-rename, 03-image-rename, 06-patch-garage,
07-init-cnpg-gke-sa-wiring, 13-podmonitor, 15-remove-dead-dashboard,
k3d-knoe-auth-dev-loop, phase2-oidc-gke-deploy, todo-1-cfg-save-path-bug).
docs/completed/README.md summarises all shipped work with dates/refs.
docs/plans/junie/README.md — updated to 2026-05-23; active/pending tables
reflect current state; shipped section now points to docs/completed/.
conf/service/knoe.cfg — new unattended deploy config for the service/k3s
environment (used by ./deploy.sh).
Co-authored-by: Junie <junie@jetbrains.com>
infrastructure/playbooks/pihole_flush_dns.yml — Ansible playbook to
flush Pi-hole DNS cache across inventory hosts.
infrastructure/logs/ — log directory for Ansible run output.
Co-authored-by: Junie <junie@jetbrains.com>
knoe/knoe_conf.py: normalize_environment() now recognises the gke_ prefix
(e.g. gke_plenary-truck-485623-p7_us-west3_knoe-dev-0) as the 'prod'
environment, matching real GKE kubecontext naming.
status.py: add _current_kubecontext() and _cfg_path_from_kubecontext()
so status.py auto-selects the correct cfg file based on the active
kubectl context without manual KNOE_CONF overrides.
Co-authored-by: Junie <junie@jetbrains.com>
Add _validate_cfg_values() to knoe/ui/screens/cfg.py — called before
_filter_cfg_values_for_persistence so MagicMock (or any non-str) widget
values raise TypeError instead of being silently str()-coerced into
conf/<mode>.cfg.
Also re-raise TypeError/ValueError from _save_knoe_cfg so the error
escapes the outer broad except-Exception handler.
New test: tests/installer/test_cfg_save_refuses_mock_values.py
- test_save_knoe_cfg_refuses_non_string_widget_values: MagicMock vars → TypeError
- test_save_knoe_cfg_real_strings_produce_clean_cfg: real _Var stubs → clean cfg
Fixes TODO-1 / tracked in docs/completed/todo-1-cfg-save-path-bug.md.
conf/k3d.cfg and conf/k3s.cfg still contain stale MagicMock values from
before this fix and must be regenerated before committing.
Co-authored-by: Junie <junie@jetbrains.com>
vault_samba_dns_admin_pass was stale — samba-tool commands were failing
with NT_STATUS_LOGON_FAILURE. Updated to match current AD Administrator
credential so the samba_dns role can manage records end-to-end.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All samba-tool dns commands were using -P (machine account Kerberos) which
silently failed with no error — tasks reported changed=0 but records were
never written. Switch to -U Administrator --password={{ samba_dns_admin_pass }}
which uses the vault-protected admin credentials that were already defined
but never wired up.
Also fix regex patterns in record parsers: samba-tool output uses `A: IP`
and `PTR: fqdn` format, not `A IP` / `PTR fqdn` (space-separated), so
updated regex_findall patterns to match `TYPE:\s+value`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- knoe-db.yaml: switch to CNPG-managed TLS cert with serverAltDNSNames
(pg.prole.org + knoe-db-rw cluster service) — removes static serverTLSSecret/serverCASecret
- dns.yml: add pg.prole.org A record to prole_k3s_dns_records (10.0.0.3, 10.0.0.6)
for Ansible-managed split-horizon DNS via Samba AD DC
- k3s.cfg: align KNOE_HOME paths to ~/dev/prole, add PROLE_KDC_* vars, remove
hardcoded KUBECTL_CONTEXT (kubeconfig current-context is authoritative)
- prod.cfg: add PROLE_KDC_STORAGE_CLASS = prole-iscsi
- onepassword.py: skip vault check gracefully when no 1Password session active (non-TTY)
- CLAUDE.md: document production postgres connection string and DNS/CA cert ops
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mermaid 10.7.0 chokes on bare <placeholder> in message text, parsing
it as HTML. Swap to square brackets in the cross-realm §4.3 sequence
diagram and the Gitea SPNEGO §5 diagram:
<svc-host>, <SPNEGO> (§4.3)
<blob> (§5)
Matches the same fix landed in knoe-db's docs/knoe-auth.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Companion to the canonical architecture doc at
~/dev/knoe-db/docs/knoe-auth.md. Documents what's specific to the
prole.org homelab deployment:
- Two upstream identity layers (Samba AD on myrddin for PROLE.ORG
Kerberos, Google Workspace knoey.com for OIDC)
- Three deploy targets (k3s on myrddin/merlin/gandalf, k3d for
engineer dev loop, GKE for production API endpoint)
- Hostname map (myrddin/morgana/zinfandel/db.prole.org/
git.prole.org/api.knoe.dev/db.knoe.dev/etc.)
- Cross-realm Kerberos trust setup (KNOE.LOCAL ↔ PROLE.ORG) via
infrastructure/playbooks/kerberos_trust_setup.yml — RC4 pin
explanation, kvno sync notes
- Gitea SPNEGO at git.prole.org with Mermaid flow diagram
- Supabase Studio status (db.prole.org Google OAuth working;
db.knoe.dev knoe-auth OIDC blocked on pg_oauth image)
- Engineer onboarding checklist (prole-specific steps)
- Known fragility / current incidents table
Mermaid sequence diagram covers the cross-realm flow end-to-end
(workstation → Samba KDC → MIT KDC → service ticket), distinct from
the canonical doc's generic version.
References the matching Junie briefs in knoe-db for the open
fragility items (pg_oauth-image-install, init-user-schemas-heredoc-fix,
deploy-pipeline-restore).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Five fixes Junie surfaced while running the kdc-trust-reset-repeatable
Junie brief end-to-end (companion to commit 6f99f95). All hit during
the unattended `install.sh --mode k3s --reset` pipeline.
- knoe/core/milestones.py (KerberosMilestone):
For k3s and k3d modes, deploy the KDC pod via `init_kdc.sh start`
before running init_kerberos.sh. init_kerberos.sh only chains into
init_kdc.sh when PROLE_KDC_STANDALONE=1; without this hook the
cluster came up with no KDC pod and the cross-realm trust principals
had nowhere to land.
- knoe/milestone.py (Milestone._get_script_env):
Clear KUBECTL_CONTEXT in addition to KUBECONTEXT so stale entries
from a different machine's cfg don't override the kubeconfig's
own current-context.
- etc/knoe_cfg.sh (_knoe_read_cfg):
Skip KUBECTL_CONTEXT / KUBE_CONTEXT_NAME / KUBECONTEXT entries when
reading cfg in k3s mode. Same theme: kubeconfig current-context is
authoritative.
- etc/init_1password.sh + knoe/core/onepassword.py:
When running non-interactively (no TTY on stdin) and no `op`
session exists, skip rather than hang on `op signin`. Lets the
unattended pipeline proceed for k3s/k3d where in-cluster secrets
are managed separately from 1Password.
Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
