The ansible excision in 5a769f2 was correct in scope -- no more host
inventory, no playbooks, no roles -- except it took out the one file
update.sh depends on for master DB password rotation:
infrastructure/inventory/group_vars/all/vault_db_master.yml
update.sh(1) reads this file (ansible-vault encrypted) at the start of
every run to decrypt the master password used to rotate k8s DB secrets
(knoe-db-user/superuser/knoe), ALTER ROLE on CNPG primary for the
prole/postgres/knoe/authenticator users, and reset Grafana admin creds.
With the vault file gone, update.sh failed with
ERROR: Vault file not found: .../vault_db_master.yml
and took down the tail of deploy.sh.
Restored from the parent commit of 5a769f2 verbatim -- content is
already ansible-vault AES256 encrypted, so it's safe to carry in a
private repo. The `.vault_pass` file that decrypts it remains
gitignored (never committed). Broader `infrastructure/` stays gone.
TODO (deferred): migrate this one remaining ansible-vault secret to
either OpenBao or a k8s Secret so knoe-db doesn't retain a residual
ansible-vault dependency surface. Tracked in our pending work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
knoe-db deploys cluster-native (mode k3d for local dev, k8s for the
shared company cluster, k3s still supported for individual homelab
collaborators via conf/k3s.cfg). The ansible-managed distributed cluster
of prole.org machines is no longer the target topology, and the Ansible
inventory in this tree targets only the original author's specific hosts
(merlin/gandalf/myrddin/pi/raspberry/retropie.prole.org). It is neither
portable nor useful for the new deployment shape.
Removed:
- infrastructure/inventory/ (hosts.ini + 7 *.prole.org host_vars)
- infrastructure/playbooks/ (11 playbooks, all hardcoded to *.prole.org)
- infrastructure/roles/ (20 roles: pihole, samba AD DC, mariadb,
netplan, iscsi, prole_ssl, etc. -- all
tied to the bare-metal prole.org topology)
- infrastructure/setup/ (cr_ansible_user.sh, cr_samba_family_users.sh,
filesystem.txt -- prole.org bootstrap)
- infrastructure/deployments/svc-check-helm/
(legacy; etc/init_kong.sh already tears
down svc-check resources on every run)
- infrastructure/ansible.cfg (references deleted inventory)
- ansible.cfg, ansible.sh, ansible_min.cfg, ansible_recovery.log (root;
orphaned without inventory)
- vault_backup/ (ansible-vault encrypted prole.org secrets
backup: ad_dc_vault.yml, all_prole_vault.yml,
iscsi_vault.yml)
- tools/ (dashboard.sh, ghcr_docker_login.sh,
k3s-cluster-rename.sh, manage-node.sh,
export_mariadb_synology.sh,
prepare_mariadb_usb.sh, and a dumped
synology.prole.org mariadb .sql file --
all prole.org-era deployment helpers)
Kept: scripts/ (cluster ops), bin/ (prole-env.sh, prole-kpf.sh dev
shims), conf/k3s.cfg / conf/k3d.cfg (still supported modes; any
*.prole.org defaults in those files are legitimate mode fallbacks for
collaborators who keep running their own k3s/k3d setups). Test
fixtures referencing *.prole.org hostnames are left as-is (regression
data).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summarize CLAUDE.md updates:
- Document both live GKE clusters as e2-standard-2 x3 in us-west3 and note stale app default in code.
- Call out DB cluster naming mismatch between conf (knoe-dev-cnpg-0) and code default (knoe-cnpg-0).
- Add SSD quota guidance: 300GB fully consumed by CNPG, so non-CNPG PVCs must use standard/pd-standard.
- Refresh reset script reference and quota section details to match current runtime state.
Co-authored-by: Junie <junie@jetbrains.com>
- fix conf/prod/prole.cfg by replacing hardcoded /Users/chrisfu paths with /Users/chrisfu
- remove injected [update.sh] log lines and stray password artifacts so config is executable
- include latest pending updates across deployment config, UI/core flow, vault/network artifacts, and helper scripts
## GCP / Cluster Environment Screen
- Auto-populate Cloud tab from conf/prod/gcp.cfg on screen open (org_id,
billing_account, billing_project, project_id)
- gcloud auth validity checked on screen startup; friendly modal dialog
streams gcloud auth login output live so user never leaves the app
- Live GKE cluster browser: fetches clusters via gcloud container clusters
list, displays with checkmark selector, auto-selects saved cluster
- Selecting a cluster runs get-credentials, sets KUBECONFIG/KUBECONTEXT,
and syncs the region dropdown to the selected cluster's location
- Region dropdown populated live from gcloud compute regions list with
checkmark on currently selected region; graceful fallback when offline
- New 'GCP Storage' tab with workload->StorageClass mapping (CNPG->premium-rwo,
Redis/Monitoring->standard-rwo, Garage->garage-hdd) and Fetch from Cluster
- Provider readonly field styled correctly (no solid-black on macOS)
- Stale prole.cfg/conf/prole.cfg symlinks removed; all config I/O now
resolves env-specific paths via prole_conf.entrypoint_path()
## GKE Autopilot Compatibility (Common Services)
- Synology iSCSI StorageClass and static PVs guarded behind PROLE_MODE!=k8s
in init_openbao.sh (GKE Autopilot forbids hostPath/iSCSI volumes)
- In-cluster Docker registry (hostPath) skipped in k8s mode; GCP Artifact
Registry used instead
- Kong renamed knoe-svc-kong in k8s mode; all health-check kubectl calls in
init_common_services.sh and status_common_services.sh updated accordingly
- DNS endpoints switched from *.prole.org to *.knoe.dev in k8s mode
(api.knoe.dev, git.knoe.dev, svc.knoe.dev); ingress uses gce class
- New GKE-clean Kong manifests under deploy/opentofu/k8s/manifests/prole/:
no k3s node affinity, explicit Autopilot resource requests/limits
## Garage S3 Store (GKE)
- New garage-statefulset-gcp.yaml targeting garage-hdd StorageClass
(pd-standard, avoids SSD_TOTAL_GB quota exhaustion in us-west3)
- New storageclass-gcp-hdd.yaml (pd-standard, Retain, WaitForFirstConsumer)
- GCP StorageClass manifests skipped on re-runs (Autopilot built-ins are
immutable; skip-if-exists guard added)
- PVC deletion guard extended to cover any storageClass (not just synology)
so stale claims are cleaned before StatefulSet recreation
## Topology (GKE Autopilot)
- DaemonSet collector skipped in prod mode (forbidden in kube-system by
GKE Warden); Kubernetes-only node facts path used instead
- All ready GKE nodes assumed cnpg-eligible and monitoring-eligible without
taint/synology-mount checks (skip_collector + assume_nodes_eligible flags)
## KUBECONFIG / kubectl (k8s mode)
- actions.py: new elif mode==k8s branch sets KUBECONFIG=~/.kube/config
and injects KUBECONTEXT from prole_cfg_data into script env
- _build_kubectl_cmd falls back to Global.KUBECONTEXT when
init_cluster.selected_kubectx is empty
- _activate_selected_gke_cluster persists KUBECONFIG/KUBECONTEXT to
prole_cfg_data and saves prole.cfg immediately after get-credentials
## Database Build Screen (GKE)
- Registry display shows correct Artifact Registry URL
(<region>-docker.pkg.dev/<project>/<namespace>/knoe-db) in green
- Build+push: gcloud auth configure-docker, auto-creates AR repository
named after SERVICE_NAMESPACE (e.g. knoe-system) if missing, then
docker tag + push; falls back to gcr.io if region unavailable
- GCP config loaded from conf/prod/gcp.cfg on every screen entry;
keys normalised to lowercase so project_id lookup is always consistent
## Config / Namespace persistence
- prole_conf.py activate_environment: symlink creation removed; sets
CLUSTER_ENV env-var so all subsequent calls resolve correct env directory
- knoe/ui/screens/__init__.py: startup config load uses entrypoint_path()
instead of hardcoded conf/prole.cfg; seeds SERVICE_NAMESPACE=knoe-system
for managed envs so Common Services never defaults to 'default'
- cfg.py _save_prole_cfg: saves to env-specific path via entrypoint_path()
- etc/prole_cfg.sh: removed all ln -snf symlink creation
Co-authored-by: Junie <junie@jetbrains.com>
- add tools/manage-node.sh for k3s node add/remove automation with inventory reconciliation
- include kubectl drain flag compatibility and --ssh-host targeting for uninstall workflows
- persist installer cfg/env values and update deploy/database/cfg UI handling
- refresh related inventory/network artifacts and expand installer/database test coverage
Co-authored-by: Junie <junie@jetbrains.com>
- Introduced `knoe.core.prod_config` with production configuration schema and helper methods.
- Added validation logic to enforce required fields and expected formats.
- Implemented in-process API (`ProdConfigApi`) for managing production configs.
- Added test coverage for default config behavior, validation, YAML generation, and API workflow.
- Updated cluster UI layout and tests to integrate production config with new navigation flows.
- Added host inventory and updated service flow to reflect production setup changes.
STABLE BUILD — all 35+ pods Running, zero non-Running pods, CNPG ready=3.
## Architecture changes (Supabase slim-down)
- Removed local Supabase Kong, Storage, MinIO, Imgproxy from deployed stack
- Rewired API routing to shared Kong in kube-system
- Rewired object storage to shared Garage in knoe-system
- Supabase footprint reduced to control-plane only: auth, rest, meta,
analytics, realtime (studio port-forward retained)
- Supabase pods renamed: stripped redundant '-prole-supabase-' prefix via
per-component fullnameOverride (e.g. supabase-auth, supabase-rest, ...)
- Supabase workloads pinned to merlin.prole.org (memory-safe node)
## File changes
- conf/service/prole.cfg
* Replaced all hardcoded /Users/chrisfu/dev/prole paths with ${HOME}/dev/prole
so config is portable when run from myrddin.prole.org or any user home
* SUPABASE_ENABLED=True, SUPABASE_PRIMARY_NODE=merlin.prole.org in
both [Global] and [Optional Features]
* supabase_enabled=true in [Inputs]/init_cluster section
- supabase/helm/render_supabase.py
* Explicitly disabled: kong, storage, minio, imgproxy
* Active components pinned via nodeSelector to merlin.prole.org
* fullnameOverride per component to clean up pod names
* Default primary node changed from pi.prole.org -> merlin.prole.org
- supabase/deploy.sh
* Removed svc/kong port-forward; API via shared Kong in kube-system
* Removed imgproxy/storage from PV/PVC prep and helm --set persistence args
* Fixed kubectl exec missing -i flag for psql heredoc stdin in
setup_knoe_db_for_supabase
* Updated Studio access messaging to reference shared platform Kong
- k8s/prole/knoe-db.yaml
* instances: 3
* Added control-plane/master tolerations for myrddin scheduling
- deploy/opentofu/k3s/manifests/prole/knoe-db.yaml
* instances: 3, added control-plane tolerations (mirrors k8s/prole/)
- k8s/prole/iscsi-pvs.yaml
* Expanded d003-data/d003-wal nodeAffinity to [pi, merlin, myrddin]
to reflect post-OOM-recovery live state
- k8s/registry/deployment.yaml
* nodeSelector restored to myrddin.prole.org (hostPort 5000 owner)
* Added control-plane + master tolerations so pod schedules correctly
- modes/k3s/knoe-db/.version
* Aligned to 140 (image tag 18-140) matching repo-root version
- etc/init_common_services.sh, etc/init_registry.sh
* Minor fixes aligned with slimmed service layout and node targeting
- infrastructure/inventory/host_vars/pi.prole.org.yml
* Updated host vars to reflect post-recovery pi state
- knoe/core/topology.py, knoe/ui/screens/cluster.py,
knoe/ui/screens/cluster_nodes.py
* Topology and UI updates for 3-node cluster representation
- scan/network_description.txt
* Refreshed network scan output post-recovery
- conf/port-mapping.cfg
* Updated port mapping to reflect removal of local Kong/storage forwards
- scripts/cleanup_stale_rs.sh (new)
* Utility script to prune orphaned ReplicaSets after rollouts
## Verified stable state
- CNPG: Cluster in healthy state, ready=3
knoe-db-2 (merlin, primary), knoe-db-3 (pi), knoe-db-5 (myrddin)
- Supabase: 5 pods Running on merlin.prole.org
supabase-auth, supabase-rest, supabase-meta,
supabase-analytics, supabase-realtime
- Monitoring: all prometheus/grafana/alertmanager pods Running on merlin
- Registry: Running on myrddin.prole.org (hostPort 5000)
- Zero non-Running / non-Completed pods across all namespaces
Co-authored-by: Junie <junie@jetbrains.com>
- Rename host mount namespace from /prole/d00x to /synology/d00x
- Make LUN ownership explicit per host in inventory (myrddin=d001; merlin=d002,d004; pi=d003)
- Add safe migration cleanup for legacy /prole/d00x mounts and stale /etc/fstab entries
- Update k3s/ArgoCD/OpenTofu manifests and PV node affinities to match new mount paths
- Improve iSCSI role check-mode behavior and add quiesce/detach flow in ansible.sh
- Pin Prometheus/Alertmanager + Grafana to merlin.prole.org in k3s to avoid PV node-affinity issues
- Taint myrddin.prole.org as control-plane-only and add CNPG toleration
- Prevent init_cloudnative_pg from deploying prole-svc-kong into the DB namespace
- GitOps-manage Kong declarative config via ConfigMap + add placement/init tests
Prefer infra-managed manifests for bootstrap; reconcile CNPG instances based on Ready+schedulable labeled db nodes; update manifests to use node-role affinity + anti-affinity; add policy/tests and config touch-ups (incl. prole.cfg).
Update inventory/roles to keep K3s state on local storage and prevent iSCSI from managing /var/lib/rancher.
Also add a single-host k3s install playbook, a systemd override template, and docs describing the installer ↔ Ansible boundary and slow-storage knobs.
Co-authored-by: Junie <junie@jetbrains.com>
- k3s_sync: choose any server as sync source fallback; refresh kubeconfig and validate nodes\n- fetch_kubeconfig: save to prole-k3s.kubeconfig and etc/secrets/k3s.kubeconfig\n- k3s sync: sync server URL into config.yaml; restore TLS bundle only on servers\n- cert-manager/ACME: wait for CNI/webhook readiness; retry ClusterIssuer/Certificate apply\n- inventory: tune myrddin control-plane timing args; add render test\n- misc: update port mappings and prole-db manifests/version
Co-authored-by: Junie <junie@jetbrains.com>
- Ensure dashboard service doesn't create an ordering cycle with `multi-user.target`.
- Verify service safely attaches to a virtual terminal (VT) and passes VT to Xorg.
- Ensure no `su` usage by default in the service.
- Validate script behavior for passing VT and `-keeptty` options to Xorg.
chore(dashboard): update systemd unit to avoid VT conflicts and improve logging
- Remove `After` and `Wants` directives for `multi-user.target` to prevent conflicts.
- Add `TTYPath`, `StandardInput`, and related options for VT attachment.
- Ensure `ExecStopPost` restores getty service for the dashboard VT.
fix(dashboard): improve Xorg resolution and clean up script logic
- Dynamically determine Xorg executable location in `dashboard.sh`.
- Pass VT, display, and `-keeptty` to Xorg to improve script reliability.
fix(mariadb): correct exFAT mount options for external storage
- Avoid `chown` failures by setting root ownership with group access for `mysql`.
- Update external mount options to `uid=0,gid=mysql,umask=0002`.
cleanup(mariadb): remove unnecessary owner/group permissions from migration marker task
- Enable mounting borrowed USB disk (exFAT) persistently at `/external`.
- Migrate MariaDB datadir to `/external/mariadb` with bind mount at `/srv/mariadb/mariadb`.
- Add migration marker and rollback support to prevent accidental remigration.
- Include Ansible tasks for external storage setup and migration validation.
- Add ssl_hosts group and prole_ssl role to deploy registry cert/key to /etc/ssl/certs/prole
- Configure k8s registry deployment to serve HTTPS using mounted host certs
- Switch k3s/containerd registry mirror config to https:// and prefer HTTPS checks with HTTP fallback
Co-authored-by: Junie <junie@jetbrains.com>
- Added tasks and playbooks for K3s datastore export/import using MariaDB Tools role, with associated tests and defaults.
- Enhanced iSCSI role to support mkfs-once logic and safer re-initialization of block storage.
- Migrated iSCSI-backed Rancher data from retropie to merlin.prole.org.
- Updated k3s roles/playbooks to relax Rancher storage preflight checks, supporting PARTUUID-based mounts.
- Adjusted Samba AD NetBIOS name derivation to use uppercase short hostname by default.
- Incremented prole DB version to 104, updated generated prole.cfg, inventory, and recovery manifest templates.
ArgoCD IO load distribution: Updated k8s/argocd/install.yaml to use hostPath volumes for application-controller (/prole/d001), redis (/prole/d002), and repo-server (/prole/d003). Updated Ansible configure task to create hostPath directories on myrddin.prole.org.
K3s configuration and testing: Added test playbooks for kubeconfig rewriting and argument validation. Updated K3s install, configure, and image import tasks. Configured required mounts and host-specific variables for myrddin.prole.org.
Miscellaneous: Updated conf/prole.cfg with K3S_TOKEN. Bumped prole-db version and updated recovery manifest templates. Minor UI adjustments in cluster screen.
Co-authored-by: Junie <junie@jetbrains.com>
- Fixed YAML syntax and Python version parsing in CNPG operator tasks.
- Added automated detection of active network interface (eth0 vs wlan0) for Retropie and Pi nodes to ensure correct K3s configuration.
- Implemented persistent node taints for retropie.prole.org to exclude CloudNative-PG workloads.
- Added a rollout restart mechanism for cnpg-controller-manager to recover from scheduling on tainted or unstable nodes.
- Removed redundant kubectl-cnpg dependency from the Python installer as it is now managed by Ansible.
- Improved prole.sh to force color output for Ansible playbooks.
- Updated prole.cfg with active K3s token and CNPG version information.
- Implement proactive CA hash verification and automated repair for mismatches
- Ensure agents prioritize discovered server tokens over stale vault values
- Fix K3s service hangs with explicit stop and killall before reinstall
- Add Retropie/Pi networking fixes (WiFi power save, wlan0 priority)
- Pin pre-staged images to stable, architecture-aware versions (arm64)
- Remove obsolete init-port-forwards and prole.cfg sync tasks
- Update k8s manifests and installer core logic with new tests
Repairs and improvements:
- iSCSI: Added cleanup tasks to remove stale mounts and fstab entries. Improved robustness of iSCSI target management and added 'iscsi_absent_mounts' support.
- K3s:
- Updated service start logic to accept 'activating' state, preventing premature failure during slow startups.
- Improved service stop logic to safely handle missing or not-found services.
- Ensured 'prole-installer' ServiceAccount and ClusterRoleBinding exist for K8s administration.
- Added leader election and etcd tuning arguments (forgiving leases) to config.yaml.j2.
- Removed deprecated 'prole-port-forwards' systemd service.
- Installer & Scripts:
- Updated legacy_tk.py to support K3s mode, secret resolution for passwords, and better environment management (including ~/.prole/env.sh for service mode).
- Updated init_ansible.sh to support PROLE_VAULT_PASS_FILE and ANSIBLE_VAULT_PASSWORD_FILE.
- Improved directory and kubeconfig path resolution in prole_cfg.sh to support fallback to ~/.prole.
- Enhanced Grafana password resolution in init_monitoring.sh.
- Added automatic application of iSCSI StorageClass and PersistentVolumes in init_openbao.sh.
- General: Switched conf/prole.cfg to k3s deployment mode and updated vault_k3s.yml token.
New Ansible Tasks and Playbooks:
- infrastructure/playbooks/iscsi_cleanup.yml: Automates logout and removal of stale iSCSI node records.
- infrastructure/playbooks/prole_logs_migrate.yml: Orchestrates /prole/logs migration to iSCSI storage.
- infrastructure/playbooks/tmp_bao_dir.yml: Ensures host-level storage directories for OpenBao.
- infrastructure/playbooks/tmp_mount.yml: Utility to verify and enforce host-level mounts.
- infrastructure/playbooks/k3s_sync.yml: Added tasks to start K3s after sync and update local kubeconfig on the controller.
- Added 'Unmount stale iSCSI mounts' and 'Remove stale iSCSI fstab entries' to the iscsi role.
- Added 'Ensure prole-installer service account exists' to the k3s role.
- install.py: Major update including configuration variable expansion, improved k3s/k3d handling, and enhanced installation logic.
- etc/ scripts: Significant refactoring of initialization scripts (Kerberos, Port Forwards, Garage Store, etc.).
- Port Forwards: Transitioned from XML to port-mappings.conf for managing kubectl port-forwards.
- Status Reporting: Improved status checking for common services.
- Infrastructure: Updated Ansible inventory and rsyslog role configurations.
- Tests: Added a comprehensive suite of tests for 'etc' initialization scripts in prole/tests/etc/.
- Documentation: Added prole-db-documentation-mcp-architecture.md.
- General: Updated Dockerfiles and various helper scripts.
- Installer: Updated k3s deployment logic and configuration generation.
- k3s Role: Implemented token drift protection to verify Vault secrets against live node tokens.
- DNS: Enhanced samba_reverse_dns role to support multiple reverse zones.
- Service Init: Updated initialization scripts and status reporting.
- Infrastructure: Added prole management role and k3s diagnostic playbook.
- Configuration: Updated prole.cfg and added vaulted group variables.
- Introduced cgroup kernel parameter management for `k3s_hosts` via Ansible playbook.
- Added Kubernetes PersistentVolumes for iSCSI storage with specific node affinity.
- Defined `prole-iscsi` StorageClass for volume provisioning.
- Added `nodeSelector` for multiple Kubernetes resources to ensure scheduling on `myrddin.prole.org`.
- Modified `storage` requests and set `storageClassName` in `garage-statefulset.yaml`.
- Enhanced `install.py` for dynamic environment configuration and kubeconfig handling.
- Improved cgroup management tasks in Ansible with conflict resolution and parameter updates.
- Simplified vault token update process in playbooks and updated encryption checks.
