- Added tasks and playbooks for K3s datastore export/import using MariaDB Tools role, with associated tests and defaults.
- Enhanced iSCSI role to support mkfs-once logic and safer re-initialization of block storage.
- Migrated iSCSI-backed Rancher data from retropie to merlin.prole.org.
- Updated k3s roles/playbooks to relax Rancher storage preflight checks, supporting PARTUUID-based mounts.
- Adjusted Samba AD NetBIOS name derivation to use uppercase short hostname by default.
- Incremented prole DB version to 104, updated generated prole.cfg, inventory, and recovery manifest templates.
- Detect/remove legacy prole-svc-kong resources (deploy/svc/configmap) in default during repair
- Ensure canonical Kong exists in configured SERVICE_NAMESPACE before cleanup
- Remove hardcoded knoe-system defaults; require non-default SERVICE_NAMESPACE in k3s
- Avoid silent installer hangs under CI/pytest by auto-generating DB master password
- Update tests for namespace/repair behavior and DB version defaults
Co-authored-by: Junie <junie@jetbrains.com>
ArgoCD IO load distribution: Updated k8s/argocd/install.yaml to use hostPath volumes for application-controller (/prole/d001), redis (/prole/d002), and repo-server (/prole/d003). Updated Ansible configure task to create hostPath directories on myrddin.prole.org.
K3s configuration and testing: Added test playbooks for kubeconfig rewriting and argument validation. Updated K3s install, configure, and image import tasks. Configured required mounts and host-specific variables for myrddin.prole.org.
Miscellaneous: Updated conf/prole.cfg with K3S_TOKEN. Bumped prole-db version and updated recovery manifest templates. Minor UI adjustments in cluster screen.
Co-authored-by: Junie <junie@jetbrains.com>
* CNPG: Implement version pinning, automatic manifest URL derivation, and optional operator upgrades.
* CNPG: Add installation and management of the kubectl-cnpg plugin with architecture detection.
* k3s: Update etc/init_cloudnative_pg.sh to support configurable CNPG operator versions.
* Ansible: Enhance cgroups and k3s roles with improved check mode support and more robust variable defaults.
* Ansible: Improve kernel command-line path selection logic in the cgroups role.
Co-authored-by: Junie <junie@jetbrains.com>
We can now complete a full K3s deployment run and deliver a working CloudNativePG (CNPG) stack from the standalone installer.\n\nHighlights:\n- Installer scripts hardened for CNPG deploy + recovery templating + rollout control\n- OpenTofu/K3s manifests aligned with in-repo k8s templates for prole-db\n- Supporting init flows updated (OpenBao, monitoring, Kerberos, OpenTofu)
Co-authored-by: Junie <junie@jetbrains.com>
- Removed all logic that read or set NAMESPACE/PROLE_NAMESPACE from the shell environment or wrote it to env.sh.
- prole.cfg is now the sole source of truth for the namespace value, loaded exclusively by prole_cfg.sh.
- etc/prole_cfg.sh & mock_val/prole_cfg.sh: Removed kubectl-context fallback and default for PROLE_NAMESPACE.
- Shell scripts (init_*.sh): Replaced NAMESPACE=${NAMESPACE:-...} with NAMESPACE="${PROLE_NAMESPACE}".
- Python (actions, environment, milestone): Removed env["PROLE_NAMESPACE"] from subprocess env dicts and env.sh.
- etc/init_cloudnative_pg.sh: Removed DB_PASSWORD env fallback for secret creation.
Co-authored-by: Junie <junie@jetbrains.com>
- Renamed etc/init_prole-db-backup.sh to etc/init_cnpg_backup.sh and updated all references.
- Enhanced CNPG backup script with authoritative status validation.
- Implemented SupabaseImagePreloadMilestone to front-load heavy image transfers.
- Updated supabase/deploy.sh with --prefetch-images-only and --skip-prefetch.
- Updated unit tests for CNPG backups and Supabase silent milestone sequencing.
- Included incidental environment updates from installer execution.
Co-authored-by: Junie <junie@jetbrains.com>
Summary:
Removed the prole-db-manager microservice and simplified deployment to use
prole-authority as the internal management and authorization point. Fixed two
blocking bugs that prevented silent install from completing on knoe-dev-cluster.
Removed: prole-db-manager
- Deleted db-manager-deployment.yaml and db-manager-service.yaml from opentofu manifests
- Deleted src/db-manager/ (Dockerfile, server.js, package.json, tests)
- Removed prole-db-manager port-forward mapping from installer/core/env.py
- Removed init_db_manager.sh from Initialization Scripts (milestones.py, actions.py)
- Removed init_certmgr.sh and init_db_manager.sh tabs from services screen (services.py)
- Removed live k8s Deployment/Service from knoe-dev-cluster
Fixed: PostgreSQL version downgrade error (pg17 -> pg18)
- Created conf/postgresql/.version with value 18
- Updated k8s/prole/prole-db.yaml and prole-db-recovery.yaml.tpl imageName to prole-db:18-089
- Fixed _init_database_options_state() to restore saved version_type from prole.cfg
so db_version_type defaults to v18 (pg18) instead of silently reverting to pg17
- Added database_options.* keys to _collect_input_snapshot() in cfg.py so
distribution, version_type, and all extension toggles persist to prole.cfg
Fixed: Cluster name inconsistency
- Removed stale prole-dev-cluster references; all scripts now use knoe-dev-cluster
- Added knoe-dev-cluster to mode-detection case in etc/prole_cfg.sh
Config: conf/prole.cfg
- Set kerberos_config.enabled = False, KERBEROS_AUTO_ENABLED = False
- Added database_options.distribution = percona, version_type = v18
- Added all 13 extension flags set to True (postgis, pgvector, pgcrypto, pgaudit,
pg_repack, pg_stat_statements, pg_buffercache, pg_freespacemap, pgrowlocks,
postgres_fdw, dblink, pg_stat_monitor, pgbadger)
Verification:
./install.py -s -l -v -c conf/prole.cfg completed successfully.
CNPG deployed prole-db:18-089 to knoe-dev-cluster; all milestones passed.
Co-authored-by: Junie <junie@jetbrains.com>
- Resolve namespace drift in Kerberos test flow and OpenBao deployment.
- Fix OpenAPI credential-validation failures in etc scripts by using --validate=false with kubectl.
- Introduce preferred_service_namespace() to accurately target namespaces for KDC policy/test paths.
- Default Kerberos test user to 'administrator' and prefill from Ansible vault secrets.
- Generalized OpenBao deployment logic to correctly handle multi-namespace environments.
- Added regression test 'tests/etc/test_init_kerberos_k3s_real_kdc.sh'.
- Synchronized SERVICE_NAMESPACE extraction in prole_cfg.sh.
Co-authored-by: Junie <junie@jetbrains.com>
- Dockerfile: Resolved interactive tzdata prompts with debconf and fixed Percona package names (percona-pg-stat-monitor18 and percona-postgresql-contrib).
- Configuration: Updated prole.cfg with specific namespace (prole-hq0-db0), user (chrisfu), and encrypted secrets.
- Installer: Enhanced initialization scripts (init_openbao.sh, init_certmgr.sh) and updated service layers for cluster deployment.
- Tests: Added coverage for dependent image collection in installer core.
- Version: Bumped prole-db version to 77.
Co-authored-by: Junie <junie@jetbrains.com>
Align Kong k3s manifests with etc/init_kong.sh and svc-check Helm values (prole-svc-kong + prole-svc-kong-config) so rollout waits for the correct deployment.
- Tighten Cluster Environment screen layout; switch Service/Prod to kubectx context selection; keep namespace and key controls on one line; ensure Repair button remains reachable.
- Add UI layout regression test to render with large mock data and assert key widgets remain visible and console is scrollable.
- Make kubeconfig generation deterministic under tests by avoiding overwriting cert-based kubeconfigs; write token sidecar kubeconfig when needed.
- Update common-services init scripts and add k3s/Helm deployment bits (svc-check, Kong/CertMgr tasks).
- Refactor k3s/kube_config.sh to fetch remote kubeconfig and configure local context.
- Add Ansible task svc_check.yml to deploy a status site via Kong and cert-manager.
- Include svc_check.yml in k3s role's main tasks.
- Deleted PostgREST Kubernetes deployment and service manifests.
- Removed `init_postgrest.sh` initialization script and PostgREST references across UI, installer, and config.
- Updated Kong API Gateway to exclude `/rest/*` routing to PostgREST.
- Simplified `supabase/deploy.sh` by removing references to PostgREST scaling and alias creation.
- Adjusted cluster configuration and UI to reflect the absence of PostgREST.
- Include new `prole.spec` for build configurations and dependencies.
- Add Terraform state handling for OpenTofu in `k3s` cluster.
- Provision multiple Kubernetes resources in `prole-db` namespace: namespace, services, ConfigMaps, StatefulSets, Ingress rules, and PersistentVolumes.
- Integrate deployment and configuration enhancements for `garage`, `prole`, and related components.
Kong API Gateway (replacing prole nginx):
- Add etc/init_kong.sh provisioning script (DB-less mode, prole-db namespace)
- Add kong-deployment.yaml and kong-service.yaml manifests
- Rewire ingress rules (svc/git/api.prole.org) to prole-db-kong:8000
- Update kustomization.yaml to reference kong manifests
PostgREST & DB Manager in prole-db namespace:
- Add etc/init_postgrest.sh and etc/init_db_manager.sh scripts
- Add postgrest/db-manager deployment and service manifests
- Add src/db-manager/ Node.js REST endpoint for backup triggers
- Default NAMESPACE changed to prole-db in both scripts
Docker image pre-load from PROLE_DATA/docker-import:
- Add _preload_docker_images() to init_common_services.sh
- Scan for .tar files exported by final_deployment.sh
- Import via k3d image import (k3d) or ctr (k3s) before deployments
- Increase rollout timeouts to 300s (configurable via ROLLOUT_TIMEOUT) in init_openbao.sh, init_opentofu.sh, init_garage_store.sh, init_registry.sh
OpenTofu password resolution fix:
- Add Kubernetes secret fallback in resolve_admin_password()
- Change hard exit 1 to graceful return 1 with warning
- Wrap call in if-guard so set -e doesn't abort the script chain
Milestone fix (init scripts not running):
- Add init_kong.sh, init_postgrest.sh, init_db_manager.sh to InitializationScriptsMilestone.execute() script list and arg branches
- Previously only actions.py had these; milestones.py was missing them
Installer integration:
- Add Kong/PostgREST/DB Manager to silent installer _step_init_scripts
- Add corresponding tabs and execution blocks in UI services.py
Separation of concerns: merge silent/UI actions & modularize screens.
ProleInstallerBase (actions.py): Created shared base class with 37 deduplicated methods previously duplicated between ProleSilentInstaller and ProleInstaller. Namespace, environment, secret, deployment, port-forward, authority/repair, image, and logging helpers now defined once. Subclasses override _get_input() to bridge their data-access layers.
screens.py -> screens/ package (18 mixin modules): Split 10,234-line monolithic screens.py into focused mixin modules: base, navigation, welcome, dependencies, network, environment, database, cluster, services, security, ollama, supabase, docker, build, packaging, deploy, validate, cfg. __init__.py composes ProleInstaller from all mixins and re-exports has_display(), main() for full backward compatibility.
All 37 tests pass with no regressions.
- Monitoring: Migrated from manual Grafana/Prometheus manifests to kube-prometheus-stack based setup in etc/init_monitoring.sh. Removed old manifest files from deploy/ and k8s/.
- Installer Core: Refactored installer with new modules for actions, environment handling, and UI screens. Enhanced Milestone logic to support advanced configuration (ArgoCD, Registry namespaces, Kerberos flags, etc.).
- Service & Init Scripts: Updated multiple initialization scripts (init_*.sh) for better integration with OpenBao, Kerberos, and the new monitoring stack. Added new scripts for Nginx Ingress, Ollama parsing, and K3D route fixes.
- Infrastructure: Enhanced Samba AD DC Ansible role with realm derivation, provisioning guidance, and group management. Updated K3s role tasks.
- Configuration: Refined default settings in conf/ to align with the new deployment architecture.
- App & Tools: Updated prole-app Swift code and prole.sh for improved environment variable handling and installation flow.
- Fixed YAML syntax and Python version parsing in CNPG operator tasks.
- Added automated detection of active network interface (eth0 vs wlan0) for Retropie and Pi nodes to ensure correct K3s configuration.
- Implemented persistent node taints for retropie.prole.org to exclude CloudNative-PG workloads.
- Added a rollout restart mechanism for cnpg-controller-manager to recover from scheduling on tainted or unstable nodes.
- Removed redundant kubectl-cnpg dependency from the Python installer as it is now managed by Ansible.
- Improved prole.sh to force color output for Ansible playbooks.
- Updated prole.cfg with active K3s token and CNPG version information.
- Prevent connmand from flapping routes by ignoring veth* and cni0 interfaces
- Ensure wlan0 maintains routing priority over eth0 on Retropie and Pi nodes
- Increase image pull retries and delay to improve resilience against transient network instability
- Implement proactive CA hash verification and automated repair for mismatches
- Ensure agents prioritize discovered server tokens over stale vault values
- Fix K3s service hangs with explicit stop and killall before reinstall
- Add Retropie/Pi networking fixes (WiFi power save, wlan0 priority)
- Pin pre-staged images to stable, architecture-aware versions (arm64)
- Remove obsolete init-port-forwards and prole.cfg sync tasks
- Update k8s manifests and installer core logic with new tests
Repairs and improvements:
- iSCSI: Added cleanup tasks to remove stale mounts and fstab entries. Improved robustness of iSCSI target management and added 'iscsi_absent_mounts' support.
- K3s:
- Updated service start logic to accept 'activating' state, preventing premature failure during slow startups.
- Improved service stop logic to safely handle missing or not-found services.
- Ensured 'prole-installer' ServiceAccount and ClusterRoleBinding exist for K8s administration.
- Added leader election and etcd tuning arguments (forgiving leases) to config.yaml.j2.
- Removed deprecated 'prole-port-forwards' systemd service.
- Installer & Scripts:
- Updated legacy_tk.py to support K3s mode, secret resolution for passwords, and better environment management (including ~/.prole/env.sh for service mode).
- Updated init_ansible.sh to support PROLE_VAULT_PASS_FILE and ANSIBLE_VAULT_PASSWORD_FILE.
- Improved directory and kubeconfig path resolution in prole_cfg.sh to support fallback to ~/.prole.
- Enhanced Grafana password resolution in init_monitoring.sh.
- Added automatic application of iSCSI StorageClass and PersistentVolumes in init_openbao.sh.
- General: Switched conf/prole.cfg to k3s deployment mode and updated vault_k3s.yml token.
New Ansible Tasks and Playbooks:
- infrastructure/playbooks/iscsi_cleanup.yml: Automates logout and removal of stale iSCSI node records.
- infrastructure/playbooks/prole_logs_migrate.yml: Orchestrates /prole/logs migration to iSCSI storage.
- infrastructure/playbooks/tmp_bao_dir.yml: Ensures host-level storage directories for OpenBao.
- infrastructure/playbooks/tmp_mount.yml: Utility to verify and enforce host-level mounts.
- infrastructure/playbooks/k3s_sync.yml: Added tasks to start K3s after sync and update local kubeconfig on the controller.
- Added 'Unmount stale iSCSI mounts' and 'Remove stale iSCSI fstab entries' to the iscsi role.
- Added 'Ensure prole-installer service account exists' to the k3s role.
- Finalized stable, repeatable reset logic for the k3d pipeline.
- Refactored installer into modular components: core, milestone, runner, and state.
- Introduced new UI abstractions with support for ncurses and Tkinter.
- Updated initialization scripts and configurations for CloudNativePG, Kerberos, OpenBao, and Monitoring.
- Improved pipeline repair and port-forwarding mechanisms.
- Moved files from 'prole/' subdirectory to root level or appropriate subdirectories (tests, authority, infrastructure) to flatten the project structure.
- Updated 'install.py' and initialization scripts in 'etc/' to reflect the new directory layout.
- Added 'etc/repair_pipeline.sh' for automated pipeline repairs.
- Updated configuration files including 'conf/prole.cfg' and 'env.sh'.
- Integrated ArgoCD manifests in 'k8s/argocd/'.
- Updated 'prole-app' environment and properties.
- Moved and updated test scripts for better organization and reliability.
- Added 'tests/silent_install_test.sh' for automated installation testing.
- install.py: Major update including configuration variable expansion, improved k3s/k3d handling, and enhanced installation logic.
- etc/ scripts: Significant refactoring of initialization scripts (Kerberos, Port Forwards, Garage Store, etc.).
- Port Forwards: Transitioned from XML to port-mappings.conf for managing kubectl port-forwards.
- Status Reporting: Improved status checking for common services.
- Infrastructure: Updated Ansible inventory and rsyslog role configurations.
- Tests: Added a comprehensive suite of tests for 'etc' initialization scripts in prole/tests/etc/.
- Documentation: Added prole-db-documentation-mcp-architecture.md.
- General: Updated Dockerfiles and various helper scripts.
- Installer: Updated k3s deployment logic and configuration generation.
- k3s Role: Implemented token drift protection to verify Vault secrets against live node tokens.
- DNS: Enhanced samba_reverse_dns role to support multiple reverse zones.
- Service Init: Updated initialization scripts and status reporting.
- Infrastructure: Added prole management role and k3s diagnostic playbook.
- Configuration: Updated prole.cfg and added vaulted group variables.
- Consolidated and split initialization scripts in etc/:
- Removed init_prole-db.sh and init_authority.sh.
- Added init_kdc.sh for in-cluster MIT Kerberos KDC (prole-authority).
- Added init_ollama.sh for Ollama AI service integration.
- Added init_service_layer.sh for high-level service orchestration.
- Added init_k3s_registry.sh for private registry management.
- Major updates to install.py:
- Support for new Ollama and KDC configuration.
- Improved prole.cfg rendering and namespace handling.
- Updated unattended install flags.
- Infrastructure and Deployment:
- Updated K3s Ansible role with private registry support (registries.yaml template).
- Added prole-authority Dockerfile.
- Updated OpenBao Kerberos ConfigMap and other K8s manifests.
- Configuration:
- Updated prole.cfg with new sections for Ollama and Monitoring.
- Refined environment variable exports in env.sh and prole_cfg.sh.
- Introduced scripts `init_common_services.sh` and `status_common_services.sh` for deploying and checking common services (OpenBao, OpenTofu, registry) within Kubernetes namespaces.
- Improved service namespace configuration in `install.py` and updated initialization logic.
- Updated `prole.cfg` and `kerberos-configmap.yaml` with necessary changes to integrate the new features.
