Commit Graph

81 Commits

Author SHA1 Message Date
chrisfu
ec2795f8a3 Checkpoint: Kerberos egress fixes, local registry provisioning for k3d/k3s, and installer GUI workflow enhancements. Key changes: AD forwarder port offsets, outbound NetworkPolicy, KDC dns_lookup disabling; k3d/k3s registry support with Docker network connect fix; GUI 'Deploy Services' button and manual pause; new test_service_layer_navigation.py. 2026-02-17 14:26:12 -08:00
chrisfu
ec8a4e98af k3s: Fix installation hangs, CA mismatches, and arm64 networking
- Implement proactive CA hash verification and automated repair for mismatches

- Ensure agents prioritize discovered server tokens over stale vault values

- Fix K3s service hangs with explicit stop and killall before reinstall

- Add Retropie/Pi networking fixes (WiFi power save, wlan0 priority)

- Pin pre-staged images to stable, architecture-aware versions (arm64)

- Remove obsolete init-port-forwards and prole.cfg sync tasks

- Update k8s manifests and installer core logic with new tests
2026-02-16 23:30:27 -08:00
chrisfu
9aa959f61d Milestone: Balanced iSCSI IO, updated prole.sh launcher with log/verbose/debug/coverage/silent options, and enhanced installer logging 2026-02-16 16:44:18 -08:00
chrisfu
23cf4ac33d Configure K3s for retropie and add Traefik Ingress. Updated K3s templates for node-ip/flannel-iface, configured retropie host vars, enabled Traefik, added Ingress resources for Grafana, Prole, Kong, and Supabase, and set Postgres to LoadBalancer. 2026-02-16 04:28:40 -08:00
chrisfu
6b89ea23d7 Stabilize deliverable k3d pipeline and refactor installer components
- Finalized stable, repeatable reset logic for the k3d pipeline.

- Refactored installer into modular components: core, milestone, runner, and state.

- Introduced new UI abstractions with support for ncurses and Tkinter.

- Updated initialization scripts and configurations for CloudNativePG, Kerberos, OpenBao, and Monitoring.

- Improved pipeline repair and port-forwarding mechanisms.
2026-02-15 00:03:24 -08:00
chrisfu
d2efb835b0 Refactor project structure and update initialization scripts
- Moved files from 'prole/' subdirectory to root level or appropriate subdirectories (tests, authority, infrastructure) to flatten the project structure.

- Updated 'install.py' and initialization scripts in 'etc/' to reflect the new directory layout.

- Added 'etc/repair_pipeline.sh' for automated pipeline repairs.

- Updated configuration files including 'conf/prole.cfg' and 'env.sh'.

- Integrated ArgoCD manifests in 'k8s/argocd/'.

- Updated 'prole-app' environment and properties.

- Moved and updated test scripts for better organization and reliability.

- Added 'tests/silent_install_test.sh' for automated installation testing.
2026-02-14 13:44:49 -08:00
chrisfu
96f594fd3c Refactor initialization scripts and add new service components
- Consolidated and split initialization scripts in etc/:
    - Removed init_prole-db.sh and init_authority.sh.
    - Added init_kdc.sh for in-cluster MIT Kerberos KDC (prole-authority).
    - Added init_ollama.sh for Ollama AI service integration.
    - Added init_service_layer.sh for high-level service orchestration.
    - Added init_k3s_registry.sh for private registry management.
- Major updates to install.py:
    - Support for new Ollama and KDC configuration.
    - Improved prole.cfg rendering and namespace handling.
    - Updated unattended install flags.
- Infrastructure and Deployment:
    - Updated K3s Ansible role with private registry support (registries.yaml template).
    - Added prole-authority Dockerfile.
    - Updated OpenBao Kerberos ConfigMap and other K8s manifests.
- Configuration:
    - Updated prole.cfg with new sections for Ollama and Monitoring.
    - Refined environment variable exports in env.sh and prole_cfg.sh.
2026-02-11 13:09:31 -08:00
chrisfu
87e2f5d385 Checkpoint: Refactor common services and monitoring initialization. Updates: etc/init_monitoring.sh (Grafana naming, legacy cleanup), etc/init_common_services.sh (migration/cleanup logic), added etc/init_registry.sh, updated etc/init_cloudnative_pg.sh (manifest filtering), updated configs, and enhanced tests. Fixed: excluded .ansible logs and added to .gitignore. 2026-02-09 18:50:44 -08:00
chrisfu
d88f5b6424 Add scripts for managing common services and enhance service namespace handling
- Introduced scripts `init_common_services.sh` and `status_common_services.sh` for deploying and checking common services (OpenBao, OpenTofu, registry) within Kubernetes namespaces.
- Improved service namespace configuration in `install.py` and updated initialization logic.
- Updated `prole.cfg` and `kerberos-configmap.yaml` with necessary changes to integrate the new features.
2026-02-07 23:20:08 -08:00
chrisfu
f47c18fef7 feat(infrastructure): enhance k3s automation and OpenTofu integration
- Infrastructure:
    - Updated k3s Ansible role with mountpoint preflight checks and better permission management.
    - Automated deployment of prole configuration and port-forwarding scripts to cluster hosts.
    - Added systemd service for managing port forwards on k3s nodes.
    - Added prole-installer service account token automation.
- K8s Manifests:
    - Renamed and added Persistent Volumes in iscsi-pvs.yaml (including OpenBao support).
    - Updated StatefulSets for garage and openbao.
    - Migrated prole-db to CloudNativePG-based configuration.
    - Added comprehensive OpenTofu manifests for cluster deployment.
- Configuration:
    - Added cluster-specific configurations (k3d, k3s-hosts).
    - Added PostgreSQL configuration templates.
    - Updated .gitignore to track the conf/ directory.
- Tools:
    - Updated install.py and port-forwarding scripts.
    - Added render_manifest.py for manifest generation.
2026-02-07 22:53:18 -08:00
chrisfu
58faab34fb Add playbooks and manifests for cgroups and iSCSI storage setup
- Introduced cgroup kernel parameter management for `k3s_hosts` via Ansible playbook.
- Added Kubernetes PersistentVolumes for iSCSI storage with specific node affinity.
- Defined `prole-iscsi` StorageClass for volume provisioning.
2026-02-06 01:19:59 -08:00
chrisfu
6b6e5e2aec nodeSelector and storage updates across k8s manifests
- Added `nodeSelector` for multiple Kubernetes resources to ensure scheduling on `myrddin.prole.org`.
- Modified `storage` requests and set `storageClassName` in `garage-statefulset.yaml`.
- Enhanced `install.py` for dynamic environment configuration and kubeconfig handling.
- Improved cgroup management tasks in Ansible with conflict resolution and parameter updates.
- Simplified vault token update process in playbooks and updated encryption checks.
2026-02-06 01:17:54 -08:00
chrisfu
da2f6600ba feat: infrastructure and installer updates for k3s, OpenTofu, and prole-db
- Add k3s start/stop Ansible playbooks and roles.

- Implement OpenTofu initialization scripts and k8s manifests.

- Update ncurses installer with OpenTofu support and improved k3s integration.

- Add mode support (--mode) to etc/ initialization scripts.

- Update prole-db with recovery, barman objectstore, and SSH OpenBao support.

- Refine k8s manifests for OpenBao and prole-db.
2026-02-05 21:27:18 -08:00
chrisfu
620ce63190 checkpoint: update infrastructure scripts, manifests, and installer. etc/init_*.sh: added namespace support and OpenBao monitoring configuration; k8s/prole: added grafana-pvc, updated prole-db with S3 region support; install.py: implemented config secret encryption and improved ARM64 platform detection; .gitignore: expanded ignore patterns; prole-net: updated prole-agent binary 2026-02-04 01:37:46 -08:00
chrisfu
bcc8f23a0d Enhance secret management and k8s infrastructure
- Secret Management: Integrated AESGCM for temporary secret handling in install.py and enhanced OpenBao (Vault) support with namespace injection and additional secret paths (Grafana, Kerberos, TDE).
- Infrastructure & K8s:
    - Added Barman Object Store backup configuration (S3) to prole-db.yaml.
    - Updated Prometheus deployment with PVC and persistent configuration.
    - Updated k3s cluster/registry creation scripts.
    - Added etc/build-a-bao.sh for OpenBao setup.
- MSSQL Integration: Updated docker scripts and k8s deployments for Prole MSSQL database.
- Documentation: Added docs/PROLE-CFG-SECRETS.md explaining the new secret handling.
- General: Refined initialization scripts (init_authority.sh, init_openbao.sh, etc.) and updated the ncurses installer.
2026-02-03 22:39:50 -08:00
chrisfu
edbc6c5385 Checkpoint: Refactor installer UI and update Supabase deployment strategy
- Refactored install.py and installer package for improved UI and navigation.

- Replaced Supabase k8s manifests with a dedicated deployment script and port-wiring logic.

- Added new deployment pipeline and finalization scripts in etc/.

- Updated initialization scripts for Kerberos, authority, and port forwards.

- Updated port mappings and tests.
2026-02-03 14:55:05 -08:00
chrisfu
5ab6381982 Fix YAML parsing errors in init_authority.sh and improve SSH key management. Resolved 'invalid Yaml document separator' and 'could not find expected :' errors in etc/init_authority.sh by using printf to inject multi-line SSH keys safely. Added 'ansible' user setup. Updated etc/init_openbao.sh to store ansible keys. Created prole/tests/test_init_authority.sh for validation. 2026-02-02 13:15:31 -08:00
chrisfu
96aaf488d6 Bump Prole-DB image version to 17.7-059, enable at-rest encryption, and comment out Kerberos and pg_prolelog configurations. Update kdc and admin_server IPs in Kerberos config.
'Add CNPG build/init test and skip tiger geocoder' -m 'Test script usage: scripts/test-cnpg-prole-db.sh' -m 'Env overrides: CNPG_TEST_NAMESPACE, CNPG_CLUSTER_NAME, CNPG_MANIFEST_URL, CNPG_VERSION, IMAGE_LOAD (kind|minikube|k3d), IMAGE_LOAD_CMD, CNPG_INSTANCES'
2026-02-02 01:18:22 -08:00
chrisfu
7e9e6d0c11 Add CNPG build/init test and skip tiger geocoder
Test script usage: scripts/test-cnpg-prole-db.sh

Env overrides: CNPG_TEST_NAMESPACE, CNPG_CLUSTER_NAME, CNPG_MANIFEST_URL, CNPG_VERSION, IMAGE_LOAD (kind|minikube|k3d), IMAGE_LOAD_CMD, CNPG_INSTANCES
2026-02-02 01:15:11 -08:00
chrisfu
fff18fdbe4 Update Prole-DB and improve Supabase integration
- Bumped Prole-DB image version to 17.7-053 in scripts, Dockerfile, and manifests.
- Replaced `prole-scan` with `prole-agent` throughout scripts and tests.
- Refined Kubernetes setup for Supabase to use namespace 'supabase'.
- Introduced conversion of Supabase Docker Compose to Kubernetes manifests with `kompose`.
- Added support for Kerberos toggle via environment variables in `init_kerberos.sh`.
- Improved error handling and logging in scripts for better maintainability.
2026-02-01 23:56:25 -08:00
chrisfu
9c1f58caf4 Checkpoint: namespace management improvements and prole-db updates. Enhanced namespace management with automated namespace detection and configuration in etc/prole_cfg.sh and etc/init_prole-db.sh. Added namespace reset and recovery scripts: scripts/reset-ns.sh, etc/init_prole-db-reset.sh, and k8s/prole/prole-db-recovery.yaml.tpl. Updated prole-db image versioning to use release files and bumped version to 17.7-041. Improved installer UI and configuration handling in install.py and installer/screen.py. Updated Kerberos and CloudNativePG initialization scripts for better namespace support. Added /prole/backup/ to .gitignore. 2026-01-31 19:22:05 -08:00
chrisfu
f0e75f6107 feat: complete end-to-end installation and storage integration. This commit marks a significant milestone where the end-to-end installation process is now fully functional. Key changes: Integrated Garage storage service (S3-compatible); Implemented Prole DB backup; Enhanced Kerberos integration; Updated default namespace to prole-chrisfu-deadbeef; Streamlined dependencies (removed Ollama); Added installation validation and testing scripts; Improved installer UI and deployment logic. 2026-01-29 22:50:05 -08:00
chrisfu
dda938ee95 Integrate namespace setup and management into scripts, Dockerfile, and update installer UI. 2026-01-28 23:02:50 -08:00
chrisfu
5e823f82a8 feat(supabase): add optional init flow, k8s manifests, and docs
• install.py
• etc/init_supabase.sh
• k8s/prole/kustomization.yaml
• k8s/prole/prole-db-postgres-service.yaml
• k8s/prole/supabase-configmap.yaml
• k8s/prole/supabase-deployment.yaml
• k8s/prole/supabase-service.yaml
• prole-db/supabase.md
2026-01-27 13:06:35 -08:00
chrisfu
a4d0a2e0a3 Update prole-db docker tag to 17.7-037 and update pg_prolelog deb package 2026-01-15 11:58:39 -08:00
chrisfu
906392d462 feat(installer): improve UI and add test coverage for core features
- Refactored installer UI with updated canvas rendering, sidebar navigation, and footer buttons.
- Enhanced styling for macOS compatibility and consistent design across controls.
- Added Pytest-based unit tests for `screen.py` and `config.py`.
- Expanded dependency catalog with new tools like `tshark` and `pyshark`.
- Improved error tolerance for background rendering and added placeholders for Kerberos configuration.
2026-01-08 21:51:30 -08:00
chrisfu
10adb0e53a feat(app, k8s): add OpenBAO status check and enhance service integration
- Added OpenBAO readiness check via `PFScriptBridge`.
- Merged service defaults with indexed configuration in `Config.swift`.
- Improved service checker to handle OpenBAO readiness using `kubectl`.
- Refined `StatusView` to support traffic light interactivity and case-insensitive service matching.
- Expanded `prole-db.yaml` with additional PostgreSQL extensions and function grants.
2025-12-30 23:47:22 -08:00
chrisfu
73bc5bfede feat: implement encrypted development database with OpenBao and CNPG - Update install.py with password matching visual feedback and OpenBao integration - Enhance etc/init_openbao.sh to store database password in OpenBao KV - Update etc/init_cloudnative_pg.sh to synchronize database password from OpenBao to K8s secrets - Configure CNPG cluster in k8s/prole/prole-db.yaml with pg_hba for password authentication - Update various scripts and app sources for better service integration 2025-12-30 22:18:55 -08:00
chrisfu
528e362d95 feat(scripts): add create, delete, and recreate actions for CloudNative-PG initialization
- Introduced new script commands to manage CloudNative-PG clusters (`create`, `delete`, `recreate`).
- Updated default namespace to `prole` and refined variable handling for cluster name.
- Adjusted kustomization.yaml to reference `prole-db.yaml` for consistency in manifests.
2025-12-24 03:02:57 -08:00
chrisfu
940a48a237 refactor(scripts, installer): replace deprecated init_primary_domain.sh with init_openbao.sh
- Removed `init_primary_domain.sh` entirely; introduced `init_openbao.sh` with updated logic and streamlined functionality.
- Standardized script names to align with the OpenBao-centric workflow.
- Updated installer to dynamically fetch Prole database versions and reflect script changes.
- Improved TLS generation and k8s manifest handling for CloudNative-PG.
- Added new references to OpenBao initialization across scripts and UI components.
2025-12-23 22:40:03 -08:00
chrisfu
3a1f5ea444 Milestone: Integrated CloudNative-PG initialization and build flow into Prole installer. Added new initialization screens, refactored scripts, and updated k8s manifests. 2025-12-20 20:33:19 -08:00