Commit Graph

24 Commits

Author SHA1 Message Date
chrisfu
e3c2e625f1 refactor(config): separate k3d k3s and gke config entrypoints
Rename env config files from conf/*/prole.cfg to conf/k3d.cfg, conf/k3s.cfg, and conf/gke.cfg. Update shell/Python loaders and etc/deploy scripts to resolve named configs cleanly while keeping legacy fallback behavior. Align k3s Ansible tasks, docs, and regression coverage with the new configuration layout.

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-11 22:20:45 -07:00
chrisfu
c65d336d1f Make init-script secret checks warning-only and unblock Next
Co-authored-by: Junie <junie@jetbrains.com>
2026-03-31 23:15:54 -07:00
chrisfu
a069989315 Rename prole-db to knoe-db, add knoe-auth as cluster-internal KDC
Itemized changes:

1. knoe-auth: New cluster-internal KDC and SSO gateway service
   - Created etc/init_knoe_auth.sh based on init_kdc.sh with knoe-auth naming
   - Namespace defaults to SERVICE_NAMESPACE (knoe-system)
   - ConfigMap: knoe-auth-kdc-config, Secret: knoe-auth-secrets
   - Legacy cleanup removes old auth/dog/authority deployments

2. Orchestration: knoe-auth initializes before CloudNativePG
   - Updated prole.sh to insert init_knoe_auth.sh as step 2 (before CNPG)
   - Renumbered all subsequent initialization steps

3. Kong routing: Updated init_kong.sh to route to knoe-auth in SERVICE_NAMESPACE

4. Comment/reference updates for knoe-auth
   - Updated init_common_services.sh, init_service_layer.sh, init_kerberos.sh

5. prole-db renamed to knoe-db across the entire codebase
   - Renamed prole-db/ directory to knoe-db/
   - Renamed all prole-db Kubernetes manifests (deploy/opentofu, k8s/)
   - Renamed scripts: docker-root-knoe-db.sh, docker-run-knoe-db.sh, test-cnpg-knoe-db.sh
   - Renamed etc/init_prole-db-reset.sh to etc/init_knoe-db-reset.sh
   - Renamed etc/prole-db-passwwd.sh to etc/knoe-db-passwwd.sh
   - Renamed mock_val counterparts accordingly
   - Renamed tests/etc/test_init_prole-db-reset.sh to test_init_knoe-db-reset.sh
   - Renamed docs/prole-db-documentation-mcp-architecture.md to knoe-db variant
   - Renamed modes/k3d/prole-db/ to modes/k3d/knoe-db/
   - Renamed prole-db.iml to knoe-db.iml

6. Configuration updates
   - Updated conf/dev, conf/prod, conf/test, conf/service prole.cfg files
   - Updated conf/port-mapping.cfg
   - Updated etc/prole_cfg.sh and mock_val/prole_cfg.sh
   - Updated service/prole.cfg

7. Kubernetes manifests and deploy configuration
   - Updated deploy/opentofu/k3s ArgoCD application YAMLs
   - Updated kong-configmap.yaml and kustomization.yaml
   - Updated k3s/kong-config.yml and prole-resources.yaml
   - Updated prole-mssql-db deployment YAMLs
   - Updated supabase helm render and deploy scripts

8. Infrastructure and GCP Terraform
   - Updated deploy/gcp/terraform: folders, groups, IAM, service-projects

9. Python/installer code updates
   - Updated knoe/core: actions, build_context, controller, env, milestones
   - Updated knoe/milestone.py
   - Updated knoe/ui/screens: cfg, database, database_options, deploy, docker,
     navigation, security, services, validate
   - Updated knoe.spec, status.py

10. Shell script updates
    - Updated etc/: build_db, init_cloudnative_pg, init_cnpg_backup,
      init_db_manager, init_forgejo, init_gitlab, init_monitoring, init_openbao,
      init_port_forwards, init_postgrest, init_supabase_ports, status
    - Updated mock_val/ counterparts for all above scripts
    - Updated prole-net/init-prole-dns.sh
    - Updated bin/prole-kpf.sh, gitea/deploy.sh, supabase/deploy.sh

11. Test updates
    - Updated tests/etc/: test_init_cloudnative_pg*, test_init_cnpg_backup*,
      test_init_kdc*, test_init_kerberos*, test_init_kong*, test_prole_cfg*
    - Updated tests/installer/: test_actions_helpers, test_cfg_save_kubecontext,
      test_controller, test_core_classes, test_milestones, test_milestones_extended,
      test_namespace_propagation
    - Updated tests/: test_database_options, test_navigation,
      test_render_supabase_hostname, test_docker_build_fix,
      test_all_prole_home_fixes, silent_install_test, final_test

12. Documentation updates
    - Updated docs/: DOCKER-BUILD-FIX, PROLE-CFG-SECRETS, PROLE-HOME-DIRECTORY,
      build-system, patent
    - Updated scan/network_description.txt
    - Updated pom.xml

13. Miscellaneous script updates
    - Updated root-level: _adopt_replica_pvcs, _fix_replica_merlin, _import_pi,
      _patch_cluster, _prebind_pvcs, _rebind_d002, _rebind_d002b, test_resolve
    - Updated scripts/generate_spec.py

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 22:16:21 -07:00
chrisfu
4ee2b259c9 Checkpoint: rename installer to knoe + harden db build context
- Add build-context helper to copy Docker context safely (ignore runtime data, keep symlinks)

- Update UI and core actions to use ~/.prole/build and shared copy helper

- Add/adjust tests and scripts; introduce knoe ops helpers and update manifests

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 01:45:21 -07:00
chrisfu
ecacf01266 Remove legacy secrets, configs, and scripts; introduce prole-auth service and Grafana SSO proxy configuration. 2026-03-20 13:00:36 -07:00
chrisfu
ce5ff83eb4 checkpoint: improve init scripts, installer flows, and kube context handling
- Fix kube context switching for k3s single-context kubeconfigs and k3d shorthand prefixes

- Update common init/status scripts (registry, kerberos, cnpg backup, service layer, common services)

- Add Gitea init script and installer ArgoCD screen

- Add Supabase realtime probe patching plus regression tests

- Extend installer core/UI test coverage
2026-03-14 20:07:54 -07:00
chrisfu
909e92109e Refactor shell layout; add shellspec + authority
- Introduce lib/shell helpers and keep etc/* scripts thin via compatibility shims

- Move Kerberos validation to scripts/validation/check_kerberos.sh and update callers

- Add deterministic shellspec unit tests under tests/shellspec/ and wire Maven to run them

- Add minimal Spring Boot authority module with startup + /health endpoint and Maven wiring

- Document the new layout in docs/layout.md

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-13 09:19:56 -07:00
chrisfu
86cf99cf76 milestone(k3s+cnpg): end-to-end deploy run via standalone installer
We can now complete a full K3s deployment run and deliver a working CloudNativePG (CNPG) stack from the standalone installer.\n\nHighlights:\n- Installer scripts hardened for CNPG deploy + recovery templating + rollout control\n- OpenTofu/K3s manifests aligned with in-repo k8s templates for prole-db\n- Supporting init flows updated (OpenBao, monitoring, Kerberos, OpenTofu)

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-03 20:03:15 -08:00
chrisfu
46f31843c2 Fix Kerberos test flow and improve multi-namespace support in k3s
- Resolve namespace drift in Kerberos test flow and OpenBao deployment.
- Fix OpenAPI credential-validation failures in etc scripts by using --validate=false with kubectl.
- Introduce preferred_service_namespace() to accurately target namespaces for KDC policy/test paths.
- Default Kerberos test user to 'administrator' and prefill from Ansible vault secrets.
- Generalized OpenBao deployment logic to correctly handle multi-namespace environments.
- Added regression test 'tests/etc/test_init_kerberos_k3s_real_kdc.sh'.
- Synchronized SERVICE_NAMESPACE extraction in prole_cfg.sh.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-01 00:57:10 -08:00
chrisfu
765aa0b926 Checkpoint: cluster env UI layout + k3s common services
- Tighten Cluster Environment screen layout; switch Service/Prod to kubectx context selection; keep namespace and key controls on one line; ensure Repair button remains reachable.

- Add UI layout regression test to render with large mock data and assert key widgets remain visible and console is scrollable.

- Make kubeconfig generation deterministic under tests by avoiding overwriting cert-based kubeconfigs; write token sidecar kubeconfig when needed.

- Update common-services init scripts and add k3s/Helm deployment bits (svc-check, Kong/CertMgr tasks).
2026-02-27 14:35:14 -08:00
chrisfu
d68925bbc5 Add TLS certificates to CNPG cluster and restore missing Optional Features config - Add certificates section (prole-db-tls, prole-db-ca) to prole-db.yaml and prole-db-recovery.yaml.tpl - Expand generate_tls_if_missing() to create both CA and server TLS secrets with proper SANs - Add detect_and_reprovision_unencrypted_cluster() for existing clusters needing encryption conversion - Propagate AT_REST_ENCRYPTION_ENABLED to shell scripts via _script_env_for_namespace() - Restore AT_REST_ENCRYPTION_ENABLED, SUPABASE_ENABLED, KERBEROS_ENABLED in prole.cfg - Update installer milestones, docker screens, port-mapping, init_kerberos, init_postgrest, prole.sh 2026-02-22 18:58:01 -08:00
chrisfu
6e2d3e9011 refactor: modernize installer and monitoring setup
- Monitoring: Migrated from manual Grafana/Prometheus manifests to kube-prometheus-stack based setup in etc/init_monitoring.sh. Removed old manifest files from deploy/ and k8s/.
- Installer Core: Refactored installer with new modules for actions, environment handling, and UI screens. Enhanced Milestone logic to support advanced configuration (ArgoCD, Registry namespaces, Kerberos flags, etc.).
- Service & Init Scripts: Updated multiple initialization scripts (init_*.sh) for better integration with OpenBao, Kerberos, and the new monitoring stack. Added new scripts for Nginx Ingress, Ollama parsing, and K3D route fixes.
- Infrastructure: Enhanced Samba AD DC Ansible role with realm derivation, provisioning guidance, and group management. Updated K3s role tasks.
- Configuration: Refined default settings in conf/ to align with the new deployment architecture.
- App & Tools: Updated prole-app Swift code and prole.sh for improved environment variable handling and installation flow.
2026-02-19 21:07:39 -08:00
chrisfu
ec2795f8a3 Checkpoint: Kerberos egress fixes, local registry provisioning for k3d/k3s, and installer GUI workflow enhancements. Key changes: AD forwarder port offsets, outbound NetworkPolicy, KDC dns_lookup disabling; k3d/k3s registry support with Docker network connect fix; GUI 'Deploy Services' button and manual pause; new test_service_layer_navigation.py. 2026-02-17 14:26:12 -08:00
chrisfu
6b89ea23d7 Stabilize deliverable k3d pipeline and refactor installer components
- Finalized stable, repeatable reset logic for the k3d pipeline.

- Refactored installer into modular components: core, milestone, runner, and state.

- Introduced new UI abstractions with support for ncurses and Tkinter.

- Updated initialization scripts and configurations for CloudNativePG, Kerberos, OpenBao, and Monitoring.

- Improved pipeline repair and port-forwarding mechanisms.
2026-02-15 00:03:24 -08:00
chrisfu
f2c9012cce Refactor installation and initialization logic, and expand test coverage
- install.py: Major update including configuration variable expansion, improved k3s/k3d handling, and enhanced installation logic.

- etc/ scripts: Significant refactoring of initialization scripts (Kerberos, Port Forwards, Garage Store, etc.).

- Port Forwards: Transitioned from XML to port-mappings.conf for managing kubectl port-forwards.

- Status Reporting: Improved status checking for common services.

- Infrastructure: Updated Ansible inventory and rsyslog role configurations.

- Tests: Added a comprehensive suite of tests for 'etc' initialization scripts in prole/tests/etc/.

- Documentation: Added prole-db-documentation-mcp-architecture.md.

- General: Updated Dockerfiles and various helper scripts.
2026-02-13 21:36:39 -08:00
chrisfu
96f594fd3c Refactor initialization scripts and add new service components
- Consolidated and split initialization scripts in etc/:
    - Removed init_prole-db.sh and init_authority.sh.
    - Added init_kdc.sh for in-cluster MIT Kerberos KDC (prole-authority).
    - Added init_ollama.sh for Ollama AI service integration.
    - Added init_service_layer.sh for high-level service orchestration.
    - Added init_k3s_registry.sh for private registry management.
- Major updates to install.py:
    - Support for new Ollama and KDC configuration.
    - Improved prole.cfg rendering and namespace handling.
    - Updated unattended install flags.
- Infrastructure and Deployment:
    - Updated K3s Ansible role with private registry support (registries.yaml template).
    - Added prole-authority Dockerfile.
    - Updated OpenBao Kerberos ConfigMap and other K8s manifests.
- Configuration:
    - Updated prole.cfg with new sections for Ollama and Monitoring.
    - Refined environment variable exports in env.sh and prole_cfg.sh.
2026-02-11 13:09:31 -08:00
chrisfu
da2f6600ba feat: infrastructure and installer updates for k3s, OpenTofu, and prole-db
- Add k3s start/stop Ansible playbooks and roles.

- Implement OpenTofu initialization scripts and k8s manifests.

- Update ncurses installer with OpenTofu support and improved k3s integration.

- Add mode support (--mode) to etc/ initialization scripts.

- Update prole-db with recovery, barman objectstore, and SSH OpenBao support.

- Refine k8s manifests for OpenBao and prole-db.
2026-02-05 21:27:18 -08:00
chrisfu
bcc8f23a0d Enhance secret management and k8s infrastructure
- Secret Management: Integrated AESGCM for temporary secret handling in install.py and enhanced OpenBao (Vault) support with namespace injection and additional secret paths (Grafana, Kerberos, TDE).
- Infrastructure & K8s:
    - Added Barman Object Store backup configuration (S3) to prole-db.yaml.
    - Updated Prometheus deployment with PVC and persistent configuration.
    - Updated k3s cluster/registry creation scripts.
    - Added etc/build-a-bao.sh for OpenBao setup.
- MSSQL Integration: Updated docker scripts and k8s deployments for Prole MSSQL database.
- Documentation: Added docs/PROLE-CFG-SECRETS.md explaining the new secret handling.
- General: Refined initialization scripts (init_authority.sh, init_openbao.sh, etc.) and updated the ncurses installer.
2026-02-03 22:39:50 -08:00
chrisfu
edbc6c5385 Checkpoint: Refactor installer UI and update Supabase deployment strategy
- Refactored install.py and installer package for improved UI and navigation.

- Replaced Supabase k8s manifests with a dedicated deployment script and port-wiring logic.

- Added new deployment pipeline and finalization scripts in etc/.

- Updated initialization scripts for Kerberos, authority, and port forwards.

- Updated port mappings and tests.
2026-02-03 14:55:05 -08:00
chrisfu
fff18fdbe4 Update Prole-DB and improve Supabase integration
- Bumped Prole-DB image version to 17.7-053 in scripts, Dockerfile, and manifests.
- Replaced `prole-scan` with `prole-agent` throughout scripts and tests.
- Refined Kubernetes setup for Supabase to use namespace 'supabase'.
- Introduced conversion of Supabase Docker Compose to Kubernetes manifests with `kompose`.
- Added support for Kerberos toggle via environment variables in `init_kerberos.sh`.
- Improved error handling and logging in scripts for better maintainability.
2026-02-01 23:56:25 -08:00
chrisfu
9c1f58caf4 Checkpoint: namespace management improvements and prole-db updates. Enhanced namespace management with automated namespace detection and configuration in etc/prole_cfg.sh and etc/init_prole-db.sh. Added namespace reset and recovery scripts: scripts/reset-ns.sh, etc/init_prole-db-reset.sh, and k8s/prole/prole-db-recovery.yaml.tpl. Updated prole-db image versioning to use release files and bumped version to 17.7-041. Improved installer UI and configuration handling in install.py and installer/screen.py. Updated Kerberos and CloudNativePG initialization scripts for better namespace support. Added /prole/backup/ to .gitignore. 2026-01-31 19:22:05 -08:00
chrisfu
9060518877 Sourced prole_cfg.sh for environment and config loading across all init scripts, consolidating configuration management. Removed redundant env.sh loading logic. 2026-01-31 01:24:18 -08:00
chrisfu
a7cb5b5fca Add prole_cfg.sh sourcing for configuration management
- Consistently source `prole_cfg.sh` across multiple init scripts for unified configuration handling.
- Enhanced error handling in `screen.py` to manage potential UI exceptions gracefully.
- Improved asynchronous function scheduling using the new `safe_after` method.
2026-01-30 23:37:32 -08:00
chrisfu
f0e75f6107 feat: complete end-to-end installation and storage integration. This commit marks a significant milestone where the end-to-end installation process is now fully functional. Key changes: Integrated Garage storage service (S3-compatible); Implemented Prole DB backup; Enhanced Kerberos integration; Updated default namespace to prole-chrisfu-deadbeef; Streamlined dependencies (removed Ollama); Added installation validation and testing scripts; Improved installer UI and deployment logic. 2026-01-29 22:50:05 -08:00