Commit Graph

26 Commits

Author SHA1 Message Date
chrisfu
55b6a6aff3 Complete rebranding from prole to knoe and fix macOS application identity. Bulk renamed 'prole' to 'knoe' across code, scripts, and manifests. Updated Makefile with 'knoe', 'build', and 'start' targets. Implemented macOS Application Bundle (.app) support for correct identity. Fixed macOS 'Python' process name to 'Knoe.DB Installer' via Objective-C bridge. Standardized application name to 'Knoe.DB Installer' across all interfaces.
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-22 15:08:35 -07:00
chrisfu
666ed5c1c3 chore: add secretref resolution for auth OIDC inputs
- Added `_resolve_secretref_value` to handle `secretref://` values for `auth.clientId` and `auth.clientSecret`.
- Extended environment and file-based lookups for secretref resolution.
- Updated tests to validate new secretref resolution logic in config and GitOps workflows.
2026-04-18 18:08:16 -07:00
chrisfu
f2677b3eaf chore: add auth OIDC secret refs handling in config loader
- Implemented `auth.clientId` and `auth.clientSecret` backfilling from Global section and environment variables.
- Added tests to verify correct hydration of OIDC inputs in configuration defaults.
2026-04-18 17:28:34 -07:00
chrisfu
090b0e882a chore: update prole.cfg defaults for dev cluster, refine config overrides
- Updated default paths, namespaces, and cluster names for k3d/dev configuration.
- Added `skip_names` set to exclude `prod.cfg` and `gcp.cfg` from override processing.
- Refined init parameters for Supabase, Kubernetes, and database deployments to align with dev-specific settings.
- Enhanced test coverage for excluded config files (`prod.cfg`, `gcp.cfg`) within `test_prole_conf.py`.
2026-04-12 13:26:59 -07:00
chrisfu
138a01c181 chore: update prole.cfg defaults for dev cluster, refine config overrides
- Updated default paths, namespaces, and cluster names for k3d/dev configuration.
- Added `skip_names` set to exclude `prod.cfg` and `gcp.cfg` from override processing.
- Refined init parameters for Supabase, Kubernetes, and database deployments to align with dev-specific settings.
- Enhanced test coverage for excluded config files (`prod.cfg`, `gcp.cfg`) within `test_prole_conf.py`.
2026-04-12 12:52:43 -07:00
chrisfu
65369811f7 Harden ingress targeting and normalize HOME paths
- enforce app-cluster-only rendering for public ingress hosts with DB-cluster guardrails\n- fix Supabase/GitLab/authority ingress host ownership and ingress-class safety checks\n- normalize persisted home-directory paths to /Users/chrisfu and update gke config defaults\n- add/adjust regression tests for ingress placement/hostname and cfg path normalization

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-12 12:32:09 -07:00
chrisfu
c94c62e1bb Harden prod deploy namespace/context routing and vault password handling
- persist and load DB master password via Ansible Vault bootstrap flow

- enforce knoe-system service namespace and explicit app/db kubecontext targeting

- improve OpenBao/CNPG deploy reliability and logging; add retries/readiness diagnostics

- tighten reset/delete cluster behavior and expand installer/deploy pipeline test coverage

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-10 00:43:07 -07:00
chrisfu
c0a7d0c5dc Refine installer orchestration and Supabase rendering paths
- improve installer/action/controller flow and shell-variable expansion handling across screens\n- adjust Supabase Helm rendering and storage deployment templates\n- align monitoring, cloudnative-pg and repair pipeline behavior with updated config paths\n- refresh and expand installer/core regression tests around milestones, navigation and repair logic

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-31 23:15:54 -07:00
chrisfu
761d80486b feat: add storage probing and operational service updates
- add reusable storage probing subsystem with discovery, bounded probe execution, IO classification, caching, and topology integration

- render per-node storage inventory in Cluster Nodes UI and extend installer test coverage for topology/storage behavior

- introduce core service operation modules and align actions, milestones, services, and supporting configs/scripts for repair/update workflows

- update CNPG/Supabase/database artifacts, placement and port mapping configs, plus related integration tests

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-26 09:44:23 -07:00
chrisfu
ad36f4ee54 Stage 2: complete CNPG shell-to-Python cutover, remove init_cloudnative_pg.sh
All active runtime paths that previously shelled out to etc/init_cloudnative_pg.sh
now dispatch through Python. The shell script has been removed from git.

New files:
- knoe/core/ops/cloudnative_pg.py: canonical Python owner for CNPG lifecycle with
  public API (initialize, deploy, rollout, ensure_operator, pin_controller,
  install_barman_plugin) and internal helpers (_apply_manifest, _wait_cnpg_pods,
  _reconcile_instances, etc.)
- prole/tools/run_cnpg_coverage.py: coverage entry points for k3d/k3s modes,
  report sub-command, and check-shell scanner to confirm no live Python dispatch
  to the removed shell script

Modified files:
- knoe/core/actions.py: replace shell dispatch in _step_init_scripts,
  _step_cnpg_deploy, and repair pipeline with Python calls
- knoe/core/milestones.py: replace shell dispatch in InitializationScriptsMilestone
  and DeploymentMilestone
- knoe/ui/screens/services.py: replace shell dispatch for init-scripts step,
  deploy button, and rollout button
- prole/deployment.py: replace shell dispatch in _run_post_apply_scripts
- status.py: remove init_cloudnative_pg.sh from _STATUS_SCRIPTS list
- tests/installer/test_actions_helpers.py: mock Python functions, assert shell
  script is never dispatched
- tests/installer/test_milestones.py: same
- tests/installer/test_services_init_scripts.py: same

Removed:
- etc/init_cloudnative_pg.sh (git rm)

Verification:
- prole.tools.run_cnpg_coverage check-shell reports clean
- All 61 tests in the affected test files pass

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-23 19:28:04 -07:00
chrisfu
0e806a9233 checkpoint: tighten CNPG k3s storage flow and policy updates
- normalize K3S/CNPG/OpenTofu env keys across installer and configs; - harden CNPG storage validation/provisioning and fail fast in init script; - update DB manifests/recovery template and port/network mapping data; - expand cluster node policy UI coverage and shell/python tests; - add k3s mode version marker files

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-23 12:39:03 -07:00
chrisfu
b09319ffaf feat: pre-provision cluster-scoped CNPG Synology PVs
Add storage ops for deterministic namespace+cluster local PV paths, labels/selectors, host-path preparation, and idempotent reconciliation before CNPG cluster apply.

Wire selector injection and validation into CNPG deploy flow/script, and extend installer tests for provisioning orchestration and failure handling.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-23 00:32:35 -07:00
chrisfu
59c8ea3f8f Add CNPG placement planner with persistence and integration tests
- Introduce `cnpg_placement.py` to handle round-robin node assignments for CloudNativePG clusters.
- Implement persistent placement plans with schema normalization and hashing.
- Add `load_cnpg_placement_plan` and `save_cnpg_placement_plan` for plan persistence.
- Integrate planner with installer to support node-based topology configuration.
- Update `actions.py` with placement planning logic, including rebalance support and node eligibility checks.
- Extend shell utilities (`init_cloudnative_pg.sh`) for placement-aware CNPG topology adjustments.
- Add comprehensive unit tests and integration tests for planner functionality, persistence, and shell environment exports.
2026-03-23 00:06:47 -07:00
chrisfu
a069989315 Rename prole-db to knoe-db, add knoe-auth as cluster-internal KDC
Itemized changes:

1. knoe-auth: New cluster-internal KDC and SSO gateway service
   - Created etc/init_knoe_auth.sh based on init_kdc.sh with knoe-auth naming
   - Namespace defaults to SERVICE_NAMESPACE (knoe-system)
   - ConfigMap: knoe-auth-kdc-config, Secret: knoe-auth-secrets
   - Legacy cleanup removes old auth/dog/authority deployments

2. Orchestration: knoe-auth initializes before CloudNativePG
   - Updated prole.sh to insert init_knoe_auth.sh as step 2 (before CNPG)
   - Renumbered all subsequent initialization steps

3. Kong routing: Updated init_kong.sh to route to knoe-auth in SERVICE_NAMESPACE

4. Comment/reference updates for knoe-auth
   - Updated init_common_services.sh, init_service_layer.sh, init_kerberos.sh

5. prole-db renamed to knoe-db across the entire codebase
   - Renamed prole-db/ directory to knoe-db/
   - Renamed all prole-db Kubernetes manifests (deploy/opentofu, k8s/)
   - Renamed scripts: docker-root-knoe-db.sh, docker-run-knoe-db.sh, test-cnpg-knoe-db.sh
   - Renamed etc/init_prole-db-reset.sh to etc/init_knoe-db-reset.sh
   - Renamed etc/prole-db-passwwd.sh to etc/knoe-db-passwwd.sh
   - Renamed mock_val counterparts accordingly
   - Renamed tests/etc/test_init_prole-db-reset.sh to test_init_knoe-db-reset.sh
   - Renamed docs/prole-db-documentation-mcp-architecture.md to knoe-db variant
   - Renamed modes/k3d/prole-db/ to modes/k3d/knoe-db/
   - Renamed prole-db.iml to knoe-db.iml

6. Configuration updates
   - Updated conf/dev, conf/prod, conf/test, conf/service prole.cfg files
   - Updated conf/port-mapping.cfg
   - Updated etc/prole_cfg.sh and mock_val/prole_cfg.sh
   - Updated service/prole.cfg

7. Kubernetes manifests and deploy configuration
   - Updated deploy/opentofu/k3s ArgoCD application YAMLs
   - Updated kong-configmap.yaml and kustomization.yaml
   - Updated k3s/kong-config.yml and prole-resources.yaml
   - Updated prole-mssql-db deployment YAMLs
   - Updated supabase helm render and deploy scripts

8. Infrastructure and GCP Terraform
   - Updated deploy/gcp/terraform: folders, groups, IAM, service-projects

9. Python/installer code updates
   - Updated knoe/core: actions, build_context, controller, env, milestones
   - Updated knoe/milestone.py
   - Updated knoe/ui/screens: cfg, database, database_options, deploy, docker,
     navigation, security, services, validate
   - Updated knoe.spec, status.py

10. Shell script updates
    - Updated etc/: build_db, init_cloudnative_pg, init_cnpg_backup,
      init_db_manager, init_forgejo, init_gitlab, init_monitoring, init_openbao,
      init_port_forwards, init_postgrest, init_supabase_ports, status
    - Updated mock_val/ counterparts for all above scripts
    - Updated prole-net/init-prole-dns.sh
    - Updated bin/prole-kpf.sh, gitea/deploy.sh, supabase/deploy.sh

11. Test updates
    - Updated tests/etc/: test_init_cloudnative_pg*, test_init_cnpg_backup*,
      test_init_kdc*, test_init_kerberos*, test_init_kong*, test_prole_cfg*
    - Updated tests/installer/: test_actions_helpers, test_cfg_save_kubecontext,
      test_controller, test_core_classes, test_milestones, test_milestones_extended,
      test_namespace_propagation
    - Updated tests/: test_database_options, test_navigation,
      test_render_supabase_hostname, test_docker_build_fix,
      test_all_prole_home_fixes, silent_install_test, final_test

12. Documentation updates
    - Updated docs/: DOCKER-BUILD-FIX, PROLE-CFG-SECRETS, PROLE-HOME-DIRECTORY,
      build-system, patent
    - Updated scan/network_description.txt
    - Updated pom.xml

13. Miscellaneous script updates
    - Updated root-level: _adopt_replica_pvcs, _fix_replica_merlin, _import_pi,
      _patch_cluster, _prebind_pvcs, _rebind_d002, _rebind_d002b, test_resolve
    - Updated scripts/generate_spec.py

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 22:16:21 -07:00
chrisfu
4ee2b259c9 Checkpoint: rename installer to knoe + harden db build context
- Add build-context helper to copy Docker context safely (ignore runtime data, keep symlinks)

- Update UI and core actions to use ~/.prole/build and shared copy helper

- Add/adjust tests and scripts; introduce knoe ops helpers and update manifests

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 01:45:21 -07:00
chrisfu
7efb2331f6 checkpoint: service env config + installer updates
- Expand generated service config (conf/service/prole.cfg) with captured inputs and derived vars

- Update installer core (actions/controller/env) and UI screens (services/argocd)

- Update monitoring init script and port mapping

- Update k8s prole-db manifests; add monitoring PV + StorageClass

- Add dev/test config templates and postgresql version file

- Update supabase generated helm values; bump prole-db version

- Extend installer tests incl. ArgoCD render smoke
2026-03-17 09:29:08 -07:00
chrisfu
ce5ff83eb4 checkpoint: improve init scripts, installer flows, and kube context handling
- Fix kube context switching for k3s single-context kubeconfigs and k3d shorthand prefixes

- Update common init/status scripts (registry, kerberos, cnpg backup, service layer, common services)

- Add Gitea init script and installer ArgoCD screen

- Add Supabase realtime probe patching plus regression tests

- Extend installer core/UI test coverage
2026-03-14 20:07:54 -07:00
chrisfu
92e4101403 installer: preflight DB secrets + bootstrap OpenTofu password
- Materialize required DB Kubernetes secrets from user-entered password before CNPG init

- Auto-generate a temporary OpenTofu admin password in non-interactive runs when none is available

- Update installer UI flows and tests to cover the new preflights/rotation

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-13 16:38:38 -07:00
chrisfu
b905cbe584 Bootstrap env-based prole.cfg entrypoint (safe legacy migration)
- Add centralized config resolver/activator with per-environment layering and stable symlink entrypoint\n- Bootstrap missing env dirs and preserve non-symlink legacy prole.cfg by seeding into inferred env\n- Wire installer UI/backend + shell helpers to shared resolution path for explicit, safe env switching\n- Harden k3d: registry network/DNS wiring and ArgoCD repo-server hostPath permission init\n- Add docs + regression tests for env switching, namespace stability, k3d registry, and ArgoCD rollout

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-13 12:23:15 -07:00
chrisfu
caa2685fc2 Fix installer path portability (avoid hardcoded /Users paths)
- Rewrite stale foreign home prefixes when loading env defaults (macOS <-> Linux)

- Write /Users/chrisfu-relative values to env.sh; sanitize prole.cfg paths to use ${HOME}

- Remove remaining /Users/chrisfu defaults from helper scripts and OpenTofu kubeconfig path

- Make installer tests more reliable by forcing repo root to front of sys.path

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-09 16:55:49 -07:00
chrisfu
d397b3000e Fix installer ctx apply flow and registry namespace
- Add explicit Apply button for kubectl context switching to avoid half-applied changes

- Allow kubeconfig/context-based auth without requiring K3S_TOKEN when kubeconfig is valid

- Prompt cleanup/reset of previous Common Core services namespace to prevent resource collisions

- Remove hardcoded 'common-services' registry namespace; default registry deploy/check to SERVICE_NAMESPACE/REGISTRY_NAMESPACE

- Update mocks and add regression tests for namespace resolution and installer behavior
2026-03-07 10:35:27 -08:00
chrisfu
436c6214df checkpoint: k3s agent config + node labels + registry optimizations
- k3s: render server-only config keys only for servers; add regression test; make guardrail test non-sudo

- k3s/site: add post-provision node labeling tasks + playbook tier; extend inventory node label mapping

- registry: default namespace to common-services and skip redundant image pushes by probing registry manifests (installer + init scripts)
2026-03-07 03:14:26 -08:00
chrisfu
125da5669b Fix k3s CNPG image registry resolution and import flows
- In k3s mode, resolve CNPG images to a cluster-reachable internal registry and ignore k3d/localhost-ish registry values.

- Avoid any k3d cluster/containerd interactions when PROLE_MODE=k3s; add a preflight-image action.

- Extend init scripts and Ansible k3s/mariadb import tasks/playbooks for the updated k3s flow.

- Add/extend installer UI + tests around services and action/milestone helpers.

- Add merlin MariaDB provisioning playbook, services init-script test, and a k3s datastore SQL snapshot.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-07 00:23:00 -08:00
chrisfu
f20df131df Fix Percona PG 18 build and update installer configuration.
- Dockerfile: Resolved interactive tzdata prompts with debconf and fixed Percona package names (percona-pg-stat-monitor18 and percona-postgresql-contrib).

- Configuration: Updated prole.cfg with specific namespace (prole-hq0-db0), user (chrisfu), and encrypted secrets.

- Installer: Enhanced initialization scripts (init_openbao.sh, init_certmgr.sh) and updated service layers for cluster deployment.

- Tests: Added coverage for dependent image collection in installer core.

- Version: Bumped prole-db version to 77.

Co-authored-by: Junie <junie@jetbrains.com>
2026-02-28 23:11:32 -08:00
chrisfu
dafed53810 Enhance build system with 'make test' and fix installer test debris. Added 'make test' to Makefile with 'pyconv' (black) integration and coverage summary. Reformatted codebase with black. Fixed 'install.py' test bug where MagicMock objects created directory debris by improving mocking and patching in 'tests/test_navigation.py', 'tests/test_service_layer_navigation.py', and 'tests/test_install_logic.py'. 2026-02-25 14:55:05 -08:00
chrisfu
1573bb59c2 Add prole.spec and extend OpenTofu k3s configuration
- Include new `prole.spec` for build configurations and dependencies.
- Add Terraform state handling for OpenTofu in `k3s` cluster.
- Provision multiple Kubernetes resources in `prole-db` namespace: namespace, services, ConfigMaps, StatefulSets, Ingress rules, and PersistentVolumes.
- Integrate deployment and configuration enhancements for `garage`, `prole`, and related components.
2026-02-24 21:47:02 -08:00