Commit Graph

29 Commits

Author SHA1 Message Date
chrisfu
de7bb8d0ff Refactor shell library structure; relocate helpers under etc/lib/shell/ and update references 2026-03-16 21:44:58 -07:00
chrisfu
f051d1d42a iscsi/installer: provision merlin d004; fix etc script runtime
- Add merlin iSCSI config for PROLE-DATA-4 mounted at /prole/d004 (xfs, _netdev,noatime)

- Refine iscsi role login/mount flow (device resolution, mkfs-if-missing, UUID fstab, tags)

- Ensure installer run_script stages lib/shell into PROLE_HOME so etc scripts can source common libs; add regression test

- Add init scripts for Forgejo/GitLab; ignore generated conf/prole.cfg

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-16 21:30:52 -07:00
chrisfu
a445c77233 checkpoint: installer env var expansion + k3s registry + CNPG backups
- Expand nested $VAR/${VAR} (and ~) when resolving PROLE_* paths in installer core/UI to avoid literal $HOME dirs\n- Improve k3s registry host:port derivation (explicit cfg/env, server URL, kubeconfig fallback) and registry status UX\n- Update CNPG backup/init scripts for plugin-based backups + ScheduledBackup; improve plugin socket readiness and status output\n- Add/extend tests covering env var expansion, registry hostport resolution, and CNPG backup behavior
2026-03-14 22:55:32 -07:00
chrisfu
ce5ff83eb4 checkpoint: improve init scripts, installer flows, and kube context handling
- Fix kube context switching for k3s single-context kubeconfigs and k3d shorthand prefixes

- Update common init/status scripts (registry, kerberos, cnpg backup, service layer, common services)

- Add Gitea init script and installer ArgoCD screen

- Add Supabase realtime probe patching plus regression tests

- Extend installer core/UI test coverage
2026-03-14 20:07:54 -07:00
chrisfu
92e4101403 installer: preflight DB secrets + bootstrap OpenTofu password
- Materialize required DB Kubernetes secrets from user-entered password before CNPG init

- Auto-generate a temporary OpenTofu admin password in non-interactive runs when none is available

- Update installer UI flows and tests to cover the new preflights/rotation

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-13 16:38:38 -07:00
chrisfu
b905cbe584 Bootstrap env-based prole.cfg entrypoint (safe legacy migration)
- Add centralized config resolver/activator with per-environment layering and stable symlink entrypoint\n- Bootstrap missing env dirs and preserve non-symlink legacy prole.cfg by seeding into inferred env\n- Wire installer UI/backend + shell helpers to shared resolution path for explicit, safe env switching\n- Harden k3d: registry network/DNS wiring and ArgoCD repo-server hostPath permission init\n- Add docs + regression tests for env switching, namespace stability, k3d registry, and ArgoCD rollout

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-13 12:23:15 -07:00
chrisfu
caa2685fc2 Fix installer path portability (avoid hardcoded /Users paths)
- Rewrite stale foreign home prefixes when loading env defaults (macOS <-> Linux)

- Write /Users/chrisfu-relative values to env.sh; sanitize prole.cfg paths to use ${HOME}

- Remove remaining /Users/chrisfu defaults from helper scripts and OpenTofu kubeconfig path

- Make installer tests more reliable by forcing repo root to front of sys.path

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-09 16:55:49 -07:00
chrisfu
e7650427e5 Refactor dependency resolution to replace bash -lc calls with shutil.which and improve robustness
- Eliminated reliance on `bash -lc` for dependency lookups, using `shutil.which` for better cross-platform support.
- Added `_resolve_brew_bin` fallback logic to handle specific Homebrew scenarios.
- Updated tests to align with the new streamlined lookup mechanism.
- Refactored subprocess checks to respect augmented environment configurations and reduce PATH issues.
2026-03-08 22:35:15 -07:00
chrisfu
5fbf03eaae Remove obsolete configurations and improve dependency handling
- Deleted `prole.cfg`, `port-mapping.cfg`, and PostgreSQL version file as part of configuration cleanup.
- Enhanced Homebrew dependency resolution with `_resolve_brew_bin` and `_augment_env_for_brew` methods to manage PATH automatically.
- Updated subprocess calls to respect augmented environment configurations.
- Added unit tests for Homebrew PATH augmentation and dependency validation logic.
2026-03-08 22:11:04 -07:00
chrisfu
396cea60e7 installer: stabilize retries + improve k3s inventory detection
- Reuse namespace from existing prole.cfg unless env overrides

- Support k3s_hosts children groups and prefer k3s_servers as default

- Add headless pytest tkinter stubs and blocked-cluster reconciliation tests

- Update registry mirror endpoint, port mappings, and prole-db manifests
2026-03-07 23:57:21 -08:00
chrisfu
d397b3000e Fix installer ctx apply flow and registry namespace
- Add explicit Apply button for kubectl context switching to avoid half-applied changes

- Allow kubeconfig/context-based auth without requiring K3S_TOKEN when kubeconfig is valid

- Prompt cleanup/reset of previous Common Core services namespace to prevent resource collisions

- Remove hardcoded 'common-services' registry namespace; default registry deploy/check to SERVICE_NAMESPACE/REGISTRY_NAMESPACE

- Update mocks and add regression tests for namespace resolution and installer behavior
2026-03-07 10:35:27 -08:00
chrisfu
436c6214df checkpoint: k3s agent config + node labels + registry optimizations
- k3s: render server-only config keys only for servers; add regression test; make guardrail test non-sudo

- k3s/site: add post-provision node labeling tasks + playbook tier; extend inventory node label mapping

- registry: default namespace to common-services and skip redundant image pushes by probing registry manifests (installer + init scripts)
2026-03-07 03:14:26 -08:00
chrisfu
125da5669b Fix k3s CNPG image registry resolution and import flows
- In k3s mode, resolve CNPG images to a cluster-reachable internal registry and ignore k3d/localhost-ish registry values.

- Avoid any k3d cluster/containerd interactions when PROLE_MODE=k3s; add a preflight-image action.

- Extend init scripts and Ansible k3s/mariadb import tasks/playbooks for the updated k3s flow.

- Add/extend installer UI + tests around services and action/milestone helpers.

- Add merlin MariaDB provisioning playbook, services init-script test, and a k3s datastore SQL snapshot.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-07 00:23:00 -08:00
chrisfu
6dc3560dcf refactor: rename backup script to etc/init_cnpg_backup.sh and optimize Supabase image prefetch
- Renamed etc/init_prole-db-backup.sh to etc/init_cnpg_backup.sh and updated all references.

- Enhanced CNPG backup script with authoritative status validation.

- Implemented SupabaseImagePreloadMilestone to front-load heavy image transfers.

- Updated supabase/deploy.sh with --prefetch-images-only and --skip-prefetch.

- Updated unit tests for CNPG backups and Supabase silent milestone sequencing.

- Included incidental environment updates from installer execution.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-02 00:34:01 -08:00
chrisfu
5618b662dd Remove prole-db-manager; simplify deployment via prole-authority; fix pg18 downgrade & cluster name
Summary:
Removed the prole-db-manager microservice and simplified deployment to use
prole-authority as the internal management and authorization point. Fixed two
blocking bugs that prevented silent install from completing on knoe-dev-cluster.

Removed: prole-db-manager
- Deleted db-manager-deployment.yaml and db-manager-service.yaml from opentofu manifests
- Deleted src/db-manager/ (Dockerfile, server.js, package.json, tests)
- Removed prole-db-manager port-forward mapping from installer/core/env.py
- Removed init_db_manager.sh from Initialization Scripts (milestones.py, actions.py)
- Removed init_certmgr.sh and init_db_manager.sh tabs from services screen (services.py)
- Removed live k8s Deployment/Service from knoe-dev-cluster

Fixed: PostgreSQL version downgrade error (pg17 -> pg18)
- Created conf/postgresql/.version with value 18
- Updated k8s/prole/prole-db.yaml and prole-db-recovery.yaml.tpl imageName to prole-db:18-089
- Fixed _init_database_options_state() to restore saved version_type from prole.cfg
  so db_version_type defaults to v18 (pg18) instead of silently reverting to pg17
- Added database_options.* keys to _collect_input_snapshot() in cfg.py so
  distribution, version_type, and all extension toggles persist to prole.cfg

Fixed: Cluster name inconsistency
- Removed stale prole-dev-cluster references; all scripts now use knoe-dev-cluster
- Added knoe-dev-cluster to mode-detection case in etc/prole_cfg.sh

Config: conf/prole.cfg
- Set kerberos_config.enabled = False, KERBEROS_AUTO_ENABLED = False
- Added database_options.distribution = percona, version_type = v18
- Added all 13 extension flags set to True (postgis, pgvector, pgcrypto, pgaudit,
  pg_repack, pg_stat_statements, pg_buffercache, pg_freespacemap, pgrowlocks,
  postgres_fdw, dblink, pg_stat_monitor, pgbadger)

Verification:
./install.py -s -l -v -c conf/prole.cfg completed successfully.
CNPG deployed prole-db:18-089 to knoe-dev-cluster; all milestones passed.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-01 20:40:44 -08:00
chrisfu
46f31843c2 Fix Kerberos test flow and improve multi-namespace support in k3s
- Resolve namespace drift in Kerberos test flow and OpenBao deployment.
- Fix OpenAPI credential-validation failures in etc scripts by using --validate=false with kubectl.
- Introduce preferred_service_namespace() to accurately target namespaces for KDC policy/test paths.
- Default Kerberos test user to 'administrator' and prefill from Ansible vault secrets.
- Generalized OpenBao deployment logic to correctly handle multi-namespace environments.
- Added regression test 'tests/etc/test_init_kerberos_k3s_real_kdc.sh'.
- Synchronized SERVICE_NAMESPACE extraction in prole_cfg.sh.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-01 00:57:10 -08:00
chrisfu
f20df131df Fix Percona PG 18 build and update installer configuration.
- Dockerfile: Resolved interactive tzdata prompts with debconf and fixed Percona package names (percona-pg-stat-monitor18 and percona-postgresql-contrib).

- Configuration: Updated prole.cfg with specific namespace (prole-hq0-db0), user (chrisfu), and encrypted secrets.

- Installer: Enhanced initialization scripts (init_openbao.sh, init_certmgr.sh) and updated service layers for cluster deployment.

- Tests: Added coverage for dependent image collection in installer core.

- Version: Bumped prole-db version to 77.

Co-authored-by: Junie <junie@jetbrains.com>
2026-02-28 23:11:32 -08:00
chrisfu
a40496f128 Implement real-time command executor for network scan: added run_streaming_cmd for unbuffered output, integrated with NetworkScreen UI, and added unit tests.
Co-authored-by: Junie <junie@jetbrains.com>
2026-02-28 17:58:58 -08:00
chrisfu
765aa0b926 Checkpoint: cluster env UI layout + k3s common services
- Tighten Cluster Environment screen layout; switch Service/Prod to kubectx context selection; keep namespace and key controls on one line; ensure Repair button remains reachable.

- Add UI layout regression test to render with large mock data and assert key widgets remain visible and console is scrollable.

- Make kubeconfig generation deterministic under tests by avoiding overwriting cert-based kubeconfigs; write token sidecar kubeconfig when needed.

- Update common-services init scripts and add k3s/Helm deployment bits (svc-check, Kong/CertMgr tasks).
2026-02-27 14:35:14 -08:00
chrisfu
dafed53810 Enhance build system with 'make test' and fix installer test debris. Added 'make test' to Makefile with 'pyconv' (black) integration and coverage summary. Reformatted codebase with black. Fixed 'install.py' test bug where MagicMock objects created directory debris by improving mocking and patching in 'tests/test_navigation.py', 'tests/test_service_layer_navigation.py', and 'tests/test_install_logic.py'. 2026-02-25 14:55:05 -08:00
chrisfu
1573bb59c2 Add prole.spec and extend OpenTofu k3s configuration
- Include new `prole.spec` for build configurations and dependencies.
- Add Terraform state handling for OpenTofu in `k3s` cluster.
- Provision multiple Kubernetes resources in `prole-db` namespace: namespace, services, ConfigMaps, StatefulSets, Ingress rules, and PersistentVolumes.
- Integrate deployment and configuration enhancements for `garage`, `prole`, and related components.
2026-02-24 21:47:02 -08:00
chrisfu
51c91595b9 Refactor infrastructure, installer, and deployment configs: expanded CloudNativePG and PostgREST init scripts, updated ArgoCD/OpenTofu/K8s manifests (ingress, kustomization, postgrest, prole-db), added installer config helpers and revised cluster UI, updated prole.cfg and prole.sh, simplified Makefile, added env helper tests, removed stale spec/log files, added prole/ directory 2026-02-23 21:19:19 -08:00
chrisfu
4bbd3b2dd3 fix: UI runtime fixes, idempotent k3d KUBECONFIG, Ollama screen redesign, mode-conditional init scripts, Grafana restart, real-time docker build output, shell script fixes, test updates 2026-02-20 23:19:48 -08:00
chrisfu
312fb27df5 test: add unit tests for core helpers, milestone base, controller, and build - 141 tests for env.py/config.py pure functions (bool, cluster env, ollama, port-forwards, yaml, render_prole_cfg, etc.) - 48 tests for state, milestone base class, controller, build, and milestone subclasses - Combined coverage improved from 15% to 20% (2414/12349 stmts) 2026-02-20 19:30:13 -08:00
chrisfu
ec50f82cc8 refactor: deduplicate installer business logic & split screens.py into package
Separation of concerns: merge silent/UI actions & modularize screens.

ProleInstallerBase (actions.py): Created shared base class with 37 deduplicated methods previously duplicated between ProleSilentInstaller and ProleInstaller. Namespace, environment, secret, deployment, port-forward, authority/repair, image, and logging helpers now defined once. Subclasses override _get_input() to bridge their data-access layers.

screens.py -> screens/ package (18 mixin modules): Split 10,234-line monolithic screens.py into focused mixin modules: base, navigation, welcome, dependencies, network, environment, database, cluster, services, security, ollama, supabase, docker, build, packaging, deploy, validate, cfg. __init__.py composes ProleInstaller from all mixins and re-exports has_display(), main() for full backward compatibility.

All 37 tests pass with no regressions.
2026-02-20 14:43:20 -08:00
chrisfu
ec8a4e98af k3s: Fix installation hangs, CA mismatches, and arm64 networking
- Implement proactive CA hash verification and automated repair for mismatches

- Ensure agents prioritize discovered server tokens over stale vault values

- Fix K3s service hangs with explicit stop and killall before reinstall

- Add Retropie/Pi networking fixes (WiFi power save, wlan0 priority)

- Pin pre-staged images to stable, architecture-aware versions (arm64)

- Remove obsolete init-port-forwards and prole.cfg sync tasks

- Update k8s manifests and installer core logic with new tests
2026-02-16 23:30:27 -08:00
chrisfu
15a60575ad remove background files 2026-01-18 16:14:09 -08:00
chrisfu
a4d0a2e0a3 Update prole-db docker tag to 17.7-037 and update pg_prolelog deb package 2026-01-15 11:58:39 -08:00
chrisfu
906392d462 feat(installer): improve UI and add test coverage for core features
- Refactored installer UI with updated canvas rendering, sidebar navigation, and footer buttons.
- Enhanced styling for macOS compatibility and consistent design across controls.
- Added Pytest-based unit tests for `screen.py` and `config.py`.
- Expanded dependency catalog with new tools like `tshark` and `pyshark`.
- Improved error tolerance for background rendering and added placeholders for Kerberos configuration.
2026-01-08 21:51:30 -08:00