Commit Graph

21 Commits

Author SHA1 Message Date
chrisfu
e3c2e625f1 refactor(config): separate k3d k3s and gke config entrypoints
Rename env config files from conf/*/prole.cfg to conf/k3d.cfg, conf/k3s.cfg, and conf/gke.cfg. Update shell/Python loaders and etc/deploy scripts to resolve named configs cleanly while keeping legacy fallback behavior. Align k3s Ansible tasks, docs, and regression coverage with the new configuration layout.

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-11 22:20:45 -07:00
chrisfu
3312c39b1f feat: GCP/GKE CNPG hardening, Artifact Registry traffic light, and knoe-system namespace fixes
UI screens
- database.py: fix mode detection to use env_key priority (prod→k8s, service→k3s) so stale DEPLOYMENT_MODE never overrides the user's chosen environment
- database.py: Registry status reads ARTIFACT_REGISTRY_AVAILABLE persisted by cluster screen; uses SERVICE_NAMESPACE for Artifact Registry repo name
- cluster.py: add Artifact Registry traffic light (amber→green/red) to prod section; _check_artifact_registry_async persists ARTIFACT_REGISTRY_AVAILABLE into Global cfg
- cluster.py: re-trigger Artifact Registry check after GKE cluster selection so the light re-evaluates once region is available from KUBECONTEXT
- cluster_nodes.py: fix TclError on Python 3.14 — pady=(2,0) tuple → pady=2 scalar
- __init__.py: seed knoe-system namespace when saved value is "default", not only when empty
- services.py: replace hardcoded "Prole DB" log string with dynamic cnpg_cluster name

Core ops
- cloudnative_pg.py: replace one-shot Barman plugin retry with 6-attempt loop; first cert-manager/x509 failure triggers rollout restart + 30 s CA propagation wait; subsequent failures back off up to 60 s per attempt
- cloudnative_pg.py: TLS CA CN now uses cluster_name instead of hardcoded "Prole CNPG CA"
- registry.py, garage_store.py: refactored into per-mode modules (k3d/k3s/k8s registry and garage store, shared _garage_common)

Deploy / config
- deploy/gcp/gke/knoe-db.yaml: GKE-specific CNPG cluster manifest (rw/ro/r on separate nodes with premium-rwo storage)
- etc/init_common_services.sh, modes/k8s/knoe-db/.version: updated for current deploy
- kong-deployment.yaml: updated manifest

Tests
- test_cluster_nodes_render_smoke.py: add pack/grid, winfo_children, winfo_reqheight, update_idletasks, grid_slaves to dummy widgets; monkeypatch tk.Label so CNPG placement render completes without a real Tkinter root

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-04 19:36:08 -07:00
chrisfu
d02aadae89 Add node management workflow and installer config persistence updates
- add tools/manage-node.sh for k3s node add/remove automation with inventory reconciliation

- include kubectl drain flag compatibility and --ssh-host targeting for uninstall workflows

- persist installer cfg/env values and update deploy/database/cfg UI handling

- refresh related inventory/network artifacts and expand installer/database test coverage

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-29 11:22:33 -07:00
chrisfu
a069989315 Rename prole-db to knoe-db, add knoe-auth as cluster-internal KDC
Itemized changes:

1. knoe-auth: New cluster-internal KDC and SSO gateway service
   - Created etc/init_knoe_auth.sh based on init_kdc.sh with knoe-auth naming
   - Namespace defaults to SERVICE_NAMESPACE (knoe-system)
   - ConfigMap: knoe-auth-kdc-config, Secret: knoe-auth-secrets
   - Legacy cleanup removes old auth/dog/authority deployments

2. Orchestration: knoe-auth initializes before CloudNativePG
   - Updated prole.sh to insert init_knoe_auth.sh as step 2 (before CNPG)
   - Renumbered all subsequent initialization steps

3. Kong routing: Updated init_kong.sh to route to knoe-auth in SERVICE_NAMESPACE

4. Comment/reference updates for knoe-auth
   - Updated init_common_services.sh, init_service_layer.sh, init_kerberos.sh

5. prole-db renamed to knoe-db across the entire codebase
   - Renamed prole-db/ directory to knoe-db/
   - Renamed all prole-db Kubernetes manifests (deploy/opentofu, k8s/)
   - Renamed scripts: docker-root-knoe-db.sh, docker-run-knoe-db.sh, test-cnpg-knoe-db.sh
   - Renamed etc/init_prole-db-reset.sh to etc/init_knoe-db-reset.sh
   - Renamed etc/prole-db-passwwd.sh to etc/knoe-db-passwwd.sh
   - Renamed mock_val counterparts accordingly
   - Renamed tests/etc/test_init_prole-db-reset.sh to test_init_knoe-db-reset.sh
   - Renamed docs/prole-db-documentation-mcp-architecture.md to knoe-db variant
   - Renamed modes/k3d/prole-db/ to modes/k3d/knoe-db/
   - Renamed prole-db.iml to knoe-db.iml

6. Configuration updates
   - Updated conf/dev, conf/prod, conf/test, conf/service prole.cfg files
   - Updated conf/port-mapping.cfg
   - Updated etc/prole_cfg.sh and mock_val/prole_cfg.sh
   - Updated service/prole.cfg

7. Kubernetes manifests and deploy configuration
   - Updated deploy/opentofu/k3s ArgoCD application YAMLs
   - Updated kong-configmap.yaml and kustomization.yaml
   - Updated k3s/kong-config.yml and prole-resources.yaml
   - Updated prole-mssql-db deployment YAMLs
   - Updated supabase helm render and deploy scripts

8. Infrastructure and GCP Terraform
   - Updated deploy/gcp/terraform: folders, groups, IAM, service-projects

9. Python/installer code updates
   - Updated knoe/core: actions, build_context, controller, env, milestones
   - Updated knoe/milestone.py
   - Updated knoe/ui/screens: cfg, database, database_options, deploy, docker,
     navigation, security, services, validate
   - Updated knoe.spec, status.py

10. Shell script updates
    - Updated etc/: build_db, init_cloudnative_pg, init_cnpg_backup,
      init_db_manager, init_forgejo, init_gitlab, init_monitoring, init_openbao,
      init_port_forwards, init_postgrest, init_supabase_ports, status
    - Updated mock_val/ counterparts for all above scripts
    - Updated prole-net/init-prole-dns.sh
    - Updated bin/prole-kpf.sh, gitea/deploy.sh, supabase/deploy.sh

11. Test updates
    - Updated tests/etc/: test_init_cloudnative_pg*, test_init_cnpg_backup*,
      test_init_kdc*, test_init_kerberos*, test_init_kong*, test_prole_cfg*
    - Updated tests/installer/: test_actions_helpers, test_cfg_save_kubecontext,
      test_controller, test_core_classes, test_milestones, test_milestones_extended,
      test_namespace_propagation
    - Updated tests/: test_database_options, test_navigation,
      test_render_supabase_hostname, test_docker_build_fix,
      test_all_prole_home_fixes, silent_install_test, final_test

12. Documentation updates
    - Updated docs/: DOCKER-BUILD-FIX, PROLE-CFG-SECRETS, PROLE-HOME-DIRECTORY,
      build-system, patent
    - Updated scan/network_description.txt
    - Updated pom.xml

13. Miscellaneous script updates
    - Updated root-level: _adopt_replica_pvcs, _fix_replica_merlin, _import_pi,
      _patch_cluster, _prebind_pvcs, _rebind_d002, _rebind_d002b, test_resolve
    - Updated scripts/generate_spec.py

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 22:16:21 -07:00
chrisfu
0262bd4be2 Refactor to support mode-scoped DB build contexts and versioning
- Introduce mode-scoped directories under `~/.prole` for build contexts, runtime data, and versioning.
- Update DB image version generation and Dockerfile creation to respect deployment modes (e.g., k3d, k3s, k8s).
- Adjust UI, core actions, and scripts to avoid cross-mode conflicts and ensure correct version isolation.
- Add tests for mode-aware behaviors and update existing tests for new paths and markers.
- Bump PostgreSQL and image versions to `18` and `138`, respectively.
- Update Kubernetes manifests to align with new image and version configuration.
2026-03-22 05:47:50 -07:00
chrisfu
1353882260 Update PROLE_HOME resolution and enhance iSCSI PV handling
- Implement centralized `resolve_prole_home` utility for consistent environment-based `PROLE_HOME` resolution across modules
- Replace hardcoded home paths with `resolve_prole_home`
- Refactor PV management to support iSCSI mounts and node placement from Ansible manifests
- Improve Kubernetes manifest handling to dynamically apply namespaces per document
- Adjust `knoe-db` build context path and related tests
- Add utilities for detecting and applying Ansible-defined node labels and PVs
2026-03-22 02:43:52 -07:00
chrisfu
4ee2b259c9 Checkpoint: rename installer to knoe + harden db build context
- Add build-context helper to copy Docker context safely (ignore runtime data, keep symlinks)

- Update UI and core actions to use ~/.prole/build and shared copy helper

- Add/adjust tests and scripts; introduce knoe ops helpers and update manifests

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 01:45:21 -07:00
chrisfu
ecacf01266 Remove legacy secrets, configs, and scripts; introduce prole-auth service and Grafana SSO proxy configuration. 2026-03-20 13:00:36 -07:00
chrisfu
de7bb8d0ff Refactor shell library structure; relocate helpers under etc/lib/shell/ and update references 2026-03-16 21:44:58 -07:00
chrisfu
5dcd3b9581 ansible: treat /var/lib/rancher as local K3s state
Update inventory/roles to keep K3s state on local storage and prevent iSCSI from managing /var/lib/rancher.

Also add a single-host k3s install playbook, a systemd override template, and docs describing the installer ↔ Ansible boundary and slow-storage knobs.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-15 23:20:55 -07:00
chrisfu
b905cbe584 Bootstrap env-based prole.cfg entrypoint (safe legacy migration)
- Add centralized config resolver/activator with per-environment layering and stable symlink entrypoint\n- Bootstrap missing env dirs and preserve non-symlink legacy prole.cfg by seeding into inferred env\n- Wire installer UI/backend + shell helpers to shared resolution path for explicit, safe env switching\n- Harden k3d: registry network/DNS wiring and ArgoCD repo-server hostPath permission init\n- Add docs + regression tests for env switching, namespace stability, k3d registry, and ArgoCD rollout

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-13 12:23:15 -07:00
chrisfu
909e92109e Refactor shell layout; add shellspec + authority
- Introduce lib/shell helpers and keep etc/* scripts thin via compatibility shims

- Move Kerberos validation to scripts/validation/check_kerberos.sh and update callers

- Add deterministic shellspec unit tests under tests/shellspec/ and wire Maven to run them

- Add minimal Spring Boot authority module with startup + /health endpoint and Maven wiring

- Document the new layout in docs/layout.md

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-13 09:19:56 -07:00
chrisfu
bd54755218 remove owner/group/mode enforcement for external mountpoint in Ansible tasks
- Update documentation to note `/external` may already be mounted before Ansible runs.
- Avoid redundant ownership and permissions enforcement in `mariadb_primary` role.
2026-03-13 02:15:05 -07:00
chrisfu
17cb827be5 test(dashboard): add unit tests for systemd unit and script validation
- Ensure dashboard service doesn't create an ordering cycle with `multi-user.target`.
- Verify service safely attaches to a virtual terminal (VT) and passes VT to Xorg.
- Ensure no `su` usage by default in the service.
- Validate script behavior for passing VT and `-keeptty` options to Xorg.

chore(dashboard): update systemd unit to avoid VT conflicts and improve logging

- Remove `After` and `Wants` directives for `multi-user.target` to prevent conflicts.
- Add `TTYPath`, `StandardInput`, and related options for VT attachment.
- Ensure `ExecStopPost` restores getty service for the dashboard VT.

fix(dashboard): improve Xorg resolution and clean up script logic

- Dynamically determine Xorg executable location in `dashboard.sh`.
- Pass VT, display, and `-keeptty` to Xorg to improve script reliability.

fix(mariadb): correct exFAT mount options for external storage

- Avoid `chown` failures by setting root ownership with group access for `mysql`.
- Update external mount options to `uid=0,gid=mysql,umask=0002`.

cleanup(mariadb): remove unnecessary owner/group permissions from migration marker task
2026-03-13 01:49:57 -07:00
chrisfu
3c2a2d23a1 feat(mariadb): add temporary external storage support for Merlin
- Enable mounting borrowed USB disk (exFAT) persistently at `/external`.
- Migrate MariaDB datadir to `/external/mariadb` with bind mount at `/srv/mariadb/mariadb`.
- Add migration marker and rollback support to prevent accidental remigration.
- Include Ansible tasks for external storage setup and migration validation.
2026-03-13 00:44:08 -07:00
chrisfu
f2c9012cce Refactor installation and initialization logic, and expand test coverage
- install.py: Major update including configuration variable expansion, improved k3s/k3d handling, and enhanced installation logic.

- etc/ scripts: Significant refactoring of initialization scripts (Kerberos, Port Forwards, Garage Store, etc.).

- Port Forwards: Transitioned from XML to port-mappings.conf for managing kubectl port-forwards.

- Status Reporting: Improved status checking for common services.

- Infrastructure: Updated Ansible inventory and rsyslog role configurations.

- Tests: Added a comprehensive suite of tests for 'etc' initialization scripts in prole/tests/etc/.

- Documentation: Added prole-db-documentation-mcp-architecture.md.

- General: Updated Dockerfiles and various helper scripts.
2026-02-13 21:36:39 -08:00
chrisfu
bcc8f23a0d Enhance secret management and k8s infrastructure
- Secret Management: Integrated AESGCM for temporary secret handling in install.py and enhanced OpenBao (Vault) support with namespace injection and additional secret paths (Grafana, Kerberos, TDE).
- Infrastructure & K8s:
    - Added Barman Object Store backup configuration (S3) to prole-db.yaml.
    - Updated Prometheus deployment with PVC and persistent configuration.
    - Updated k3s cluster/registry creation scripts.
    - Added etc/build-a-bao.sh for OpenBao setup.
- MSSQL Integration: Updated docker scripts and k8s deployments for Prole MSSQL database.
- Documentation: Added docs/PROLE-CFG-SECRETS.md explaining the new secret handling.
- General: Refined initialization scripts (init_authority.sh, init_openbao.sh, etc.) and updated the ncurses installer.
2026-02-03 22:39:50 -08:00
chrisfu
fff18fdbe4 Update Prole-DB and improve Supabase integration
- Bumped Prole-DB image version to 17.7-053 in scripts, Dockerfile, and manifests.
- Replaced `prole-scan` with `prole-agent` throughout scripts and tests.
- Refined Kubernetes setup for Supabase to use namespace 'supabase'.
- Introduced conversion of Supabase Docker Compose to Kubernetes manifests with `kompose`.
- Added support for Kerberos toggle via environment variables in `init_kerberos.sh`.
- Improved error handling and logging in scripts for better maintainability.
2026-02-01 23:56:25 -08:00
chrisfu
dda938ee95 Integrate namespace setup and management into scripts, Dockerfile, and update installer UI. 2026-01-28 23:02:50 -08:00
chrisfu
cbfe930b78 feat: add ncurses interface, build system, and embedded resources
Major feature additions and infrastructure improvements for the Prole
Database Installer, enabling command-line operation and packaged binary
distribution.

## Ncurses Terminal Interface

- Add installer/ncurses_ui.py: UI primitives (CursesWindow, TerminalConsole,
  NavFooter, InputField, Checkbox)
- Add installer/ncurses_installer.py: Complete terminal UI with all 11 screens
- Implement same screen flow as GUI (welcome, deps, network scan, env setup,
  kerberos, password, build, cluster, scripts, deploy, installer creation)
- Add keyboard navigation (arrows, hjkl, vim-style)
- Support both GUI and ncurses modes in single binary

## Automatic Display Detection

- Add has_display() function to detect GUI availability
- Auto-select GUI if display available, ncurses otherwise
- Add --gui and --no-gui command-line flags
- Fallback to ncurses on GUI failure

## Build System and Packaging

- Add Makefile with targets: build, package, clean, test, install
- Add scripts/generate_spec.py: PyInstaller spec generator
- Add installer.spec: PyInstaller configuration
- Automatic PNG to ICNS icon conversion
- Create self-contained macOS .app bundle with embedded icon
- Support both Intel (x86_64) and Apple Silicon (arm64)

## Embedded Resources

- Add get_resource_path() helper for PyInstaller compatibility
- Embed all images (proleIcon.png, proleLogo.png, proleLogoSepia.png)
- Embed prole-net/prole-scan binary (6.8 MB universal binary)
- Embed prole-app/dist/Prole Tools.app (12 MB app bundle)
- Embed prole-db/ Docker build context

## Writable Directory Fixes

- Create ~/.prole/build/prole-db/ for Docker builds (fixes read-only _MEIPASS)
- Create ~/.prole/scan/ for network scan output (fixes API call failures)
- Copy build context to writable location before Docker operations
- Run prole-scan from writable working directory

## Documentation

- docs/build-system.md: Complete build system guide
- docs/ncurses-installer.md: Ncurses interface documentation
- docs/RELEASE-NOTES.md: Feature overview and release notes
- docs/IMAGE-RESOURCES.md: Image resource management
- docs/EMBEDDED-RESOURCES.md: Binary and app bundle embedding
- docs/DOCKER-BUILD-FIX.md: Docker build hang solution
- docs/PROLE-HOME-DIRECTORY.md: ~/.prole directory structure
- BUILD.md: Quick build reference

## Key Changes

install.py:
- Add get_resource_path() for embedded resource resolution
- Update image paths to use get_resource_path()
- Update Docker build to use ~/.prole/build/prole-db/
- Update network scan to use ~/.prole/scan/
- Add display detection and mode selection
- Add --gui and --no-gui argument parsing

## Testing

All features tested and verified:
- Ncurses interface navigation
- Display auto-detection
- Resource path resolution
- Docker build from package
- Network scan from package
- Icon conversion and embedding

Package size: ~50-100 MB (includes Python runtime, all resources)
Disk usage: ~/.prole/ uses ~2-6 MB

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-19 23:10:37 -08:00
chrisfu
92fe6fba95 docs and history. UE Dockerfile 2025-12-02 18:42:00 -08:00