Commit Graph

17 Commits

Author SHA1 Message Date
chrisfu
ce5ff83eb4 checkpoint: improve init scripts, installer flows, and kube context handling
- Fix kube context switching for k3s single-context kubeconfigs and k3d shorthand prefixes

- Update common init/status scripts (registry, kerberos, cnpg backup, service layer, common services)

- Add Gitea init script and installer ArgoCD screen

- Add Supabase realtime probe patching plus regression tests

- Extend installer core/UI test coverage
2026-03-14 20:07:54 -07:00
chrisfu
b905cbe584 Bootstrap env-based prole.cfg entrypoint (safe legacy migration)
- Add centralized config resolver/activator with per-environment layering and stable symlink entrypoint\n- Bootstrap missing env dirs and preserve non-symlink legacy prole.cfg by seeding into inferred env\n- Wire installer UI/backend + shell helpers to shared resolution path for explicit, safe env switching\n- Harden k3d: registry network/DNS wiring and ArgoCD repo-server hostPath permission init\n- Add docs + regression tests for env switching, namespace stability, k3d registry, and ArgoCD rollout

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-13 12:23:15 -07:00
chrisfu
d397b3000e Fix installer ctx apply flow and registry namespace
- Add explicit Apply button for kubectl context switching to avoid half-applied changes

- Allow kubeconfig/context-based auth without requiring K3S_TOKEN when kubeconfig is valid

- Prompt cleanup/reset of previous Common Core services namespace to prevent resource collisions

- Remove hardcoded 'common-services' registry namespace; default registry deploy/check to SERVICE_NAMESPACE/REGISTRY_NAMESPACE

- Update mocks and add regression tests for namespace resolution and installer behavior
2026-03-07 10:35:27 -08:00
chrisfu
436c6214df checkpoint: k3s agent config + node labels + registry optimizations
- k3s: render server-only config keys only for servers; add regression test; make guardrail test non-sudo

- k3s/site: add post-provision node labeling tasks + playbook tier; extend inventory node label mapping

- registry: default namespace to common-services and skip redundant image pushes by probing registry manifests (installer + init scripts)
2026-03-07 03:14:26 -08:00
chrisfu
125da5669b Fix k3s CNPG image registry resolution and import flows
- In k3s mode, resolve CNPG images to a cluster-reachable internal registry and ignore k3d/localhost-ish registry values.

- Avoid any k3d cluster/containerd interactions when PROLE_MODE=k3s; add a preflight-image action.

- Extend init scripts and Ansible k3s/mariadb import tasks/playbooks for the updated k3s flow.

- Add/extend installer UI + tests around services and action/milestone helpers.

- Add merlin MariaDB provisioning playbook, services init-script test, and a k3s datastore SQL snapshot.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-07 00:23:00 -08:00
chrisfu
494d9600b1 Repair: dedupe Kong from default namespace
- Detect/remove legacy prole-svc-kong resources (deploy/svc/configmap) in default during repair

- Ensure canonical Kong exists in configured SERVICE_NAMESPACE before cleanup

- Remove hardcoded knoe-system defaults; require non-default SERVICE_NAMESPACE in k3s

- Avoid silent installer hangs under CI/pytest by auto-generating DB master password

- Update tests for namespace/repair behavior and DB version defaults

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-05 18:38:43 -08:00
chrisfu
a43aed7134 k3s: eliminate localhost registry + dedupe common-core
- Ensure k3s mode uses the k3s registry endpoint and avoid localhost/k3d image prefixes.

- Make ArgoCD repo-server cmp symlink creation idempotent.

- Normalize common-core provisioning to knoe-system and add repair-time dedupe of stray default-namespace installs.

- Add k3s MariaDB datastore/refresh playbooks and regression tests.
2026-03-05 14:31:33 -08:00
chrisfu
5618b662dd Remove prole-db-manager; simplify deployment via prole-authority; fix pg18 downgrade & cluster name
Summary:
Removed the prole-db-manager microservice and simplified deployment to use
prole-authority as the internal management and authorization point. Fixed two
blocking bugs that prevented silent install from completing on knoe-dev-cluster.

Removed: prole-db-manager
- Deleted db-manager-deployment.yaml and db-manager-service.yaml from opentofu manifests
- Deleted src/db-manager/ (Dockerfile, server.js, package.json, tests)
- Removed prole-db-manager port-forward mapping from installer/core/env.py
- Removed init_db_manager.sh from Initialization Scripts (milestones.py, actions.py)
- Removed init_certmgr.sh and init_db_manager.sh tabs from services screen (services.py)
- Removed live k8s Deployment/Service from knoe-dev-cluster

Fixed: PostgreSQL version downgrade error (pg17 -> pg18)
- Created conf/postgresql/.version with value 18
- Updated k8s/prole/prole-db.yaml and prole-db-recovery.yaml.tpl imageName to prole-db:18-089
- Fixed _init_database_options_state() to restore saved version_type from prole.cfg
  so db_version_type defaults to v18 (pg18) instead of silently reverting to pg17
- Added database_options.* keys to _collect_input_snapshot() in cfg.py so
  distribution, version_type, and all extension toggles persist to prole.cfg

Fixed: Cluster name inconsistency
- Removed stale prole-dev-cluster references; all scripts now use knoe-dev-cluster
- Added knoe-dev-cluster to mode-detection case in etc/prole_cfg.sh

Config: conf/prole.cfg
- Set kerberos_config.enabled = False, KERBEROS_AUTO_ENABLED = False
- Added database_options.distribution = percona, version_type = v18
- Added all 13 extension flags set to True (postgis, pgvector, pgcrypto, pgaudit,
  pg_repack, pg_stat_statements, pg_buffercache, pg_freespacemap, pgrowlocks,
  postgres_fdw, dblink, pg_stat_monitor, pgbadger)

Verification:
./install.py -s -l -v -c conf/prole.cfg completed successfully.
CNPG deployed prole-db:18-089 to knoe-dev-cluster; all milestones passed.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-01 20:40:44 -08:00
chrisfu
f20df131df Fix Percona PG 18 build and update installer configuration.
- Dockerfile: Resolved interactive tzdata prompts with debconf and fixed Percona package names (percona-pg-stat-monitor18 and percona-postgresql-contrib).

- Configuration: Updated prole.cfg with specific namespace (prole-hq0-db0), user (chrisfu), and encrypted secrets.

- Installer: Enhanced initialization scripts (init_openbao.sh, init_certmgr.sh) and updated service layers for cluster deployment.

- Tests: Added coverage for dependent image collection in installer core.

- Version: Bumped prole-db version to 77.

Co-authored-by: Junie <junie@jetbrains.com>
2026-02-28 23:11:32 -08:00
chrisfu
b03efa8f69 Kong API gateway, docker-import preload, OpenTofu graceful fallback, milestone fix
Kong API Gateway (replacing prole nginx):
- Add etc/init_kong.sh provisioning script (DB-less mode, prole-db namespace)
- Add kong-deployment.yaml and kong-service.yaml manifests
- Rewire ingress rules (svc/git/api.prole.org) to prole-db-kong:8000
- Update kustomization.yaml to reference kong manifests

PostgREST & DB Manager in prole-db namespace:
- Add etc/init_postgrest.sh and etc/init_db_manager.sh scripts
- Add postgrest/db-manager deployment and service manifests
- Add src/db-manager/ Node.js REST endpoint for backup triggers
- Default NAMESPACE changed to prole-db in both scripts

Docker image pre-load from PROLE_DATA/docker-import:
- Add _preload_docker_images() to init_common_services.sh
- Scan for .tar files exported by final_deployment.sh
- Import via k3d image import (k3d) or ctr (k3s) before deployments
- Increase rollout timeouts to 300s (configurable via ROLLOUT_TIMEOUT) in init_openbao.sh, init_opentofu.sh, init_garage_store.sh, init_registry.sh

OpenTofu password resolution fix:
- Add Kubernetes secret fallback in resolve_admin_password()
- Change hard exit 1 to graceful return 1 with warning
- Wrap call in if-guard so set -e doesn't abort the script chain

Milestone fix (init scripts not running):
- Add init_kong.sh, init_postgrest.sh, init_db_manager.sh to InitializationScriptsMilestone.execute() script list and arg branches
- Previously only actions.py had these; milestones.py was missing them

Installer integration:
- Add Kong/PostgREST/DB Manager to silent installer _step_init_scripts
- Add corresponding tabs and execution blocks in UI services.py
2026-02-22 00:57:49 -08:00
chrisfu
6e2d3e9011 refactor: modernize installer and monitoring setup
- Monitoring: Migrated from manual Grafana/Prometheus manifests to kube-prometheus-stack based setup in etc/init_monitoring.sh. Removed old manifest files from deploy/ and k8s/.
- Installer Core: Refactored installer with new modules for actions, environment handling, and UI screens. Enhanced Milestone logic to support advanced configuration (ArgoCD, Registry namespaces, Kerberos flags, etc.).
- Service & Init Scripts: Updated multiple initialization scripts (init_*.sh) for better integration with OpenBao, Kerberos, and the new monitoring stack. Added new scripts for Nginx Ingress, Ollama parsing, and K3D route fixes.
- Infrastructure: Enhanced Samba AD DC Ansible role with realm derivation, provisioning guidance, and group management. Updated K3s role tasks.
- Configuration: Refined default settings in conf/ to align with the new deployment architecture.
- App & Tools: Updated prole-app Swift code and prole.sh for improved environment variable handling and installation flow.
2026-02-19 21:07:39 -08:00
chrisfu
ec2795f8a3 Checkpoint: Kerberos egress fixes, local registry provisioning for k3d/k3s, and installer GUI workflow enhancements. Key changes: AD forwarder port offsets, outbound NetworkPolicy, KDC dns_lookup disabling; k3d/k3s registry support with Docker network connect fix; GUI 'Deploy Services' button and manual pause; new test_service_layer_navigation.py. 2026-02-17 14:26:12 -08:00
chrisfu
23cf4ac33d Configure K3s for retropie and add Traefik Ingress. Updated K3s templates for node-ip/flannel-iface, configured retropie host vars, enabled Traefik, added Ingress resources for Grafana, Prole, Kong, and Supabase, and set Postgres to LoadBalancer. 2026-02-16 04:28:40 -08:00
chrisfu
d2efb835b0 Refactor project structure and update initialization scripts
- Moved files from 'prole/' subdirectory to root level or appropriate subdirectories (tests, authority, infrastructure) to flatten the project structure.

- Updated 'install.py' and initialization scripts in 'etc/' to reflect the new directory layout.

- Added 'etc/repair_pipeline.sh' for automated pipeline repairs.

- Updated configuration files including 'conf/prole.cfg' and 'env.sh'.

- Integrated ArgoCD manifests in 'k8s/argocd/'.

- Updated 'prole-app' environment and properties.

- Moved and updated test scripts for better organization and reliability.

- Added 'tests/silent_install_test.sh' for automated installation testing.
2026-02-14 13:44:49 -08:00
chrisfu
e228dd9243 checkpoint: installer refinements, k3s drift protection, and infra updates
- Installer: Updated k3s deployment logic and configuration generation.

- k3s Role: Implemented token drift protection to verify Vault secrets against live node tokens.

- DNS: Enhanced samba_reverse_dns role to support multiple reverse zones.

- Service Init: Updated initialization scripts and status reporting.

- Infrastructure: Added prole management role and k3s diagnostic playbook.

- Configuration: Updated prole.cfg and added vaulted group variables.
2026-02-12 02:30:44 -08:00
chrisfu
96f594fd3c Refactor initialization scripts and add new service components
- Consolidated and split initialization scripts in etc/:
    - Removed init_prole-db.sh and init_authority.sh.
    - Added init_kdc.sh for in-cluster MIT Kerberos KDC (prole-authority).
    - Added init_ollama.sh for Ollama AI service integration.
    - Added init_service_layer.sh for high-level service orchestration.
    - Added init_k3s_registry.sh for private registry management.
- Major updates to install.py:
    - Support for new Ollama and KDC configuration.
    - Improved prole.cfg rendering and namespace handling.
    - Updated unattended install flags.
- Infrastructure and Deployment:
    - Updated K3s Ansible role with private registry support (registries.yaml template).
    - Added prole-authority Dockerfile.
    - Updated OpenBao Kerberos ConfigMap and other K8s manifests.
- Configuration:
    - Updated prole.cfg with new sections for Ollama and Monitoring.
    - Refined environment variable exports in env.sh and prole_cfg.sh.
2026-02-11 13:09:31 -08:00
chrisfu
87e2f5d385 Checkpoint: Refactor common services and monitoring initialization. Updates: etc/init_monitoring.sh (Grafana naming, legacy cleanup), etc/init_common_services.sh (migration/cleanup logic), added etc/init_registry.sh, updated etc/init_cloudnative_pg.sh (manifest filtering), updated configs, and enhanced tests. Fixed: excluded .ansible logs and added to .gitignore. 2026-02-09 18:50:44 -08:00