Commit Graph

25 Commits

Author SHA1 Message Date
chrisfu
0262bd4be2 Refactor to support mode-scoped DB build contexts and versioning
- Introduce mode-scoped directories under `~/.prole` for build contexts, runtime data, and versioning.
- Update DB image version generation and Dockerfile creation to respect deployment modes (e.g., k3d, k3s, k8s).
- Adjust UI, core actions, and scripts to avoid cross-mode conflicts and ensure correct version isolation.
- Add tests for mode-aware behaviors and update existing tests for new paths and markers.
- Bump PostgreSQL and image versions to `18` and `138`, respectively.
- Update Kubernetes manifests to align with new image and version configuration.
2026-03-22 05:47:50 -07:00
chrisfu
4ee2b259c9 Checkpoint: rename installer to knoe + harden db build context
- Add build-context helper to copy Docker context safely (ignore runtime data, keep symlinks)

- Update UI and core actions to use ~/.prole/build and shared copy helper

- Add/adjust tests and scripts; introduce knoe ops helpers and update manifests

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 01:45:21 -07:00
chrisfu
2244d6acd6 Refactor iSCSI Synology mounts to /synology/d00x
- Rename host mount namespace from /prole/d00x to /synology/d00x

- Make LUN ownership explicit per host in inventory (myrddin=d001; merlin=d002,d004; pi=d003)

- Add safe migration cleanup for legacy /prole/d00x mounts and stale /etc/fstab entries

- Update k3s/ArgoCD/OpenTofu manifests and PV node affinities to match new mount paths

- Improve iSCSI role check-mode behavior and add quiesce/detach flow in ansible.sh
2026-03-21 10:43:06 -07:00
chrisfu
ecacf01266 Remove legacy secrets, configs, and scripts; introduce prole-auth service and Grafana SSO proxy configuration. 2026-03-20 13:00:36 -07:00
chrisfu
b114099e17 Use prole.cfg hostnames for Supabase front-door
- Persist and export supabase_hostname for canonical external Supabase entrypoint (db.prole.org)

- Render Supabase Helm ingress host and public URLs (API_EXTERNAL_URL/GOTRUE_SITE_URL/SUPABASE_PUBLIC_URL) from config

- Align tracked Traefik ingress manifests to db.prole.org

- Generalize service hostname/TLS wiring in k3s automation and refresh Kong/monitoring behavior

- Make optional workload policy checks deterministic when kube context is absent; add render tests
2026-03-19 04:19:38 -07:00
chrisfu
16a3dce584 k3s: enforce node placement for monitoring/CNPG/Supabase
- Pin Prometheus/Alertmanager + Grafana to merlin.prole.org in k3s to avoid PV node-affinity issues

- Taint myrddin.prole.org as control-plane-only and add CNPG toleration

- Prevent init_cloudnative_pg from deploying prole-svc-kong into the DB namespace

- GitOps-manage Kong declarative config via ConfigMap + add placement/init tests
2026-03-18 23:16:06 -07:00
chrisfu
133b719479 Align CNPG bootstrap placement with Ansible policy
Prefer infra-managed manifests for bootstrap; reconcile CNPG instances based on Ready+schedulable labeled db nodes; update manifests to use node-role affinity + anti-affinity; add policy/tests and config touch-ups (incl. prole.cfg).
2026-03-18 22:06:41 -07:00
chrisfu
b7fd89b373 Fix instance count enforcement for CNPG to always default to 3 and update manifests/tests accordingly 2026-03-11 13:21:50 -07:00
chrisfu
396cea60e7 installer: stabilize retries + improve k3s inventory detection
- Reuse namespace from existing prole.cfg unless env overrides

- Support k3s_hosts children groups and prefer k3s_servers as default

- Add headless pytest tkinter stubs and blocked-cluster reconciliation tests

- Update registry mirror endpoint, port mappings, and prole-db manifests
2026-03-07 23:57:21 -08:00
chrisfu
a43aed7134 k3s: eliminate localhost registry + dedupe common-core
- Ensure k3s mode uses the k3s registry endpoint and avoid localhost/k3d image prefixes.

- Make ArgoCD repo-server cmp symlink creation idempotent.

- Normalize common-core provisioning to knoe-system and add repair-time dedupe of stray default-namespace installs.

- Add k3s MariaDB datastore/refresh playbooks and regression tests.
2026-03-05 14:31:33 -08:00
chrisfu
86cf99cf76 milestone(k3s+cnpg): end-to-end deploy run via standalone installer
We can now complete a full K3s deployment run and deliver a working CloudNativePG (CNPG) stack from the standalone installer.\n\nHighlights:\n- Installer scripts hardened for CNPG deploy + recovery templating + rollout control\n- OpenTofu/K3s manifests aligned with in-repo k8s templates for prole-db\n- Supporting init flows updated (OpenBao, monitoring, Kerberos, OpenTofu)

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-03 20:03:15 -08:00
chrisfu
5618b662dd Remove prole-db-manager; simplify deployment via prole-authority; fix pg18 downgrade & cluster name
Summary:
Removed the prole-db-manager microservice and simplified deployment to use
prole-authority as the internal management and authorization point. Fixed two
blocking bugs that prevented silent install from completing on knoe-dev-cluster.

Removed: prole-db-manager
- Deleted db-manager-deployment.yaml and db-manager-service.yaml from opentofu manifests
- Deleted src/db-manager/ (Dockerfile, server.js, package.json, tests)
- Removed prole-db-manager port-forward mapping from installer/core/env.py
- Removed init_db_manager.sh from Initialization Scripts (milestones.py, actions.py)
- Removed init_certmgr.sh and init_db_manager.sh tabs from services screen (services.py)
- Removed live k8s Deployment/Service from knoe-dev-cluster

Fixed: PostgreSQL version downgrade error (pg17 -> pg18)
- Created conf/postgresql/.version with value 18
- Updated k8s/prole/prole-db.yaml and prole-db-recovery.yaml.tpl imageName to prole-db:18-089
- Fixed _init_database_options_state() to restore saved version_type from prole.cfg
  so db_version_type defaults to v18 (pg18) instead of silently reverting to pg17
- Added database_options.* keys to _collect_input_snapshot() in cfg.py so
  distribution, version_type, and all extension toggles persist to prole.cfg

Fixed: Cluster name inconsistency
- Removed stale prole-dev-cluster references; all scripts now use knoe-dev-cluster
- Added knoe-dev-cluster to mode-detection case in etc/prole_cfg.sh

Config: conf/prole.cfg
- Set kerberos_config.enabled = False, KERBEROS_AUTO_ENABLED = False
- Added database_options.distribution = percona, version_type = v18
- Added all 13 extension flags set to True (postgis, pgvector, pgcrypto, pgaudit,
  pg_repack, pg_stat_statements, pg_buffercache, pg_freespacemap, pgrowlocks,
  postgres_fdw, dblink, pg_stat_monitor, pgbadger)

Verification:
./install.py -s -l -v -c conf/prole.cfg completed successfully.
CNPG deployed prole-db:18-089 to knoe-dev-cluster; all milestones passed.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-01 20:40:44 -08:00
chrisfu
e64bbfb07f Fix k3s Kong manifest names
Align Kong k3s manifests with etc/init_kong.sh and svc-check Helm values (prole-svc-kong + prole-svc-kong-config) so rollout waits for the correct deployment.
2026-02-27 14:59:08 -08:00
chrisfu
f589cbd397 Remove PostgREST deployment, service, and related configs from prole-db namespace
- Deleted PostgREST Kubernetes deployment and service manifests.
- Removed `init_postgrest.sh` initialization script and PostgREST references across UI, installer, and config.
- Updated Kong API Gateway to exclude `/rest/*` routing to PostgREST.
- Simplified `supabase/deploy.sh` by removing references to PostgREST scaling and alias creation.
- Adjusted cluster configuration and UI to reflect the absence of PostgREST.
2026-02-25 00:38:07 -08:00
chrisfu
1573bb59c2 Add prole.spec and extend OpenTofu k3s configuration
- Include new `prole.spec` for build configurations and dependencies.
- Add Terraform state handling for OpenTofu in `k3s` cluster.
- Provision multiple Kubernetes resources in `prole-db` namespace: namespace, services, ConfigMaps, StatefulSets, Ingress rules, and PersistentVolumes.
- Integrate deployment and configuration enhancements for `garage`, `prole`, and related components.
2026-02-24 21:47:02 -08:00
chrisfu
51c91595b9 Refactor infrastructure, installer, and deployment configs: expanded CloudNativePG and PostgREST init scripts, updated ArgoCD/OpenTofu/K8s manifests (ingress, kustomization, postgrest, prole-db), added installer config helpers and revised cluster UI, updated prole.cfg and prole.sh, simplified Makefile, added env helper tests, removed stale spec/log files, added prole/ directory 2026-02-23 21:19:19 -08:00
chrisfu
a98c6d9d1c Fix Kong OOM crash: increase memory limit to 1Gi, mount ConfigMap to subdirectory 2026-02-22 01:02:00 -08:00
chrisfu
b03efa8f69 Kong API gateway, docker-import preload, OpenTofu graceful fallback, milestone fix
Kong API Gateway (replacing prole nginx):
- Add etc/init_kong.sh provisioning script (DB-less mode, prole-db namespace)
- Add kong-deployment.yaml and kong-service.yaml manifests
- Rewire ingress rules (svc/git/api.prole.org) to prole-db-kong:8000
- Update kustomization.yaml to reference kong manifests

PostgREST & DB Manager in prole-db namespace:
- Add etc/init_postgrest.sh and etc/init_db_manager.sh scripts
- Add postgrest/db-manager deployment and service manifests
- Add src/db-manager/ Node.js REST endpoint for backup triggers
- Default NAMESPACE changed to prole-db in both scripts

Docker image pre-load from PROLE_DATA/docker-import:
- Add _preload_docker_images() to init_common_services.sh
- Scan for .tar files exported by final_deployment.sh
- Import via k3d image import (k3d) or ctr (k3s) before deployments
- Increase rollout timeouts to 300s (configurable via ROLLOUT_TIMEOUT) in init_openbao.sh, init_opentofu.sh, init_garage_store.sh, init_registry.sh

OpenTofu password resolution fix:
- Add Kubernetes secret fallback in resolve_admin_password()
- Change hard exit 1 to graceful return 1 with warning
- Wrap call in if-guard so set -e doesn't abort the script chain

Milestone fix (init scripts not running):
- Add init_kong.sh, init_postgrest.sh, init_db_manager.sh to InitializationScriptsMilestone.execute() script list and arg branches
- Previously only actions.py had these; milestones.py was missing them

Installer integration:
- Add Kong/PostgREST/DB Manager to silent installer _step_init_scripts
- Add corresponding tabs and execution blocks in UI services.py
2026-02-22 00:57:49 -08:00
chrisfu
6e2d3e9011 refactor: modernize installer and monitoring setup
- Monitoring: Migrated from manual Grafana/Prometheus manifests to kube-prometheus-stack based setup in etc/init_monitoring.sh. Removed old manifest files from deploy/ and k8s/.
- Installer Core: Refactored installer with new modules for actions, environment handling, and UI screens. Enhanced Milestone logic to support advanced configuration (ArgoCD, Registry namespaces, Kerberos flags, etc.).
- Service & Init Scripts: Updated multiple initialization scripts (init_*.sh) for better integration with OpenBao, Kerberos, and the new monitoring stack. Added new scripts for Nginx Ingress, Ollama parsing, and K3D route fixes.
- Infrastructure: Enhanced Samba AD DC Ansible role with realm derivation, provisioning guidance, and group management. Updated K3s role tasks.
- Configuration: Refined default settings in conf/ to align with the new deployment architecture.
- App & Tools: Updated prole-app Swift code and prole.sh for improved environment variable handling and installation flow.
2026-02-19 21:07:39 -08:00
chrisfu
9aa959f61d Milestone: Balanced iSCSI IO, updated prole.sh launcher with log/verbose/debug/coverage/silent options, and enhanced installer logging 2026-02-16 16:44:18 -08:00
chrisfu
23cf4ac33d Configure K3s for retropie and add Traefik Ingress. Updated K3s templates for node-ip/flannel-iface, configured retropie host vars, enabled Traefik, added Ingress resources for Grafana, Prole, Kong, and Supabase, and set Postgres to LoadBalancer. 2026-02-16 04:28:40 -08:00
chrisfu
96f594fd3c Refactor initialization scripts and add new service components
- Consolidated and split initialization scripts in etc/:
    - Removed init_prole-db.sh and init_authority.sh.
    - Added init_kdc.sh for in-cluster MIT Kerberos KDC (prole-authority).
    - Added init_ollama.sh for Ollama AI service integration.
    - Added init_service_layer.sh for high-level service orchestration.
    - Added init_k3s_registry.sh for private registry management.
- Major updates to install.py:
    - Support for new Ollama and KDC configuration.
    - Improved prole.cfg rendering and namespace handling.
    - Updated unattended install flags.
- Infrastructure and Deployment:
    - Updated K3s Ansible role with private registry support (registries.yaml template).
    - Added prole-authority Dockerfile.
    - Updated OpenBao Kerberos ConfigMap and other K8s manifests.
- Configuration:
    - Updated prole.cfg with new sections for Ollama and Monitoring.
    - Refined environment variable exports in env.sh and prole_cfg.sh.
2026-02-11 13:09:31 -08:00
chrisfu
87e2f5d385 Checkpoint: Refactor common services and monitoring initialization. Updates: etc/init_monitoring.sh (Grafana naming, legacy cleanup), etc/init_common_services.sh (migration/cleanup logic), added etc/init_registry.sh, updated etc/init_cloudnative_pg.sh (manifest filtering), updated configs, and enhanced tests. Fixed: excluded .ansible logs and added to .gitignore. 2026-02-09 18:50:44 -08:00
chrisfu
f47c18fef7 feat(infrastructure): enhance k3s automation and OpenTofu integration
- Infrastructure:
    - Updated k3s Ansible role with mountpoint preflight checks and better permission management.
    - Automated deployment of prole configuration and port-forwarding scripts to cluster hosts.
    - Added systemd service for managing port forwards on k3s nodes.
    - Added prole-installer service account token automation.
- K8s Manifests:
    - Renamed and added Persistent Volumes in iscsi-pvs.yaml (including OpenBao support).
    - Updated StatefulSets for garage and openbao.
    - Migrated prole-db to CloudNativePG-based configuration.
    - Added comprehensive OpenTofu manifests for cluster deployment.
- Configuration:
    - Added cluster-specific configurations (k3d, k3s-hosts).
    - Added PostgreSQL configuration templates.
    - Updated .gitignore to track the conf/ directory.
- Tools:
    - Updated install.py and port-forwarding scripts.
    - Added render_manifest.py for manifest generation.
2026-02-07 22:53:18 -08:00
chrisfu
da2f6600ba feat: infrastructure and installer updates for k3s, OpenTofu, and prole-db
- Add k3s start/stop Ansible playbooks and roles.

- Implement OpenTofu initialization scripts and k8s manifests.

- Update ncurses installer with OpenTofu support and improved k3s integration.

- Add mode support (--mode) to etc/ initialization scripts.

- Update prole-db with recovery, barman objectstore, and SSH OpenBao support.

- Refine k8s manifests for OpenBao and prole-db.
2026-02-05 21:27:18 -08:00