- separate default app and db cluster contexts (knoe-dev-0 and knoe-cnpg-0) across actions, ops flows, and installer config\n- update CNPG placement and GKE manifests to keep PostgreSQL on the db cluster while app services stay on the app cluster\n- add GCS-backed Barman ObjectStore manifest and cross-cluster Garage patch workflow\n- refresh UI cluster/database/service screens and monitoring wiring for dual-cluster operation\n- add reset/patch scripts, db context selection test coverage, and architecture/network documentation updates
Co-authored-by: Junie <junie@jetbrains.com>
- harden CNPG backup ObjectStore readiness fallback in statusless clusters by validating controller availability and expected spec
- improve backup/plugin reconciliation paths and related test coverage for barman-cloud flows
- include ongoing GKE/CNPG deployment and configuration updates from this checkpoint
Co-authored-by: Junie <junie@jetbrains.com>
## GCP / Cluster Environment Screen
- Auto-populate Cloud tab from conf/prod/gcp.cfg on screen open (org_id,
billing_account, billing_project, project_id)
- gcloud auth validity checked on screen startup; friendly modal dialog
streams gcloud auth login output live so user never leaves the app
- Live GKE cluster browser: fetches clusters via gcloud container clusters
list, displays with checkmark selector, auto-selects saved cluster
- Selecting a cluster runs get-credentials, sets KUBECONFIG/KUBECONTEXT,
and syncs the region dropdown to the selected cluster's location
- Region dropdown populated live from gcloud compute regions list with
checkmark on currently selected region; graceful fallback when offline
- New 'GCP Storage' tab with workload->StorageClass mapping (CNPG->premium-rwo,
Redis/Monitoring->standard-rwo, Garage->garage-hdd) and Fetch from Cluster
- Provider readonly field styled correctly (no solid-black on macOS)
- Stale prole.cfg/conf/prole.cfg symlinks removed; all config I/O now
resolves env-specific paths via prole_conf.entrypoint_path()
## GKE Autopilot Compatibility (Common Services)
- Synology iSCSI StorageClass and static PVs guarded behind PROLE_MODE!=k8s
in init_openbao.sh (GKE Autopilot forbids hostPath/iSCSI volumes)
- In-cluster Docker registry (hostPath) skipped in k8s mode; GCP Artifact
Registry used instead
- Kong renamed knoe-svc-kong in k8s mode; all health-check kubectl calls in
init_common_services.sh and status_common_services.sh updated accordingly
- DNS endpoints switched from *.prole.org to *.knoe.dev in k8s mode
(api.knoe.dev, git.knoe.dev, svc.knoe.dev); ingress uses gce class
- New GKE-clean Kong manifests under deploy/opentofu/k8s/manifests/prole/:
no k3s node affinity, explicit Autopilot resource requests/limits
## Garage S3 Store (GKE)
- New garage-statefulset-gcp.yaml targeting garage-hdd StorageClass
(pd-standard, avoids SSD_TOTAL_GB quota exhaustion in us-west3)
- New storageclass-gcp-hdd.yaml (pd-standard, Retain, WaitForFirstConsumer)
- GCP StorageClass manifests skipped on re-runs (Autopilot built-ins are
immutable; skip-if-exists guard added)
- PVC deletion guard extended to cover any storageClass (not just synology)
so stale claims are cleaned before StatefulSet recreation
## Topology (GKE Autopilot)
- DaemonSet collector skipped in prod mode (forbidden in kube-system by
GKE Warden); Kubernetes-only node facts path used instead
- All ready GKE nodes assumed cnpg-eligible and monitoring-eligible without
taint/synology-mount checks (skip_collector + assume_nodes_eligible flags)
## KUBECONFIG / kubectl (k8s mode)
- actions.py: new elif mode==k8s branch sets KUBECONFIG=~/.kube/config
and injects KUBECONTEXT from prole_cfg_data into script env
- _build_kubectl_cmd falls back to Global.KUBECONTEXT when
init_cluster.selected_kubectx is empty
- _activate_selected_gke_cluster persists KUBECONFIG/KUBECONTEXT to
prole_cfg_data and saves prole.cfg immediately after get-credentials
## Database Build Screen (GKE)
- Registry display shows correct Artifact Registry URL
(<region>-docker.pkg.dev/<project>/<namespace>/knoe-db) in green
- Build+push: gcloud auth configure-docker, auto-creates AR repository
named after SERVICE_NAMESPACE (e.g. knoe-system) if missing, then
docker tag + push; falls back to gcr.io if region unavailable
- GCP config loaded from conf/prod/gcp.cfg on every screen entry;
keys normalised to lowercase so project_id lookup is always consistent
## Config / Namespace persistence
- prole_conf.py activate_environment: symlink creation removed; sets
CLUSTER_ENV env-var so all subsequent calls resolve correct env directory
- knoe/ui/screens/__init__.py: startup config load uses entrypoint_path()
instead of hardcoded conf/prole.cfg; seeds SERVICE_NAMESPACE=knoe-system
for managed envs so Common Services never defaults to 'default'
- cfg.py _save_prole_cfg: saves to env-specific path via entrypoint_path()
- etc/prole_cfg.sh: removed all ln -snf symlink creation
Co-authored-by: Junie <junie@jetbrains.com>
STABLE BUILD — all 35+ pods Running, zero non-Running pods, CNPG ready=3.
## Architecture changes (Supabase slim-down)
- Removed local Supabase Kong, Storage, MinIO, Imgproxy from deployed stack
- Rewired API routing to shared Kong in kube-system
- Rewired object storage to shared Garage in knoe-system
- Supabase footprint reduced to control-plane only: auth, rest, meta,
analytics, realtime (studio port-forward retained)
- Supabase pods renamed: stripped redundant '-prole-supabase-' prefix via
per-component fullnameOverride (e.g. supabase-auth, supabase-rest, ...)
- Supabase workloads pinned to merlin.prole.org (memory-safe node)
## File changes
- conf/service/prole.cfg
* Replaced all hardcoded /Users/chrisfu/dev/prole paths with ${HOME}/dev/prole
so config is portable when run from myrddin.prole.org or any user home
* SUPABASE_ENABLED=True, SUPABASE_PRIMARY_NODE=merlin.prole.org in
both [Global] and [Optional Features]
* supabase_enabled=true in [Inputs]/init_cluster section
- supabase/helm/render_supabase.py
* Explicitly disabled: kong, storage, minio, imgproxy
* Active components pinned via nodeSelector to merlin.prole.org
* fullnameOverride per component to clean up pod names
* Default primary node changed from pi.prole.org -> merlin.prole.org
- supabase/deploy.sh
* Removed svc/kong port-forward; API via shared Kong in kube-system
* Removed imgproxy/storage from PV/PVC prep and helm --set persistence args
* Fixed kubectl exec missing -i flag for psql heredoc stdin in
setup_knoe_db_for_supabase
* Updated Studio access messaging to reference shared platform Kong
- k8s/prole/knoe-db.yaml
* instances: 3
* Added control-plane/master tolerations for myrddin scheduling
- deploy/opentofu/k3s/manifests/prole/knoe-db.yaml
* instances: 3, added control-plane tolerations (mirrors k8s/prole/)
- k8s/prole/iscsi-pvs.yaml
* Expanded d003-data/d003-wal nodeAffinity to [pi, merlin, myrddin]
to reflect post-OOM-recovery live state
- k8s/registry/deployment.yaml
* nodeSelector restored to myrddin.prole.org (hostPort 5000 owner)
* Added control-plane + master tolerations so pod schedules correctly
- modes/k3s/knoe-db/.version
* Aligned to 140 (image tag 18-140) matching repo-root version
- etc/init_common_services.sh, etc/init_registry.sh
* Minor fixes aligned with slimmed service layout and node targeting
- infrastructure/inventory/host_vars/pi.prole.org.yml
* Updated host vars to reflect post-recovery pi state
- knoe/core/topology.py, knoe/ui/screens/cluster.py,
knoe/ui/screens/cluster_nodes.py
* Topology and UI updates for 3-node cluster representation
- scan/network_description.txt
* Refreshed network scan output post-recovery
- conf/port-mapping.cfg
* Updated port mapping to reflect removal of local Kong/storage forwards
- scripts/cleanup_stale_rs.sh (new)
* Utility script to prune orphaned ReplicaSets after rollouts
## Verified stable state
- CNPG: Cluster in healthy state, ready=3
knoe-db-2 (merlin, primary), knoe-db-3 (pi), knoe-db-5 (myrddin)
- Supabase: 5 pods Running on merlin.prole.org
supabase-auth, supabase-rest, supabase-meta,
supabase-analytics, supabase-realtime
- Monitoring: all prometheus/grafana/alertmanager pods Running on merlin
- Registry: Running on myrddin.prole.org (hostPort 5000)
- Zero non-Running / non-Completed pods across all namespaces
Co-authored-by: Junie <junie@jetbrains.com>
- render_supabase.py: read SUPABASE_ADDITIONAL_REDIRECT_URLS from cfg,
normalise each entry to https://.../** wildcard, inject as GOTRUE_URI_ALLOW_LIST
in GoTrue auth env so logins from svc.prole.org are accepted
- conf/service/prole.cfg: set SUPABASE_ADDITIONAL_REDIRECT_URLS=svc.prole.org
- actions.py: preserve SUPABASE_ADDITIONAL_REDIRECT_URLS in globals_to_save
so pipeline write-back never clobbers it
- k8s/prole/merlin-local-iscsi-storageclass.yaml: register StorageClass
merlin-local-iscsi-d002 (no-provisioner/Immediate) to silence
ProvisioningFailed warnings on static Supabase PVs
Result: all 9 Supabase pods Running; svc.prole.org login origin accepted by GoTrue
Co-authored-by: Junie <junie@jetbrains.com>
- Rename iSCSI storage class and PV/PVC selectors/labels from prole to synology across k8s and OpenTofu manifests\n- Update CNPG/OpenBao/Garage/monitoring init flows, render helpers, and mock scripts for synology-backed storage objects\n- Integrate related UI/core/service config/version updates and add supporting regression tests for CNPG storage/image behavior\n- Keep storage reconciliation tests aligned with current CNPG affinity output
Co-authored-by: Junie <junie@jetbrains.com>
- Introduce mode-scoped directories under `~/.prole` for build contexts, runtime data, and versioning.
- Update DB image version generation and Dockerfile creation to respect deployment modes (e.g., k3d, k3s, k8s).
- Adjust UI, core actions, and scripts to avoid cross-mode conflicts and ensure correct version isolation.
- Add tests for mode-aware behaviors and update existing tests for new paths and markers.
- Bump PostgreSQL and image versions to `18` and `138`, respectively.
- Update Kubernetes manifests to align with new image and version configuration.
- Rename host mount namespace from /prole/d00x to /synology/d00x
- Make LUN ownership explicit per host in inventory (myrddin=d001; merlin=d002,d004; pi=d003)
- Add safe migration cleanup for legacy /prole/d00x mounts and stale /etc/fstab entries
- Update k3s/ArgoCD/OpenTofu manifests and PV node affinities to match new mount paths
- Improve iSCSI role check-mode behavior and add quiesce/detach flow in ansible.sh
- Persist and export supabase_hostname for canonical external Supabase entrypoint (db.prole.org)
- Render Supabase Helm ingress host and public URLs (API_EXTERNAL_URL/GOTRUE_SITE_URL/SUPABASE_PUBLIC_URL) from config
- Align tracked Traefik ingress manifests to db.prole.org
- Generalize service hostname/TLS wiring in k3s automation and refresh Kong/monitoring behavior
- Make optional workload policy checks deterministic when kube context is absent; add render tests
Prefer infra-managed manifests for bootstrap; reconcile CNPG instances based on Ready+schedulable labeled db nodes; update manifests to use node-role affinity + anti-affinity; add policy/tests and config touch-ups (incl. prole.cfg).
- Add diagnostics/reset/retry logic for stuck monitoring Helm installs\n- Add optional cluster-side node/lease deletion during k3s reset via delegated control-plane\n- Bump prole-db image tag/version to 18-123\n- Update monitoring init unit test for reset/diagnostics path
- k3s_sync: choose any server as sync source fallback; refresh kubeconfig and validate nodes\n- fetch_kubeconfig: save to prole-k3s.kubeconfig and etc/secrets/k3s.kubeconfig\n- k3s sync: sync server URL into config.yaml; restore TLS bundle only on servers\n- cert-manager/ACME: wait for CNI/webhook readiness; retry ClusterIssuer/Certificate apply\n- inventory: tune myrddin control-plane timing args; add render test\n- misc: update port mappings and prole-db manifests/version
Co-authored-by: Junie <junie@jetbrains.com>
- Materialize required DB Kubernetes secrets from user-entered password before CNPG init
- Auto-generate a temporary OpenTofu admin password in non-interactive runs when none is available
- Update installer UI flows and tests to cover the new preflights/rotation
Co-authored-by: Junie <junie@jetbrains.com>
- Add ssl_hosts group and prole_ssl role to deploy registry cert/key to /etc/ssl/certs/prole
- Configure k8s registry deployment to serve HTTPS using mounted host certs
- Switch k3s/containerd registry mirror config to https:// and prefer HTTPS checks with HTTP fallback
Co-authored-by: Junie <junie@jetbrains.com>
- Reuse namespace from existing prole.cfg unless env overrides
- Support k3s_hosts children groups and prefer k3s_servers as default
- Add headless pytest tkinter stubs and blocked-cluster reconciliation tests
- Update registry mirror endpoint, port mappings, and prole-db manifests
ArgoCD IO load distribution: Updated k8s/argocd/install.yaml to use hostPath volumes for application-controller (/prole/d001), redis (/prole/d002), and repo-server (/prole/d003). Updated Ansible configure task to create hostPath directories on myrddin.prole.org.
K3s configuration and testing: Added test playbooks for kubeconfig rewriting and argument validation. Updated K3s install, configure, and image import tasks. Configured required mounts and host-specific variables for myrddin.prole.org.
Miscellaneous: Updated conf/prole.cfg with K3S_TOKEN. Bumped prole-db version and updated recovery manifest templates. Minor UI adjustments in cluster screen.
Co-authored-by: Junie <junie@jetbrains.com>
We can now complete a full K3s deployment run and deliver a working CloudNativePG (CNPG) stack from the standalone installer.\n\nHighlights:\n- Installer scripts hardened for CNPG deploy + recovery templating + rollout control\n- OpenTofu/K3s manifests aligned with in-repo k8s templates for prole-db\n- Supporting init flows updated (OpenBao, monitoring, Kerberos, OpenTofu)
Co-authored-by: Junie <junie@jetbrains.com>
- Removed all logic that read or set NAMESPACE/PROLE_NAMESPACE from the shell environment or wrote it to env.sh.
- prole.cfg is now the sole source of truth for the namespace value, loaded exclusively by prole_cfg.sh.
- etc/prole_cfg.sh & mock_val/prole_cfg.sh: Removed kubectl-context fallback and default for PROLE_NAMESPACE.
- Shell scripts (init_*.sh): Replaced NAMESPACE=${NAMESPACE:-...} with NAMESPACE="${PROLE_NAMESPACE}".
- Python (actions, environment, milestone): Removed env["PROLE_NAMESPACE"] from subprocess env dicts and env.sh.
- etc/init_cloudnative_pg.sh: Removed DB_PASSWORD env fallback for secret creation.
Co-authored-by: Junie <junie@jetbrains.com>
- Renamed etc/init_prole-db-backup.sh to etc/init_cnpg_backup.sh and updated all references.
- Enhanced CNPG backup script with authoritative status validation.
- Implemented SupabaseImagePreloadMilestone to front-load heavy image transfers.
- Updated supabase/deploy.sh with --prefetch-images-only and --skip-prefetch.
- Updated unit tests for CNPG backups and Supabase silent milestone sequencing.
- Included incidental environment updates from installer execution.
Co-authored-by: Junie <junie@jetbrains.com>
Summary:
Removed the prole-db-manager microservice and simplified deployment to use
prole-authority as the internal management and authorization point. Fixed two
blocking bugs that prevented silent install from completing on knoe-dev-cluster.
Removed: prole-db-manager
- Deleted db-manager-deployment.yaml and db-manager-service.yaml from opentofu manifests
- Deleted src/db-manager/ (Dockerfile, server.js, package.json, tests)
- Removed prole-db-manager port-forward mapping from installer/core/env.py
- Removed init_db_manager.sh from Initialization Scripts (milestones.py, actions.py)
- Removed init_certmgr.sh and init_db_manager.sh tabs from services screen (services.py)
- Removed live k8s Deployment/Service from knoe-dev-cluster
Fixed: PostgreSQL version downgrade error (pg17 -> pg18)
- Created conf/postgresql/.version with value 18
- Updated k8s/prole/prole-db.yaml and prole-db-recovery.yaml.tpl imageName to prole-db:18-089
- Fixed _init_database_options_state() to restore saved version_type from prole.cfg
so db_version_type defaults to v18 (pg18) instead of silently reverting to pg17
- Added database_options.* keys to _collect_input_snapshot() in cfg.py so
distribution, version_type, and all extension toggles persist to prole.cfg
Fixed: Cluster name inconsistency
- Removed stale prole-dev-cluster references; all scripts now use knoe-dev-cluster
- Added knoe-dev-cluster to mode-detection case in etc/prole_cfg.sh
Config: conf/prole.cfg
- Set kerberos_config.enabled = False, KERBEROS_AUTO_ENABLED = False
- Added database_options.distribution = percona, version_type = v18
- Added all 13 extension flags set to True (postgis, pgvector, pgcrypto, pgaudit,
pg_repack, pg_stat_statements, pg_buffercache, pg_freespacemap, pgrowlocks,
postgres_fdw, dblink, pg_stat_monitor, pgbadger)
Verification:
./install.py -s -l -v -c conf/prole.cfg completed successfully.
CNPG deployed prole-db:18-089 to knoe-dev-cluster; all milestones passed.
Co-authored-by: Junie <junie@jetbrains.com>
- Resolve namespace drift in Kerberos test flow and OpenBao deployment.
- Fix OpenAPI credential-validation failures in etc scripts by using --validate=false with kubectl.
- Introduce preferred_service_namespace() to accurately target namespaces for KDC policy/test paths.
- Default Kerberos test user to 'administrator' and prefill from Ansible vault secrets.
- Generalized OpenBao deployment logic to correctly handle multi-namespace environments.
- Added regression test 'tests/etc/test_init_kerberos_k3s_real_kdc.sh'.
- Synchronized SERVICE_NAMESPACE extraction in prole_cfg.sh.
Co-authored-by: Junie <junie@jetbrains.com>
- Tighten Cluster Environment screen layout; switch Service/Prod to kubectx context selection; keep namespace and key controls on one line; ensure Repair button remains reachable.
- Add UI layout regression test to render with large mock data and assert key widgets remain visible and console is scrollable.
- Make kubeconfig generation deterministic under tests by avoiding overwriting cert-based kubeconfigs; write token sidecar kubeconfig when needed.
- Update common-services init scripts and add k3s/Helm deployment bits (svc-check, Kong/CertMgr tasks).
- Monitoring: Migrated from manual Grafana/Prometheus manifests to kube-prometheus-stack based setup in etc/init_monitoring.sh. Removed old manifest files from deploy/ and k8s/.
- Installer Core: Refactored installer with new modules for actions, environment handling, and UI screens. Enhanced Milestone logic to support advanced configuration (ArgoCD, Registry namespaces, Kerberos flags, etc.).
- Service & Init Scripts: Updated multiple initialization scripts (init_*.sh) for better integration with OpenBao, Kerberos, and the new monitoring stack. Added new scripts for Nginx Ingress, Ollama parsing, and K3D route fixes.
- Infrastructure: Enhanced Samba AD DC Ansible role with realm derivation, provisioning guidance, and group management. Updated K3s role tasks.
- Configuration: Refined default settings in conf/ to align with the new deployment architecture.
- App & Tools: Updated prole-app Swift code and prole.sh for improved environment variable handling and installation flow.
- Implement proactive CA hash verification and automated repair for mismatches
- Ensure agents prioritize discovered server tokens over stale vault values
- Fix K3s service hangs with explicit stop and killall before reinstall
- Add Retropie/Pi networking fixes (WiFi power save, wlan0 priority)
- Pin pre-staged images to stable, architecture-aware versions (arm64)
- Remove obsolete init-port-forwards and prole.cfg sync tasks
- Update k8s manifests and installer core logic with new tests
- Finalized stable, repeatable reset logic for the k3d pipeline.
- Refactored installer into modular components: core, milestone, runner, and state.
- Introduced new UI abstractions with support for ncurses and Tkinter.
- Updated initialization scripts and configurations for CloudNativePG, Kerberos, OpenBao, and Monitoring.
- Improved pipeline repair and port-forwarding mechanisms.
- Moved files from 'prole/' subdirectory to root level or appropriate subdirectories (tests, authority, infrastructure) to flatten the project structure.
- Updated 'install.py' and initialization scripts in 'etc/' to reflect the new directory layout.
- Added 'etc/repair_pipeline.sh' for automated pipeline repairs.
- Updated configuration files including 'conf/prole.cfg' and 'env.sh'.
- Integrated ArgoCD manifests in 'k8s/argocd/'.
- Updated 'prole-app' environment and properties.
- Moved and updated test scripts for better organization and reliability.
- Added 'tests/silent_install_test.sh' for automated installation testing.
- Consolidated and split initialization scripts in etc/:
- Removed init_prole-db.sh and init_authority.sh.
- Added init_kdc.sh for in-cluster MIT Kerberos KDC (prole-authority).
- Added init_ollama.sh for Ollama AI service integration.
- Added init_service_layer.sh for high-level service orchestration.
- Added init_k3s_registry.sh for private registry management.
- Major updates to install.py:
- Support for new Ollama and KDC configuration.
- Improved prole.cfg rendering and namespace handling.
- Updated unattended install flags.
- Infrastructure and Deployment:
- Updated K3s Ansible role with private registry support (registries.yaml template).
- Added prole-authority Dockerfile.
- Updated OpenBao Kerberos ConfigMap and other K8s manifests.
- Configuration:
- Updated prole.cfg with new sections for Ollama and Monitoring.
- Refined environment variable exports in env.sh and prole_cfg.sh.
- Introduced scripts `init_common_services.sh` and `status_common_services.sh` for deploying and checking common services (OpenBao, OpenTofu, registry) within Kubernetes namespaces.
- Improved service namespace configuration in `install.py` and updated initialization logic.
- Updated `prole.cfg` and `kerberos-configmap.yaml` with necessary changes to integrate the new features.
