- remove hardcoded physical host assumptions in UI/init flows and rely on config-driven selectors\n- optimize GitLab reruns to skip unnecessary operator upgrades/long reconcile waits when unchanged\n- add and update regression/shell tests for fast-path and k8s context/ingress/garage behaviors
Co-authored-by: Junie <junie@jetbrains.com>
- Refactored `init_gitlab.sh` and `init_gitea.sh` to use configurable node selector keys, removing hardcoded defaults like `gandalf.prole.org`.
- Enhanced scheduling logic to validate required fields and prevent stale node constraints during reruns.
- Added fast-path guards in GitLab init to skip redundant operations when no changes occur in operator or CR specifications.
- Updated Supabase deployment to clear stale topology spread constraints alongside node selectors and affinity.
- Added tests for configurable node selector keys, fast-path guards, and stale constraint cleanups.
- Changed default storage class for GKE to `standard` to reduce pressure on SSD/balanced quotas.
- Disabled `enforceGeneralNodeRole` scheduling for APP/DB values to support split cluster setups.
- Updated tests to validate storage class and scheduling logic changes.
- Introduced `SERVICE_INGRESS_TLS_ENABLED` for configurable svc ingress TLS rendering in `init_kong.sh`.
- Updated ingress templates to include or exclude TLS annotations and blocks dynamically.
- Improved PVC and pod readiness checks in `supabase/deploy.sh` with enriched logging and detailed blockers.
- Added support for configurable storage class resolution for Supabase DB frontdoor PVCs.
- Enhanced deployment workflows to handle split APP/DB cluster setups with context-aware readiness checks.
- Updated test cases to validate svc ingress TLS configuration and storage class logic.
- Introduced `enforceGeneralNodeRole` to make node affinity and selector enforcement configurable.
- Updated `_split_frontdoor_docs` to handle custom release labels for Kong services.
- Enhanced Helm templates to support dynamic scheduling configurations.
- Added `reconcile_db_frontdoor_studio_pvcs` to migrate PVCs to the target storage class.
- Updated tests to validate custom release labels and scheduling logic.
- Updated `_extract_k8s_docs` to use `yaml.safe_load_all` for improved parsing and reliability.
- Enhanced `_split_frontdoor_docs` with stricter validation of Deployment, Service, and Ingress specs.
- Added namespace enforcement for frontdoor resources during manifest splitting.
- Refactored and optimized test cases to cover new validation and splitting functionality.
- Updated `_extract_k8s_docs` to use `yaml.safe_load_all` for improved parsing and reliability.
- Enhanced `_split_frontdoor_docs` with stricter validation of Deployment, Service, and Ingress specs.
- Added namespace enforcement for frontdoor resources during manifest splitting.
- Refactored and optimized test cases to cover new validation and splitting functionality.
- Added validation and explicit handling for Garage S3 endpoints in split APP/DB cluster setups.
- Improved dynamic role detection and authoritative context enforcement in initialization scripts.
- Updated Supabase and Garage configurations to support private cross-cluster endpoints, ensuring cluster interoperability and DNS constraints.
- Enhanced error handling and logging for invalid or missing configurations.
- Introduced jemalloc hostPath optimizations with configurable modes (`auto`, `off`, `force`).
- Integrated jemalloc setup with best-effort and forced validation flows for ensuring cluster compatibility.
- Enhanced monitoring storage class logic with mode-specific handling (`k3s`, `k3d`, `gke`) and improved validation of required classes.
- Added safeguards and detailed logging for unsupported configurations and failure scenarios.
Rename env config files from conf/*/prole.cfg to conf/k3d.cfg, conf/k3s.cfg, and conf/gke.cfg. Update shell/Python loaders and etc/deploy scripts to resolve named configs cleanly while keeping legacy fallback behavior. Align k3s Ansible tasks, docs, and regression coverage with the new configuration layout.
Co-authored-by: Junie <junie@jetbrains.com>
- Introduced support for dynamic GitLab and Supabase domain and ingress configuration based on deployment mode (k8s vs local).
- Default GitLab domain and ingress class aligned with `prole.cfg` deployment settings.
- Expanded Supabase support:
- Added configurable API and Studio hostnames with default fallbacks.
- Introduced separate ingress class configuration for API and Studio.
- Standardized public URL normalization for both services.
- Updated tests to validate domain, ingress, and configuration behaviors across environments.
- harden CNPG backup ObjectStore readiness fallback in statusless clusters by validating controller availability and expected spec
- improve backup/plugin reconciliation paths and related test coverage for barman-cloud flows
- include ongoing GKE/CNPG deployment and configuration updates from this checkpoint
Co-authored-by: Junie <junie@jetbrains.com>
- update Kong/Supabase ingress and service manifests for k3s
- expand init scripts and runtime config/topology handling
- refresh KNØE UI screens and Supabase deployment/render logic
- include related env helper test adjustments
Co-authored-by: Junie <junie@jetbrains.com>
- add reusable storage probing subsystem with discovery, bounded probe execution, IO classification, caching, and topology integration
- render per-node storage inventory in Cluster Nodes UI and extend installer test coverage for topology/storage behavior
- introduce core service operation modules and align actions, milestones, services, and supporting configs/scripts for repair/update workflows
- update CNPG/Supabase/database artifacts, placement and port mapping configs, plus related integration tests
Co-authored-by: Junie <junie@jetbrains.com>
STABLE BUILD — all 35+ pods Running, zero non-Running pods, CNPG ready=3.
## Architecture changes (Supabase slim-down)
- Removed local Supabase Kong, Storage, MinIO, Imgproxy from deployed stack
- Rewired API routing to shared Kong in kube-system
- Rewired object storage to shared Garage in knoe-system
- Supabase footprint reduced to control-plane only: auth, rest, meta,
analytics, realtime (studio port-forward retained)
- Supabase pods renamed: stripped redundant '-prole-supabase-' prefix via
per-component fullnameOverride (e.g. supabase-auth, supabase-rest, ...)
- Supabase workloads pinned to merlin.prole.org (memory-safe node)
## File changes
- conf/service/prole.cfg
* Replaced all hardcoded /Users/chrisfu/dev/prole paths with ${HOME}/dev/prole
so config is portable when run from myrddin.prole.org or any user home
* SUPABASE_ENABLED=True, SUPABASE_PRIMARY_NODE=merlin.prole.org in
both [Global] and [Optional Features]
* supabase_enabled=true in [Inputs]/init_cluster section
- supabase/helm/render_supabase.py
* Explicitly disabled: kong, storage, minio, imgproxy
* Active components pinned via nodeSelector to merlin.prole.org
* fullnameOverride per component to clean up pod names
* Default primary node changed from pi.prole.org -> merlin.prole.org
- supabase/deploy.sh
* Removed svc/kong port-forward; API via shared Kong in kube-system
* Removed imgproxy/storage from PV/PVC prep and helm --set persistence args
* Fixed kubectl exec missing -i flag for psql heredoc stdin in
setup_knoe_db_for_supabase
* Updated Studio access messaging to reference shared platform Kong
- k8s/prole/knoe-db.yaml
* instances: 3
* Added control-plane/master tolerations for myrddin scheduling
- deploy/opentofu/k3s/manifests/prole/knoe-db.yaml
* instances: 3, added control-plane tolerations (mirrors k8s/prole/)
- k8s/prole/iscsi-pvs.yaml
* Expanded d003-data/d003-wal nodeAffinity to [pi, merlin, myrddin]
to reflect post-OOM-recovery live state
- k8s/registry/deployment.yaml
* nodeSelector restored to myrddin.prole.org (hostPort 5000 owner)
* Added control-plane + master tolerations so pod schedules correctly
- modes/k3s/knoe-db/.version
* Aligned to 140 (image tag 18-140) matching repo-root version
- etc/init_common_services.sh, etc/init_registry.sh
* Minor fixes aligned with slimmed service layout and node targeting
- infrastructure/inventory/host_vars/pi.prole.org.yml
* Updated host vars to reflect post-recovery pi state
- knoe/core/topology.py, knoe/ui/screens/cluster.py,
knoe/ui/screens/cluster_nodes.py
* Topology and UI updates for 3-node cluster representation
- scan/network_description.txt
* Refreshed network scan output post-recovery
- conf/port-mapping.cfg
* Updated port mapping to reflect removal of local Kong/storage forwards
- scripts/cleanup_stale_rs.sh (new)
* Utility script to prune orphaned ReplicaSets after rollouts
## Verified stable state
- CNPG: Cluster in healthy state, ready=3
knoe-db-2 (merlin, primary), knoe-db-3 (pi), knoe-db-5 (myrddin)
- Supabase: 5 pods Running on merlin.prole.org
supabase-auth, supabase-rest, supabase-meta,
supabase-analytics, supabase-realtime
- Monitoring: all prometheus/grafana/alertmanager pods Running on merlin
- Registry: Running on myrddin.prole.org (hostPort 5000)
- Zero non-Running / non-Completed pods across all namespaces
Co-authored-by: Junie <junie@jetbrains.com>
- render_supabase.py: read SUPABASE_ADDITIONAL_REDIRECT_URLS from cfg,
normalise each entry to https://.../** wildcard, inject as GOTRUE_URI_ALLOW_LIST
in GoTrue auth env so logins from svc.prole.org are accepted
- conf/service/prole.cfg: set SUPABASE_ADDITIONAL_REDIRECT_URLS=svc.prole.org
- actions.py: preserve SUPABASE_ADDITIONAL_REDIRECT_URLS in globals_to_save
so pipeline write-back never clobbers it
- k8s/prole/merlin-local-iscsi-storageclass.yaml: register StorageClass
merlin-local-iscsi-d002 (no-provisioner/Immediate) to silence
ProvisioningFailed warnings on static Supabase PVs
Result: all 9 Supabase pods Running; svc.prole.org login origin accepted by GoTrue
Co-authored-by: Junie <junie@jetbrains.com>
Fix supabase-storage migration deadlock: ALTER ROLE search_path + BYPASSRLS
Root cause: supabase_storage_admin had no rolconfig search_path, so the connection
string search_path=auth caused the service to query auth.migrations (not found →
currentMigrations=[]) but insert migration tracking rows into public.migrations
(via PostgreSQL implicit public fallback). On restart, currentMigrations was always
[] despite public.migrations having rows, causing duplicate-key on migration 0.
Fix (added to deploy.sh db init block, idempotent on every reinstall):
ALTER ROLE supabase_storage_admin SET search_path = storage, auth, public;
ALTER ROLE supabase_storage_admin BYPASSRLS;
TRUNCATE TABLE IF EXISTS public.migrations;
DROP TABLE IF EXISTS storage.migrations CASCADE;
This aligns migration reads and writes to the same schema (storage first, then auth,
then public) and ensures a clean slate on every redeploy.
Result: all 9 Supabase pods Running on merlin.prole.org (merlin-local-iscsi-d002 iSCSI
storage), db.prole.org returns HTTP 401 (Supabase Studio auth gate = live and working),
CNPG knoe-db 2/2 Ready on merlin + myrddin with Percona PostgreSQL 18 + Barman backup.
Co-authored-by: Junie <junie@jetbrains.com>
Root cause: render_supabase.py wrote to deployment.*.storageClass (ignored by Helm chart PVC
templates) instead of persistence.*.storageClassName (_pvc.tpl:43 is the real path). Also
deploy.sh --set persistence.* args only ran in k8s mode, and stale values.generated.json
was reused across deploys. Together these caused Supabase PVCs to fall through to
pi-local-iscsi (pi SD card), crashing the node under memory pressure.
- supabase/helm/render_supabase.py:
- write persistence.{functions,imgproxy,snippets,storage,deno,minio}.storageClassName
using the correct Helm chart path (_pvc.tpl:43)
- add synology guard: raise SystemExit if storage class is not synology/merlin/myrddin
- remove broken deployment.*.storageClass writes
- supabase/deploy.sh:
- delete stale values.generated.json before every helm_render_values call
- fall back to reading SUPABASE_STORAGE_CLASS from cfg file via python3 if env not set
- add bash synology guard: die if storage class is not iSCSI/NFS mounted
- extend helm_set_args to include persistence.minio.storageClassName and apply
--set persistence.* for ALL modes (not just k8s)
Co-authored-by: Junie <junie@jetbrains.com>
Memory pressure on merlin (95% OOM with monitoring + CNPG replica + Supabase) resolved
by moving Supabase workload to pi.prole.org which has headroom (49% memory, 2 pods).
- conf/service/prole.cfg: set SUPABASE_PV_NODE, SUPABASE_PRIMARY_NODE=pi.prole.org,
SUPABASE_STORAGE_CLASS=pi-local-iscsi, SUPABASE_PV_BASE_DIR for rancher local-path
- supabase/helm/render_supabase.py: resolve SUPABASE_STORAGE_CLASS from env/cfg,
apply pi-local-iscsi to storage/analytics PVCs, resolve DATABASE_NAMESPACE for
correct db_host, DB_PASSWORD env override takes priority for live cluster password
- supabase/deploy.sh: default SUPABASE_PV_NODE to pi.prole.org, default base dir to
/var/lib/rancher/k3s/storage/supabase, add idempotent ALTER ROLE password sync,
live DB_PASSWORD resolution in helm_render_values from knoe-db-superuser secret
- knoe/core/actions.py: stop overriding init_cluster.supabase_enabled to False in
silent mode so prole.cfg value is honored end-to-end
- etc/init_monitoring.sh: derive pv_prom/pv_am/pv_graf inline before Released-PV
loop to fix unbound variable error on idempotent monitoring installs
Co-authored-by: Junie <junie@jetbrains.com>
- actions.py: stop overriding init_cluster.supabase_enabled to False in silent mode
- supabase/deploy.sh: change SUPABASE_PV_NODE default from myrddin to merlin.prole.org
so prole-supabase-dirprep job runs on the node that owns /synology/d002
- conf/service/prole.cfg: add SUPABASE_PV_NODE and SUPABASE_PV_BASE_DIR to [Global]
so prole_cfg.sh exports them explicitly on every run
- init_monitoring.sh: derive pv_prom/pv_am/pv_graf inline before Released-PV loop
to fix unbound variable error on idempotent monitoring installs
Co-authored-by: Junie <junie@jetbrains.com>
- Rename iSCSI storage class and PV/PVC selectors/labels from prole to synology across k8s and OpenTofu manifests\n- Update CNPG/OpenBao/Garage/monitoring init flows, render helpers, and mock scripts for synology-backed storage objects\n- Integrate related UI/core/service config/version updates and add supporting regression tests for CNPG storage/image behavior\n- Keep storage reconciliation tests aligned with current CNPG affinity output
Co-authored-by: Junie <junie@jetbrains.com>
- Persist and export supabase_hostname for canonical external Supabase entrypoint (db.prole.org)
- Render Supabase Helm ingress host and public URLs (API_EXTERNAL_URL/GOTRUE_SITE_URL/SUPABASE_PUBLIC_URL) from config
- Align tracked Traefik ingress manifests to db.prole.org
- Generalize service hostname/TLS wiring in k3s automation and refresh Kong/monitoring behavior
- Make optional workload policy checks deterministic when kube context is absent; add render tests
- Pin Prometheus/Alertmanager + Grafana to merlin.prole.org in k3s to avoid PV node-affinity issues
- Taint myrddin.prole.org as control-plane-only and add CNPG toleration
- Prevent init_cloudnative_pg from deploying prole-svc-kong into the DB namespace
- GitOps-manage Kong declarative config via ConfigMap + add placement/init tests
Prefer infra-managed manifests for bootstrap; reconcile CNPG instances based on Ready+schedulable labeled db nodes; update manifests to use node-role affinity + anti-affinity; add policy/tests and config touch-ups (incl. prole.cfg).
- Renamed etc/init_prole-db-backup.sh to etc/init_cnpg_backup.sh and updated all references.
- Enhanced CNPG backup script with authoritative status validation.
- Implemented SupabaseImagePreloadMilestone to front-load heavy image transfers.
- Updated supabase/deploy.sh with --prefetch-images-only and --skip-prefetch.
- Updated unit tests for CNPG backups and Supabase silent milestone sequencing.
- Included incidental environment updates from installer execution.
Co-authored-by: Junie <junie@jetbrains.com>
- Deleted PostgREST Kubernetes deployment and service manifests.
- Removed `init_postgrest.sh` initialization script and PostgREST references across UI, installer, and config.
- Updated Kong API Gateway to exclude `/rest/*` routing to PostgREST.
- Simplified `supabase/deploy.sh` by removing references to PostgREST scaling and alias creation.
- Adjusted cluster configuration and UI to reflect the absence of PostgREST.
- Include new `prole.spec` for build configurations and dependencies.
- Add Terraform state handling for OpenTofu in `k3s` cluster.
- Provision multiple Kubernetes resources in `prole-db` namespace: namespace, services, ConfigMaps, StatefulSets, Ingress rules, and PersistentVolumes.
- Integrate deployment and configuration enhancements for `garage`, `prole`, and related components.
Kong API Gateway (replacing prole nginx):
- Add etc/init_kong.sh provisioning script (DB-less mode, prole-db namespace)
- Add kong-deployment.yaml and kong-service.yaml manifests
- Rewire ingress rules (svc/git/api.prole.org) to prole-db-kong:8000
- Update kustomization.yaml to reference kong manifests
PostgREST & DB Manager in prole-db namespace:
- Add etc/init_postgrest.sh and etc/init_db_manager.sh scripts
- Add postgrest/db-manager deployment and service manifests
- Add src/db-manager/ Node.js REST endpoint for backup triggers
- Default NAMESPACE changed to prole-db in both scripts
Docker image pre-load from PROLE_DATA/docker-import:
- Add _preload_docker_images() to init_common_services.sh
- Scan for .tar files exported by final_deployment.sh
- Import via k3d image import (k3d) or ctr (k3s) before deployments
- Increase rollout timeouts to 300s (configurable via ROLLOUT_TIMEOUT) in init_openbao.sh, init_opentofu.sh, init_garage_store.sh, init_registry.sh
OpenTofu password resolution fix:
- Add Kubernetes secret fallback in resolve_admin_password()
- Change hard exit 1 to graceful return 1 with warning
- Wrap call in if-guard so set -e doesn't abort the script chain
Milestone fix (init scripts not running):
- Add init_kong.sh, init_postgrest.sh, init_db_manager.sh to InitializationScriptsMilestone.execute() script list and arg branches
- Previously only actions.py had these; milestones.py was missing them
Installer integration:
- Add Kong/PostgREST/DB Manager to silent installer _step_init_scripts
- Add corresponding tabs and execution blocks in UI services.py
- install.py: Major update including configuration variable expansion, improved k3s/k3d handling, and enhanced installation logic.
- etc/ scripts: Significant refactoring of initialization scripts (Kerberos, Port Forwards, Garage Store, etc.).
- Port Forwards: Transitioned from XML to port-mappings.conf for managing kubectl port-forwards.
- Status Reporting: Improved status checking for common services.
- Infrastructure: Updated Ansible inventory and rsyslog role configurations.
- Tests: Added a comprehensive suite of tests for 'etc' initialization scripts in prole/tests/etc/.
- Documentation: Added prole-db-documentation-mcp-architecture.md.
- General: Updated Dockerfiles and various helper scripts.
- Add k3s start/stop Ansible playbooks and roles.
- Implement OpenTofu initialization scripts and k8s manifests.
- Update ncurses installer with OpenTofu support and improved k3s integration.
- Add mode support (--mode) to etc/ initialization scripts.
- Update prole-db with recovery, barman objectstore, and SSH OpenBao support.
- Refine k8s manifests for OpenBao and prole-db.
- Refactored install.py and installer package for improved UI and navigation.
- Replaced Supabase k8s manifests with a dedicated deployment script and port-wiring logic.
- Added new deployment pipeline and finalization scripts in etc/.
- Updated initialization scripts for Kerberos, authority, and port forwards.
- Updated port mappings and tests.
This commit introduces automated Supabase deployment and refines the installation process.
Key improvements:
- Added supabase/deploy.sh: A comprehensive script to deploy the full Supabase stack.
- Enhanced etc/init_supabase.sh: Systematically resolved database permission issues.
- Updated install.py: Integrated Supabase setup into the main installer and added monitoring configuration.
- Updated port mappings and initialization scripts to support the Supabase service stack.
