- add tools/manage-node.sh for k3s node add/remove automation with inventory reconciliation
- include kubectl drain flag compatibility and --ssh-host targeting for uninstall workflows
- persist installer cfg/env values and update deploy/database/cfg UI handling
- refresh related inventory/network artifacts and expand installer/database test coverage
Co-authored-by: Junie <junie@jetbrains.com>
Update conf/service/prole.cfg to persist the k3s kube context as knoe.dev.prole.org and replace host-specific absolute paths with $HOME-based paths for portability.
Include all pending project changes: local_user role package and screenrc provisioning updates, plus tools/k3s-cluster-rename.sh and tests/etc/test_k3s_cluster_rename.sh.
Co-authored-by: Junie <junie@jetbrains.com>
- Introduced `knoe.core.prod_config` with production configuration schema and helper methods.
- Added validation logic to enforce required fields and expected formats.
- Implemented in-process API (`ProdConfigApi`) for managing production configs.
- Added test coverage for default config behavior, validation, YAML generation, and API workflow.
- Updated cluster UI layout and tests to integrate production config with new navigation flows.
- Added host inventory and updated service flow to reflect production setup changes.
- Introduced `knoe.core.prod_config` with production configuration schema and helper methods.
- Added validation logic to enforce required fields and expected formats.
- Implemented in-process API (`ProdConfigApi`) for managing production configs.
- Added test coverage for default config behavior, validation, YAML generation, and API workflow.
- Updated cluster UI layout and tests to integrate production config with new navigation flows.
- Added host inventory and updated service flow to reflect production setup changes.
- add reusable storage probing subsystem with discovery, bounded probe execution, IO classification, caching, and topology integration
- render per-node storage inventory in Cluster Nodes UI and extend installer test coverage for topology/storage behavior
- introduce core service operation modules and align actions, milestones, services, and supporting configs/scripts for repair/update workflows
- update CNPG/Supabase/database artifacts, placement and port mapping configs, plus related integration tests
Co-authored-by: Junie <junie@jetbrains.com>
STABLE BUILD — all 35+ pods Running, zero non-Running pods, CNPG ready=3.
## Architecture changes (Supabase slim-down)
- Removed local Supabase Kong, Storage, MinIO, Imgproxy from deployed stack
- Rewired API routing to shared Kong in kube-system
- Rewired object storage to shared Garage in knoe-system
- Supabase footprint reduced to control-plane only: auth, rest, meta,
analytics, realtime (studio port-forward retained)
- Supabase pods renamed: stripped redundant '-prole-supabase-' prefix via
per-component fullnameOverride (e.g. supabase-auth, supabase-rest, ...)
- Supabase workloads pinned to merlin.prole.org (memory-safe node)
## File changes
- conf/service/prole.cfg
* Replaced all hardcoded /Users/chrisfu/dev/prole paths with ${HOME}/dev/prole
so config is portable when run from myrddin.prole.org or any user home
* SUPABASE_ENABLED=True, SUPABASE_PRIMARY_NODE=merlin.prole.org in
both [Global] and [Optional Features]
* supabase_enabled=true in [Inputs]/init_cluster section
- supabase/helm/render_supabase.py
* Explicitly disabled: kong, storage, minio, imgproxy
* Active components pinned via nodeSelector to merlin.prole.org
* fullnameOverride per component to clean up pod names
* Default primary node changed from pi.prole.org -> merlin.prole.org
- supabase/deploy.sh
* Removed svc/kong port-forward; API via shared Kong in kube-system
* Removed imgproxy/storage from PV/PVC prep and helm --set persistence args
* Fixed kubectl exec missing -i flag for psql heredoc stdin in
setup_knoe_db_for_supabase
* Updated Studio access messaging to reference shared platform Kong
- k8s/prole/knoe-db.yaml
* instances: 3
* Added control-plane/master tolerations for myrddin scheduling
- deploy/opentofu/k3s/manifests/prole/knoe-db.yaml
* instances: 3, added control-plane tolerations (mirrors k8s/prole/)
- k8s/prole/iscsi-pvs.yaml
* Expanded d003-data/d003-wal nodeAffinity to [pi, merlin, myrddin]
to reflect post-OOM-recovery live state
- k8s/registry/deployment.yaml
* nodeSelector restored to myrddin.prole.org (hostPort 5000 owner)
* Added control-plane + master tolerations so pod schedules correctly
- modes/k3s/knoe-db/.version
* Aligned to 140 (image tag 18-140) matching repo-root version
- etc/init_common_services.sh, etc/init_registry.sh
* Minor fixes aligned with slimmed service layout and node targeting
- infrastructure/inventory/host_vars/pi.prole.org.yml
* Updated host vars to reflect post-recovery pi state
- knoe/core/topology.py, knoe/ui/screens/cluster.py,
knoe/ui/screens/cluster_nodes.py
* Topology and UI updates for 3-node cluster representation
- scan/network_description.txt
* Refreshed network scan output post-recovery
- conf/port-mapping.cfg
* Updated port mapping to reflect removal of local Kong/storage forwards
- scripts/cleanup_stale_rs.sh (new)
* Utility script to prune orphaned ReplicaSets after rollouts
## Verified stable state
- CNPG: Cluster in healthy state, ready=3
knoe-db-2 (merlin, primary), knoe-db-3 (pi), knoe-db-5 (myrddin)
- Supabase: 5 pods Running on merlin.prole.org
supabase-auth, supabase-rest, supabase-meta,
supabase-analytics, supabase-realtime
- Monitoring: all prometheus/grafana/alertmanager pods Running on merlin
- Registry: Running on myrddin.prole.org (hostPort 5000)
- Zero non-Running / non-Completed pods across all namespaces
Co-authored-by: Junie <junie@jetbrains.com>
- render_supabase.py: read SUPABASE_ADDITIONAL_REDIRECT_URLS from cfg,
normalise each entry to https://.../** wildcard, inject as GOTRUE_URI_ALLOW_LIST
in GoTrue auth env so logins from svc.prole.org are accepted
- conf/service/prole.cfg: set SUPABASE_ADDITIONAL_REDIRECT_URLS=svc.prole.org
- actions.py: preserve SUPABASE_ADDITIONAL_REDIRECT_URLS in globals_to_save
so pipeline write-back never clobbers it
- k8s/prole/merlin-local-iscsi-storageclass.yaml: register StorageClass
merlin-local-iscsi-d002 (no-provisioner/Immediate) to silence
ProvisioningFailed warnings on static Supabase PVs
Result: all 9 Supabase pods Running; svc.prole.org login origin accepted by GoTrue
Co-authored-by: Junie <junie@jetbrains.com>
Fix supabase-storage migration deadlock: ALTER ROLE search_path + BYPASSRLS
Root cause: supabase_storage_admin had no rolconfig search_path, so the connection
string search_path=auth caused the service to query auth.migrations (not found →
currentMigrations=[]) but insert migration tracking rows into public.migrations
(via PostgreSQL implicit public fallback). On restart, currentMigrations was always
[] despite public.migrations having rows, causing duplicate-key on migration 0.
Fix (added to deploy.sh db init block, idempotent on every reinstall):
ALTER ROLE supabase_storage_admin SET search_path = storage, auth, public;
ALTER ROLE supabase_storage_admin BYPASSRLS;
TRUNCATE TABLE IF EXISTS public.migrations;
DROP TABLE IF EXISTS storage.migrations CASCADE;
This aligns migration reads and writes to the same schema (storage first, then auth,
then public) and ensures a clean slate on every redeploy.
Result: all 9 Supabase pods Running on merlin.prole.org (merlin-local-iscsi-d002 iSCSI
storage), db.prole.org returns HTTP 401 (Supabase Studio auth gate = live and working),
CNPG knoe-db 2/2 Ready on merlin + myrddin with Percona PostgreSQL 18 + Barman backup.
Co-authored-by: Junie <junie@jetbrains.com>
Root cause: render_supabase.py wrote to deployment.*.storageClass (ignored by Helm chart PVC
templates) instead of persistence.*.storageClassName (_pvc.tpl:43 is the real path). Also
deploy.sh --set persistence.* args only ran in k8s mode, and stale values.generated.json
was reused across deploys. Together these caused Supabase PVCs to fall through to
pi-local-iscsi (pi SD card), crashing the node under memory pressure.
- supabase/helm/render_supabase.py:
- write persistence.{functions,imgproxy,snippets,storage,deno,minio}.storageClassName
using the correct Helm chart path (_pvc.tpl:43)
- add synology guard: raise SystemExit if storage class is not synology/merlin/myrddin
- remove broken deployment.*.storageClass writes
- supabase/deploy.sh:
- delete stale values.generated.json before every helm_render_values call
- fall back to reading SUPABASE_STORAGE_CLASS from cfg file via python3 if env not set
- add bash synology guard: die if storage class is not iSCSI/NFS mounted
- extend helm_set_args to include persistence.minio.storageClassName and apply
--set persistence.* for ALL modes (not just k8s)
Co-authored-by: Junie <junie@jetbrains.com>
Root cause: render_supabase.py wrote to deployment.*.storageClass (ignored by Helm chart PVC
templates) instead of persistence.*.storageClassName (_pvc.tpl:43 is the real path). Also
deploy.sh --set persistence.* args only ran in k8s mode, and stale values.generated.json
was reused across deploys. Together these caused Supabase PVCs to fall through to
pi-local-iscsi (pi SD card), crashing the node under memory pressure.
- supabase/helm/render_supabase.py:
- write persistence.{functions,imgproxy,snippets,storage,deno,minio}.storageClassName
using the correct Helm chart path (_pvc.tpl:43)
- add synology guard: raise SystemExit if storage class is not synology/merlin/myrddin
- remove broken deployment.*.storageClass writes
- supabase/deploy.sh:
- delete stale values.generated.json before every helm_render_values call
- fall back to reading SUPABASE_STORAGE_CLASS from cfg file via python3 if env not set
- add bash synology guard: die if storage class is not iSCSI/NFS mounted
- extend helm_set_args to include persistence.minio.storageClassName and apply
--set persistence.* for ALL modes (not just k8s)
Co-authored-by: Junie <junie@jetbrains.com>
Memory pressure on merlin (95% OOM with monitoring + CNPG replica + Supabase) resolved
by moving Supabase workload to pi.prole.org which has headroom (49% memory, 2 pods).
- conf/service/prole.cfg: set SUPABASE_PV_NODE, SUPABASE_PRIMARY_NODE=pi.prole.org,
SUPABASE_STORAGE_CLASS=pi-local-iscsi, SUPABASE_PV_BASE_DIR for rancher local-path
- supabase/helm/render_supabase.py: resolve SUPABASE_STORAGE_CLASS from env/cfg,
apply pi-local-iscsi to storage/analytics PVCs, resolve DATABASE_NAMESPACE for
correct db_host, DB_PASSWORD env override takes priority for live cluster password
- supabase/deploy.sh: default SUPABASE_PV_NODE to pi.prole.org, default base dir to
/var/lib/rancher/k3s/storage/supabase, add idempotent ALTER ROLE password sync,
live DB_PASSWORD resolution in helm_render_values from knoe-db-superuser secret
- knoe/core/actions.py: stop overriding init_cluster.supabase_enabled to False in
silent mode so prole.cfg value is honored end-to-end
- etc/init_monitoring.sh: derive pv_prom/pv_am/pv_graf inline before Released-PV
loop to fix unbound variable error on idempotent monitoring installs
Co-authored-by: Junie <junie@jetbrains.com>
- actions.py: stop overriding init_cluster.supabase_enabled to False in silent mode
- supabase/deploy.sh: change SUPABASE_PV_NODE default from myrddin to merlin.prole.org
so prole-supabase-dirprep job runs on the node that owns /synology/d002
- conf/service/prole.cfg: add SUPABASE_PV_NODE and SUPABASE_PV_BASE_DIR to [Global]
so prole_cfg.sh exports them explicitly on every run
- init_monitoring.sh: derive pv_prom/pv_am/pv_graf inline before Released-PV loop
to fix unbound variable error on idempotent monitoring installs
Co-authored-by: Junie <junie@jetbrains.com>
Remove hardcoded override that reset init_cluster.supabase_enabled to
False inside _prepare_k3s_pipeline before _write_cfg() persisted the
value back to prole.cfg. The [Inputs] section in prole.cfg correctly
carries supabase_enabled = true; the pipeline must not clobber it.
- render_supabase.py already defaults ingress host to db.prole.org
- SupabaseMilestone is already wired in the run() milestones list
- No other changes required; single-line removal unblocks Supabase deploy
Co-authored-by: Junie <junie@jetbrains.com>
All active runtime paths that previously shelled out to etc/init_cloudnative_pg.sh
now dispatch through Python. The shell script has been removed from git.
New files:
- knoe/core/ops/cloudnative_pg.py: canonical Python owner for CNPG lifecycle with
public API (initialize, deploy, rollout, ensure_operator, pin_controller,
install_barman_plugin) and internal helpers (_apply_manifest, _wait_cnpg_pods,
_reconcile_instances, etc.)
- prole/tools/run_cnpg_coverage.py: coverage entry points for k3d/k3s modes,
report sub-command, and check-shell scanner to confirm no live Python dispatch
to the removed shell script
Modified files:
- knoe/core/actions.py: replace shell dispatch in _step_init_scripts,
_step_cnpg_deploy, and repair pipeline with Python calls
- knoe/core/milestones.py: replace shell dispatch in InitializationScriptsMilestone
and DeploymentMilestone
- knoe/ui/screens/services.py: replace shell dispatch for init-scripts step,
deploy button, and rollout button
- prole/deployment.py: replace shell dispatch in _run_post_apply_scripts
- status.py: remove init_cloudnative_pg.sh from _STATUS_SCRIPTS list
- tests/installer/test_actions_helpers.py: mock Python functions, assert shell
script is never dispatched
- tests/installer/test_milestones.py: same
- tests/installer/test_services_init_scripts.py: same
Removed:
- etc/init_cloudnative_pg.sh (git rm)
Verification:
- prole.tools.run_cnpg_coverage check-shell reports clean
- All 61 tests in the affected test files pass
Co-authored-by: Junie <junie@jetbrains.com>
- Replace ambiguous NAMESPACE handling with SERVICE_NAMESPACE, DATABASE_NAMESPACE, and CLUSTER_NAME across config parsing/defaults, environment setup, actions, and UI bindings.
- Persist DATABASE_NAMESPACE and CLUSTER_NAME from Database Browser on Next and use them consistently in CNPG deployment/runtime logic.
- Update milestones/tests and include related config/tooling artifacts (cleanup_cnpg_storage tool, service config/version/network updates).
Co-authored-by: Junie <junie@jetbrains.com>
- detect local k3s node kubeconfig and skip kubectx/use-context mutation when already targeting local API\n- add configurable KUBE_CONTEXT_NAME resolution with compatibility fallbacks and switch only when required\n- update init scripts to use ensure_kube_context helper naming\n- broaden monitoring eligibility to discovered /synology/d### mounts so /synology/d004 qualifies\n- add focused kube-context and topology tests covering local/remote and read-only kubeconfig cases
Co-authored-by: Junie <junie@jetbrains.com>
- Introduced `test_init_cloudnative_pg_manifest_validation_fail_fast.sh` script.
- Ensures runtime validation and pod readiness wait are skipped if manifest validation fails.
- Added comprehensive mocks for tools (e.g., `kubectl`, `findmnt`) to simulate behavior.
- Updated script to include detailed error and debug logs during manifest validation steps.
- Adjusted function logic to handle manifest validation failures gracefully without proceeding further.
- set global MODE/DEPLOYMENT_MODE to k3s and remove cross-environment/prod cluster entries from active dev/service config paths\n- switch k3s prole-auth image references to myrddin.prole.org:5000\n- strengthen CNPG image resolution and test coverage for rewriting k3d/localhost registries in k3s mode\n- include updated port mapping/network snapshot and current k3d mode version metadata
Co-authored-by: Junie <junie@jetbrains.com>
Add storage ops for deterministic namespace+cluster local PV paths, labels/selectors, host-path preparation, and idempotent reconciliation before CNPG cluster apply.
Wire selector injection and validation into CNPG deploy flow/script, and extend installer tests for provisioning orchestration and failure handling.
Co-authored-by: Junie <junie@jetbrains.com>
- Introduce `cnpg_placement.py` to handle round-robin node assignments for CloudNativePG clusters.
- Implement persistent placement plans with schema normalization and hashing.
- Add `load_cnpg_placement_plan` and `save_cnpg_placement_plan` for plan persistence.
- Integrate planner with installer to support node-based topology configuration.
- Update `actions.py` with placement planning logic, including rebalance support and node eligibility checks.
- Extend shell utilities (`init_cloudnative_pg.sh`) for placement-aware CNPG topology adjustments.
- Add comprehensive unit tests and integration tests for planner functionality, persistence, and shell environment exports.
- Integrate `init_registry.sh` into service-layer initialization and cleanup flows.
- Update dependency order: Registry initializes first to support downstream services.
- Refactor cleanup sequence: Registry stops last in reverse dependency order.
- Enhance action mapping for registry lifecycle operations.
- Establish deployment sequence: Registry → OpenBao → Garage → OpenTofu.
- Relocate `init_registry.sh` call earlier in flow to align with dependencies.
- Update warning messages for missing script scenarios.
- Reorganize `init_opentofu.sh` to execute last as it depends on prior services.
- Introduce dynamic endpoint candidate selection for registry pushes, prioritizing localhost:5000 when applicable.
- Refactor to remove reliance on port-forwarding; fallback to direct node import for k3s.
- Enhance registry endpoint probing using both HTTP and HTTPS.
- Consolidate docker and skopeo push logic for robustness in multi-endpoint scenarios.
- Update network discovery to reflect restructured data.
- Use `scp` to transfer image tar files to remote nodes before importing, ensuring reliability for large images.
- Clean up temporary files on remote nodes post-import or in case of failure.
- Extend readiness probe to support HTTP alongside HTTPS endpoints.
- _prole_cfg_bootstrap_env_layout now detects when conf_dir is already
an env-specific directory (dev, service, prod, test, k3d, k3s, k8s)
and skips creating env subdirectories to avoid conf/service/service/.
- When conf_dir is an env dir with a stale symlink, resolve it to a
regular file instead of creating nested paths.
- Remove spurious conf/service/service/prole.cfg from the repo.
- Replace conf/service/prole.cfg symlink with the actual config file.
Co-authored-by: Junie <junie@jetbrains.com>
- Rename iSCSI storage class and PV/PVC selectors/labels from prole to synology across k8s and OpenTofu manifests\n- Update CNPG/OpenBao/Garage/monitoring init flows, render helpers, and mock scripts for synology-backed storage objects\n- Integrate related UI/core/service config/version updates and add supporting regression tests for CNPG storage/image behavior\n- Keep storage reconciliation tests aligned with current CNPG affinity output
Co-authored-by: Junie <junie@jetbrains.com>
- Introduce mode-scoped directories under `~/.prole` for build contexts, runtime data, and versioning.
- Update DB image version generation and Dockerfile creation to respect deployment modes (e.g., k3d, k3s, k8s).
- Adjust UI, core actions, and scripts to avoid cross-mode conflicts and ensure correct version isolation.
- Add tests for mode-aware behaviors and update existing tests for new paths and markers.
- Bump PostgreSQL and image versions to `18` and `138`, respectively.
- Update Kubernetes manifests to align with new image and version configuration.
- Implement centralized `resolve_prole_home` utility for consistent environment-based `PROLE_HOME` resolution across modules
- Replace hardcoded home paths with `resolve_prole_home`
- Refactor PV management to support iSCSI mounts and node placement from Ansible manifests
- Improve Kubernetes manifest handling to dynamically apply namespaces per document
- Adjust `knoe-db` build context path and related tests
- Add utilities for detecting and applying Ansible-defined node labels and PVs
- Rename host mount namespace from /prole/d00x to /synology/d00x
- Make LUN ownership explicit per host in inventory (myrddin=d001; merlin=d002,d004; pi=d003)
- Add safe migration cleanup for legacy /prole/d00x mounts and stale /etc/fstab entries
- Update k3s/ArgoCD/OpenTofu manifests and PV node affinities to match new mount paths
- Improve iSCSI role check-mode behavior and add quiesce/detach flow in ansible.sh
- Add Cluster Nodes spreadsheet screen for host/service placement (Primary Host vs Enabled Here)
- Wire navigation to show Cluster Nodes after Cluster Environment (multi-node, non-k3d)
- Update screen registry/base helpers and related layout/navigation tests
- Refresh port mappings and config fixtures
