Summarize CLAUDE.md updates:
- Document both live GKE clusters as e2-standard-2 x3 in us-west3 and note stale app default in code.
- Call out DB cluster naming mismatch between conf (knoe-dev-cnpg-0) and code default (knoe-cnpg-0).
- Add SSD quota guidance: 300GB fully consumed by CNPG, so non-CNPG PVCs must use standard/pd-standard.
- Refresh reset script reference and quota section details to match current runtime state.
Co-authored-by: Junie <junie@jetbrains.com>
- Added skipping logic for aged or bound PVC events in `deploy.sh` to reduce noise in diagnostics.
- Enforced replica target of 1 for specific GitLab workloads to ensure compliance with requirements.
- Improved PVC error diagnostics in `deploy.sh` with additional context.
- Added robust StorageClass reconciliation logic, including dynamic creation and parameter validation.
- Updated Helm templates to support advanced StorageClass attributes like `reclaimPolicy` and `volumeBindingMode`.
- Redirected error messages in `deploy.sh` and `render_supabase.py` to `stderr` for better logging clarity.
- Extended allowed prefixes for Supabase StorageClass to include `pd-standard`.
- Added hard validation for Supabase StorageClass configuration in `deploy.sh` and Helm templates.
- Implemented detailed reconciliation logic for GitLab workloads, addressing over-deployment scenarios.
- Enhanced `deploy.sh` with explicit namespace handling and PVC compliance checks.
- Added PVC quota error detection with detailed logging in `deploy.sh`.
- Enhanced ingress readiness checks with additional Supabase ingress resources.
- Updated Helm templates to standardize and always enable ingress for API and Studio.
- Added PVC quota error detection with detailed logging in `deploy.sh`.
- Enhanced ingress readiness checks with additional Supabase ingress resources.
- Updated Helm templates to standardize and always enable ingress for API and Studio.
- Simplified public APP endpoint fallback logic in `deploy.sh`.
- Enhanced timeout handling with detailed Supabase ingress reconciliation diagnostics.
- Removed redundant ingress class override in Helm template for Supabase.
- Standardized naming from "Prole" to "Knoe" in README, scripts, and deployment files for consistency.
- Refined ingress configuration to adjust class and annotations based on mode.
- Enhanced deploy.sh to handle public APP endpoint status for specific hosts.
- Standardized naming from "Prole" to "Knoe" in README, scripts, and deployment files for consistency.
- Refined ingress configuration to adjust class and annotations based on mode.
- Enhanced deploy.sh to handle public APP endpoint status for specific hosts.
- Resolved context mismatches by updating DB references to APP where appropriate.
- Updated Supabase deployment to ensure proper handling of cross-cluster DB hosts.
- Simplified ingress logic by setting `split_frontdoor_to_db` to `false` for APP clusters lacking GCE Ingress support.
- Resolved context mismatches by updating DB references to APP where appropriate.
- Updated Supabase deployment to ensure proper handling of cross-cluster DB hosts.
- Simplified ingress logic by setting `split_frontdoor_to_db` to `false` for APP clusters lacking GCE Ingress support.
- Added automatic persistence disabling when deployment is disabled.
- Enhanced readiness checks with support for selectors and detailed status logging.
- Improved retry logic to avoid unnecessary namespace resets for healthy app clusters.
- Refined helm deployment flow with clearer readiness and health diagnostics.
- Introduced diagnostics for MinIO failures due to file access issues, with suggested fixes for pod security settings.
- Added podSecurityContext and securityContext configurations to enforce non-root execution for MinIO.
- Updated resource requests/limits for MinIO pods to ensure optimal resource usage.
- Standardized file path references in manifest-summary.json to relative paths.
- Changed default GKE non-DB PVCs to use CSI `pd-standard` with WFFC mode, deprecating legacy `standard`.
- Added reconciliation logic for APP PVCs to align with updated storage class.
- Enforced single replica for Supabase functions on GKE, disabling autoscaling.
- Updated tests to validate storage class changes, PVC reconciliation, and single-replica logic.
- remove hardcoded physical host assumptions in UI/init flows and rely on config-driven selectors\n- optimize GitLab reruns to skip unnecessary operator upgrades/long reconcile waits when unchanged\n- add and update regression/shell tests for fast-path and k8s context/ingress/garage behaviors
Co-authored-by: Junie <junie@jetbrains.com>
- Refactored `init_gitlab.sh` and `init_gitea.sh` to use configurable node selector keys, removing hardcoded defaults like `gandalf.prole.org`.
- Enhanced scheduling logic to validate required fields and prevent stale node constraints during reruns.
- Added fast-path guards in GitLab init to skip redundant operations when no changes occur in operator or CR specifications.
- Updated Supabase deployment to clear stale topology spread constraints alongside node selectors and affinity.
- Added tests for configurable node selector keys, fast-path guards, and stale constraint cleanups.
- Changed default storage class for GKE to `standard` to reduce pressure on SSD/balanced quotas.
- Disabled `enforceGeneralNodeRole` scheduling for APP/DB values to support split cluster setups.
- Updated tests to validate storage class and scheduling logic changes.
- Introduced `SERVICE_INGRESS_TLS_ENABLED` for configurable svc ingress TLS rendering in `init_kong.sh`.
- Updated ingress templates to include or exclude TLS annotations and blocks dynamically.
- Improved PVC and pod readiness checks in `supabase/deploy.sh` with enriched logging and detailed blockers.
- Added support for configurable storage class resolution for Supabase DB frontdoor PVCs.
- Enhanced deployment workflows to handle split APP/DB cluster setups with context-aware readiness checks.
- Updated test cases to validate svc ingress TLS configuration and storage class logic.
- Introduced `enforceGeneralNodeRole` to make node affinity and selector enforcement configurable.
- Updated `_split_frontdoor_docs` to handle custom release labels for Kong services.
- Enhanced Helm templates to support dynamic scheduling configurations.
- Added `reconcile_db_frontdoor_studio_pvcs` to migrate PVCs to the target storage class.
- Updated tests to validate custom release labels and scheduling logic.
- Updated `_extract_k8s_docs` to use `yaml.safe_load_all` for improved parsing and reliability.
- Enhanced `_split_frontdoor_docs` with stricter validation of Deployment, Service, and Ingress specs.
- Added namespace enforcement for frontdoor resources during manifest splitting.
- Refactored and optimized test cases to cover new validation and splitting functionality.
- Updated `_extract_k8s_docs` to use `yaml.safe_load_all` for improved parsing and reliability.
- Enhanced `_split_frontdoor_docs` with stricter validation of Deployment, Service, and Ingress specs.
- Added namespace enforcement for frontdoor resources during manifest splitting.
- Refactored and optimized test cases to cover new validation and splitting functionality.
- Added validation and explicit handling for Garage S3 endpoints in split APP/DB cluster setups.
- Improved dynamic role detection and authoritative context enforcement in initialization scripts.
- Updated Supabase and Garage configurations to support private cross-cluster endpoints, ensuring cluster interoperability and DNS constraints.
- Enhanced error handling and logging for invalid or missing configurations.
- Introduced jemalloc hostPath optimizations with configurable modes (`auto`, `off`, `force`).
- Integrated jemalloc setup with best-effort and forced validation flows for ensuring cluster compatibility.
- Enhanced monitoring storage class logic with mode-specific handling (`k3s`, `k3d`, `gke`) and improved validation of required classes.
- Added safeguards and detailed logging for unsupported configurations and failure scenarios.
Rename env config files from conf/*/prole.cfg to conf/k3d.cfg, conf/k3s.cfg, and conf/gke.cfg. Update shell/Python loaders and etc/deploy scripts to resolve named configs cleanly while keeping legacy fallback behavior. Align k3s Ansible tasks, docs, and regression coverage with the new configuration layout.
Co-authored-by: Junie <junie@jetbrains.com>
- Introduced support for dynamic GitLab and Supabase domain and ingress configuration based on deployment mode (k8s vs local).
- Default GitLab domain and ingress class aligned with `prole.cfg` deployment settings.
- Expanded Supabase support:
- Added configurable API and Studio hostnames with default fallbacks.
- Introduced separate ingress class configuration for API and Studio.
- Standardized public URL normalization for both services.
- Updated tests to validate domain, ingress, and configuration behaviors across environments.
- harden CNPG backup ObjectStore readiness fallback in statusless clusters by validating controller availability and expected spec
- improve backup/plugin reconciliation paths and related test coverage for barman-cloud flows
- include ongoing GKE/CNPG deployment and configuration updates from this checkpoint
Co-authored-by: Junie <junie@jetbrains.com>
- update Kong/Supabase ingress and service manifests for k3s
- expand init scripts and runtime config/topology handling
- refresh KNØE UI screens and Supabase deployment/render logic
- include related env helper test adjustments
Co-authored-by: Junie <junie@jetbrains.com>
- add reusable storage probing subsystem with discovery, bounded probe execution, IO classification, caching, and topology integration
- render per-node storage inventory in Cluster Nodes UI and extend installer test coverage for topology/storage behavior
- introduce core service operation modules and align actions, milestones, services, and supporting configs/scripts for repair/update workflows
- update CNPG/Supabase/database artifacts, placement and port mapping configs, plus related integration tests
Co-authored-by: Junie <junie@jetbrains.com>
STABLE BUILD — all 35+ pods Running, zero non-Running pods, CNPG ready=3.
## Architecture changes (Supabase slim-down)
- Removed local Supabase Kong, Storage, MinIO, Imgproxy from deployed stack
- Rewired API routing to shared Kong in kube-system
- Rewired object storage to shared Garage in knoe-system
- Supabase footprint reduced to control-plane only: auth, rest, meta,
analytics, realtime (studio port-forward retained)
- Supabase pods renamed: stripped redundant '-prole-supabase-' prefix via
per-component fullnameOverride (e.g. supabase-auth, supabase-rest, ...)
- Supabase workloads pinned to merlin.prole.org (memory-safe node)
## File changes
- conf/service/prole.cfg
* Replaced all hardcoded /Users/chrisfu/dev/prole paths with ${HOME}/dev/prole
so config is portable when run from myrddin.prole.org or any user home
* SUPABASE_ENABLED=True, SUPABASE_PRIMARY_NODE=merlin.prole.org in
both [Global] and [Optional Features]
* supabase_enabled=true in [Inputs]/init_cluster section
- supabase/helm/render_supabase.py
* Explicitly disabled: kong, storage, minio, imgproxy
* Active components pinned via nodeSelector to merlin.prole.org
* fullnameOverride per component to clean up pod names
* Default primary node changed from pi.prole.org -> merlin.prole.org
- supabase/deploy.sh
* Removed svc/kong port-forward; API via shared Kong in kube-system
* Removed imgproxy/storage from PV/PVC prep and helm --set persistence args
* Fixed kubectl exec missing -i flag for psql heredoc stdin in
setup_knoe_db_for_supabase
* Updated Studio access messaging to reference shared platform Kong
- k8s/prole/knoe-db.yaml
* instances: 3
* Added control-plane/master tolerations for myrddin scheduling
- deploy/opentofu/k3s/manifests/prole/knoe-db.yaml
* instances: 3, added control-plane tolerations (mirrors k8s/prole/)
- k8s/prole/iscsi-pvs.yaml
* Expanded d003-data/d003-wal nodeAffinity to [pi, merlin, myrddin]
to reflect post-OOM-recovery live state
- k8s/registry/deployment.yaml
* nodeSelector restored to myrddin.prole.org (hostPort 5000 owner)
* Added control-plane + master tolerations so pod schedules correctly
- modes/k3s/knoe-db/.version
* Aligned to 140 (image tag 18-140) matching repo-root version
- etc/init_common_services.sh, etc/init_registry.sh
* Minor fixes aligned with slimmed service layout and node targeting
- infrastructure/inventory/host_vars/pi.prole.org.yml
* Updated host vars to reflect post-recovery pi state
- knoe/core/topology.py, knoe/ui/screens/cluster.py,
knoe/ui/screens/cluster_nodes.py
* Topology and UI updates for 3-node cluster representation
- scan/network_description.txt
* Refreshed network scan output post-recovery
- conf/port-mapping.cfg
* Updated port mapping to reflect removal of local Kong/storage forwards
- scripts/cleanup_stale_rs.sh (new)
* Utility script to prune orphaned ReplicaSets after rollouts
## Verified stable state
- CNPG: Cluster in healthy state, ready=3
knoe-db-2 (merlin, primary), knoe-db-3 (pi), knoe-db-5 (myrddin)
- Supabase: 5 pods Running on merlin.prole.org
supabase-auth, supabase-rest, supabase-meta,
supabase-analytics, supabase-realtime
- Monitoring: all prometheus/grafana/alertmanager pods Running on merlin
- Registry: Running on myrddin.prole.org (hostPort 5000)
- Zero non-Running / non-Completed pods across all namespaces
Co-authored-by: Junie <junie@jetbrains.com>
- render_supabase.py: read SUPABASE_ADDITIONAL_REDIRECT_URLS from cfg,
normalise each entry to https://.../** wildcard, inject as GOTRUE_URI_ALLOW_LIST
in GoTrue auth env so logins from svc.prole.org are accepted
- conf/service/prole.cfg: set SUPABASE_ADDITIONAL_REDIRECT_URLS=svc.prole.org
- actions.py: preserve SUPABASE_ADDITIONAL_REDIRECT_URLS in globals_to_save
so pipeline write-back never clobbers it
- k8s/prole/merlin-local-iscsi-storageclass.yaml: register StorageClass
merlin-local-iscsi-d002 (no-provisioner/Immediate) to silence
ProvisioningFailed warnings on static Supabase PVs
Result: all 9 Supabase pods Running; svc.prole.org login origin accepted by GoTrue
Co-authored-by: Junie <junie@jetbrains.com>
Fix supabase-storage migration deadlock: ALTER ROLE search_path + BYPASSRLS
Root cause: supabase_storage_admin had no rolconfig search_path, so the connection
string search_path=auth caused the service to query auth.migrations (not found →
currentMigrations=[]) but insert migration tracking rows into public.migrations
(via PostgreSQL implicit public fallback). On restart, currentMigrations was always
[] despite public.migrations having rows, causing duplicate-key on migration 0.
Fix (added to deploy.sh db init block, idempotent on every reinstall):
ALTER ROLE supabase_storage_admin SET search_path = storage, auth, public;
ALTER ROLE supabase_storage_admin BYPASSRLS;
TRUNCATE TABLE IF EXISTS public.migrations;
DROP TABLE IF EXISTS storage.migrations CASCADE;
This aligns migration reads and writes to the same schema (storage first, then auth,
then public) and ensures a clean slate on every redeploy.
Result: all 9 Supabase pods Running on merlin.prole.org (merlin-local-iscsi-d002 iSCSI
storage), db.prole.org returns HTTP 401 (Supabase Studio auth gate = live and working),
CNPG knoe-db 2/2 Ready on merlin + myrddin with Percona PostgreSQL 18 + Barman backup.
Co-authored-by: Junie <junie@jetbrains.com>
Root cause: render_supabase.py wrote to deployment.*.storageClass (ignored by Helm chart PVC
templates) instead of persistence.*.storageClassName (_pvc.tpl:43 is the real path). Also
deploy.sh --set persistence.* args only ran in k8s mode, and stale values.generated.json
was reused across deploys. Together these caused Supabase PVCs to fall through to
pi-local-iscsi (pi SD card), crashing the node under memory pressure.
- supabase/helm/render_supabase.py:
- write persistence.{functions,imgproxy,snippets,storage,deno,minio}.storageClassName
using the correct Helm chart path (_pvc.tpl:43)
- add synology guard: raise SystemExit if storage class is not synology/merlin/myrddin
- remove broken deployment.*.storageClass writes
- supabase/deploy.sh:
- delete stale values.generated.json before every helm_render_values call
- fall back to reading SUPABASE_STORAGE_CLASS from cfg file via python3 if env not set
- add bash synology guard: die if storage class is not iSCSI/NFS mounted
- extend helm_set_args to include persistence.minio.storageClassName and apply
--set persistence.* for ALL modes (not just k8s)
Co-authored-by: Junie <junie@jetbrains.com>
Memory pressure on merlin (95% OOM with monitoring + CNPG replica + Supabase) resolved
by moving Supabase workload to pi.prole.org which has headroom (49% memory, 2 pods).
- conf/service/prole.cfg: set SUPABASE_PV_NODE, SUPABASE_PRIMARY_NODE=pi.prole.org,
SUPABASE_STORAGE_CLASS=pi-local-iscsi, SUPABASE_PV_BASE_DIR for rancher local-path
- supabase/helm/render_supabase.py: resolve SUPABASE_STORAGE_CLASS from env/cfg,
apply pi-local-iscsi to storage/analytics PVCs, resolve DATABASE_NAMESPACE for
correct db_host, DB_PASSWORD env override takes priority for live cluster password
- supabase/deploy.sh: default SUPABASE_PV_NODE to pi.prole.org, default base dir to
/var/lib/rancher/k3s/storage/supabase, add idempotent ALTER ROLE password sync,
live DB_PASSWORD resolution in helm_render_values from knoe-db-superuser secret
- knoe/core/actions.py: stop overriding init_cluster.supabase_enabled to False in
silent mode so prole.cfg value is honored end-to-end
- etc/init_monitoring.sh: derive pv_prom/pv_am/pv_graf inline before Released-PV
loop to fix unbound variable error on idempotent monitoring installs
Co-authored-by: Junie <junie@jetbrains.com>
- actions.py: stop overriding init_cluster.supabase_enabled to False in silent mode
- supabase/deploy.sh: change SUPABASE_PV_NODE default from myrddin to merlin.prole.org
so prole-supabase-dirprep job runs on the node that owns /synology/d002
- conf/service/prole.cfg: add SUPABASE_PV_NODE and SUPABASE_PV_BASE_DIR to [Global]
so prole_cfg.sh exports them explicitly on every run
- init_monitoring.sh: derive pv_prom/pv_am/pv_graf inline before Released-PV loop
to fix unbound variable error on idempotent monitoring installs
Co-authored-by: Junie <junie@jetbrains.com>
- Rename iSCSI storage class and PV/PVC selectors/labels from prole to synology across k8s and OpenTofu manifests\n- Update CNPG/OpenBao/Garage/monitoring init flows, render helpers, and mock scripts for synology-backed storage objects\n- Integrate related UI/core/service config/version updates and add supporting regression tests for CNPG storage/image behavior\n- Keep storage reconciliation tests aligned with current CNPG affinity output
Co-authored-by: Junie <junie@jetbrains.com>
