k3d clusters only have local-path provisioner; defaulting to synology-iscsi
caused all Supabase PVCs to get stuck Pending. Also add local-path to the
validation allowlist so it is not rejected as an unknown storage class.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Second instance of the "chart Ingress ghosts out of the cluster shortly
after helm install" pattern. Earlier we worked around it for the kong
ingress by emitting a standalone manifest; studio/ingress.yaml now shows
the same symptom -- helm creates it, cluster reports `supabase-studio
is missing` seconds later, db.0.knoe.dev returns ERR_CONNECTION_CLOSED.
Rather than duplicate the externally-managed workaround for a second
ingress, this retires studio's ingress entirely and merges db.0.knoe.dev
onto the already-working supabase-kong ingress.
Kong's declarative config has always had a `dashboard` service block
that forwards `/` to supabase-studio:3000 (templates/kong/config.yaml).
So traffic for db.0.knoe.dev flowing through supabase-kong lands on
Studio exactly the same way it would have through the separate ingress,
just with one extra hop through Kong. From a user perspective:
identical. From our perspective: one ingress, one static IP, one
BackendConfig, no ghosting for Studio.
GCE supports multiple ManagedCertificates per ingress via a
comma-separated `networking.gke.io/managed-certificates` annotation;
each cert covers its own SAN. We already had both
supabase-api-managed-cert (api.0.knoe.dev) and
supabase-studio-managed-cert (db.0.knoe.dev) Active, so stitch them
both onto the shared kong ingress and SNI routes cleanly.
Changes in supabase/helm/render_supabase.py:
- `studioIngress.enabled` flips to false in k8s mode (chart's
studio/ingress.yaml is already gated on this value, so it no-ops)
- `service.studio.backendConfigName` cleared in k8s mode (chart's
studio/backendconfig.yaml and the studio Service annotation both
gate on this, so they no-op too -- single source of truth stays
in render_supabase.py)
- Standalone kong-ingress generator now merges studioIngress.hosts as
additional rules, same supabase-kong:8000 backend (Kong handles the
internal routing to Studio)
- Same generator merges studio's ManagedCertificate into the kong
ingress annotation as a comma-separated cert list
Studio's Service keeps existing (needed for Kong's in-cluster
`http://supabase-studio:3000` upstream). Only the public-facing Ingress
and BackendConfig go away.
Deployment-side follow-ups (not code):
- db.0.knoe.dev DNS must be repointed from 136.110.189.6
(the supabase-studio static IP) to 34.120.221.5 (the supabase-api
static IP, which is what this merged ingress is bound to).
- Once DNS is cut over, the supabase-studio global static IP is
orphaned and can be released to free the quota slot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GCE L7 Ingresses for knoe-svc-kong (svc.knoe.dev / api.knoe.dev),
supabase-kong (api.0.knoe.dev) and supabase-studio (db.0.knoe.dev) were
all stuck UNHEALTHY: the default GCE healthCheck is HTTP GET `/` on the
backend port, but Kong returns 404 on any unrouted path and Studio
returns a 301 redirect -- neither passes the default probe, so the LB
serves "Server Error" instead of reaching the pod.
Replicate the pattern already working for gitlab-webservice-default:
emit a BackendConfig CRD with a TCP healthCheck on the service port and
annotate the Service with cloud.google.com/backend-config so GCE picks
it up. TCP is sufficient for LB-level liveness -- the backend is "alive"
as long as the process is accepting connections.
- etc/init_kong.sh: new SVC_KNOE_BACKEND_CONFIG_NAME; apply BackendConfig
inside k8s/GCE branch; annotate Service post-apply.
- knoe-supabase chart: new kong/backendconfig.yaml + studio/backendconfig.yaml
(TCP on 8000 / 3000), gated on service.{kong,studio}.backendConfigName.
- knoe-supabase chart: kong/service.yaml + studio/service.yaml pick up
cloud.google.com/backend-config when backendConfigName is set.
- render_supabase.py: sets service.{kong,studio}.backendConfigName in k8s
mode so the above wires up automatically.
Separately, the chart-managed supabase-kong Ingress was being reaped
from the cluster seconds after helm install (manifest present in the
release, gone via `kubectl get`). Root cause TBD -- suspected
meta.helm.sh/* annotation ownership colliding with a GKE/Anthos audit
controller. Workaround: render_supabase.py now emits a standalone
public-ingress-kong.yaml (no helm metadata) that supabase/deploy.sh
applies alongside public-ingress-tls.yaml, and the chart template gains
an `ingress.externallyManaged` guard so it no-ops in k8s mode. Default
`false` keeps k3d/k3s behavior unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reserved 5 named external static IPs and threaded the
kubernetes.io/ingress.global-static-ip-name annotation through every
public GCE L7 ingress the installer renders, so ingress delete/recreate
stops churning DNS. Also turned on a dedicated regional L4
LoadBalancer for gitlab-shell (port 22) so SSH git workflows work.
New config surface in conf/gke.cfg:
GITLAB_GLOBAL_STATIC_IP_NAME = git-knoe-dev (34.102.141.87)
SVC_KNOE_GLOBAL_STATIC_IP_NAME = svc-knoe (34.111.197.33)
SUPABASE_API_GLOBAL_STATIC_IP_NAME = supabase-api (34.120.221.5)
SUPABASE_STUDIO_GLOBAL_STATIC_IP_NAME = supabase-studio (136.110.189.6)
GITLAB_SSH_HOST = git-ssh.knoe.dev
GITLAB_SHELL_LOADBALANCER_IP = 34.106.243.154 (regional us-west3)
GITLAB_SHELL_EXTERNAL_TRAFFIC_POLICY = Local (default)
Why SSH lives on a separate hostname+IP: Google-managed certs require
a GCE global L7 IP; port 22 needs a regional Network LB. Those can not
share an IP on GCP, so git.knoe.dev stays on HTTPS and git-ssh.knoe.dev
takes SSH.
Wiring per surface:
- etc/init_gitlab.sh
* Added GITLAB_SSH_HOST (default git-ssh.<root> in k8s mode),
GITLAB_SHELL_LOADBALANCER_IP, GITLAB_SHELL_EXTERNAL_TRAFFIC_POLICY,
GITLAB_GLOBAL_STATIC_IP_NAME config keys.
* CR global.hosts.ssh now reads ${GITLAB_SSH_HOST}.
* gitlab-shell block conditionally renders service: {type:
LoadBalancer, loadBalancerIP, externalTrafficPolicy} when the LB IP
is set.
* GITLAB_GCE_TLS_ANNOTATIONS_YAML gains
kubernetes.io/ingress.global-static-ip-name: "<name>" when
GITLAB_GLOBAL_STATIC_IP_NAME is set.
- etc/init_kong.sh
* Added SVC_KNOE_GLOBAL_STATIC_IP_NAME var near other SERVICE_TLS_*
defaults.
* gce_tls_annotations heredoc gets the static-IP line appended on the
same condition.
- supabase/helm/render_supabase.py
* Two new _first(env, cfg, default) extractions for
SUPABASE_API_GLOBAL_STATIC_IP_NAME and
SUPABASE_STUDIO_GLOBAL_STATIC_IP_NAME near the existing managed-cert
/ frontend-config vars.
* Two new dict-spread blocks in the Kong + Studio ingress annotations
that emit the static-IP annotation only when mode=k8s and the value
is non-empty.
All three ingress surfaces already use kubectl apply (merge-friendly);
re-running init_gitlab.sh / init_kong.sh / deploy.sh (supabase step) is
enough to pick up the new annotation. The GKE LB controller will swap
each ingress's forwarding rule from the auto-generated k8s2-fr-*
reservation to the named reservation, then release the old ephemeral.
Ingress IPs change; DNS records need updating (TTL <=300s while
iterating). Google-managed cert for gitlab-managed-cert will briefly
Provision again during the swap; the other three were already
Provisioning.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Introduced dependency validation for ManagedCertificate and FrontendConfig annotations in Supabase ingresses.
- Enhanced `deploy.sh` with public ingress TLS resource application and improved error diagnostics.
- Updated `render_supabase.py` to generate GKE TLS manifests and log rendered resources.
- Refined Supabase deployment flow to include detailed ingress dependency validation and reconciliation.
- Improved handling of GKE preload TLS annotations by ignoring `SERVICE_PRE_SHARED_CERT` in k8s mode with managed TLS resources.
- Added drift detection for ingress class, ManagedCertificate, and FrontendConfig annotations, with forced replacement for GCLB binding consistency.
- Enhanced TLS diagnostics with detailed path state, failure reasons, and HTTPS probe handling for GKE configurations.
- Updated tests and scripts to validate GKE-specific behavior and ingress rendering.
- Updated `deploy.sh` and `init_kong.sh` to support GKE ManagedCertificate and FrontendConfig reconciliation in k8s mode with GCE ingress class.
- Enhanced TLS path diagnostics to distinguish between missing, attached-but-not-serving, and actively serving configurations.
- Added HTTPS probing for GKE-managed public ingress paths and validation for managed certificate statuses.
- Updated `render_supabase.py` to generate and attach TLS annotations for Supabase API and Studio ingresses in k8s mode.
- Added tests to validate TLS path handling, probing, and manifest generation for GKE deployments.
Summarize CLAUDE.md updates:
- Document both live GKE clusters as e2-standard-2 x3 in us-west3 and note stale app default in code.
- Call out DB cluster naming mismatch between conf (knoe-dev-cnpg-0) and code default (knoe-cnpg-0).
- Add SSD quota guidance: 300GB fully consumed by CNPG, so non-CNPG PVCs must use standard/pd-standard.
- Refresh reset script reference and quota section details to match current runtime state.
Co-authored-by: Junie <junie@jetbrains.com>
- Improved PVC error diagnostics in `deploy.sh` with additional context.
- Added robust StorageClass reconciliation logic, including dynamic creation and parameter validation.
- Updated Helm templates to support advanced StorageClass attributes like `reclaimPolicy` and `volumeBindingMode`.
- Redirected error messages in `deploy.sh` and `render_supabase.py` to `stderr` for better logging clarity.
- Extended allowed prefixes for Supabase StorageClass to include `pd-standard`.
- Added hard validation for Supabase StorageClass configuration in `deploy.sh` and Helm templates.
- Implemented detailed reconciliation logic for GitLab workloads, addressing over-deployment scenarios.
- Enhanced `deploy.sh` with explicit namespace handling and PVC compliance checks.
- Added PVC quota error detection with detailed logging in `deploy.sh`.
- Enhanced ingress readiness checks with additional Supabase ingress resources.
- Updated Helm templates to standardize and always enable ingress for API and Studio.
- Added PVC quota error detection with detailed logging in `deploy.sh`.
- Enhanced ingress readiness checks with additional Supabase ingress resources.
- Updated Helm templates to standardize and always enable ingress for API and Studio.
- Simplified public APP endpoint fallback logic in `deploy.sh`.
- Enhanced timeout handling with detailed Supabase ingress reconciliation diagnostics.
- Removed redundant ingress class override in Helm template for Supabase.
- Standardized naming from "Prole" to "Knoe" in README, scripts, and deployment files for consistency.
- Refined ingress configuration to adjust class and annotations based on mode.
- Enhanced deploy.sh to handle public APP endpoint status for specific hosts.
- Standardized naming from "Prole" to "Knoe" in README, scripts, and deployment files for consistency.
- Refined ingress configuration to adjust class and annotations based on mode.
- Enhanced deploy.sh to handle public APP endpoint status for specific hosts.
- Introduced diagnostics for MinIO failures due to file access issues, with suggested fixes for pod security settings.
- Added podSecurityContext and securityContext configurations to enforce non-root execution for MinIO.
- Updated resource requests/limits for MinIO pods to ensure optimal resource usage.
- Standardized file path references in manifest-summary.json to relative paths.
- Changed default GKE non-DB PVCs to use CSI `pd-standard` with WFFC mode, deprecating legacy `standard`.
- Added reconciliation logic for APP PVCs to align with updated storage class.
- Enforced single replica for Supabase functions on GKE, disabling autoscaling.
- Updated tests to validate storage class changes, PVC reconciliation, and single-replica logic.
- Introduced `enforceGeneralNodeRole` to make node affinity and selector enforcement configurable.
- Updated `_split_frontdoor_docs` to handle custom release labels for Kong services.
- Enhanced Helm templates to support dynamic scheduling configurations.
- Added `reconcile_db_frontdoor_studio_pvcs` to migrate PVCs to the target storage class.
- Updated tests to validate custom release labels and scheduling logic.
- Updated `_extract_k8s_docs` to use `yaml.safe_load_all` for improved parsing and reliability.
- Enhanced `_split_frontdoor_docs` with stricter validation of Deployment, Service, and Ingress specs.
- Added namespace enforcement for frontdoor resources during manifest splitting.
- Refactored and optimized test cases to cover new validation and splitting functionality.
- Updated `_extract_k8s_docs` to use `yaml.safe_load_all` for improved parsing and reliability.
- Enhanced `_split_frontdoor_docs` with stricter validation of Deployment, Service, and Ingress specs.
- Added namespace enforcement for frontdoor resources during manifest splitting.
- Refactored and optimized test cases to cover new validation and splitting functionality.
- Added validation and explicit handling for Garage S3 endpoints in split APP/DB cluster setups.
- Improved dynamic role detection and authoritative context enforcement in initialization scripts.
- Updated Supabase and Garage configurations to support private cross-cluster endpoints, ensuring cluster interoperability and DNS constraints.
- Enhanced error handling and logging for invalid or missing configurations.
- Introduced jemalloc hostPath optimizations with configurable modes (`auto`, `off`, `force`).
- Integrated jemalloc setup with best-effort and forced validation flows for ensuring cluster compatibility.
- Enhanced monitoring storage class logic with mode-specific handling (`k3s`, `k3d`, `gke`) and improved validation of required classes.
- Added safeguards and detailed logging for unsupported configurations and failure scenarios.
- Introduced support for dynamic GitLab and Supabase domain and ingress configuration based on deployment mode (k8s vs local).
- Default GitLab domain and ingress class aligned with `prole.cfg` deployment settings.
- Expanded Supabase support:
- Added configurable API and Studio hostnames with default fallbacks.
- Introduced separate ingress class configuration for API and Studio.
- Standardized public URL normalization for both services.
- Updated tests to validate domain, ingress, and configuration behaviors across environments.
- update Kong/Supabase ingress and service manifests for k3s
- expand init scripts and runtime config/topology handling
- refresh KNØE UI screens and Supabase deployment/render logic
- include related env helper test adjustments
Co-authored-by: Junie <junie@jetbrains.com>
STABLE BUILD — all 35+ pods Running, zero non-Running pods, CNPG ready=3.
## Architecture changes (Supabase slim-down)
- Removed local Supabase Kong, Storage, MinIO, Imgproxy from deployed stack
- Rewired API routing to shared Kong in kube-system
- Rewired object storage to shared Garage in knoe-system
- Supabase footprint reduced to control-plane only: auth, rest, meta,
analytics, realtime (studio port-forward retained)
- Supabase pods renamed: stripped redundant '-prole-supabase-' prefix via
per-component fullnameOverride (e.g. supabase-auth, supabase-rest, ...)
- Supabase workloads pinned to merlin.prole.org (memory-safe node)
## File changes
- conf/service/prole.cfg
* Replaced all hardcoded /Users/chrisfu/dev/prole paths with ${HOME}/dev/prole
so config is portable when run from myrddin.prole.org or any user home
* SUPABASE_ENABLED=True, SUPABASE_PRIMARY_NODE=merlin.prole.org in
both [Global] and [Optional Features]
* supabase_enabled=true in [Inputs]/init_cluster section
- supabase/helm/render_supabase.py
* Explicitly disabled: kong, storage, minio, imgproxy
* Active components pinned via nodeSelector to merlin.prole.org
* fullnameOverride per component to clean up pod names
* Default primary node changed from pi.prole.org -> merlin.prole.org
- supabase/deploy.sh
* Removed svc/kong port-forward; API via shared Kong in kube-system
* Removed imgproxy/storage from PV/PVC prep and helm --set persistence args
* Fixed kubectl exec missing -i flag for psql heredoc stdin in
setup_knoe_db_for_supabase
* Updated Studio access messaging to reference shared platform Kong
- k8s/prole/knoe-db.yaml
* instances: 3
* Added control-plane/master tolerations for myrddin scheduling
- deploy/opentofu/k3s/manifests/prole/knoe-db.yaml
* instances: 3, added control-plane tolerations (mirrors k8s/prole/)
- k8s/prole/iscsi-pvs.yaml
* Expanded d003-data/d003-wal nodeAffinity to [pi, merlin, myrddin]
to reflect post-OOM-recovery live state
- k8s/registry/deployment.yaml
* nodeSelector restored to myrddin.prole.org (hostPort 5000 owner)
* Added control-plane + master tolerations so pod schedules correctly
- modes/k3s/knoe-db/.version
* Aligned to 140 (image tag 18-140) matching repo-root version
- etc/init_common_services.sh, etc/init_registry.sh
* Minor fixes aligned with slimmed service layout and node targeting
- infrastructure/inventory/host_vars/pi.prole.org.yml
* Updated host vars to reflect post-recovery pi state
- knoe/core/topology.py, knoe/ui/screens/cluster.py,
knoe/ui/screens/cluster_nodes.py
* Topology and UI updates for 3-node cluster representation
- scan/network_description.txt
* Refreshed network scan output post-recovery
- conf/port-mapping.cfg
* Updated port mapping to reflect removal of local Kong/storage forwards
- scripts/cleanup_stale_rs.sh (new)
* Utility script to prune orphaned ReplicaSets after rollouts
## Verified stable state
- CNPG: Cluster in healthy state, ready=3
knoe-db-2 (merlin, primary), knoe-db-3 (pi), knoe-db-5 (myrddin)
- Supabase: 5 pods Running on merlin.prole.org
supabase-auth, supabase-rest, supabase-meta,
supabase-analytics, supabase-realtime
- Monitoring: all prometheus/grafana/alertmanager pods Running on merlin
- Registry: Running on myrddin.prole.org (hostPort 5000)
- Zero non-Running / non-Completed pods across all namespaces
Co-authored-by: Junie <junie@jetbrains.com>
- render_supabase.py: read SUPABASE_ADDITIONAL_REDIRECT_URLS from cfg,
normalise each entry to https://.../** wildcard, inject as GOTRUE_URI_ALLOW_LIST
in GoTrue auth env so logins from svc.prole.org are accepted
- conf/service/prole.cfg: set SUPABASE_ADDITIONAL_REDIRECT_URLS=svc.prole.org
- actions.py: preserve SUPABASE_ADDITIONAL_REDIRECT_URLS in globals_to_save
so pipeline write-back never clobbers it
- k8s/prole/merlin-local-iscsi-storageclass.yaml: register StorageClass
merlin-local-iscsi-d002 (no-provisioner/Immediate) to silence
ProvisioningFailed warnings on static Supabase PVs
Result: all 9 Supabase pods Running; svc.prole.org login origin accepted by GoTrue
Co-authored-by: Junie <junie@jetbrains.com>
Root cause: render_supabase.py wrote to deployment.*.storageClass (ignored by Helm chart PVC
templates) instead of persistence.*.storageClassName (_pvc.tpl:43 is the real path). Also
deploy.sh --set persistence.* args only ran in k8s mode, and stale values.generated.json
was reused across deploys. Together these caused Supabase PVCs to fall through to
pi-local-iscsi (pi SD card), crashing the node under memory pressure.
- supabase/helm/render_supabase.py:
- write persistence.{functions,imgproxy,snippets,storage,deno,minio}.storageClassName
using the correct Helm chart path (_pvc.tpl:43)
- add synology guard: raise SystemExit if storage class is not synology/merlin/myrddin
- remove broken deployment.*.storageClass writes
- supabase/deploy.sh:
- delete stale values.generated.json before every helm_render_values call
- fall back to reading SUPABASE_STORAGE_CLASS from cfg file via python3 if env not set
- add bash synology guard: die if storage class is not iSCSI/NFS mounted
- extend helm_set_args to include persistence.minio.storageClassName and apply
--set persistence.* for ALL modes (not just k8s)
Co-authored-by: Junie <junie@jetbrains.com>
Memory pressure on merlin (95% OOM with monitoring + CNPG replica + Supabase) resolved
by moving Supabase workload to pi.prole.org which has headroom (49% memory, 2 pods).
- conf/service/prole.cfg: set SUPABASE_PV_NODE, SUPABASE_PRIMARY_NODE=pi.prole.org,
SUPABASE_STORAGE_CLASS=pi-local-iscsi, SUPABASE_PV_BASE_DIR for rancher local-path
- supabase/helm/render_supabase.py: resolve SUPABASE_STORAGE_CLASS from env/cfg,
apply pi-local-iscsi to storage/analytics PVCs, resolve DATABASE_NAMESPACE for
correct db_host, DB_PASSWORD env override takes priority for live cluster password
- supabase/deploy.sh: default SUPABASE_PV_NODE to pi.prole.org, default base dir to
/var/lib/rancher/k3s/storage/supabase, add idempotent ALTER ROLE password sync,
live DB_PASSWORD resolution in helm_render_values from knoe-db-superuser secret
- knoe/core/actions.py: stop overriding init_cluster.supabase_enabled to False in
silent mode so prole.cfg value is honored end-to-end
- etc/init_monitoring.sh: derive pv_prom/pv_am/pv_graf inline before Released-PV
loop to fix unbound variable error on idempotent monitoring installs
Co-authored-by: Junie <junie@jetbrains.com>
- Persist and export supabase_hostname for canonical external Supabase entrypoint (db.prole.org)
- Render Supabase Helm ingress host and public URLs (API_EXTERNAL_URL/GOTRUE_SITE_URL/SUPABASE_PUBLIC_URL) from config
- Align tracked Traefik ingress manifests to db.prole.org
- Generalize service hostname/TLS wiring in k3s automation and refresh Kong/monitoring behavior
- Make optional workload policy checks deterministic when kube context is absent; add render tests
- Pin Prometheus/Alertmanager + Grafana to merlin.prole.org in k3s to avoid PV node-affinity issues
- Taint myrddin.prole.org as control-plane-only and add CNPG toleration
- Prevent init_cloudnative_pg from deploying prole-svc-kong into the DB namespace
- GitOps-manage Kong declarative config via ConfigMap + add placement/init tests
Prefer infra-managed manifests for bootstrap; reconcile CNPG instances based on Ready+schedulable labeled db nodes; update manifests to use node-role affinity + anti-affinity; add policy/tests and config touch-ups (incl. prole.cfg).
