Follow-up to 23f3f14. With the checksum annotation in place, `helm
upgrade` did start rolling Kong forward, but the new pod never became
Ready: `rollout status` timed out with
"1 old replicas are pending termination"
Kong's ConfigMap contains the /healthz route correctly, the BackendConfig
is HTTP /healthz, but the Kong process itself refuses to start cleanly
when one of its services has `url: http://127.0.0.1:8000/` -- Kong's own
proxy port. This self-reference apparently trips the declarative-config
parser (undocumented; symptomatic only).
Switch to a RFC-2606 `.invalid` placeholder. The request-termination
plugin short-circuits before any DNS lookup or upstream connection, so
a non-resolvable URL is functionally equivalent and Kong's parser has
nothing to complain about.
Same change in both places we define the /healthz route:
- supabase/helm/knoe-supabase/templates/kong/config.yaml (supabase-kong)
- etc/init_kong.sh inline kong.yml heredoc (knoe-svc-kong)
The knoe-svc-kong backend was already HEALTHY with the self-referential
URL -- different Kong instance, possibly different parser path -- but
keeping the two configs aligned so future edits don't drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to 0f2fe93. Live post-deploy diagnostics showed that even
though helm upgrade rewrote the supabase-kong ConfigMap with the new
/healthz route, the existing Kong pod kept serving the old config: the
Deployment spec itself never changed, so no rolling restart happened.
Kong loads /usr/local/kong/kong.yml at startup and doesn't watch the
file, so the healthcheck probe against /healthz got 404 and the GCE
backend stayed UNHEALTHY. api.0.knoe.dev stayed broken.
Standard helm workaround: annotate the pod template with a sha256 of
the ConfigMap template. Any content change bumps the hash, which
changes the Deployment spec, which triggers a rolling restart. Pattern
matches what vector/deployment.yaml in this chart already does and is
widely used in the bitnami / ingress-nginx charts.
Studio has no ConfigMap volume mount (verified via grep), so no
equivalent annotation is needed there.
This lands the permanent B.1 fix from the plan. Whether the live Kong
pod needs a manual `kubectl rollout restart` to pick up the current
REVISION 2 config -- or whether the /healthz service URL itself is
tripping Kong's parser -- is still to be determined from Part A
diagnostics before any B.2 / B.3 edits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to 391c4f5. Live deploy showed GCE's L7 BackendConfig CRD
hard-rejects type: TCP with:
Error syncing to GCP: error running backend syncing routine:
error ensuring health check:
Protocol "TCP" is not valid, must be one of [HTTP,HTTPS,HTTP2]
The sync never completes, so the LB has no healthy backend and TCP
connections to the public endpoint just close (ERR_CONNECTION_CLOSED).
Fix: switch all three BackendConfigs to type: HTTP with request paths
that return 200:
- supabase-kong & knoe-svc-kong: add a dedicated /healthz route to the
Kong declarative config via the request-termination plugin, which
returns 200 synchronously with no upstream call. Equivalent liveness
semantics to the TCP check we wanted (backend is alive as long as Kong
accepts connections) but over HTTP, which GCE actually accepts.
- supabase/helm/knoe-supabase/templates/kong/config.yaml
- etc/init_kong.sh (inline kong.yml heredoc)
- supabase-studio: Studio returns 301 on / (Next.js default) so we
point the probe at /favicon.ico -- Next.js serves it as a static asset
with 200 unconditionally. Not as clean as a real readiness endpoint
but Studio does not expose one that returns 200 without auth.
- supabase/helm/knoe-supabase/templates/studio/backendconfig.yaml
Verified locally via helm template -f values.generated.json: the
rendered BackendConfigs come out with the HTTP protocol + correct paths,
and the Kong ConfigMap has the healthz service block before the
auth-v1-open service.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GCE L7 Ingresses for knoe-svc-kong (svc.knoe.dev / api.knoe.dev),
supabase-kong (api.0.knoe.dev) and supabase-studio (db.0.knoe.dev) were
all stuck UNHEALTHY: the default GCE healthCheck is HTTP GET `/` on the
backend port, but Kong returns 404 on any unrouted path and Studio
returns a 301 redirect -- neither passes the default probe, so the LB
serves "Server Error" instead of reaching the pod.
Replicate the pattern already working for gitlab-webservice-default:
emit a BackendConfig CRD with a TCP healthCheck on the service port and
annotate the Service with cloud.google.com/backend-config so GCE picks
it up. TCP is sufficient for LB-level liveness -- the backend is "alive"
as long as the process is accepting connections.
- etc/init_kong.sh: new SVC_KNOE_BACKEND_CONFIG_NAME; apply BackendConfig
inside k8s/GCE branch; annotate Service post-apply.
- knoe-supabase chart: new kong/backendconfig.yaml + studio/backendconfig.yaml
(TCP on 8000 / 3000), gated on service.{kong,studio}.backendConfigName.
- knoe-supabase chart: kong/service.yaml + studio/service.yaml pick up
cloud.google.com/backend-config when backendConfigName is set.
- render_supabase.py: sets service.{kong,studio}.backendConfigName in k8s
mode so the above wires up automatically.
Separately, the chart-managed supabase-kong Ingress was being reaped
from the cluster seconds after helm install (manifest present in the
release, gone via `kubectl get`). Root cause TBD -- suspected
meta.helm.sh/* annotation ownership colliding with a GKE/Anthos audit
controller. Workaround: render_supabase.py now emits a standalone
public-ingress-kong.yaml (no helm metadata) that supabase/deploy.sh
applies alongside public-ingress-tls.yaml, and the chart template gains
an `ingress.externallyManaged` guard so it no-ops in k8s mode. Default
`false` keeps k3d/k3s behavior unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Introduced `enforceGeneralNodeRole` to make node affinity and selector enforcement configurable.
- Updated `_split_frontdoor_docs` to handle custom release labels for Kong services.
- Enhanced Helm templates to support dynamic scheduling configurations.
- Added `reconcile_db_frontdoor_studio_pvcs` to migrate PVCs to the target storage class.
- Updated tests to validate custom release labels and scheduling logic.
