chrisfu/prole
1
0
Fork 0
mirror of https://github.com/dredx/prole.git synced 2026-09-23 11:03:59 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

fix(net): BackendConfig healthchecks + externally-managed supabase-kong ingress

Browse Source 
GCE L7 Ingresses for knoe-svc-kong (svc.knoe.dev / api.knoe.dev),
supabase-kong (api.0.knoe.dev) and supabase-studio (db.0.knoe.dev) were
all stuck UNHEALTHY: the default GCE healthCheck is HTTP GET `/` on the
backend port, but Kong returns 404 on any unrouted path and Studio
returns a 301 redirect -- neither passes the default probe, so the LB
serves "Server Error" instead of reaching the pod.

Replicate the pattern already working for gitlab-webservice-default:
emit a BackendConfig CRD with a TCP healthCheck on the service port and
annotate the Service with cloud.google.com/backend-config so GCE picks
it up. TCP is sufficient for LB-level liveness -- the backend is "alive"
as long as the process is accepting connections.

- etc/init_kong.sh: new SVC_KNOE_BACKEND_CONFIG_NAME; apply BackendConfig
  inside k8s/GCE branch; annotate Service post-apply.
- knoe-supabase chart: new kong/backendconfig.yaml + studio/backendconfig.yaml
  (TCP on 8000 / 3000), gated on service.{kong,studio}.backendConfigName.
- knoe-supabase chart: kong/service.yaml + studio/service.yaml pick up
  cloud.google.com/backend-config when backendConfigName is set.
- render_supabase.py: sets service.{kong,studio}.backendConfigName in k8s
  mode so the above wires up automatically.

Separately, the chart-managed supabase-kong Ingress was being reaped
from the cluster seconds after helm install (manifest present in the
release, gone via `kubectl get`). Root cause TBD -- suspected
meta.helm.sh/* annotation ownership colliding with a GKE/Anthos audit
controller. Workaround: render_supabase.py now emits a standalone
public-ingress-kong.yaml (no helm metadata) that supabase/deploy.sh
applies alongside public-ingress-tls.yaml, and the chart template gains
an `ingress.externallyManaged` guard so it no-ops in k8s mode. Default
`false` keeps k3d/k3s behavior unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
chrisfu 2026-04-21 10:31:06 -07:00
parent 47d906cc28
commit 391c4f5fc9
8 changed files with 262 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

46
etc/init_kong.sh
View File

@ -153,6 +153,15 @@ SERVICE_TLS_CLUSTER_ISSUER="${SERVICE_TLS_CLUSTER_ISSUER:-letsencrypt-prod}"
# ingress delete/recreate. Leave blank to let GCE assign ephemerally.
SVC_KNOE_GLOBAL_STATIC_IP_NAME="${SVC_KNOE_GLOBAL_STATIC_IP_NAME:-}"
# BackendConfig name for the knoe-svc-kong Service. GCE's default L7 health
# check hits HTTP `/` on the backend port, which Kong responds to with 404
# (no route) -- marking the backend UNHEALTHY and causing the LB to return
# "Server Error" instead of reaching Kong. We instead point the GCE LB at a
# TCP health check (port-level liveness) so the backend passes as long as
# Kong is accepting connections, which is sufficient for our traffic shape.
# Mirrors the pattern in etc/init_gitlab.sh (gitlab-webservice-backendconfig).
SVC_KNOE_BACKEND_CONFIG_NAME="${SVC_KNOE_BACKEND_CONFIG_NAME:-knoe-svc-kong-backendconfig}"
# Legacy: svc-check used to own svc.prole.org. We now route the service hostname
# to Grafana, so remove any leftover svc-check resources to avoid conflicts.
SVC_CHECK_NAMESPACE="${SVC_CHECK_NAMESPACE:-svc-check}"
@ -596,6 +605,31 @@ EOF
return 1
fi
echo "Confirmed: ManagedCertificate/${service_managed_cert_name} and FrontendConfig/${service_frontend_config_name} exist in ns=${NAMESPACE}."
# BackendConfig: GCE default healthCheck is HTTP GET / on the backend port
# and Kong returns 404 on an unrouted path, so the backend never goes
# HEALTHY. A TCP healthCheck on the proxy port is sufficient for our case
# (backend is alive as long as Kong is accepting connections). Service is
# annotated below with cloud.google.com/backend-config so GCE picks this up.
echo "Reconciling BackendConfig (${SVC_KNOE_BACKEND_CONFIG_NAME}) for ${KONG_NAME} in ns=${NAMESPACE} ..."
kubectl apply -f - <<EOF
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: ${SVC_KNOE_BACKEND_CONFIG_NAME}
namespace: ${NAMESPACE}
spec:
healthCheck:
type: TCP
port: ${KONG_PROXY_PORT}
checkIntervalSec: 15
timeoutSec: 5
healthyThreshold: 1
unhealthyThreshold: 3
connectionDraining:
drainingTimeoutSec: 30
EOF
gce_tls_annotations=$(cat <<EOF
networking.gke.io/managed-certificates: ${service_managed_cert_name}
networking.gke.io/v1beta1.FrontendConfig: ${service_frontend_config_name}
@ -770,6 +804,18 @@ deploy() {
svc_out=$(kubectl_apply_retry -f "$manifests_dir/kong-service.yaml" -n "$NAMESPACE" 2>&1)
echo "$svc_out"
# In k8s/GCE mode, annotate the Service so GCE LB picks up the BackendConfig
# with the TCP health check. Matches etc/init_gitlab.sh's pattern for
# gitlab-webservice-default. Additive annotation; survives manifest
# re-applies (the YAML in deploy/opentofu/ doesn't set it).
if [[ "${PROLE_MODE:-}" == "k8s" ]]; then
echo "Annotating Service ${KONG_NAME} with cloud.google.com/backend-config=${SVC_KNOE_BACKEND_CONFIG_NAME}..."
kubectl -n "$NAMESPACE" annotate svc "$KONG_NAME" \
"cloud.google.com/backend-config={\"default\":\"${SVC_KNOE_BACKEND_CONFIG_NAME}\"}" \
--overwrite >/dev/null || \
echo "WARN: Failed to annotate ${KONG_NAME} with backend-config; GCE LB will fall back to default healthcheck (likely UNHEALTHY)." >&2
fi
echo "Waiting for $KONG_NAME rollout ..."
kubectl rollout status deployment/"$KONG_NAME" -n "$NAMESPACE" --timeout=120s

32
supabase/deploy.sh
View File

@ -817,6 +817,33 @@ apply_supabase_public_ingress_tls_resources() {
fi
}
# Apply the standalone supabase-kong Ingress (api.0.knoe.dev) in k8s/GKE mode.
# render_supabase.py emits this manifest outside the helm chart because the
# chart-rendered ingress kept getting reaped from the cluster minutes after
# helm install. Gated by the chart's `ingress.externallyManaged` flag which
# render_supabase.py sets to true in k8s mode, so the chart no-ops
# kong/ingress.yaml and this standalone manifest is the single source of
# truth for the api.0 ingress.
apply_supabase_public_ingress_kong_resource() {
local kube_context="$1"
local namespace="$2"
local manifest_path="$3"
if [[ -z "$manifest_path" || ! -f "$manifest_path" ]]; then
return 0
fi
local ctx_args=()
if [[ -n "$kube_context" ]]; then
ctx_args+=(--context "$kube_context")
fi
log "Applying standalone supabase-kong Ingress from '$manifest_path'..."
if ! kubectl "${ctx_args[@]}" -n "$namespace" apply -f "$manifest_path"; then
die "Failed applying standalone supabase-kong Ingress from '$manifest_path'."
fi
}
apply_defaults() {
DEV_HOME="${DEV_HOME:-$HOME/dev}"
DEV_HOME="${DEV_HOME/#\~/$HOME}"
@ -2949,13 +2976,16 @@ print(val)" 2>/dev/null || true)
local values="$PROJECT_ROOT/supabase/helm/generated/values.generated.json"
local db_frontdoor_manifest_path=""
local public_ingress_tls_manifest_path=""
local public_ingress_kong_manifest_path=""
local ns="supabase"
ns="$(load_manifest_summary_path supabase_namespace)"
ns="${ns:-supabase}"
db_frontdoor_manifest_path="$(load_manifest_summary_path manifests_frontdoor_db)"
public_ingress_tls_manifest_path="$(load_manifest_summary_path manifests_public_ingress_tls)"
public_ingress_kong_manifest_path="$(load_manifest_summary_path manifests_public_ingress_kong)"
apply_supabase_public_ingress_tls_resources "" "$ns" "$public_ingress_tls_manifest_path"
apply_supabase_public_ingress_kong_resource "" "$ns" "$public_ingress_kong_manifest_path"
# Pre-apply storage preflight: rendered PVC-bearing components must all use
# the resolved storage class source-of-truth.
@ -3093,6 +3123,8 @@ print((data.get("info") or {}).get("status") or "unknown")' || true)"
ensure_k8s_supabase_static_pvs "$storage_class"
public_ingress_tls_manifest_path="$(load_manifest_summary_path manifests_public_ingress_tls)"
apply_supabase_public_ingress_tls_resources "" "$ns" "$public_ingress_tls_manifest_path"
public_ingress_kong_manifest_path="$(load_manifest_summary_path manifests_public_ingress_kong)"
apply_supabase_public_ingress_kong_resource "" "$ns" "$public_ingress_kong_manifest_path"
fi
check_supabase_retained_pv_blocked "$ns"

34
supabase/helm/knoe-supabase/templates/kong/backendconfig.yaml Normal file
View File

@ -0,0 +1,34 @@
{{- /*
GCE L7 BackendConfig for supabase-kong.
GCE's default Ingress healthCheck is HTTP GET `/` on the backend port; Kong
returns 404 on any unrouted path, so the backend never goes HEALTHY and the
LB serves "Server Error" instead of reaching the proxy. A TCP healthCheck on
the Kong proxy port is sufficient for our deployment shape -- the backend is
"alive" as long as Kong is accepting connections, and GCE only probes for
reachability; per-route health is not required here.
The companion Service template (kong/service.yaml) annotates the Service
with cloud.google.com/backend-config so GCE picks this up.
Only rendered when .Values.service.kong.backendConfigName is set
(render_supabase.py populates it in k8s/GKE mode).
*/ -}}
{{- if and .Values.deployment.kong.enabled (.Values.service.kong.backendConfigName | default "") -}}
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: {{ .Values.service.kong.backendConfigName | quote }}
labels:
{{- include "supabase.labels" . | nindent 4 }}
spec:
healthCheck:
type: TCP
port: {{ .Values.service.kong.port | default 8000 }}
checkIntervalSec: 15
timeoutSec: 5
healthyThreshold: 1
unhealthyThreshold: 3
connectionDraining:
drainingTimeoutSec: 30
{{- end }}

12
supabase/helm/knoe-supabase/templates/kong/ingress.yaml
View File

@ -1,5 +1,16 @@
{{- /*
In k8s/GKE mode the supabase-kong Ingress is owned by render_supabase.py as
a standalone manifest (supabase/k8s/public-ingress-kong.yaml) applied by
deploy.sh outside the helm chart. The chart-rendered ingress was getting
silently reaped from the cluster within seconds of install; root cause is
suspected to involve meta.helm.sh/* annotation ownership colliding with a
GKE/Anthos audit controller, but diagnosis is deferred. externallyManaged
defaults to false so `helm install` in local k3d/k3s modes (where
render_supabase.py is not in the path) keeps the historical behavior.
*/ -}}
{{- if .Values.deployment.kong.enabled -}}
{{- if .Values.ingress.enabled -}}
{{- if not (.Values.ingress.externallyManaged | default false) -}}
{{- $fullName := include "supabase.kong.fullname" . -}}
{{- $svcPort := .Values.service.kong.port -}}
{{- if and .Values.ingress.className (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) }}
@ -55,3 +66,4 @@ spec:
{{- end }}
{{- end }}
{{- end }}
{{- end }}

11
supabase/helm/knoe-supabase/templates/kong/service.yaml
View File

@ -5,6 +5,17 @@ metadata:
name: {{ include "supabase.kong.fullname" . }}
labels:
{{- include "supabase.labels" . | nindent 4 }}
{{- $backendConfigName := .Values.service.kong.backendConfigName | default "" }}
{{- $extraAnnotations := .Values.service.kong.annotations | default dict }}
{{- if or $backendConfigName $extraAnnotations }}
annotations:
{{- if $backendConfigName }}
cloud.google.com/backend-config: '{"default":"{{ $backendConfigName }}"}'
{{- end }}
{{- with $extraAnnotations }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}
spec:
type: {{ .Values.service.kong.type }}
ports:

28
supabase/helm/knoe-supabase/templates/studio/backendconfig.yaml Normal file
View File

@ -0,0 +1,28 @@
{{- /*
GCE L7 BackendConfig for supabase-studio.
Mirror of kong/backendconfig.yaml. Studio is a Next.js app on port 3000
that returns 301 redirects on `/` (not 200), which trips GCE's default
HTTP healthCheck and marks the backend UNHEALTHY. A TCP healthCheck is
sufficient for LB-level liveness.
Rendered only when .Values.service.studio.backendConfigName is set.
*/ -}}
{{- if and .Values.deployment.studio.enabled (.Values.service.studio.backendConfigName | default "") -}}
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: {{ .Values.service.studio.backendConfigName | quote }}
labels:
{{- include "supabase.labels" . | nindent 4 }}
spec:
healthCheck:
type: TCP
port: {{ .Values.service.studio.port | default 3000 }}
checkIntervalSec: 15
timeoutSec: 5
healthyThreshold: 1
unhealthyThreshold: 3
connectionDraining:
drainingTimeoutSec: 30
{{- end }}

11
supabase/helm/knoe-supabase/templates/studio/service.yaml
View File

@ -5,6 +5,17 @@ metadata:
name: {{ include "supabase.studio.fullname" . }}
labels:
{{- include "supabase.labels" . | nindent 4 }}
{{- $backendConfigName := .Values.service.studio.backendConfigName | default "" }}
{{- $extraAnnotations := .Values.service.studio.annotations | default dict }}
{{- if or $backendConfigName $extraAnnotations }}
annotations:
{{- if $backendConfigName }}
cloud.google.com/backend-config: '{"default":"{{ $backendConfigName }}"}'
{{- end }}
{{- with $extraAnnotations }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}
spec:
type: {{ .Values.service.studio.type }}
ports:

88
supabase/helm/render_supabase.py
View File

@ -991,6 +991,13 @@ def _build_overlay(cfg: configparser.ConfigParser, args: argparse.Namespace) ->
},
"ingress": {
"enabled": True,
# In k8s/GKE mode the chart-rendered supabase-kong Ingress kept
# getting deleted from the cluster minutes after install (helm
# thought it owned it, something on the cluster kept reaping it).
# We sidestep by emitting a standalone public-ingress-kong.yaml
# below and telling the chart NOT to render its own ingress:
# kong/ingress.yaml has a top-level `if not externallyManaged` guard.
"externallyManaged": mode == "k8s",
"className": "" if mode == "k8s" else api_ingress_class,
"hosts": [{"host": host, "paths": [{"path": "/", "pathType": "Prefix"}]} for host in api_host_entries],
"annotations": {
@ -1024,6 +1031,20 @@ def _build_overlay(cfg: configparser.ConfigParser, args: argparse.Namespace) ->
} if frontdoor_auth_enabled and mode != "k8s" else {}),
}
},
# BackendConfig wiring for GCE LB -- Kong's default "/" returns 404
# and Studio returns 301, both fail GCE's default HTTP healthCheck
# and mark the backends UNHEALTHY. Our chart templates emit a
# BackendConfig with a TCP healthCheck when backendConfigName is set,
# and the Service gets a cloud.google.com/backend-config annotation
# referencing that name.
"service": {
"kong": {
"backendConfigName": "supabase-kong-backendconfig" if mode == "k8s" else "",
},
"studio": {
"backendConfigName": "supabase-studio-backendconfig" if mode == "k8s" else "",
},
},
}
if supabase_storage_class:
@ -1188,6 +1209,72 @@ def render(args: argparse.Namespace) -> None:
elif ingress_tls_manifest_path.exists():
ingress_tls_manifest_path.unlink()
# Standalone supabase-kong Ingress for GCE L7 (api.0.knoe.dev).
# The chart's templates/kong/ingress.yaml had the ingress vanish out from
# under us even though helm's manifest still showed it (root cause TBD --
# suspected meta.helm.sh annotation mismatch interacting with a GKE audit
# reaper). We sidestep by:
# 1. Setting overlay.ingress.externallyManaged=true (k8s only), which
# makes the chart no-op its kong/ingress.yaml template.
# 2. Emitting the ingress here as a standalone manifest that deploy.sh
# applies alongside public-ingress-tls.yaml. No helm ownership labels
# or meta annotations -- a plain Kubernetes Ingress the chart won't
# try to adopt.
ingress_kong_manifest_path = k8s_dir / "public-ingress-kong.yaml"
if meta.get("mode") == "k8s":
api_ingress_cfg = overlay.get("ingress") or {}
hosts = api_ingress_cfg.get("hosts") or []
api_annotations = dict(api_ingress_cfg.get("annotations") or {})
# Drop chart-internal keys before emitting
api_annotations.pop("externallyManaged", None)
kong_ingress_rules: list[dict[str, Any]] = []
for host_entry in hosts:
if not isinstance(host_entry, dict):
continue
host = str(host_entry.get("host") or "").strip()
if not host:
continue
kong_ingress_rules.append(
{
"host": host,
"http": {
"paths": [
{
"path": p.get("path", "/"),
"pathType": p.get("pathType", "Prefix"),
"backend": {
"service": {
"name": "supabase-kong",
"port": {"number": 8000},
}
},
}
for p in (host_entry.get("paths") or [{"path": "/", "pathType": "Prefix"}])
]
},
}
)
if kong_ingress_rules and api_annotations:
kong_ingress_doc = {
"apiVersion": "networking.k8s.io/v1",
"kind": "Ingress",
"metadata": {
"name": "supabase-kong",
"namespace": meta["supabase_namespace"],
"annotations": api_annotations,
},
"spec": {"rules": kong_ingress_rules},
}
ingress_kong_manifest_path.write_text(
yaml.safe_dump(kong_ingress_doc, sort_keys=False)
)
print(f"\nGenerated Supabase kong public Ingress manifest: {ingress_kong_manifest_path}")
print(yaml.safe_dump(kong_ingress_doc, sort_keys=False).rstrip())
elif ingress_kong_manifest_path.exists():
ingress_kong_manifest_path.unlink()
elif ingress_kong_manifest_path.exists():
ingress_kong_manifest_path.unlink()
# Render chart to a single manifest file (Helm accepts JSON values)
rendered_path = k8s_dir / "supabase-helm.yaml"
cmd = [
@ -1246,6 +1333,7 @@ def render(args: argparse.Namespace) -> None:
"manifests_app": str(app_rendered_path),
"manifests_frontdoor_db": str(frontdoor_rendered_path),
"manifests_public_ingress_tls": str(ingress_tls_manifest_path),
"manifests_public_ingress_kong": str(ingress_kong_manifest_path),
"db_host": meta["db_host"],
"supabase_namespace": meta["supabase_namespace"],
}