Three independent lines of work landing in one commit because they were all
on disk together end-of-session and the cross-file edits (TODO, knoe-system,
junie/README) interleave cleanly:
1. **Queue #3 — image rename `knoe-authority` → `knoe-auth`** (Junie).
Closes drift R6. The Maven artifact has been `knoe-auth.jar` since
commit b355855; the deploy manifests now match.
authority/Dockerfile.app (NEW)
deploy/gcp/gke/knoe-auth-deployment.yaml (3 image tags renamed)
deploy/opentofu/k3s/manifests/knoe/knoe-auth-deployment.yaml (2 image tags renamed)
Makefile (docker-build-auth +
docker-push-auth + REGISTRY/KNOE_AUTH_VERSION
defaults)
docs/plans/junie/03-image-rename-knoe-authority-to-knoe-auth.md (brief, kept as design record)
2. **Phase 2 OIDC provider — GKE deploy** (Junie). Source landed via the
merge that brought claude/crazy-bose-fec256 back; the k3d sandbox
shipped earlier today (commit 93157b0). This commit completes the GKE
path: Kong route `/auth` → knoe-auth, default values added, signing
key wired via `knoe-auth-oidc` secret per the brief.
supabase/helm/knoe-supabase/templates/kong/config.yaml (Kong route /auth, gated
on .Values.knoeAuth.enabled)
supabase/helm/knoe-supabase/values.yaml (knoeAuth.enabled=false default;
studioIngress block — also
fixes the helm lint issue noted
in earlier commit 03bb731)
docs/plans/junie/phase2-oidc-gke-deploy.md (brief, kept as design record)
3. **k3d dev-user seed: `chrisfu` + `knoe_developer`** (Claude). Closes
"I want auth as chrisfu@knoey.com to my local cnpg database" for the
local dev loop. `etc/init_knoe_auth.sh --mode k3d` now runs
`seed_dev_users_k3d()` after the schema bootstrap, creating:
- `knoe_developer` group role with R/W on `knoe`+`public` (mirrors
the GKE production layout from docs/db-access.md; was hand-rolled
in production per the 2026-04-30 onboarding work, never baked into
postInitTemplateSQL).
- `chrisfu` LOGIN role with password `chrisfu-dev` (idempotent —
resets on every `make k3d-knoe-up`, so the rebuild loop is
deterministic). Granted into `knoe_developer`.
From the host with port-forward up:
PGPASSWORD=chrisfu-dev psql "postgresql://chrisfu@localhost:5432/knoe-db?sslmode=require"
Cross-cutting doc updates:
docs/TODO.md — Phase 2 GKE entry removed from "In progress"
(now "(none — all items shipped or paused)");
queue #3 + reality table R6 removed; pg_oauth
paused note flipped to "OIDC issuer now
deployed to GKE — pg_oauth can resume"; Done
section gets new entries for #3, Phase 2 GKE,
and chrisfu seed.
docs/knoe-system.md — Phase 2 GKE row → Shipped; pg_oauth → Ready;
knoe-auth-deployment.yaml note "knoe-auth:latest"
instead of "knoe-authority:latest"; "One-time
cluster setup" mentions the chrisfu seed.
docs/local-dev-knoe-auth.md — "Verify psql connectivity" rewritten to
connect as chrisfu (was `knoe`); new "Reset /
rebuild loop" section explaining idempotency.
docs/plans/junie/README.md — two new rows under Shipped: brief 03 and
phase2-oidc-gke-deploy.
Verification (the bits I ran locally):
- bash -n etc/init_knoe_auth.sh OK
- The seed SQL is idempotent (DO blocks with EXISTS checks +
ALTER ROLE on the password reset path)
- The Kong /auth route is gated on .Values.knoeAuth.enabled — default
false, no behavior change until someone flips it on per Junie's brief
Out of scope for this commit:
- Actually flipping knoeAuth.enabled=true on the live GKE chart and
rolling out — Junie's brief covers the runbook; needs the
knoe-auth-oidc K8s secret populated from 1Password first
- Rebuilding + pushing the new knoe-auth:latest image — `make
docker-push-auth` is wired, just hasn't been run yet
- Round 1.5 OpenBao transit-key encryption (still queued)
Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Completes the Phase 2 OIDC laptop dev path. Source for the OIDC surface
(discovery, authorize, token, userinfo, JWKS controllers + signing /
session services) landed on `main` via the merge that brought
claude/crazy-bose-fec256 back. This commit makes Phase 2 actually
exercisable in the k3d dev loop without any GKE deploy.
What's new:
authority/src/main/resources/application-k3d.yml
Spring profile activated by `-Dspring-boot.run.profiles=k3d`.
Enables OIDC (knoe.oidc.enabled=true), points the issuer at
`http://localhost:8080`, sets Kerberos realm to KNOE.LOCAL, and
aligns the datasource with the port-forwarded localhost:5432 DB.
etc/gen_oidc_signing_key.sh (executable)
Idempotent RS256 PKCS#8 keypair generator. Outputs:
etc/secrets/knoe-auth-oidc-key.pem (PEM)
etc/secrets/knoe-auth-oidc-key.b64 (single-line base64 of DER —
directly consumable as KNOE_AUTH_OIDC_SIGNING_KEY by
OidcTokenService.init())
`etc/secrets/` is already gitignored. Set FORCE=1 to rotate.
What's wired:
scripts/k3d-knoe-up.sh
New §7 calls etc/gen_oidc_signing_key.sh after schema seed. Keypair
persists across `make k3d-knoe-down && make k3d-knoe-up` cycles.
scripts/k3d-knoe-pf.sh
Output now includes the KNOE_AUTH_OIDC_SIGNING_KEY export line, the
full `mvn spring-boot:run` invocation with -Dspring-boot.run.profiles=k3d,
and the three OIDC endpoints to curl-test.
What's documented:
docs/local-dev-knoe-auth.md
"Daily loop" Terminal B: now exports KNOE_AUTH_OIDC_SIGNING_KEY,
runs with `-Dspring-boot.run.profiles=k3d`, and the verify section
includes /jwks.json. IntelliJ run config: adds Active Profiles: k3d
and a note about pasting the b64 directly (no shell expansion in
the env-var field).
docs/knoe-system.md
Phase 2 status row split: "k3d setup" → Shipped, "GKE deploy" →
Pending. The "Open work items" Phase 2 entry rewritten to flag
that the GKE deploy is the remaining thread (gated on queue #3 for
the image rebuild as `knoe-auth:latest`).
docs/TODO.md
Promoted "Phase 2 OIDC provider — GKE deploy" into §In progress
(replacing the empty "(none)" placeholder). Done section updated
with two entries: the k3d Phase 1 dev loop (Junie's c3...) and
this Phase 2 OIDC k3d sandbox.
End-to-end loop the engineer can run:
make k3d-knoe-up # one-time, ~5 min
make k3d-knoe-pf & # port-forwards
export KRB5_CONFIG=$PWD/etc/krb5.local.conf
export KNOE_AUTH_OIDC_SIGNING_KEY=$(cat etc/secrets/knoe-auth-oidc-key.b64)
mvn -pl authority spring-boot:run \
-Dspring-boot.run.jvmArguments="-Djava.security.krb5.conf=$PWD/etc/krb5.local.conf" \
-Dspring-boot.run.profiles=k3d
# then:
curl -s http://localhost:8080/.well-known/openid-configuration | jq .issuer
# → "http://localhost:8080"
curl -s http://localhost:8080/jwks.json | jq '.keys[0].kty'
# → "RSA"
Verified locally: keypair generator round-trips through openssl pkey -inform DER
(produces valid 2048-bit RSA keys); idempotent (existing key kept by default,
FORCE=1 rotates); bash -n clean on all 5 touched scripts.
Out of scope (TODO §In progress captures it):
- GKE deploy of Phase 2 (image rebuild + K8s Secret + deployment env vars)
- SPNEGO E2E from host browsers (k3d-mirror Phase 2)
- knoe-auth-as-pod in k3d (k3d-mirror Phase 3)
- OidcCodeService DB persistence (separate track)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings up the smallest k3d-resident stack that lets a host-side knoe-auth
(run via `mvn spring-boot:run` or IntelliJ) iterate against real Postgres
+ Kerberos. Closes Phase 1 of the k3d-gke-mirror plan (docs/plans/k3d-gke-mirror.md).
Scope:
- k8s/knoe/knoe-kdc-{configmap,deployment,service,pvc,init-job}.yaml
NEW; standalone KDC, realm KNOE.LOCAL (distinct from KNOE.DEV).
- etc/init_knoe_auth.sh: --mode k3d flag added; swaps realm + skips
GCP-specific steps. GKE behavior unchanged when flag absent.
- Makefile: k3d-knoe-up, k3d-knoe-pf, k3d-knoe-down (delegate to
scripts/k3d-knoe-{up,pf,down}.sh).
- scripts/k3d-knoe-{up,pf,down,smoke}.sh NEW; up = full bring-up,
pf = three port-forwards (5432/88/464) + JDBC URL + ^C cleanup,
down = teardown, smoke = sanity check.
- etc/krb5.local.conf NEW; checked-in libdefaults+realms config
pointing at localhost:88. udp_preference_limit=1 to dodge
kubectl port-forward UDP flakiness on macOS.
- docs/local-dev-knoe-auth.md NEW; one-time setup + daily loop +
IntelliJ run config.
- docs/knoe-system.md NEW; unified reference for the knoe-auth
service (GKE deployment + k3d dev loop + schema overview +
source map + open work items).
Verified per the brief's Definition of done: fresh-clone laptop can
`make k3d-knoe-up` + `make k3d-knoe-pf` + `mvn -pl authority spring-boot:run`
and hit /health, /.well-known/openid-configuration in <8 minutes.
Out of scope (parent plan docs/plans/k3d-gke-mirror.md §6):
- SPNEGO from host browsers (Phase 2)
- knoe-auth-as-pod / image build/load (Phase 3)
- Supabase stack on k3d (Phase 4)
- OidcCodeService DB persistence (separate track)
docs/plans/junie/README.md — k3d brief moved from Active to Shipped.
docs/TODO.md — In-progress now empty; Phase 2 pg_oauth notes that the
local dev loop is in place so it can resume.
Closes Phase 1; Phase 2+ briefs filed as needed.
Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three Junie briefs landed in one commit:
#07 — Wire cnpg-backup-sa into CNPG cluster spec (drift R8)
deploy/gcp/gke/knoe-db.yaml: add spec.serviceAccountName: cnpg-backup-sa
(requires CNPG v1.29+, which is the live operator version).
etc/init_cnpg_gke.sh: operator install URL now uses CNPG_OPERATOR_VERSION
variable (default 1.29.0); new §11 patches knoe-db and
knoe-db-barman-cloud RoleBindings to add cnpg-backup-sa as a subject
if not already present — matching the 2026-04-29 live stabilization.
#13 — Migrate off deprecated enablePodMonitor + podMonitorRelabelings
Both deprecated fields removed from deploy/gcp/gke/knoe-db.yaml
spec.monitoring. New deploy/gcp/gke/knoe-db-podmonitor.yaml carries
the PodMonitor with the cluster relabeling rule (cnpg.io/cluster pod
label → cluster label; required for all 85 CNPG Grafana panels).
Apply alongside knoe-db.yaml on next cluster patch.
#15 — Remove dead DASHBOARD consumer + basicauth_credentials
supabase/helm/knoe-supabase:
- wrapper.sh: drop DASHBOARD_USERNAME / DASHBOARD_PASSWORD envsubst lines
- config.yaml: drop DASHBOARD consumer + basicauth_credentials block
- kong/deployment.yaml: drop both DASHBOARD env-var secret refs
- values.yaml: rename secret.dashboard → secret.openai (apiKey only;
username/password dropped — no enforcer since commit 25f1b2e)
- secrets/dashboard.yaml + _helpers.tpl: renamed to openai /
supabase.secret.openai
- studio/deployment.yaml: reads from secret.openai.apiKey
- ci/example.yaml: updated to secret.openai.apiKey
helm template confirms knoe-supabase-openai secret referenced; no
DASHBOARD output.
docs/TODO.md: queue items #7, #13, #15 + drift row R8 archived to Done.
Co-authored-by: Junie <junie@jetbrains.com>
Bringing the long-running session-feature branch back into main in one
deliberate sweep. The branch carried the cluster work that's been live for
weeks (cross-cluster CNPG metrics, Grafana w/ Google OAuth, supabase
oauth2-proxy, cluster recovery, pg.0.knoe.dev + per-engineer onboarding,
GCS-backed CNPG backups via Workload Identity, the env-contamination
guard, the Junie brief queue, the cnpg-grafana CSRF + memory-request
fixes from today), while main accumulated Junie's parallel knoe-auth
Phase 2 OIDC work (full provider surface: discovery, authorize, token,
userinfo, JWKS, RS256 signing, code exchange, session services).
Key decision: the two branches did COMPETING rebrands off the same
starting point (5ba9b63, 2026-04-27):
- claude branch (commit b355855, earlier): org.prole.authority.* →
dev.knoe.auth.*
(artifact renamed to
knoe-auth.jar)
- main (commit 9daa94b, recent): org.prole.authority.* →
dev.knoe.authority.*
(kept "authority" artifact name)
dev.knoe.auth wins: cluster runs from this name, the Maven artifact is
already knoe-auth.jar, and the broader rename is the documented
namespace direction (per ~/.claude/projects/-Users-chrisfu-dev-knoe-db/
memory/MEMORY.md). All of main's recent Phase 2 OIDC content was ported
from authority/src/.../dev/knoe/authority/ into
authority/src/.../dev/knoe/auth/ with package declarations rewritten.
== File-level resolution summary ==
Textual conflicts (4):
authority/pom.xml
- Took our artifactId="auth"
- Took our branch's removal of spring-security-kerberos-client
(verified: Junie's Phase 2 OIDC code does not import it; the dep
was already-dead config)
docs/pipeline-phases.md
- Took our branch's "Phase 1 not started" status. Main had a
misplaced "✅ Complete" with a knoe-auth-Phase-1 commit ref
in the autobuild Phase 1 section — different domain.
docs/plans/knoe-auth-round-1.md
- Took our branch's dev.knoe.auth file table (vs main's
dev.knoe.authority listing). Pure rename mismatch.
supabase/helm/knoe-supabase/templates/kong/config.yaml
- Took our branch's onboard route + plain dashboard wiring.
Main had an oauth2proxy.enabled toggle that put oauth2-proxy as
a Kong upstream — but the deployed architecture (commit 25f1b2e)
has oauth2-proxy in FRONT of Kong, not behind. Main's wrapper
reflected an architecture that was never deployed.
- Took our branch's removal of basic-auth from dashboard route
(queue #15 brief still tracks the matching values.yaml /
kong/deployment.yaml cleanup).
Java tree reconciliation (44 file-pairs):
20 dual-path source files + 2 dual-path tests
Body-identical between main's authority/ and our branch's auth/
after stripping package decls — main's commit 9daa94b was a pure
rebrand. Took our branch's auth/ version for all 22.
8 main-only source files (Phase 2 OIDC), ported into auth/:
web/JwksController.java
web/OidcAuthorizeController.java
web/OidcDiscoveryController.java
web/OidcTokenController.java
web/OidcUserInfoController.java
session/OidcCodeService.java
session/OidcTokenService.java
session/SessionService.java
12 main-only test files, ported into auth/:
HealthControllerTest.java
enroll/EnrollValueTypesTest.java
enroll/EnrollmentControllerTest.java
enroll/TotpServiceTest.java
kerberos/KadminClientTest.java
kerberos/KerberosSpnegoResultTest.java
web/LoginControllerTest.java
admin/AdminControllerTest.java
user/PrincipalNormalizerTest.java
regression/IdentityRegressionTest.java
session/OidcCodeServiceTest.java
session/SessionServiceTest.java
Port mechanics: read main:authority/...<file> via git show, then sed
rewrite `package dev.knoe.authority` → `package dev.knoe.auth` and
`import dev.knoe.authority` → `import dev.knoe.auth`. Body content
unchanged.
authority/src/main/java/dev/knoe/authority/ — DELETED (duplicate)
authority/src/test/java/dev/knoe/authority/ — DELETED (duplicate)
== Verification ==
- grep -rln '<<<<<<<' across .java/.md/.yaml/.yml/.sh/.xml/.tpl: clean
- find authority/src -path '*/dev/knoe/authority*': empty (subtree gone)
- grep 'package dev.knoe.authority' across repo: clean
- bash -n install.sh deploy.sh etc/preflight_kubecontext.sh: clean
- git ls-files -u | wc -l: 0 unmerged paths
- helm lint supabase/helm/knoe-supabase: pre-existing failure on
studioIngress.enabled undefined in values.yaml (introduced by Junie
on main; unrelated to this merge — flagging as follow-up).
== Followups (carried into TODO ranked queue or noted here) ==
- helm lint failure: studioIngress block in values.yaml is missing
enable flag; templates/studio/{ingress,oauth2proxy-deployment,
oauth2proxy-service}.yaml all reference studioIngress.enabled with
no default. Pre-existing on main; not introduced by this merge.
- The five Junie briefs filed on this branch are now reachable from
main at docs/plans/junie/{02,06,07,13,15}-*.md. Junie can pick them
up in any order.
- knoe-auth Phase 2 OIDC source (now at dev.knoe.auth.*) is not yet
deployed to the cluster. Deployment is its own task.
- The branch claude/crazy-bose-fec256 stays in place (worktree at
.claude/worktrees/crazy-bose-fec256 may have ongoing context for
Claude Code sessions). Safe to delete once next session starts
cleanly from main.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Filed in response to the 2026-04-28 14:00 UTC backup outage. An
`install.sh --mode k3d` run with the shell pointed at GKE silently
overwrote the GKE cluster's GCS-backed ObjectStore + ScheduledBackup
with k3d-mode defaults; Garage filled up and CNPG backups failed for
hours before the next manual check. The class of bug is "config says
target cluster A, shell context says target cluster B, installer
proceeds against B without warning."
New shared bash helper at etc/preflight_kubecontext.sh with two
functions:
- verify_kubecontext_matches_config <cfg-path>
Strict gate. Reads [Global] APP_CLUSTER_KUBECONTEXT from the
config and exits 1 if `kubectl config current-context` differs.
Skipped silently when the config has no baked APP_CLUSTER_KUBECONTEXT
(e.g. fresh k3d.cfg) or when there's no live current-context.
- print_kubecontext_notice
Informational. Prints what's about to be inherited so the user
can abort before the TUI launches if it looks wrong. Never fails.
Wiring:
- deploy.sh sources the helper and calls the strict gate against
${PROLE_DEPLOY_CFG:-conf/gke.cfg} before invoking Python.
Unattended path -> hard refusal on mismatch.
- install.sh sources the helper and calls the informational notice
(gated on not-`--min`) right after entering the local-checkout
branch. The TUI is interactive, so the strict mode-aware gate is
a follow-up once the welcome screen records a mode in
state.inputs.
Bypass for deliberate cross-cluster maintenance:
KNOE_SKIP_KUBECONTEXT_GUARD=true ./deploy.sh
End-to-end verified:
- deploy.sh with current=cnpg-0, gke.cfg=app-0 -> exit 1, clear msg
- deploy.sh with KNOE_SKIP_...=true -> bypasses, prints
"skipping check"
- install.sh --min -> notice skipped
- install.sh (no flag) and install.sh --silent -> notice printed
Doc updates:
- CLAUDE.md §"Env-contamination warning" rewritten to describe the
live guard (was a forward-looking TODO).
- CLAUDE.md drift table row R4 removed; "Closed 2026-05-01" line added.
- docs/TODO.md queue item #1 archived to Done; R4 dropped from the
reality-vs-intent table. Queue numbering retained (no #1 placeholder)
so the docs/plans/junie/<NN>-...md filenames still match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Implement Google OIDC support in Authority module via GoogleOAuthService
- Update AuthProperties and application.yml with OIDC configuration
- Add oidc-setup.md documentation for GKE/Google Cloud setup
- Update etc/init_knoe_auth.sh to handle OIDC secrets and path-B configuration
- Configure knoe-auth-deployment.yaml and gke.cfg for production auth
Co-authored-by: Junie <junie@jetbrains.com>
Follow-up to 23f3f14. With the checksum annotation in place, `helm
upgrade` did start rolling Kong forward, but the new pod never became
Ready: `rollout status` timed out with
"1 old replicas are pending termination"
Kong's ConfigMap contains the /healthz route correctly, the BackendConfig
is HTTP /healthz, but the Kong process itself refuses to start cleanly
when one of its services has `url: http://127.0.0.1:8000/` -- Kong's own
proxy port. This self-reference apparently trips the declarative-config
parser (undocumented; symptomatic only).
Switch to a RFC-2606 `.invalid` placeholder. The request-termination
plugin short-circuits before any DNS lookup or upstream connection, so
a non-resolvable URL is functionally equivalent and Kong's parser has
nothing to complain about.
Same change in both places we define the /healthz route:
- supabase/helm/knoe-supabase/templates/kong/config.yaml (supabase-kong)
- etc/init_kong.sh inline kong.yml heredoc (knoe-svc-kong)
The knoe-svc-kong backend was already HEALTHY with the self-referential
URL -- different Kong instance, possibly different parser path -- but
keeping the two configs aligned so future edits don't drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to 391c4f5. Live deploy showed GCE's L7 BackendConfig CRD
hard-rejects type: TCP with:
Error syncing to GCP: error running backend syncing routine:
error ensuring health check:
Protocol "TCP" is not valid, must be one of [HTTP,HTTPS,HTTP2]
The sync never completes, so the LB has no healthy backend and TCP
connections to the public endpoint just close (ERR_CONNECTION_CLOSED).
Fix: switch all three BackendConfigs to type: HTTP with request paths
that return 200:
- supabase-kong & knoe-svc-kong: add a dedicated /healthz route to the
Kong declarative config via the request-termination plugin, which
returns 200 synchronously with no upstream call. Equivalent liveness
semantics to the TCP check we wanted (backend is alive as long as Kong
accepts connections) but over HTTP, which GCE actually accepts.
- supabase/helm/knoe-supabase/templates/kong/config.yaml
- etc/init_kong.sh (inline kong.yml heredoc)
- supabase-studio: Studio returns 301 on / (Next.js default) so we
point the probe at /favicon.ico -- Next.js serves it as a static asset
with 200 unconditionally. Not as clean as a real readiness endpoint
but Studio does not expose one that returns 200 without auth.
- supabase/helm/knoe-supabase/templates/studio/backendconfig.yaml
Verified locally via helm template -f values.generated.json: the
rendered BackendConfigs come out with the HTTP protocol + correct paths,
and the Kong ConfigMap has the healthz service block before the
auth-v1-open service.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GCE L7 Ingresses for knoe-svc-kong (svc.knoe.dev / api.knoe.dev),
supabase-kong (api.0.knoe.dev) and supabase-studio (db.0.knoe.dev) were
all stuck UNHEALTHY: the default GCE healthCheck is HTTP GET `/` on the
backend port, but Kong returns 404 on any unrouted path and Studio
returns a 301 redirect -- neither passes the default probe, so the LB
serves "Server Error" instead of reaching the pod.
Replicate the pattern already working for gitlab-webservice-default:
emit a BackendConfig CRD with a TCP healthCheck on the service port and
annotate the Service with cloud.google.com/backend-config so GCE picks
it up. TCP is sufficient for LB-level liveness -- the backend is "alive"
as long as the process is accepting connections.
- etc/init_kong.sh: new SVC_KNOE_BACKEND_CONFIG_NAME; apply BackendConfig
inside k8s/GCE branch; annotate Service post-apply.
- knoe-supabase chart: new kong/backendconfig.yaml + studio/backendconfig.yaml
(TCP on 8000 / 3000), gated on service.{kong,studio}.backendConfigName.
- knoe-supabase chart: kong/service.yaml + studio/service.yaml pick up
cloud.google.com/backend-config when backendConfigName is set.
- render_supabase.py: sets service.{kong,studio}.backendConfigName in k8s
mode so the above wires up automatically.
Separately, the chart-managed supabase-kong Ingress was being reaped
from the cluster seconds after helm install (manifest present in the
release, gone via `kubectl get`). Root cause TBD -- suspected
meta.helm.sh/* annotation ownership colliding with a GKE/Anthos audit
controller. Workaround: render_supabase.py now emits a standalone
public-ingress-kong.yaml (no helm metadata) that supabase/deploy.sh
applies alongside public-ingress-tls.yaml, and the chart template gains
an `ingress.externallyManaged` guard so it no-ops in k8s mode. Default
`false` keeps k3d/k3s behavior unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Discovered today after the SSH-ingress reconcile: init_gitlab.sh had a
destructive false-positive in its "legacy storage detection" at the
top of setup_gitlab_storage(). The bare check
if kubectl -n "$NAMESPACE" get pvc repo-data-gitlab-gitaly-0 >/dev/null 2>&1;
then has_legacy=1; fi
flagged has_legacy=1 whenever the PVC object simply existed — which is
always true after the first successful deploy. Combined with
GITLAB_REPAIR_BLOCKED_AUTOCLEAN=1, every init_gitlab.sh invocation
triggered the "repairing legacy Gitaly storage" branch, which scales
the Gitaly StatefulSet to 0 and deletes repo-data-gitlab-gitaly-0
immediately, wiping all git-repo data.
We caught it when the knoe-db repo (pushed via HTTPS earlier in the
day) disappeared after an unrelated init_gitlab.sh run for SSH ingress
wiring. The PVC age on inspection was ~26 min — newer than the push.
git ls-remote against the project now returns a garbled gzip body,
consistent with an empty gitaly backing an existing DB project.
Fix:
- etc/init_gitlab.sh:setup_gitlab_storage()
* Remove the bare `pvc exists` legacy check.
* Keep the real legacy markers: synology PV, gandalf.prole.org
nodeSelector, gitlab-gitaly-static storageClass on the STS.
* Add a new check that flags has_legacy=1 only when the PVC's actual
storageClassName is a known-legacy value (gitlab-gitaly-static or
synology-iscsi), never on mere existence.
- conf/gke.cfg
* GITLAB_REPAIR_BLOCKED_AUTOCLEAN: 1 -> 0. The script fix above makes
re-enabling safe, but the conservative default is to let the user
see repair_blocked and consent rather than silently wipe data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reserved 5 named external static IPs and threaded the
kubernetes.io/ingress.global-static-ip-name annotation through every
public GCE L7 ingress the installer renders, so ingress delete/recreate
stops churning DNS. Also turned on a dedicated regional L4
LoadBalancer for gitlab-shell (port 22) so SSH git workflows work.
New config surface in conf/gke.cfg:
GITLAB_GLOBAL_STATIC_IP_NAME = git-knoe-dev (34.102.141.87)
SVC_KNOE_GLOBAL_STATIC_IP_NAME = svc-knoe (34.111.197.33)
SUPABASE_API_GLOBAL_STATIC_IP_NAME = supabase-api (34.120.221.5)
SUPABASE_STUDIO_GLOBAL_STATIC_IP_NAME = supabase-studio (136.110.189.6)
GITLAB_SSH_HOST = git-ssh.knoe.dev
GITLAB_SHELL_LOADBALANCER_IP = 34.106.243.154 (regional us-west3)
GITLAB_SHELL_EXTERNAL_TRAFFIC_POLICY = Local (default)
Why SSH lives on a separate hostname+IP: Google-managed certs require
a GCE global L7 IP; port 22 needs a regional Network LB. Those can not
share an IP on GCP, so git.knoe.dev stays on HTTPS and git-ssh.knoe.dev
takes SSH.
Wiring per surface:
- etc/init_gitlab.sh
* Added GITLAB_SSH_HOST (default git-ssh.<root> in k8s mode),
GITLAB_SHELL_LOADBALANCER_IP, GITLAB_SHELL_EXTERNAL_TRAFFIC_POLICY,
GITLAB_GLOBAL_STATIC_IP_NAME config keys.
* CR global.hosts.ssh now reads ${GITLAB_SSH_HOST}.
* gitlab-shell block conditionally renders service: {type:
LoadBalancer, loadBalancerIP, externalTrafficPolicy} when the LB IP
is set.
* GITLAB_GCE_TLS_ANNOTATIONS_YAML gains
kubernetes.io/ingress.global-static-ip-name: "<name>" when
GITLAB_GLOBAL_STATIC_IP_NAME is set.
- etc/init_kong.sh
* Added SVC_KNOE_GLOBAL_STATIC_IP_NAME var near other SERVICE_TLS_*
defaults.
* gce_tls_annotations heredoc gets the static-IP line appended on the
same condition.
- supabase/helm/render_supabase.py
* Two new _first(env, cfg, default) extractions for
SUPABASE_API_GLOBAL_STATIC_IP_NAME and
SUPABASE_STUDIO_GLOBAL_STATIC_IP_NAME near the existing managed-cert
/ frontend-config vars.
* Two new dict-spread blocks in the Kong + Studio ingress annotations
that emit the static-IP annotation only when mode=k8s and the value
is non-empty.
All three ingress surfaces already use kubectl apply (merge-friendly);
re-running init_gitlab.sh / init_kong.sh / deploy.sh (supabase step) is
enough to pick up the new annotation. The GKE LB controller will swap
each ingress's forwarding rule from the auto-generated k8s2-fr-*
reservation to the named reservation, then release the old ephemeral.
Ingress IPs change; DNS records need updating (TTL <=300s while
iterating). Google-managed cert for gitlab-managed-cert will briefly
Provision again during the swap; the other three were already
Provisioning.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enables iterating on GitLab reconciles without a full deploy.sh run and
unblocks the end-to-end login flow:
- etc/init_gitlab.sh
* Pre-scan $@ for --config before sourcing prole_cfg.sh so
PROLE_DEPLOY_CFG is set for standalone invocations; previously
prole_cfg.sh auto-picked conf/k3d.cfg and leaked KUBECONTEXT=dev,
failing with `error: context "dev" does not exist`.
* _init_gitlab_resolve_secretref: bash mirror of
knoe/core/actions.py:_resolve_secretref_value so
secretref://google-oidc-client-* resolves from etc/secrets/ files
when run outside deploy.sh Python env-injection.
* Garage bucket array: drop -storage suffix from uploads/artifacts/
lfs/packages/dependency-proxy. Chart defaults for these have no
suffix, and the object_store block in the CR does not override
per-object bucket names; the prior mismatch caused first-login 500s
(NoSuchBucket on avatar PUT).
- deploy/gcp/gke/gitlab-google-oidc-secret.example.yaml
* Add discovery: true so omniauth-openid_connect fetches Google
.well-known/openid-configuration; fixes "Could not authenticate
from OpenIDConnect: No host info" on the callback.
- conf/gke.cfg
* GITLAB_WEBSERVICE_LIMITS_MEMORY 1800M -> 3Gi, REQUESTS_MEMORY
900M -> 2Gi, REQUESTS_CPU 200m -> 500m. Live pod was sitting at
1706Mi/1800M (99%) in OOMKilled loop.
- conf/port-mapping.cfg: add supabase + gitea forwards, fix postgres
namespace knoe-db -> knoe-db-0 for split-cluster CNPG layout.
Adds read-only diagnostics used to chase the above:
- etc/diag_gitlab_boot.sh, diag_gitlab_webservice_oom.sh, diag_gke_storage.sh
- etc/ensure_default_storage_class.sh + k8s/prole/storageclass-gcp-standard-hdd.yaml
(preflight + HDD-default SC manifest for SSD-quota-constrained GKE projects).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Added pre-apply validation for ManagedCertificate and FrontendConfig objects to ensure existence before ingress application.
- Improved GKE ingress diagnostics with annotations and resource state checks for pre- and post-apply phases.
- Introduced patch-only handling for annotation drifts to avoid unnecessary ingress replacements.
- Enhanced HTTPS probing logic with GCLB convergence checks for managed certificate states.
- Updated `deploy.sh`, `init_kong.sh`, and `init_gitlab.sh` with detailed ingress dependency validation and reconciliation steps.
- Improved handling of GKE preload TLS annotations by ignoring `SERVICE_PRE_SHARED_CERT` in k8s mode with managed TLS resources.
- Added drift detection for ingress class, ManagedCertificate, and FrontendConfig annotations, with forced replacement for GCLB binding consistency.
- Enhanced TLS diagnostics with detailed path state, failure reasons, and HTTPS probe handling for GKE configurations.
- Updated tests and scripts to validate GKE-specific behavior and ingress rendering.
- Updated `deploy.sh` and `init_kong.sh` to support GKE ManagedCertificate and FrontendConfig reconciliation in k8s mode with GCE ingress class.
- Enhanced TLS path diagnostics to distinguish between missing, attached-but-not-serving, and actively serving configurations.
- Added HTTPS probing for GKE-managed public ingress paths and validation for managed certificate statuses.
- Updated `render_supabase.py` to generate and attach TLS annotations for Supabase API and Studio ingresses in k8s mode.
- Added tests to validate TLS path handling, probing, and manifest generation for GKE deployments.
- Introduced `GITLAB_FALLBACK_INGRESS_NAME` for better flexibility in managing fallback ingress names.
- Added logic to clean up both current and legacy fallback ingresses (`gitlab-frontdoor-ingress`, `gitlab-kong-ingress`).
- Enhanced diagnostics for ingress host conflicts and TLS path validation.
- Updated `deploy.sh` to improve runtime configuration resolution and host diagnostics.
- Refined tests to ensure correct fallback ingress handling and conflict resolution.
- Changed GitLab ingress default ownership in GKE/k8s mode from `operator` to `fallback` for explicit GCE ingress path anchoring.
- Added logic to clean up stale operator-managed ingresses when fallback ownership is selected.
- Simplified fallback ingress creation workflow by removing redundant skip logic.
- Updated relevant tests to reflect ownership changes and ingress handling enhancements.
- Implemented `gitlab_split_cluster_ownership_diagnostics` to enforce and log split-cluster ownership policies between app and DB contexts.
- Updated workload convergence logic to prioritize readiness detection over historical restart counts for webservice pods.
- Enhanced deployment utilities with additional diagnostics for pod readiness states and replica discrepancies.
- Introduced tests to validate split-cluster enforcement and refined convergence behavior.
- Added logic to set and validate `GITLAB_TRUSTED_PROXIES` for forwarded headers behind ingress/load balancers.
- Improved `gitlab_old_replicaset_live_summary` to handle empty summaries gracefully by returning zero.
- Enhanced logging to confirm trusted proxies configuration.
- Updated tests to validate trusted proxies rendering and summary handling behavior.
- Introduced `gitlab_rendered_replica_source_fields_from_cr` to extract and log replica fields from rendered CR values.
- Enhanced logging for replica source-of-truth mismatches, including detailed diagnostics of rendered CR fields.
- Updated registry-specific key sanitization to align with chart 9+ authoritative fields.
- Added and extended tests to validate replica field rendering, sanitization, and logging enhancements.
- Introduced utilities to identify and strip deprecated top-level replica keys for chart v9 and newer.
- Added preflight validation to block CR apply if removed keys remain.
- Enhanced logging to detail stripped keys and remaining deprecated fields.
- Updated tests to ensure proper handling of top-level and hpa replica keys in rendered CR values.
- Added utilities for diagnosing and cleaning up StatefulSet template and PVC storage class mismatches in Gitaly.
- Improved logging for storage class fields in GitLab CR rendering and live StatefulSet diagnostics.
- Introduced `cleanup_gitlab_wrong_gitaly_template_storage` for automated destructive repair of misconfigured storage templates.
- Added tests to ensure authoritative Gitaly storage class enforcement and error handling for mismatches.
- Replaced `is_gke_standard_storage_equivalent` with `gitlab_storage_class_matches_expected` for stricter logic and better readability.
- Added utilities for diagnosing and cleaning up PVC/PV and StatefulSet template storage class mismatches.
- Introduced `enforce_gitlab_storage_class_target` to automate configuration enforcement and diagnostics.
- Enhanced logging and failure remediation for storage class mismatches in GitLab deployments.
- Introduced `gitlab_verify_replica_source_of_truth` and `gitlab_replica_source_of_truth_report` for replica drift validation and convergence.
- Enhanced logging and remediation logic to handle source-of-truth mismatches during deployment reconciliation.
- Added explicit `minReplicas` and `maxReplicas` handling for key components to prevent drift.
- Improved failure classification and reporting for post-remediation settle verification.
- Added `gitlab_selector_for_deployment` utility to dynamically resolve deployment selectors.
- Updated pod count and name retrieval to use resolved selectors, ensuring accuracy.
- Enhanced logging to include selector details for better debugging and remediation tracking.
- Addressed replica drift detection with CR-based reconciliation and scale-down logic.
- Added settle verification logic with configurable timeout and polling intervals for corrected deployments.
- Introduced new utility function `gitlab_non_terminal_pod_names_for_app` to retrieve non-terminal pod names.
- Improved logging for final deployment state and pod details after remediation actions.
- Added settle verification logic with configurable timeout and polling intervals for corrected deployments.
- Introduced new utility function `gitlab_non_terminal_pod_names_for_app` to retrieve non-terminal pod names.
- Improved logging for final deployment state and pod details after remediation actions.
- Introduced `gitlab_post_remediation_success` flag to streamline corrective action logging.
- Adjusted logic to skip unnecessary workload convergence gates when remediation is successful.
- Introduced `gitlab_post_remediation_success` flag to streamline corrective action logging.
- Adjusted logic to skip unnecessary workload convergence gates when remediation is successful.
Summarize CLAUDE.md updates:
- Document both live GKE clusters as e2-standard-2 x3 in us-west3 and note stale app default in code.
- Call out DB cluster naming mismatch between conf (knoe-dev-cnpg-0) and code default (knoe-cnpg-0).
- Add SSD quota guidance: 300GB fully consumed by CNPG, so non-CNPG PVCs must use standard/pd-standard.
- Refresh reset script reference and quota section details to match current runtime state.
Co-authored-by: Junie <junie@jetbrains.com>
- Added detailed logging for APP and DB contexts in `init_gitlab.sh` and `deploy.sh`.
- Enabled explicit passing of contexts and configuration paths to avoid fallbacks.
- Introduced strict environment resolution logic in `prole_cfg.sh` with fail-safe mechanisms.
- Added skipping logic for aged or bound PVC events in `deploy.sh` to reduce noise in diagnostics.
- Enforced replica target of 1 for specific GitLab workloads to ensure compliance with requirements.
- Added skipping logic for aged or bound PVC events in `deploy.sh` to reduce noise in diagnostics.
- Enforced replica target of 1 for specific GitLab workloads to ensure compliance with requirements.
