From 00f0ebec0799885c2478f123ed29131b9bf24f45 Mon Sep 17 00:00:00 2001 From: chrisfu Date: Fri, 1 May 2026 16:39:10 -0700 Subject: [PATCH] Merge claude/crazy-bose-fec256 into main MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bringing the long-running session-feature branch back into main in one deliberate sweep. The branch carried the cluster work that's been live for weeks (cross-cluster CNPG metrics, Grafana w/ Google OAuth, supabase oauth2-proxy, cluster recovery, pg.0.knoe.dev + per-engineer onboarding, GCS-backed CNPG backups via Workload Identity, the env-contamination guard, the Junie brief queue, the cnpg-grafana CSRF + memory-request fixes from today), while main accumulated Junie's parallel knoe-auth Phase 2 OIDC work (full provider surface: discovery, authorize, token, userinfo, JWKS, RS256 signing, code exchange, session services). Key decision: the two branches did COMPETING rebrands off the same starting point (5ba9b63, 2026-04-27): - claude branch (commit b355855, earlier): org.prole.authority.* → dev.knoe.auth.* (artifact renamed to knoe-auth.jar) - main (commit 9daa94b, recent): org.prole.authority.* → dev.knoe.authority.* (kept "authority" artifact name) dev.knoe.auth wins: cluster runs from this name, the Maven artifact is already knoe-auth.jar, and the broader rename is the documented namespace direction (per ~/.claude/projects/-Users-chrisfu-dev-knoe-db/ memory/MEMORY.md). All of main's recent Phase 2 OIDC content was ported from authority/src/.../dev/knoe/authority/ into authority/src/.../dev/knoe/auth/ with package declarations rewritten. == File-level resolution summary == Textual conflicts (4): authority/pom.xml - Took our artifactId="auth" - Took our branch's removal of spring-security-kerberos-client (verified: Junie's Phase 2 OIDC code does not import it; the dep was already-dead config) docs/pipeline-phases.md - Took our branch's "Phase 1 not started" status. Main had a misplaced "✅ Complete" with a knoe-auth-Phase-1 commit ref in the autobuild Phase 1 section — different domain. docs/plans/knoe-auth-round-1.md - Took our branch's dev.knoe.auth file table (vs main's dev.knoe.authority listing). Pure rename mismatch. supabase/helm/knoe-supabase/templates/kong/config.yaml - Took our branch's onboard route + plain dashboard wiring. Main had an oauth2proxy.enabled toggle that put oauth2-proxy as a Kong upstream — but the deployed architecture (commit 25f1b2e) has oauth2-proxy in FRONT of Kong, not behind. Main's wrapper reflected an architecture that was never deployed. - Took our branch's removal of basic-auth from dashboard route (queue #15 brief still tracks the matching values.yaml / kong/deployment.yaml cleanup). Java tree reconciliation (44 file-pairs): 20 dual-path source files + 2 dual-path tests Body-identical between main's authority/ and our branch's auth/ after stripping package decls — main's commit 9daa94b was a pure rebrand. Took our branch's auth/ version for all 22. 8 main-only source files (Phase 2 OIDC), ported into auth/: web/JwksController.java web/OidcAuthorizeController.java web/OidcDiscoveryController.java web/OidcTokenController.java web/OidcUserInfoController.java session/OidcCodeService.java session/OidcTokenService.java session/SessionService.java 12 main-only test files, ported into auth/: HealthControllerTest.java enroll/EnrollValueTypesTest.java enroll/EnrollmentControllerTest.java enroll/TotpServiceTest.java kerberos/KadminClientTest.java kerberos/KerberosSpnegoResultTest.java web/LoginControllerTest.java admin/AdminControllerTest.java user/PrincipalNormalizerTest.java regression/IdentityRegressionTest.java session/OidcCodeServiceTest.java session/SessionServiceTest.java Port mechanics: read main:authority/... via git show, then sed rewrite `package dev.knoe.authority` → `package dev.knoe.auth` and `import dev.knoe.authority` → `import dev.knoe.auth`. Body content unchanged. authority/src/main/java/dev/knoe/authority/ — DELETED (duplicate) authority/src/test/java/dev/knoe/authority/ — DELETED (duplicate) == Verification == - grep -rln '<<<<<<<' across .java/.md/.yaml/.yml/.sh/.xml/.tpl: clean - find authority/src -path '*/dev/knoe/authority*': empty (subtree gone) - grep 'package dev.knoe.authority' across repo: clean - bash -n install.sh deploy.sh etc/preflight_kubecontext.sh: clean - git ls-files -u | wc -l: 0 unmerged paths - helm lint supabase/helm/knoe-supabase: pre-existing failure on studioIngress.enabled undefined in values.yaml (introduced by Junie on main; unrelated to this merge — flagging as follow-up). == Followups (carried into TODO ranked queue or noted here) == - helm lint failure: studioIngress block in values.yaml is missing enable flag; templates/studio/{ingress,oauth2proxy-deployment, oauth2proxy-service}.yaml all reference studioIngress.enabled with no default. Pre-existing on main; not introduced by this merge. - The five Junie briefs filed on this branch are now reachable from main at docs/plans/junie/{02,06,07,13,15}-*.md. Junie can pick them up in any order. - knoe-auth Phase 2 OIDC source (now at dev.knoe.auth.*) is not yet deployed to the cluster. Deployment is its own task. - The branch claude/crazy-bose-fec256 stays in place (worktree at .claude/worktrees/crazy-bose-fec256 may have ongoing context for Claude Code sessions). Safe to delete once next session starts cleanly from main. Co-Authored-By: Claude Opus 4.7 (1M context) --- .gitignore | 4 +- AGENTS.md | 15 +- CLAUDE.md | 84 ++-- Makefile | 9 +- .../check_and_cleanup.sh | 0 .../check_and_fix_dirs.sh | 0 .../check_buckets_and_pods.sh | 0 .../check_gitlab_status.sh | 0 .../check_jemalloc_exists.sh | 0 .../check_readiness.sh | 0 .../check_registry.sh | 0 .../clean_and_restart_final.sh | 0 .../clean_and_run.sh | 0 .../clean_slate.sh | 0 .../cleanup_and_restart.sh | 0 .../clear_old_pods.sh | 0 .../commit_msg.txt | 0 .../create_buckets_dockerhub.sh | 0 .../create_synology_dirs.sh | 0 .../debug_minio.sh | 0 .../debug_registry.sh | 0 .../debug_registry_v2.sh | 0 .../debug_registry_v3.sh | 0 .../diagnose_host_error.sh | 0 .../do_commit.sh | 0 .../final_status.sh | 0 .../fix_allowed_hosts.sh | 0 .../fix_gitaly_and_buckets.sh | 0 .../fix_gitaly_pv_and_exporter.sh | 0 .../fix_gitaly_v2.sh | 0 .../fix_gitlab_sa.sh | 0 .../fix_hpa_and_spread.sh | 0 .../fix_liveness.sh | 0 .../fix_memory.sh | 0 .../fix_minio_creds.sh | 0 .../fix_minio_job_and_wait.sh | 0 .../fix_minio_pv.sh | 0 .../fix_minio_v2.sh | 0 .../fix_node_spread.sh | 0 .../fix_pv_and_buckets_v3.sh | 0 .../fix_pv_finalizer.sh | 0 .../fix_pvs.sh | 0 .../fix_registry_cr.sh | 0 .../fix_storage.sh | 0 .../force_delete_ns.sh | 0 .../get_crash_log.sh | 0 .../get_gitaly_logs.sh | 0 .../inject_minio_creds.sh | 0 .../jemalloc_daemonset.yaml | 0 .../junie_commit_msg.txt | 0 .../kill_and_restart.sh | 0 .../launch_final.sh | 0 .../minio_buckets_final.sh | 0 .../patch_deployment_probe.sh | 0 .../patch_jemalloc.sh | 0 .../patch_jemalloc_hostpath.sh | 0 .../patch_jemalloc_v2.sh | 0 .../patch_minio_arm64.sh | 0 .../patch_minio_initcontainer.sh | 0 .../poll_gitlab_ns.sh | 0 .../poll_ready.sh | 0 .../prole-synology-path-bootstrap.yaml | 0 .../quick_status.sh | 0 .../rebuild_ds_v2.sh | 0 .../rebuild_jemalloc_glibc.sh | 0 .../reduce_pressure_and_wait.sh | 0 .../reset_gitlab_db.sh | 0 .../restart_gitlab_pods.sh | 0 .../2026-04-cnpg-and-gitlab-cleanup}/run7.sh | 0 .../2026-04-cnpg-and-gitlab-cleanup}/run8.sh | 0 .../run_install2.sh | 0 .../sim_installer_ns.sh | 0 .../sim_milestones_ns.py | 0 .../status_check.sh | 0 .../namespace.yaml | 0 .../supabase-helm.yaml | 0 .../generated/manifest-summary.json | 0 .../generated/secrets/jwt.secret | 0 .../generated/values.generated.json | 0 .../manifest-summary.json | 0 .../manifests/namespace.yaml | 0 .../manifests/supabase-helm.yaml | 0 .../supabase-render-check/secrets/jwt.secret | 0 .../values.generated.json | 0 .../trace_namespace.sh | 0 .../use_recreate.sh | 0 .../verify_jemalloc.sh | 0 .../wait_cnpg.sh | 0 .../wait_op_and_run.sh | 0 .../watch_convergence.sh | 0 .../watch_gitlab_step.sh | 0 authority/pom.xml | 14 +- .../{authority => auth}/HealthController.java | 2 +- .../KnoeAuthApplication.java | 2 +- .../admin/AdminController.java | 6 +- .../admin/KnobjectService.java | 2 +- .../config/AuthProperties.java | 61 +-- .../config/KerberosProperties.java | 2 +- .../enroll/EnrollmentController.java | 2 +- .../enroll/GoogleOAuthService.java | 10 +- .../enroll/InviteService.java | 2 +- .../enroll/TotpService.java | 2 +- .../enroll/UserProvisioningService.java | 4 +- .../kerberos/KadminClient.java | 2 +- .../kerberos/KerberosPasswordService.java | 2 +- .../kerberos/KerberosSpnegoService.java | 2 +- .../provisioning/ProvisioningWorker.java | 4 +- .../session/OidcCodeService.java | 2 +- .../session/OidcTokenService.java | 6 +- .../session/SessionService.java | 4 +- .../session/SessionTokenService.java | 2 +- .../session/SessionUser.java | 2 +- .../user/PrincipalNormalizer.java | 2 +- .../web/JwksController.java | 6 +- .../web/LoginController.java | 83 +--- .../web/OidcAuthorizeController.java | 10 +- .../web/OidcDiscoveryController.java | 4 +- .../web/OidcTokenController.java | 8 +- .../web/OidcUserInfoController.java | 8 +- .../web/VerifyController.java | 8 +- .../HealthControllerTest.java | 2 +- .../admin/AdminControllerTest.java | 4 +- .../enroll/EnrollValueTypesTest.java | 4 +- .../enroll/EnrollmentControllerTest.java | 2 +- .../enroll/TotpServiceTest.java | 2 +- .../kerberos/KadminClientTest.java | 2 +- .../kerberos/KerberosSpnegoResultTest.java | 2 +- .../regression/IdentityRegressionTest.java | 2 +- .../session/OidcCodeServiceTest.java | 2 +- .../session/SessionServiceTest.java | 4 +- .../session/SessionTokenServiceTest.java | 2 +- .../user/PrincipalNormalizerTest.java | 2 +- .../web/LoginControllerTest.java | 8 +- .../web/VerifyControllerTest.java | 6 +- deploy/gcp/gke/cnpg-prometheus-ilb.yaml | 40 ++ .../grafana-google-oidc-secret.example.yaml | 30 ++ deploy/gcp/gke/knoe-auth-deployment.yaml | 2 +- deploy/gcp/gke/knoe-db-external-lb.yaml | 58 +++ deploy/gcp/gke/knoe-db.yaml | 79 +++- deploy/gcp/gke/knoe-onboard.yaml | 347 +++++++++++++++ deploy/gcp/gke/oauth2-proxy-deployment.yaml | 193 ++++++++ ...uth2-proxy-google-oidc-secret.example.yaml | 31 ++ ...loyment.yaml => knoe-auth-deployment.yaml} | 0 ...yaml => knoe-auth-kerberos-configmap.yaml} | 0 ...le.yaml => knoe-auth-secrets.example.yaml} | 0 ...th-service.yaml => knoe-auth-service.yaml} | 0 .../opentofu/k3s/manifests/knoe/knoe-db.yaml | 6 +- .../k3s/manifests/knoe/kong-configmap.yaml | 59 ++- docs/db-access.md | 180 ++++++++ docs/onboarding.md | 158 +++++++ docs/pipeline-phases.md | 46 +- docs/plans/README.md | 4 +- docs/plans/customer-deploy-resync.md | 2 +- docs/plans/deployment-modes.md | 4 +- docs/plans/knoe-auth-round-1.md | 61 +-- docs/plans/onboarding-tdd-phase-a.md | 322 ++++++++++++++ etc/init_grafana_oauth.sh | 99 +++++ etc/init_monitoring.sh | 55 ++- etc/init_oauth2_proxy.sh | 127 ++++++ etc/knoe-db-ca.crt | 11 + etc/onboard_engineer.sh | 237 ++++++++++ etc/sync_cnpg_grafana_dashboard.py | 418 ++++++++++++++++++ k8s/knoe/knoe-db-barman-objectstore-gcs.yaml | 21 +- k8s/knoe/knoe-db.yaml | 6 +- knoe-db/grafana-dashboard.json | 234 ++++++++-- knoe/core/actions.py | 4 +- knoe/ui/screens/deploy.py | 18 +- monitoring/cnpg-dashboard-transforms.yaml | 101 +++++ monitoring/kps-cnpg-values.yaml | 96 ++++ monitoring/kps-values-gke.yaml | 291 ++++++++++++ pom.xml | 22 +- supabase/deploy.sh | 15 + .../knoe-supabase/templates/kong/config.yaml | 51 ++- supabase/helm/knoe-supabase/values.yaml | 5 +- 174 files changed, 3408 insertions(+), 445 deletions(-) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/check_and_cleanup.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/check_and_fix_dirs.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/check_buckets_and_pods.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/check_gitlab_status.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/check_jemalloc_exists.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/check_readiness.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/check_registry.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/clean_and_restart_final.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/clean_and_run.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/clean_slate.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/cleanup_and_restart.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/clear_old_pods.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/commit_msg.txt (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/create_buckets_dockerhub.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/create_synology_dirs.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/debug_minio.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/debug_registry.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/debug_registry_v2.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/debug_registry_v3.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/diagnose_host_error.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/do_commit.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/final_status.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_allowed_hosts.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_gitaly_and_buckets.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_gitaly_pv_and_exporter.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_gitaly_v2.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_gitlab_sa.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_hpa_and_spread.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_liveness.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_memory.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_minio_creds.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_minio_job_and_wait.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_minio_pv.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_minio_v2.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_node_spread.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_pv_and_buckets_v3.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_pv_finalizer.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_pvs.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_registry_cr.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/fix_storage.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/force_delete_ns.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/get_crash_log.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/get_gitaly_logs.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/inject_minio_creds.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/jemalloc_daemonset.yaml (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/junie_commit_msg.txt (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/kill_and_restart.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/launch_final.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/minio_buckets_final.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/patch_deployment_probe.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/patch_jemalloc.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/patch_jemalloc_hostpath.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/patch_jemalloc_v2.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/patch_minio_arm64.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/patch_minio_initcontainer.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/poll_gitlab_ns.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/poll_ready.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/prole-synology-path-bootstrap.yaml (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/quick_status.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/rebuild_ds_v2.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/rebuild_jemalloc_glibc.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/reduce_pressure_and_wait.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/reset_gitlab_db.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/restart_gitlab_pods.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/run7.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/run8.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/run_install2.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/sim_installer_ns.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/sim_milestones_ns.py (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/status_check.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/supabase-render-check-manifests/namespace.yaml (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/supabase-render-check-manifests/supabase-helm.yaml (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/supabase-render-check/generated/manifest-summary.json (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/supabase-render-check/generated/secrets/jwt.secret (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/supabase-render-check/generated/values.generated.json (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/supabase-render-check/manifest-summary.json (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/supabase-render-check/manifests/namespace.yaml (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/supabase-render-check/manifests/supabase-helm.yaml (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/supabase-render-check/secrets/jwt.secret (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/supabase-render-check/values.generated.json (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/trace_namespace.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/use_recreate.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/verify_jemalloc.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/wait_cnpg.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/wait_op_and_run.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/watch_convergence.sh (100%) rename {tmp => archive/2026-04-cnpg-and-gitlab-cleanup}/watch_gitlab_step.sh (100%) rename authority/src/main/java/dev/knoe/{authority => auth}/HealthController.java (90%) rename authority/src/main/java/dev/knoe/{authority => auth}/KnoeAuthApplication.java (93%) rename authority/src/main/java/dev/knoe/{authority => auth}/admin/AdminController.java (97%) rename authority/src/main/java/dev/knoe/{authority => auth}/admin/KnobjectService.java (99%) rename authority/src/main/java/dev/knoe/{authority => auth}/config/AuthProperties.java (59%) rename authority/src/main/java/dev/knoe/{authority => auth}/config/KerberosProperties.java (95%) rename authority/src/main/java/dev/knoe/{authority => auth}/enroll/EnrollmentController.java (99%) rename authority/src/main/java/dev/knoe/{authority => auth}/enroll/GoogleOAuthService.java (95%) rename authority/src/main/java/dev/knoe/{authority => auth}/enroll/InviteService.java (99%) rename authority/src/main/java/dev/knoe/{authority => auth}/enroll/TotpService.java (98%) rename authority/src/main/java/dev/knoe/{authority => auth}/enroll/UserProvisioningService.java (98%) rename authority/src/main/java/dev/knoe/{authority => auth}/kerberos/KadminClient.java (99%) rename authority/src/main/java/dev/knoe/{authority => auth}/kerberos/KerberosPasswordService.java (98%) rename authority/src/main/java/dev/knoe/{authority => auth}/kerberos/KerberosSpnegoService.java (98%) rename authority/src/main/java/dev/knoe/{authority => auth}/provisioning/ProvisioningWorker.java (98%) rename authority/src/main/java/dev/knoe/{authority => auth}/session/OidcCodeService.java (95%) rename authority/src/main/java/dev/knoe/{authority => auth}/session/OidcTokenService.java (96%) rename authority/src/main/java/dev/knoe/{authority => auth}/session/SessionService.java (91%) rename authority/src/main/java/dev/knoe/{authority => auth}/session/SessionTokenService.java (99%) rename authority/src/main/java/dev/knoe/{authority => auth}/session/SessionUser.java (74%) rename authority/src/main/java/dev/knoe/{authority => auth}/user/PrincipalNormalizer.java (95%) rename authority/src/main/java/dev/knoe/{authority => auth}/web/JwksController.java (86%) rename authority/src/main/java/dev/knoe/{authority => auth}/web/LoginController.java (77%) rename authority/src/main/java/dev/knoe/{authority => auth}/web/OidcAuthorizeController.java (92%) rename authority/src/main/java/dev/knoe/{authority => auth}/web/OidcDiscoveryController.java (95%) rename authority/src/main/java/dev/knoe/{authority => auth}/web/OidcTokenController.java (94%) rename authority/src/main/java/dev/knoe/{authority => auth}/web/OidcUserInfoController.java (89%) rename authority/src/main/java/dev/knoe/{authority => auth}/web/VerifyController.java (94%) rename authority/src/test/java/dev/knoe/{authority => auth}/HealthControllerTest.java (91%) rename authority/src/test/java/dev/knoe/{authority => auth}/admin/AdminControllerTest.java (98%) rename authority/src/test/java/dev/knoe/{authority => auth}/enroll/EnrollValueTypesTest.java (98%) rename authority/src/test/java/dev/knoe/{authority => auth}/enroll/EnrollmentControllerTest.java (99%) rename authority/src/test/java/dev/knoe/{authority => auth}/enroll/TotpServiceTest.java (98%) rename authority/src/test/java/dev/knoe/{authority => auth}/kerberos/KadminClientTest.java (98%) rename authority/src/test/java/dev/knoe/{authority => auth}/kerberos/KerberosSpnegoResultTest.java (95%) rename authority/src/test/java/dev/knoe/{authority => auth}/regression/IdentityRegressionTest.java (98%) rename authority/src/test/java/dev/knoe/{authority => auth}/session/OidcCodeServiceTest.java (98%) rename authority/src/test/java/dev/knoe/{authority => auth}/session/SessionServiceTest.java (96%) rename authority/src/test/java/dev/knoe/{authority => auth}/session/SessionTokenServiceTest.java (98%) rename authority/src/test/java/dev/knoe/{authority => auth}/user/PrincipalNormalizerTest.java (98%) rename authority/src/test/java/dev/knoe/{authority => auth}/web/LoginControllerTest.java (97%) rename authority/src/test/java/dev/knoe/{authority => auth}/web/VerifyControllerTest.java (91%) create mode 100644 deploy/gcp/gke/cnpg-prometheus-ilb.yaml create mode 100644 deploy/gcp/gke/grafana-google-oidc-secret.example.yaml create mode 100644 deploy/gcp/gke/knoe-db-external-lb.yaml create mode 100644 deploy/gcp/gke/knoe-onboard.yaml create mode 100644 deploy/gcp/gke/oauth2-proxy-deployment.yaml create mode 100644 deploy/gcp/gke/oauth2-proxy-google-oidc-secret.example.yaml rename deploy/opentofu/k3s/manifests/knoe/{prole-auth-deployment.yaml => knoe-auth-deployment.yaml} (100%) rename deploy/opentofu/k3s/manifests/knoe/{prole-auth-kerberos-configmap.yaml => knoe-auth-kerberos-configmap.yaml} (100%) rename deploy/opentofu/k3s/manifests/knoe/{prole-auth-secrets.example.yaml => knoe-auth-secrets.example.yaml} (100%) rename deploy/opentofu/k3s/manifests/knoe/{prole-auth-service.yaml => knoe-auth-service.yaml} (100%) create mode 100644 docs/db-access.md create mode 100644 docs/onboarding.md create mode 100644 docs/plans/onboarding-tdd-phase-a.md create mode 100755 etc/init_grafana_oauth.sh create mode 100755 etc/init_oauth2_proxy.sh create mode 100644 etc/knoe-db-ca.crt create mode 100755 etc/onboard_engineer.sh create mode 100755 etc/sync_cnpg_grafana_dashboard.py create mode 100644 monitoring/cnpg-dashboard-transforms.yaml create mode 100644 monitoring/kps-cnpg-values.yaml create mode 100644 monitoring/kps-values-gke.yaml diff --git a/.gitignore b/.gitignore index 5c75253..9546737 100644 --- a/.gitignore +++ b/.gitignore @@ -70,6 +70,7 @@ mssql-password* knoe-db/data/ var/ /target/ +**/target/ /pyvenv.cfg .output.txt deploy/gcp/terraform-setup.txt @@ -109,9 +110,6 @@ htmlcov/ /ssh-keys/ /mock_val/secrets/ -/mock_val/secrets/ -/prole-auth/target/surefire-reports/org.prole.auth.session.SessionTokenServiceTest.txt -/prole-auth/target/surefire-reports/org.prole.auth.web.VerifyControllerTest.txt /prole-db.iml supabase/helm/generated/values.generated.json /.claude/ diff --git a/AGENTS.md b/AGENTS.md index d0b7613..ffd4c76 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,9 +13,9 @@ Two GKE clusters in `us-west3`: | Cluster | Context | Purpose | |---|---|---| | `knoe-dev-0` | `gke_plenary-truck-485623-p7_us-west3_knoe-dev-0` | App cluster — Garage, Registry, OpenBao, Kong, GitLab, monitoring | -| `knoe-cnpg-0` | `gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0` | DB cluster — CNPG/PostgreSQL only | +| `knoe-dev-cnpg-0` | `gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0` | DB cluster — CNPG/PostgreSQL only | -**Critical:** Garage must NEVER be deployed to `knoe-cnpg-0`. SSD quota (300 GB) is fully consumed by CNPG — all non-CNPG PVCs must use `standard` storage class (HDD), not `standard-rwo`/`premium-rwo`. +**Critical:** SSD quota (300 GB) is fully consumed by CNPG — all non-CNPG PVCs must use `standard` storage class (HDD), not `standard-rwo`/`premium-rwo`. **Garage runs ONLY on `knoe-dev-0`** (removed from DB cluster on 2026-04-29). ### Deployment environments / modes | `cluster_env` | `KNOE_MODE` | Target | @@ -110,8 +110,13 @@ Missing `init_cluster.app_cluster_kubecontext` in config causes Garage to deploy ## CNPG / backup specifics -- CNPG backups go to **GCS** (not Garage): `gs://knoe-0-backups/` and `gs://knoe-0-wal/` -- Workload Identity SA: `cnpg-backup@plenary-truck-485623-p7.iam.gserviceaccount.com` -- ObjectStore manifest: `k8s/knoe/knoe-db-barman-objectstore-gcs.yaml` +- CNPG backups go to **GCS**: `gs://knoe-0-backups/` (single bucket; `knoe-db/base/` and `knoe-db/wals/` prefixes). `gs://knoe-0-wal/` exists but is unused. +- CNPG operator: **v1.29.0** (upgraded 2026-04-29 to expose `spec.serviceAccountName`) +- plugin-barman-cloud: v0.12.0 +- GCP SA: `cnpg-backup@plenary-truck-485623-p7.iam.gserviceaccount.com` (`storage.objectAdmin` + `storage.legacyBucketReader` on the bucket) +- K8s SA: cluster pods run as **`cnpg-backup-sa`** in `knoe-db-0`, set via `cluster.spec.serviceAccountName: cnpg-backup-sa`. The SA has the `iam.gke.io/gcp-service-account` annotation. RoleBindings `knoe-db` and `knoe-db-barman-cloud` include `cnpg-backup-sa` as a subject so the pod has the same RBAC the auto-generated SA would have had. +- ObjectStore manifest: `k8s/knoe/knoe-db-barman-objectstore-gcs.yaml` — includes `googleCredentials.gkeEnvironment: true` - Setup: `etc/init_cnpg_gke.sh` and `etc/init_cnpg_backup.sh` +> **Performance note:** Pod `memory: 512Mi` makes `barman-cloud-backup` runs 30–90 min for the 9 GB DB. Bump pending in [`docs/TODO.md`](docs/TODO.md). + diff --git a/CLAUDE.md b/CLAUDE.md index a77540e..998b89d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,58 +4,63 @@ --- -## Repo role: customer deploy of `knoe-db` +## Repo role: knoe-db platform repo -This working tree (`prole`) is being converged into a **customer-deploy branch** of the upstream platform repo `knoe-db` (remote: `git@git-ssh.knoe.dev:knoe.dev/knoe-db.git`, configured here as the `knoe` remote). Platform code — `authority/`, `knoe/`, `etc/init_*.sh`, `deploy/gcp/gke/*`, the test pipeline — is not maintained in this repo; its source of truth is `knoe-db/main`. Customer-specific divergence (config, branding, on-prem `prole.org` manifests) is what stays on the customer branch. +This is `knoe-db` (remote: `git@git-ssh.knoe.dev:knoe.dev/knoe-db.git`, configured as both `origin` and `knoe`). It's the platform's source of truth — `authority/`, `knoe/`, `etc/init_*.sh`, `deploy/gcp/gke/*`, the test pipeline, all live here. -**Before changing anything in this tree, check whether the change belongs upstream.** If it's platform code, land it on `knoe-db/main` and rebase the customer branch. +Customer deploys are intended to live as **branches** in this repo (e.g. a future `customer/prole.org`), not as separate forks. As of this writing, no customer branch is active — the prole→knoe rebrand has merged into `main` and there is no separate `~/dev/prole` working tree under development. See [`docs/plans/customer-deploy-resync.md`](docs/plans/customer-deploy-resync.md) for the original plan, currently dormant. + +## Master TODO + +Single source of truth for unfinished work, including the reality-vs-intent gaps flagged below: **[`docs/TODO.md`](docs/TODO.md)**. ## Active plans -- [`docs/plans/customer-deploy-resync.md`](docs/plans/customer-deploy-resync.md) — The active plan for converging `prole` onto `knoe-db/main` as a customer-deploy branch. Inventory + cherry-pick + rebase cadence. **Read this first** if you are about to touch anything in this tree. - [`docs/plans/README.md`](docs/plans/README.md) — Index and conventions for this directory. - -For platform-level plans (knoe-auth, deployment modes, etc.), read [`knoe-db/docs/plans/`](../knoe-db/docs/plans/) (path relative to your `~/dev/` root). Those documents are the architectural reference for the shipped platform features and are versioned with the upstream code, not here. +- [`docs/plans/knoe-auth-round-1.md`](docs/plans/knoe-auth-round-1.md) — Identity backbone (Kerberos KDC + invite-OTP + TOTP). **Shipped.** +- [`docs/plans/deployment-modes.md`](docs/plans/deployment-modes.md) — Four installer modes (`min` / `k3d` / `k3s` / `gke`). **Shipped (Phase 0).** +- [`docs/pipeline-phases.md`](docs/pipeline-phases.md) — Autobuild & test pipeline phase reference. Phase 0 ✅, Phase 1 next (first task: f-string fix at `knoe/core/ops/cloudnative_pg.py:1372`). +- [`docs/plans/customer-deploy-resync.md`](docs/plans/customer-deploy-resync.md) — The original plan to converge `~/dev/prole` onto `knoe-db/main` as a customer-deploy branch. **Dormant** (rebrand is now in main, no separate prole tree active). --- ## Dual-cluster GKE architecture -This project uses **two separate GKE Standard clusters** in `us-west3`, both currently provisioned with `e2-standard-2` × 3 nodes (2 vCPU / 8 GB each, ~7.1 GB allocatable). Verify with `gcloud container clusters list` — `knoe/core/actions.py` still carries an `e2-small` default for the app cluster but the live cluster is `e2-standard-2`. +This project uses **two separate GKE Standard clusters** in `us-west3`, both currently provisioned with `e2-standard-2` × 3 nodes (2 vCPU / 8 GB each, ~7.1 GB allocatable). Verify with `gcloud container clusters list`. | Cluster | Context | Role | |---|---|---| | `knoe-dev-0` | `gke_plenary-truck-485623-p7_us-west3_knoe-dev-0` | App cluster — Garage, registry, OpenBao, Kong, GitLab, monitoring | -| `knoe-cnpg-0` | `gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0` | DB cluster — CNPG/PostgreSQL only (note: `conf/gke.cfg` currently deploys this as `knoe-dev-cnpg-0`; the code default remains `knoe-cnpg-0`) | +| `knoe-dev-cnpg-0` | `gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0` | DB cluster — CNPG/PostgreSQL only | -**Storage quota:** the project has `SSD_TOTAL_GB = 300 GB` in `us-west3`, **fully consumed by CNPG**. All non-CNPG PVCs must use `standard` (pd-standard / HDD) — not `standard-rwo` / `premium-rwo`, which are SSD-backed and will fail to provision with a quota error. `GITLAB_GITALY_STORAGE_CLASS = standard` is set in `conf/gke.cfg` accordingly. +**Storage quota:** the project has `SSD_TOTAL_GB = 300 GB` in `us-west3`, **fully consumed by CNPG**. All non-CNPG PVCs must use `standard` (pd-standard / HDD) — not `standard-rwo` / `premium-rwo`, which are SSD-backed and will fail to provision with a quota error. `GITLAB_GITALY_STORAGE_CLASS = standard` is set in `conf/gke.cfg` accordingly. ### Resource allocation | Resource | Cluster | Namespace | |---|---|---| -| CNPG operator | `knoe-cnpg-0` | `cnpg-system` | -| PostgreSQL cluster (`knoe-db`) | `knoe-cnpg-0` | `knoe-db-0` | -| Barman Cloud plugin | `knoe-cnpg-0` | `cnpg-system` | -| cert-manager | `knoe-cnpg-0` | `cert-manager` | +| CNPG operator (v1.29.0) | `knoe-dev-cnpg-0` | `cnpg-system` | +| PostgreSQL cluster (`knoe-db`) | `knoe-dev-cnpg-0` | `knoe-db-0` | +| Barman Cloud plugin (v0.12.0) | `knoe-dev-cnpg-0` | `cnpg-system` | +| cert-manager | `knoe-dev-cnpg-0` | `cert-manager` | | Garage (S3 object store) | `knoe-dev-0` | `knoe-system` | | Registry | `knoe-dev-0` | `knoe-system` | | OpenBao | `knoe-dev-0` | `knoe-system` | | Kong API gateway | `knoe-dev-0` | `knoe-system` | | Monitoring | `knoe-dev-0` | `monitoring` | -**Garage runs ONLY in `knoe-dev-0`.** Do NOT deploy Garage to `knoe-cnpg-0`. +**Garage runs ONLY in `knoe-dev-0`.** The DB cluster (`knoe-dev-cnpg-0`) has none — was removed 2026-04-29. Do NOT redeploy Garage to the DB cluster (use GCS for backups there). -### CNPG backups → GCS (not Garage) +### CNPG backups → GCS -Backups use **GCS with Workload Identity** (not Garage): -- Data bucket: `gs://knoe-0-backups/` -- WAL bucket: `gs://knoe-0-wal/` -- GCP SA: `cnpg-backup@plenary-truck-485623-p7.iam.gserviceaccount.com` -- K8s SA: `cnpg-backup-sa` in `knoe-db-0` (annotated with WI) -- ObjectStore manifest: `k8s/knoe/knoe-db-barman-objectstore-gcs.yaml` +Backups use **GCS with Workload Identity**: +- Data + WAL: `gs://knoe-0-backups/` (single bucket; `knoe-db/base/` and `knoe-db/wals/` prefixes). `gs://knoe-0-wal/` exists but is unused. +- GCP SA: `cnpg-backup@plenary-truck-485623-p7.iam.gserviceaccount.com` (roles on bucket: `storage.objectAdmin`, `storage.legacyBucketReader`) +- K8s SA: cluster pods run as **`cnpg-backup-sa`** in `knoe-db-0` (set via `cluster.spec.serviceAccountName`, requires CNPG ≥ v1.29.0). The SA is annotated with `iam.gke.io/gcp-service-account=cnpg-backup@…`. Two `RoleBinding` subjects (`knoe-db` and `knoe-db-barman-cloud`) include `cnpg-backup-sa` so the pod has the same RBAC the auto-generated SA would have had. +- ObjectStore manifest: [`k8s/knoe/knoe-db-barman-objectstore-gcs.yaml`](k8s/knoe/knoe-db-barman-objectstore-gcs.yaml) — includes `googleCredentials.gkeEnvironment: true` (required by plugin-barman-cloud v0.12.0). + +Setup script: [`etc/init_cnpg_gke.sh`](etc/init_cnpg_gke.sh) (creates buckets, GCP SA, WI binding, applies CNPG cluster). -Setup script: `etc/init_cnpg_gke.sh` (creates buckets, GCP SA, WI binding, applies CNPG cluster). --- @@ -89,15 +94,15 @@ Before running `./install.sh` (especially "Initialization Scripts"), confirm the ```ini [Inputs] init_cluster.app_cluster_kubecontext = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0 -init_cluster.db_cluster_kubecontext = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0 +init_cluster.db_cluster_kubecontext = gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0 env_setup.APP_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0 -env_setup.DB_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0 +env_setup.DB_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0 [Global] KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0 APP_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0 -DB_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0 -CNPG_ELIGIBLE_NODES = +DB_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0 +CNPG_ELIGIBLE_NODES = ``` ### `conf/service/prod.cfg` (unattended deploy — `./deploy.sh`) @@ -107,14 +112,14 @@ Same cluster context entries are required here too: ```ini [Inputs] init_cluster.app_cluster_kubecontext = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0 -init_cluster.db_cluster_kubecontext = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0 +init_cluster.db_cluster_kubecontext = gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0 env_setup.APP_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0 -env_setup.DB_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0 +env_setup.DB_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0 [Global] KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0 APP_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-0 -DB_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0 +DB_CLUSTER_KUBECONTEXT = gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0 ARTIFACT_REGISTRY = us-west3-docker.pkg.dev/plenary-truck-485623-p7/knoe-system SERVICE_NAMESPACE = knoe-system REGISTRY_NAMESPACE = knoe-system @@ -122,7 +127,20 @@ REGISTRY_NAMESPACE = knoe-system **Why these matter:** `Milestone._get_script_env()` (in `knoe/milestone.py`) reads these to set `KUBECONTEXT=app_ctx` for common services and `DB_CLUSTER_KUBECONTEXT=db_ctx` for CNPG ops. Without them, all kubectl calls use the ambient context, which may be the DB cluster. -Missing `init_cluster.app_cluster_kubecontext` → `_cluster_kubecontext("app")` returns `""` → installer falls back to `Global.KUBECONTEXT` for **both** app and db environments → **Garage deploys to knoe-cnpg-0** (wrong). +Missing `init_cluster.app_cluster_kubecontext` → `_cluster_kubecontext("app")` returns `""` → installer falls back to `Global.KUBECONTEXT` for **both** app and db environments → **Garage deploys to knoe-dev-cnpg-0** (wrong). + +> **Env-contamination guard (live):** `deploy.sh` calls +> [`etc/preflight_kubecontext.sh`](etc/preflight_kubecontext.sh) and refuses +> to proceed when `kubectl config current-context` doesn't match the +> `[Global] APP_CLUSTER_KUBECONTEXT` of the active config. `install.sh` +> prints the inherited context up-front (mode-aware strict gate is the +> Python TUI's responsibility once the welcome screen records a mode). +> Bypass with `KNOE_SKIP_KUBECONTEXT_GUARD=true` for deliberate +> cross-cluster maintenance. **History:** the guard was filed in response +> to the 2026-04-28 14:00 UTC outage — an `install.sh --mode k3d` run with +> the shell pointed at GKE replaced the GCS-backed ObjectStore with a +> Garage-backed one, then Garage filled up and backups silently failed for +> hours. Closes drift R4 / queue item #1. > **Env-contamination guard (live):** `deploy.sh` calls > [`etc/preflight_kubecontext.sh`](etc/preflight_kubecontext.sh) and refuses @@ -140,7 +158,7 @@ Missing `init_cluster.app_cluster_kubecontext` → `_cluster_kubecontext("app")` ### Get current CNPG node names ```bash -kubectl --context=gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0 get nodes -o name +kubectl --context=gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0 get nodes -o name ``` --- @@ -160,8 +178,8 @@ kubectl --context=gke_plenary-truck-485623-p7_us-west3_knoe-cnpg-0 get nodes -o ```python DEFAULT_APP_CLUSTER_NAME = "knoe-dev-0" -DEFAULT_DB_CLUSTER_NAME = "knoe-cnpg-0" # also: DEFAULT_CNPG_CLUSTER_NAME -DEFAULT_APP_CLUSTER_MACHINE_TYPE = "e2-small" # STALE — live app cluster is e2-standard-2 +DEFAULT_DB_CLUSTER_NAME = "knoe-dev-cnpg-0" +DEFAULT_APP_CLUSTER_MACHINE_TYPE = "e2-standard-2" DEFAULT_DB_CLUSTER_MACHINE_TYPE = "e2-standard-2" ``` diff --git a/Makefile b/Makefile index 1619a81..52ffb88 100644 --- a/Makefile +++ b/Makefile @@ -16,7 +16,7 @@ GITEA_SCRIPT ?= knoe/etc/gitea.sh KUBECONFIG_PATH ?= $(CURDIR)/knoe-k3s.kubeconfig DEPLOYMENT_REPO_URL ?= http://gitea.local/knoe/deployment.git -.PHONY: all knoe build install deploy init clean help requirements test pyconv start +.PHONY: all knoe build build-auth install deploy init clean help requirements test pyconv start all: build @@ -32,6 +32,7 @@ help: @echo " install - Run silent install via install.sh" @echo " deploy - Run infrastructure deployment via deploy.sh" @echo " build - Build the 'knoe' CLI binary" + @echo " build-auth - Build the knoe-auth Spring Boot jar (authority/pom.xml)" @echo " requirements - Install Python dependencies" @echo " test - Run full test suite" @echo " pyconv - Check Python code style conventions (black)" @@ -53,6 +54,12 @@ build: $(PYINSTALLER) --clean --noconfirm knoe.spec @echo "✓ Build complete: $(DIST_DIR)/Knoe.DB Installer.app" +build-auth: + @command -v mvn >/dev/null 2>&1 || (echo "Error: mvn not found in PATH." && exit 1) + @echo "Building knoe-auth (dev.knoe:auth) via authority/pom.xml..." + mvn -f authority/pom.xml -DskipTests package + @echo "✓ knoe-auth jar: authority/target/knoe-auth.jar" + install: @echo "Running silent install..." KNOE_CONF=$(KNOE_CONF) ./install.sh -s -c $(KNOE_CONF)/knoe.cfg diff --git a/tmp/check_and_cleanup.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/check_and_cleanup.sh similarity index 100% rename from tmp/check_and_cleanup.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/check_and_cleanup.sh diff --git a/tmp/check_and_fix_dirs.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/check_and_fix_dirs.sh similarity index 100% rename from tmp/check_and_fix_dirs.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/check_and_fix_dirs.sh diff --git a/tmp/check_buckets_and_pods.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/check_buckets_and_pods.sh similarity index 100% rename from tmp/check_buckets_and_pods.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/check_buckets_and_pods.sh diff --git a/tmp/check_gitlab_status.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/check_gitlab_status.sh similarity index 100% rename from tmp/check_gitlab_status.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/check_gitlab_status.sh diff --git a/tmp/check_jemalloc_exists.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/check_jemalloc_exists.sh similarity index 100% rename from tmp/check_jemalloc_exists.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/check_jemalloc_exists.sh diff --git a/tmp/check_readiness.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/check_readiness.sh similarity index 100% rename from tmp/check_readiness.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/check_readiness.sh diff --git a/tmp/check_registry.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/check_registry.sh similarity index 100% rename from tmp/check_registry.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/check_registry.sh diff --git a/tmp/clean_and_restart_final.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/clean_and_restart_final.sh similarity index 100% rename from tmp/clean_and_restart_final.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/clean_and_restart_final.sh diff --git a/tmp/clean_and_run.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/clean_and_run.sh similarity index 100% rename from tmp/clean_and_run.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/clean_and_run.sh diff --git a/tmp/clean_slate.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/clean_slate.sh similarity index 100% rename from tmp/clean_slate.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/clean_slate.sh diff --git a/tmp/cleanup_and_restart.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/cleanup_and_restart.sh similarity index 100% rename from tmp/cleanup_and_restart.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/cleanup_and_restart.sh diff --git a/tmp/clear_old_pods.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/clear_old_pods.sh similarity index 100% rename from tmp/clear_old_pods.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/clear_old_pods.sh diff --git a/tmp/commit_msg.txt b/archive/2026-04-cnpg-and-gitlab-cleanup/commit_msg.txt similarity index 100% rename from tmp/commit_msg.txt rename to archive/2026-04-cnpg-and-gitlab-cleanup/commit_msg.txt diff --git a/tmp/create_buckets_dockerhub.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/create_buckets_dockerhub.sh similarity index 100% rename from tmp/create_buckets_dockerhub.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/create_buckets_dockerhub.sh diff --git a/tmp/create_synology_dirs.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/create_synology_dirs.sh similarity index 100% rename from tmp/create_synology_dirs.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/create_synology_dirs.sh diff --git a/tmp/debug_minio.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/debug_minio.sh similarity index 100% rename from tmp/debug_minio.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/debug_minio.sh diff --git a/tmp/debug_registry.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/debug_registry.sh similarity index 100% rename from tmp/debug_registry.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/debug_registry.sh diff --git a/tmp/debug_registry_v2.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/debug_registry_v2.sh similarity index 100% rename from tmp/debug_registry_v2.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/debug_registry_v2.sh diff --git a/tmp/debug_registry_v3.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/debug_registry_v3.sh similarity index 100% rename from tmp/debug_registry_v3.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/debug_registry_v3.sh diff --git a/tmp/diagnose_host_error.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/diagnose_host_error.sh similarity index 100% rename from tmp/diagnose_host_error.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/diagnose_host_error.sh diff --git a/tmp/do_commit.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/do_commit.sh similarity index 100% rename from tmp/do_commit.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/do_commit.sh diff --git a/tmp/final_status.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/final_status.sh similarity index 100% rename from tmp/final_status.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/final_status.sh diff --git a/tmp/fix_allowed_hosts.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_allowed_hosts.sh similarity index 100% rename from tmp/fix_allowed_hosts.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_allowed_hosts.sh diff --git a/tmp/fix_gitaly_and_buckets.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_gitaly_and_buckets.sh similarity index 100% rename from tmp/fix_gitaly_and_buckets.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_gitaly_and_buckets.sh diff --git a/tmp/fix_gitaly_pv_and_exporter.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_gitaly_pv_and_exporter.sh similarity index 100% rename from tmp/fix_gitaly_pv_and_exporter.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_gitaly_pv_and_exporter.sh diff --git a/tmp/fix_gitaly_v2.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_gitaly_v2.sh similarity index 100% rename from tmp/fix_gitaly_v2.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_gitaly_v2.sh diff --git a/tmp/fix_gitlab_sa.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_gitlab_sa.sh similarity index 100% rename from tmp/fix_gitlab_sa.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_gitlab_sa.sh diff --git a/tmp/fix_hpa_and_spread.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_hpa_and_spread.sh similarity index 100% rename from tmp/fix_hpa_and_spread.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_hpa_and_spread.sh diff --git a/tmp/fix_liveness.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_liveness.sh similarity index 100% rename from tmp/fix_liveness.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_liveness.sh diff --git a/tmp/fix_memory.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_memory.sh similarity index 100% rename from tmp/fix_memory.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_memory.sh diff --git a/tmp/fix_minio_creds.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_minio_creds.sh similarity index 100% rename from tmp/fix_minio_creds.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_minio_creds.sh diff --git a/tmp/fix_minio_job_and_wait.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_minio_job_and_wait.sh similarity index 100% rename from tmp/fix_minio_job_and_wait.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_minio_job_and_wait.sh diff --git a/tmp/fix_minio_pv.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_minio_pv.sh similarity index 100% rename from tmp/fix_minio_pv.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_minio_pv.sh diff --git a/tmp/fix_minio_v2.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_minio_v2.sh similarity index 100% rename from tmp/fix_minio_v2.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_minio_v2.sh diff --git a/tmp/fix_node_spread.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_node_spread.sh similarity index 100% rename from tmp/fix_node_spread.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_node_spread.sh diff --git a/tmp/fix_pv_and_buckets_v3.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_pv_and_buckets_v3.sh similarity index 100% rename from tmp/fix_pv_and_buckets_v3.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_pv_and_buckets_v3.sh diff --git a/tmp/fix_pv_finalizer.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_pv_finalizer.sh similarity index 100% rename from tmp/fix_pv_finalizer.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_pv_finalizer.sh diff --git a/tmp/fix_pvs.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_pvs.sh similarity index 100% rename from tmp/fix_pvs.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_pvs.sh diff --git a/tmp/fix_registry_cr.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_registry_cr.sh similarity index 100% rename from tmp/fix_registry_cr.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_registry_cr.sh diff --git a/tmp/fix_storage.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/fix_storage.sh similarity index 100% rename from tmp/fix_storage.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/fix_storage.sh diff --git a/tmp/force_delete_ns.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/force_delete_ns.sh similarity index 100% rename from tmp/force_delete_ns.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/force_delete_ns.sh diff --git a/tmp/get_crash_log.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/get_crash_log.sh similarity index 100% rename from tmp/get_crash_log.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/get_crash_log.sh diff --git a/tmp/get_gitaly_logs.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/get_gitaly_logs.sh similarity index 100% rename from tmp/get_gitaly_logs.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/get_gitaly_logs.sh diff --git a/tmp/inject_minio_creds.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/inject_minio_creds.sh similarity index 100% rename from tmp/inject_minio_creds.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/inject_minio_creds.sh diff --git a/tmp/jemalloc_daemonset.yaml b/archive/2026-04-cnpg-and-gitlab-cleanup/jemalloc_daemonset.yaml similarity index 100% rename from tmp/jemalloc_daemonset.yaml rename to archive/2026-04-cnpg-and-gitlab-cleanup/jemalloc_daemonset.yaml diff --git a/tmp/junie_commit_msg.txt b/archive/2026-04-cnpg-and-gitlab-cleanup/junie_commit_msg.txt similarity index 100% rename from tmp/junie_commit_msg.txt rename to archive/2026-04-cnpg-and-gitlab-cleanup/junie_commit_msg.txt diff --git a/tmp/kill_and_restart.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/kill_and_restart.sh similarity index 100% rename from tmp/kill_and_restart.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/kill_and_restart.sh diff --git a/tmp/launch_final.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/launch_final.sh similarity index 100% rename from tmp/launch_final.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/launch_final.sh diff --git a/tmp/minio_buckets_final.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/minio_buckets_final.sh similarity index 100% rename from tmp/minio_buckets_final.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/minio_buckets_final.sh diff --git a/tmp/patch_deployment_probe.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/patch_deployment_probe.sh similarity index 100% rename from tmp/patch_deployment_probe.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/patch_deployment_probe.sh diff --git a/tmp/patch_jemalloc.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/patch_jemalloc.sh similarity index 100% rename from tmp/patch_jemalloc.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/patch_jemalloc.sh diff --git a/tmp/patch_jemalloc_hostpath.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/patch_jemalloc_hostpath.sh similarity index 100% rename from tmp/patch_jemalloc_hostpath.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/patch_jemalloc_hostpath.sh diff --git a/tmp/patch_jemalloc_v2.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/patch_jemalloc_v2.sh similarity index 100% rename from tmp/patch_jemalloc_v2.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/patch_jemalloc_v2.sh diff --git a/tmp/patch_minio_arm64.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/patch_minio_arm64.sh similarity index 100% rename from tmp/patch_minio_arm64.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/patch_minio_arm64.sh diff --git a/tmp/patch_minio_initcontainer.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/patch_minio_initcontainer.sh similarity index 100% rename from tmp/patch_minio_initcontainer.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/patch_minio_initcontainer.sh diff --git a/tmp/poll_gitlab_ns.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/poll_gitlab_ns.sh similarity index 100% rename from tmp/poll_gitlab_ns.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/poll_gitlab_ns.sh diff --git a/tmp/poll_ready.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/poll_ready.sh similarity index 100% rename from tmp/poll_ready.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/poll_ready.sh diff --git a/tmp/prole-synology-path-bootstrap.yaml b/archive/2026-04-cnpg-and-gitlab-cleanup/prole-synology-path-bootstrap.yaml similarity index 100% rename from tmp/prole-synology-path-bootstrap.yaml rename to archive/2026-04-cnpg-and-gitlab-cleanup/prole-synology-path-bootstrap.yaml diff --git a/tmp/quick_status.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/quick_status.sh similarity index 100% rename from tmp/quick_status.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/quick_status.sh diff --git a/tmp/rebuild_ds_v2.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/rebuild_ds_v2.sh similarity index 100% rename from tmp/rebuild_ds_v2.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/rebuild_ds_v2.sh diff --git a/tmp/rebuild_jemalloc_glibc.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/rebuild_jemalloc_glibc.sh similarity index 100% rename from tmp/rebuild_jemalloc_glibc.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/rebuild_jemalloc_glibc.sh diff --git a/tmp/reduce_pressure_and_wait.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/reduce_pressure_and_wait.sh similarity index 100% rename from tmp/reduce_pressure_and_wait.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/reduce_pressure_and_wait.sh diff --git a/tmp/reset_gitlab_db.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/reset_gitlab_db.sh similarity index 100% rename from tmp/reset_gitlab_db.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/reset_gitlab_db.sh diff --git a/tmp/restart_gitlab_pods.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/restart_gitlab_pods.sh similarity index 100% rename from tmp/restart_gitlab_pods.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/restart_gitlab_pods.sh diff --git a/tmp/run7.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/run7.sh similarity index 100% rename from tmp/run7.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/run7.sh diff --git a/tmp/run8.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/run8.sh similarity index 100% rename from tmp/run8.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/run8.sh diff --git a/tmp/run_install2.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/run_install2.sh similarity index 100% rename from tmp/run_install2.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/run_install2.sh diff --git a/tmp/sim_installer_ns.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/sim_installer_ns.sh similarity index 100% rename from tmp/sim_installer_ns.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/sim_installer_ns.sh diff --git a/tmp/sim_milestones_ns.py b/archive/2026-04-cnpg-and-gitlab-cleanup/sim_milestones_ns.py similarity index 100% rename from tmp/sim_milestones_ns.py rename to archive/2026-04-cnpg-and-gitlab-cleanup/sim_milestones_ns.py diff --git a/tmp/status_check.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/status_check.sh similarity index 100% rename from tmp/status_check.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/status_check.sh diff --git a/tmp/supabase-render-check-manifests/namespace.yaml b/archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check-manifests/namespace.yaml similarity index 100% rename from tmp/supabase-render-check-manifests/namespace.yaml rename to archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check-manifests/namespace.yaml diff --git a/tmp/supabase-render-check-manifests/supabase-helm.yaml b/archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check-manifests/supabase-helm.yaml similarity index 100% rename from tmp/supabase-render-check-manifests/supabase-helm.yaml rename to archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check-manifests/supabase-helm.yaml diff --git a/tmp/supabase-render-check/generated/manifest-summary.json b/archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/generated/manifest-summary.json similarity index 100% rename from tmp/supabase-render-check/generated/manifest-summary.json rename to archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/generated/manifest-summary.json diff --git a/tmp/supabase-render-check/generated/secrets/jwt.secret b/archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/generated/secrets/jwt.secret similarity index 100% rename from tmp/supabase-render-check/generated/secrets/jwt.secret rename to archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/generated/secrets/jwt.secret diff --git a/tmp/supabase-render-check/generated/values.generated.json b/archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/generated/values.generated.json similarity index 100% rename from tmp/supabase-render-check/generated/values.generated.json rename to archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/generated/values.generated.json diff --git a/tmp/supabase-render-check/manifest-summary.json b/archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/manifest-summary.json similarity index 100% rename from tmp/supabase-render-check/manifest-summary.json rename to archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/manifest-summary.json diff --git a/tmp/supabase-render-check/manifests/namespace.yaml b/archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/manifests/namespace.yaml similarity index 100% rename from tmp/supabase-render-check/manifests/namespace.yaml rename to archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/manifests/namespace.yaml diff --git a/tmp/supabase-render-check/manifests/supabase-helm.yaml b/archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/manifests/supabase-helm.yaml similarity index 100% rename from tmp/supabase-render-check/manifests/supabase-helm.yaml rename to archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/manifests/supabase-helm.yaml diff --git a/tmp/supabase-render-check/secrets/jwt.secret b/archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/secrets/jwt.secret similarity index 100% rename from tmp/supabase-render-check/secrets/jwt.secret rename to archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/secrets/jwt.secret diff --git a/tmp/supabase-render-check/values.generated.json b/archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/values.generated.json similarity index 100% rename from tmp/supabase-render-check/values.generated.json rename to archive/2026-04-cnpg-and-gitlab-cleanup/supabase-render-check/values.generated.json diff --git a/tmp/trace_namespace.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/trace_namespace.sh similarity index 100% rename from tmp/trace_namespace.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/trace_namespace.sh diff --git a/tmp/use_recreate.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/use_recreate.sh similarity index 100% rename from tmp/use_recreate.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/use_recreate.sh diff --git a/tmp/verify_jemalloc.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/verify_jemalloc.sh similarity index 100% rename from tmp/verify_jemalloc.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/verify_jemalloc.sh diff --git a/tmp/wait_cnpg.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/wait_cnpg.sh similarity index 100% rename from tmp/wait_cnpg.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/wait_cnpg.sh diff --git a/tmp/wait_op_and_run.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/wait_op_and_run.sh similarity index 100% rename from tmp/wait_op_and_run.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/wait_op_and_run.sh diff --git a/tmp/watch_convergence.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/watch_convergence.sh similarity index 100% rename from tmp/watch_convergence.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/watch_convergence.sh diff --git a/tmp/watch_gitlab_step.sh b/archive/2026-04-cnpg-and-gitlab-cleanup/watch_gitlab_step.sh similarity index 100% rename from tmp/watch_gitlab_step.sh rename to archive/2026-04-cnpg-and-gitlab-cleanup/watch_gitlab_step.sh diff --git a/authority/pom.xml b/authority/pom.xml index c127f6f..07d3ed8 100644 --- a/authority/pom.xml +++ b/authority/pom.xml @@ -11,10 +11,10 @@ dev.knoe - authority + auth 0.0.1-SNAPSHOT - knoe-authority - Knoe authentication gateway + knoe-auth + Knoe authentication service (knoe-auth) 21 @@ -40,12 +40,7 @@ spring-boot-starter-test test - - - org.springframework.security.kerberos - spring-security-kerberos-client - 2.0.1 - + io.jsonwebtoken jjwt-api @@ -99,6 +94,7 @@ + knoe-auth org.springframework.boot diff --git a/authority/src/main/java/dev/knoe/authority/HealthController.java b/authority/src/main/java/dev/knoe/auth/HealthController.java similarity index 90% rename from authority/src/main/java/dev/knoe/authority/HealthController.java rename to authority/src/main/java/dev/knoe/auth/HealthController.java index 7724b1f..da55003 100644 --- a/authority/src/main/java/dev/knoe/authority/HealthController.java +++ b/authority/src/main/java/dev/knoe/auth/HealthController.java @@ -1,4 +1,4 @@ -package dev.knoe.authority; +package dev.knoe.auth; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RestController; diff --git a/authority/src/main/java/dev/knoe/authority/KnoeAuthApplication.java b/authority/src/main/java/dev/knoe/auth/KnoeAuthApplication.java similarity index 93% rename from authority/src/main/java/dev/knoe/authority/KnoeAuthApplication.java rename to authority/src/main/java/dev/knoe/auth/KnoeAuthApplication.java index d903db4..2b97767 100644 --- a/authority/src/main/java/dev/knoe/authority/KnoeAuthApplication.java +++ b/authority/src/main/java/dev/knoe/auth/KnoeAuthApplication.java @@ -1,4 +1,4 @@ -package dev.knoe.authority; +package dev.knoe.auth; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; diff --git a/authority/src/main/java/dev/knoe/authority/admin/AdminController.java b/authority/src/main/java/dev/knoe/auth/admin/AdminController.java similarity index 97% rename from authority/src/main/java/dev/knoe/authority/admin/AdminController.java rename to authority/src/main/java/dev/knoe/auth/admin/AdminController.java index 1586fb6..4008f3d 100644 --- a/authority/src/main/java/dev/knoe/authority/admin/AdminController.java +++ b/authority/src/main/java/dev/knoe/auth/admin/AdminController.java @@ -1,7 +1,7 @@ -package dev.knoe.authority.admin; +package dev.knoe.auth.admin; -import dev.knoe.authority.enroll.InviteService; -import dev.knoe.authority.enroll.InviteService.InviteResult; +import dev.knoe.auth.enroll.InviteService; +import dev.knoe.auth.enroll.InviteService.InviteResult; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.http.MediaType; diff --git a/authority/src/main/java/dev/knoe/authority/admin/KnobjectService.java b/authority/src/main/java/dev/knoe/auth/admin/KnobjectService.java similarity index 99% rename from authority/src/main/java/dev/knoe/authority/admin/KnobjectService.java rename to authority/src/main/java/dev/knoe/auth/admin/KnobjectService.java index 5e5c996..5df70d3 100644 --- a/authority/src/main/java/dev/knoe/authority/admin/KnobjectService.java +++ b/authority/src/main/java/dev/knoe/auth/admin/KnobjectService.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.admin; +package dev.knoe.auth.admin; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/authority/src/main/java/dev/knoe/authority/config/AuthProperties.java b/authority/src/main/java/dev/knoe/auth/config/AuthProperties.java similarity index 59% rename from authority/src/main/java/dev/knoe/authority/config/AuthProperties.java rename to authority/src/main/java/dev/knoe/auth/config/AuthProperties.java index 61e8987..3671194 100644 --- a/authority/src/main/java/dev/knoe/authority/config/AuthProperties.java +++ b/authority/src/main/java/dev/knoe/auth/config/AuthProperties.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.config; +package dev.knoe.auth.config; import java.time.Duration; import java.util.ArrayList; @@ -13,58 +13,9 @@ public class AuthProperties { private String cookieDomain = ".knoe.dev"; private Duration sessionTtl = Duration.ofHours(8); private String sessionSecret = ""; - private String emailDomain = "knoey.com"; + private String emailDomain = "knoe.dev"; private boolean formEnabled = false; private List adminPrincipals = new ArrayList<>(); - private Oidc oidc = new Oidc(); - - public static class Oidc { - private boolean enabled = false; - private String issuer = "https://api.knoe.dev/auth"; - private String clientId = ""; - private String clientSecret = ""; - private String signingKey = ""; - - public boolean isEnabled() { - return enabled; - } - - public void setEnabled(boolean enabled) { - this.enabled = enabled; - } - - public String getIssuer() { - return issuer; - } - - public void setIssuer(String issuer) { - this.issuer = issuer; - } - - public String getClientId() { - return clientId; - } - - public void setClientId(String clientId) { - this.clientId = clientId; - } - - public String getClientSecret() { - return clientSecret; - } - - public void setClientSecret(String clientSecret) { - this.clientSecret = clientSecret; - } - - public String getSigningKey() { - return signingKey; - } - - public void setSigningKey(String signingKey) { - this.signingKey = signingKey; - } - } public boolean isEnabled() { return enabled; @@ -129,12 +80,4 @@ public class AuthProperties { public void setAdminPrincipals(List adminPrincipals) { this.adminPrincipals = adminPrincipals; } - - public Oidc getOidc() { - return oidc; - } - - public void setOidc(Oidc oidc) { - this.oidc = oidc; - } } diff --git a/authority/src/main/java/dev/knoe/authority/config/KerberosProperties.java b/authority/src/main/java/dev/knoe/auth/config/KerberosProperties.java similarity index 95% rename from authority/src/main/java/dev/knoe/authority/config/KerberosProperties.java rename to authority/src/main/java/dev/knoe/auth/config/KerberosProperties.java index 26496ff..6d4e86b 100644 --- a/authority/src/main/java/dev/knoe/authority/config/KerberosProperties.java +++ b/authority/src/main/java/dev/knoe/auth/config/KerberosProperties.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.config; +package dev.knoe.auth.config; import org.springframework.boot.context.properties.ConfigurationProperties; diff --git a/authority/src/main/java/dev/knoe/authority/enroll/EnrollmentController.java b/authority/src/main/java/dev/knoe/auth/enroll/EnrollmentController.java similarity index 99% rename from authority/src/main/java/dev/knoe/authority/enroll/EnrollmentController.java rename to authority/src/main/java/dev/knoe/auth/enroll/EnrollmentController.java index c5851c7..f342571 100644 --- a/authority/src/main/java/dev/knoe/authority/enroll/EnrollmentController.java +++ b/authority/src/main/java/dev/knoe/auth/enroll/EnrollmentController.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.enroll; +package dev.knoe.auth.enroll; import jakarta.servlet.http.HttpSession; import org.slf4j.Logger; diff --git a/authority/src/main/java/dev/knoe/authority/enroll/GoogleOAuthService.java b/authority/src/main/java/dev/knoe/auth/enroll/GoogleOAuthService.java similarity index 95% rename from authority/src/main/java/dev/knoe/authority/enroll/GoogleOAuthService.java rename to authority/src/main/java/dev/knoe/auth/enroll/GoogleOAuthService.java index f98560d..4fd6a01 100644 --- a/authority/src/main/java/dev/knoe/authority/enroll/GoogleOAuthService.java +++ b/authority/src/main/java/dev/knoe/auth/enroll/GoogleOAuthService.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.enroll; +package dev.knoe.auth.enroll; import com.google.api.client.googleapis.auth.oauth2.GoogleIdToken; import com.google.api.client.googleapis.auth.oauth2.GoogleIdTokenVerifier; @@ -45,10 +45,6 @@ public class GoogleOAuthService { * @param nonce nonce for id_token replay protection */ public String buildAuthorizationUrl(String state, String nonce) { - return buildAuthorizationUrl(state, nonce, this.redirectUri); - } - - public String buildAuthorizationUrl(String state, String nonce, String redirectUri) { return UriComponentsBuilder.fromHttpUrl(AUTH_ENDPOINT) .queryParam("client_id", clientId) .queryParam("redirect_uri", redirectUri) @@ -71,10 +67,6 @@ public class GoogleOAuthService { * @throws GoogleOAuthException on any error */ public GoogleIdentity exchangeCode(String code) { - return exchangeCode(code, this.redirectUri); - } - - public GoogleIdentity exchangeCode(String code, String redirectUri) { if (clientId == null || clientId.isBlank()) { throw new GoogleOAuthException("Google OAuth2 is not configured (GOOGLE_CLIENT_ID not set)"); } diff --git a/authority/src/main/java/dev/knoe/authority/enroll/InviteService.java b/authority/src/main/java/dev/knoe/auth/enroll/InviteService.java similarity index 99% rename from authority/src/main/java/dev/knoe/authority/enroll/InviteService.java rename to authority/src/main/java/dev/knoe/auth/enroll/InviteService.java index b7ab7ae..956a3b5 100644 --- a/authority/src/main/java/dev/knoe/authority/enroll/InviteService.java +++ b/authority/src/main/java/dev/knoe/auth/enroll/InviteService.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.enroll; +package dev.knoe.auth.enroll; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/authority/src/main/java/dev/knoe/authority/enroll/TotpService.java b/authority/src/main/java/dev/knoe/auth/enroll/TotpService.java similarity index 98% rename from authority/src/main/java/dev/knoe/authority/enroll/TotpService.java rename to authority/src/main/java/dev/knoe/auth/enroll/TotpService.java index a71be95..525243a 100644 --- a/authority/src/main/java/dev/knoe/authority/enroll/TotpService.java +++ b/authority/src/main/java/dev/knoe/auth/enroll/TotpService.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.enroll; +package dev.knoe.auth.enroll; import dev.samstevens.totp.code.CodeGenerator; import dev.samstevens.totp.code.CodeVerifier; diff --git a/authority/src/main/java/dev/knoe/authority/enroll/UserProvisioningService.java b/authority/src/main/java/dev/knoe/auth/enroll/UserProvisioningService.java similarity index 98% rename from authority/src/main/java/dev/knoe/authority/enroll/UserProvisioningService.java rename to authority/src/main/java/dev/knoe/auth/enroll/UserProvisioningService.java index 33f1615..8d82488 100644 --- a/authority/src/main/java/dev/knoe/authority/enroll/UserProvisioningService.java +++ b/authority/src/main/java/dev/knoe/auth/enroll/UserProvisioningService.java @@ -1,6 +1,6 @@ -package dev.knoe.authority.enroll; +package dev.knoe.auth.enroll; -import dev.knoe.authority.kerberos.KadminClient; +import dev.knoe.auth.kerberos.KadminClient; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.jdbc.core.JdbcTemplate; diff --git a/authority/src/main/java/dev/knoe/authority/kerberos/KadminClient.java b/authority/src/main/java/dev/knoe/auth/kerberos/KadminClient.java similarity index 99% rename from authority/src/main/java/dev/knoe/authority/kerberos/KadminClient.java rename to authority/src/main/java/dev/knoe/auth/kerberos/KadminClient.java index 1005352..5b9fe2f 100644 --- a/authority/src/main/java/dev/knoe/authority/kerberos/KadminClient.java +++ b/authority/src/main/java/dev/knoe/auth/kerberos/KadminClient.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.kerberos; +package dev.knoe.auth.kerberos; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/authority/src/main/java/dev/knoe/authority/kerberos/KerberosPasswordService.java b/authority/src/main/java/dev/knoe/auth/kerberos/KerberosPasswordService.java similarity index 98% rename from authority/src/main/java/dev/knoe/authority/kerberos/KerberosPasswordService.java rename to authority/src/main/java/dev/knoe/auth/kerberos/KerberosPasswordService.java index 99bef60..5018e05 100644 --- a/authority/src/main/java/dev/knoe/authority/kerberos/KerberosPasswordService.java +++ b/authority/src/main/java/dev/knoe/auth/kerberos/KerberosPasswordService.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.kerberos; +package dev.knoe.auth.kerberos; import java.io.IOException; import java.util.Map; diff --git a/authority/src/main/java/dev/knoe/authority/kerberos/KerberosSpnegoService.java b/authority/src/main/java/dev/knoe/auth/kerberos/KerberosSpnegoService.java similarity index 98% rename from authority/src/main/java/dev/knoe/authority/kerberos/KerberosSpnegoService.java rename to authority/src/main/java/dev/knoe/auth/kerberos/KerberosSpnegoService.java index 548f174..ade0b1e 100644 --- a/authority/src/main/java/dev/knoe/authority/kerberos/KerberosSpnegoService.java +++ b/authority/src/main/java/dev/knoe/auth/kerberos/KerberosSpnegoService.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.kerberos; +package dev.knoe.auth.kerberos; import java.security.PrivilegedExceptionAction; import java.util.Map; diff --git a/authority/src/main/java/dev/knoe/authority/provisioning/ProvisioningWorker.java b/authority/src/main/java/dev/knoe/auth/provisioning/ProvisioningWorker.java similarity index 98% rename from authority/src/main/java/dev/knoe/authority/provisioning/ProvisioningWorker.java rename to authority/src/main/java/dev/knoe/auth/provisioning/ProvisioningWorker.java index efe5cf1..800d787 100644 --- a/authority/src/main/java/dev/knoe/authority/provisioning/ProvisioningWorker.java +++ b/authority/src/main/java/dev/knoe/auth/provisioning/ProvisioningWorker.java @@ -1,6 +1,6 @@ -package dev.knoe.authority.provisioning; +package dev.knoe.auth.provisioning; -import dev.knoe.authority.admin.KnobjectService; +import dev.knoe.auth.admin.KnobjectService; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Value; diff --git a/authority/src/main/java/dev/knoe/authority/session/OidcCodeService.java b/authority/src/main/java/dev/knoe/auth/session/OidcCodeService.java similarity index 95% rename from authority/src/main/java/dev/knoe/authority/session/OidcCodeService.java rename to authority/src/main/java/dev/knoe/auth/session/OidcCodeService.java index bed5d43..75184ad 100644 --- a/authority/src/main/java/dev/knoe/authority/session/OidcCodeService.java +++ b/authority/src/main/java/dev/knoe/auth/session/OidcCodeService.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.session; +package dev.knoe.auth.session; import java.util.concurrent.ConcurrentHashMap; import java.util.UUID; diff --git a/authority/src/main/java/dev/knoe/authority/session/OidcTokenService.java b/authority/src/main/java/dev/knoe/auth/session/OidcTokenService.java similarity index 96% rename from authority/src/main/java/dev/knoe/authority/session/OidcTokenService.java rename to authority/src/main/java/dev/knoe/auth/session/OidcTokenService.java index d65622b..8fa8de4 100644 --- a/authority/src/main/java/dev/knoe/authority/session/OidcTokenService.java +++ b/authority/src/main/java/dev/knoe/auth/session/OidcTokenService.java @@ -1,10 +1,10 @@ -package dev.knoe.authority.session; +package dev.knoe.auth.session; import com.fasterxml.jackson.databind.ObjectMapper; import io.jsonwebtoken.Jwts; import io.jsonwebtoken.SignatureAlgorithm; -import dev.knoe.authority.config.AuthProperties; -import dev.knoe.authority.config.KerberosProperties; +import dev.knoe.auth.config.AuthProperties; +import dev.knoe.auth.config.KerberosProperties; import org.springframework.stereotype.Service; import jakarta.annotation.PostConstruct; diff --git a/authority/src/main/java/dev/knoe/authority/session/SessionService.java b/authority/src/main/java/dev/knoe/auth/session/SessionService.java similarity index 91% rename from authority/src/main/java/dev/knoe/authority/session/SessionService.java rename to authority/src/main/java/dev/knoe/auth/session/SessionService.java index c438a7c..07162c2 100644 --- a/authority/src/main/java/dev/knoe/authority/session/SessionService.java +++ b/authority/src/main/java/dev/knoe/auth/session/SessionService.java @@ -1,6 +1,6 @@ -package dev.knoe.authority.session; +package dev.knoe.auth.session; -import dev.knoe.authority.config.AuthProperties; +import dev.knoe.auth.config.AuthProperties; import org.springframework.stereotype.Service; import org.springframework.web.util.WebUtils; diff --git a/authority/src/main/java/dev/knoe/authority/session/SessionTokenService.java b/authority/src/main/java/dev/knoe/auth/session/SessionTokenService.java similarity index 99% rename from authority/src/main/java/dev/knoe/authority/session/SessionTokenService.java rename to authority/src/main/java/dev/knoe/auth/session/SessionTokenService.java index 53632e6..69fa466 100644 --- a/authority/src/main/java/dev/knoe/authority/session/SessionTokenService.java +++ b/authority/src/main/java/dev/knoe/auth/session/SessionTokenService.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.session; +package dev.knoe.auth.session; import com.fasterxml.jackson.annotation.JsonIgnoreProperties; import com.fasterxml.jackson.databind.ObjectMapper; diff --git a/authority/src/main/java/dev/knoe/authority/session/SessionUser.java b/authority/src/main/java/dev/knoe/auth/session/SessionUser.java similarity index 74% rename from authority/src/main/java/dev/knoe/authority/session/SessionUser.java rename to authority/src/main/java/dev/knoe/auth/session/SessionUser.java index 7ab6d25..ce23c1a 100644 --- a/authority/src/main/java/dev/knoe/authority/session/SessionUser.java +++ b/authority/src/main/java/dev/knoe/auth/session/SessionUser.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.session; +package dev.knoe.auth.session; import java.util.List; diff --git a/authority/src/main/java/dev/knoe/authority/user/PrincipalNormalizer.java b/authority/src/main/java/dev/knoe/auth/user/PrincipalNormalizer.java similarity index 95% rename from authority/src/main/java/dev/knoe/authority/user/PrincipalNormalizer.java rename to authority/src/main/java/dev/knoe/auth/user/PrincipalNormalizer.java index ab2ef6e..953b67d 100644 --- a/authority/src/main/java/dev/knoe/authority/user/PrincipalNormalizer.java +++ b/authority/src/main/java/dev/knoe/auth/user/PrincipalNormalizer.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.user; +package dev.knoe.auth.user; import java.util.Locale; import java.util.Optional; diff --git a/authority/src/main/java/dev/knoe/authority/web/JwksController.java b/authority/src/main/java/dev/knoe/auth/web/JwksController.java similarity index 86% rename from authority/src/main/java/dev/knoe/authority/web/JwksController.java rename to authority/src/main/java/dev/knoe/auth/web/JwksController.java index 1e5db80..afa0f77 100644 --- a/authority/src/main/java/dev/knoe/authority/web/JwksController.java +++ b/authority/src/main/java/dev/knoe/auth/web/JwksController.java @@ -1,7 +1,7 @@ -package dev.knoe.authority.web; +package dev.knoe.auth.web; -import dev.knoe.authority.config.AuthProperties; -import dev.knoe.authority.session.OidcTokenService; +import dev.knoe.auth.config.AuthProperties; +import dev.knoe.auth.session.OidcTokenService; import org.springframework.http.ResponseEntity; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RestController; diff --git a/authority/src/main/java/dev/knoe/authority/web/LoginController.java b/authority/src/main/java/dev/knoe/auth/web/LoginController.java similarity index 77% rename from authority/src/main/java/dev/knoe/authority/web/LoginController.java rename to authority/src/main/java/dev/knoe/auth/web/LoginController.java index 1b8c1e6..4d8ad62 100644 --- a/authority/src/main/java/dev/knoe/authority/web/LoginController.java +++ b/authority/src/main/java/dev/knoe/auth/web/LoginController.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.web; +package dev.knoe.auth.web; import java.net.URI; import java.util.ArrayList; @@ -10,14 +10,13 @@ import jakarta.annotation.PostConstruct; import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletResponse; -import dev.knoe.authority.config.AuthProperties; -import dev.knoe.authority.config.KerberosProperties; -import dev.knoe.authority.enroll.GoogleOAuthService; -import dev.knoe.authority.kerberos.KerberosPasswordService; -import dev.knoe.authority.kerberos.KerberosSpnegoService; -import dev.knoe.authority.session.SessionTokenService; -import dev.knoe.authority.session.SessionUser; -import dev.knoe.authority.user.PrincipalNormalizer; +import dev.knoe.auth.config.AuthProperties; +import dev.knoe.auth.config.KerberosProperties; +import dev.knoe.auth.kerberos.KerberosPasswordService; +import dev.knoe.auth.kerberos.KerberosSpnegoService; +import dev.knoe.auth.session.SessionTokenService; +import dev.knoe.auth.session.SessionUser; +import dev.knoe.auth.user.PrincipalNormalizer; import org.springframework.http.HttpHeaders; import org.springframework.http.HttpStatus; import org.springframework.http.MediaType; @@ -38,7 +37,6 @@ public class LoginController { private final KerberosPasswordService passwordAuth; private final PrincipalNormalizer normalizer; private final SessionTokenService sessionTokenService; - private final GoogleOAuthService googleOAuth; public LoginController( AuthProperties auth, @@ -46,8 +44,7 @@ public class LoginController { KerberosSpnegoService spnego, KerberosPasswordService passwordAuth, PrincipalNormalizer normalizer, - SessionTokenService sessionTokenService, - GoogleOAuthService googleOAuth + SessionTokenService sessionTokenService ) { this.auth = auth; this.kerberos = kerberos; @@ -55,7 +52,6 @@ public class LoginController { this.passwordAuth = passwordAuth; this.normalizer = normalizer; this.sessionTokenService = sessionTokenService; - this.googleOAuth = googleOAuth; } @PostConstruct @@ -83,7 +79,7 @@ public class LoginController { code { background: #f3f3f3; padding: 2px 4px; } .box { border: 1px solid #ddd; border-radius: 8px; padding: 16px; margin: 16px 0; } .row { margin: 8px 0; } - input { width: 100%%; padding: 8px; } + input { width: 100%; padding: 8px; } button { padding: 10px 14px; } @@ -95,12 +91,6 @@ public class LoginController {

-
-

Google Workspace

-

Sign in with your @knoey.com account.

-

-
-

Fallback form

Only available when enabled by configuration.

@@ -113,7 +103,7 @@ public class LoginController {
- """.formatted(escapeHtmlAttr(safeNext), escapeHtmlAttr(safeNext), escapeHtmlAttr(safeNext)); + """.formatted(escapeHtmlAttr(safeNext), escapeHtmlAttr(safeNext)); return ResponseEntity.ok(html); } @@ -159,57 +149,6 @@ public class LoginController { return builder.location(safeNext(next)).build(); } - @GetMapping("/login/google") - public ResponseEntity googleLogin( - @RequestParam(name = "next", required = false) String next, - jakarta.servlet.http.HttpSession session) { - - String state = java.util.UUID.randomUUID().toString(); - String nonce = java.util.UUID.randomUUID().toString(); - session.setAttribute("login.googleState", state); - session.setAttribute("login.googleNonce", nonce); - session.setAttribute("login.next", next); - - String redirectUri = auth.getOidc().getIssuer() + "/callback/google"; - String authUrl = googleOAuth.buildAuthorizationUrl(state, nonce, redirectUri); - - return ResponseEntity.status(HttpStatus.FOUND) - .location(URI.create(authUrl)) - .build(); - } - - @GetMapping("/callback/google") - public ResponseEntity googleCallback( - @RequestParam String code, - @RequestParam String state, - jakarta.servlet.http.HttpSession session, - HttpServletResponse response) { - - String expectedState = (String) session.getAttribute("login.googleState"); - String next = (String) session.getAttribute("login.next"); - - if (expectedState == null || !expectedState.equals(state)) { - return ResponseEntity.status(HttpStatus.BAD_REQUEST).build(); - } - - String redirectUri = auth.getOidc().getIssuer() + "/callback/google"; - var googleId = googleOAuth.exchangeCode(code, redirectUri); - - String email = googleId.email(); - String username = normalizer.normalizeUsernameFromKerberosPrincipal(email) - .orElseThrow(() -> new IllegalStateException("Failed to normalize Google email")); - - SessionUser user = new SessionUser(username, email, resolveGroups(username)); - setSessionCookie(response, user); - - session.removeAttribute("login.googleState"); - session.removeAttribute("login.next"); - - return ResponseEntity.status(HttpStatus.FOUND) - .location(safeNext(next)) - .build(); - } - @PostMapping(value = "/form", consumes = MediaType.APPLICATION_FORM_URLENCODED_VALUE) public ResponseEntity formLogin( @RequestParam MultiValueMap form, diff --git a/authority/src/main/java/dev/knoe/authority/web/OidcAuthorizeController.java b/authority/src/main/java/dev/knoe/auth/web/OidcAuthorizeController.java similarity index 92% rename from authority/src/main/java/dev/knoe/authority/web/OidcAuthorizeController.java rename to authority/src/main/java/dev/knoe/auth/web/OidcAuthorizeController.java index 330008f..155815c 100644 --- a/authority/src/main/java/dev/knoe/authority/web/OidcAuthorizeController.java +++ b/authority/src/main/java/dev/knoe/auth/web/OidcAuthorizeController.java @@ -1,9 +1,9 @@ -package dev.knoe.authority.web; +package dev.knoe.auth.web; -import dev.knoe.authority.config.AuthProperties; -import dev.knoe.authority.session.OidcCodeService; -import dev.knoe.authority.session.SessionService; -import dev.knoe.authority.session.SessionUser; +import dev.knoe.auth.config.AuthProperties; +import dev.knoe.auth.session.OidcCodeService; +import dev.knoe.auth.session.SessionService; +import dev.knoe.auth.session.SessionUser; import org.springframework.stereotype.Controller; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RequestParam; diff --git a/authority/src/main/java/dev/knoe/authority/web/OidcDiscoveryController.java b/authority/src/main/java/dev/knoe/auth/web/OidcDiscoveryController.java similarity index 95% rename from authority/src/main/java/dev/knoe/authority/web/OidcDiscoveryController.java rename to authority/src/main/java/dev/knoe/auth/web/OidcDiscoveryController.java index 6ceecfd..73e6eea 100644 --- a/authority/src/main/java/dev/knoe/authority/web/OidcDiscoveryController.java +++ b/authority/src/main/java/dev/knoe/auth/web/OidcDiscoveryController.java @@ -1,6 +1,6 @@ -package dev.knoe.authority.web; +package dev.knoe.auth.web; -import dev.knoe.authority.config.AuthProperties; +import dev.knoe.auth.config.AuthProperties; import org.springframework.http.ResponseEntity; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RestController; diff --git a/authority/src/main/java/dev/knoe/authority/web/OidcTokenController.java b/authority/src/main/java/dev/knoe/auth/web/OidcTokenController.java similarity index 94% rename from authority/src/main/java/dev/knoe/authority/web/OidcTokenController.java rename to authority/src/main/java/dev/knoe/auth/web/OidcTokenController.java index 1fdf746..1f0e4a6 100644 --- a/authority/src/main/java/dev/knoe/authority/web/OidcTokenController.java +++ b/authority/src/main/java/dev/knoe/auth/web/OidcTokenController.java @@ -1,8 +1,8 @@ -package dev.knoe.authority.web; +package dev.knoe.auth.web; -import dev.knoe.authority.config.AuthProperties; -import dev.knoe.authority.session.OidcCodeService; -import dev.knoe.authority.session.OidcTokenService; +import dev.knoe.auth.config.AuthProperties; +import dev.knoe.auth.session.OidcCodeService; +import dev.knoe.auth.session.OidcTokenService; import org.springframework.http.ResponseEntity; import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.RequestParam; diff --git a/authority/src/main/java/dev/knoe/authority/web/OidcUserInfoController.java b/authority/src/main/java/dev/knoe/auth/web/OidcUserInfoController.java similarity index 89% rename from authority/src/main/java/dev/knoe/authority/web/OidcUserInfoController.java rename to authority/src/main/java/dev/knoe/auth/web/OidcUserInfoController.java index 33976f7..853b6a9 100644 --- a/authority/src/main/java/dev/knoe/authority/web/OidcUserInfoController.java +++ b/authority/src/main/java/dev/knoe/auth/web/OidcUserInfoController.java @@ -1,8 +1,8 @@ -package dev.knoe.authority.web; +package dev.knoe.auth.web; -import dev.knoe.authority.config.AuthProperties; -import dev.knoe.authority.session.SessionService; -import dev.knoe.authority.session.SessionUser; +import dev.knoe.auth.config.AuthProperties; +import dev.knoe.auth.session.SessionService; +import dev.knoe.auth.session.SessionUser; import org.springframework.http.ResponseEntity; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RestController; diff --git a/authority/src/main/java/dev/knoe/authority/web/VerifyController.java b/authority/src/main/java/dev/knoe/auth/web/VerifyController.java similarity index 94% rename from authority/src/main/java/dev/knoe/authority/web/VerifyController.java rename to authority/src/main/java/dev/knoe/auth/web/VerifyController.java index c9ed7e0..367493f 100644 --- a/authority/src/main/java/dev/knoe/authority/web/VerifyController.java +++ b/authority/src/main/java/dev/knoe/auth/web/VerifyController.java @@ -1,13 +1,13 @@ -package dev.knoe.authority.web; +package dev.knoe.auth.web; import jakarta.servlet.http.Cookie; import jakarta.servlet.http.HttpServletRequest; import java.util.Arrays; import java.util.Optional; -import dev.knoe.authority.config.AuthProperties; -import dev.knoe.authority.session.SessionTokenService; -import dev.knoe.authority.session.SessionUser; +import dev.knoe.auth.config.AuthProperties; +import dev.knoe.auth.session.SessionTokenService; +import dev.knoe.auth.session.SessionUser; import org.springframework.http.HttpHeaders; import org.springframework.http.HttpStatus; import org.springframework.http.ResponseEntity; diff --git a/authority/src/test/java/dev/knoe/authority/HealthControllerTest.java b/authority/src/test/java/dev/knoe/auth/HealthControllerTest.java similarity index 91% rename from authority/src/test/java/dev/knoe/authority/HealthControllerTest.java rename to authority/src/test/java/dev/knoe/auth/HealthControllerTest.java index 07b9885..68d9738 100644 --- a/authority/src/test/java/dev/knoe/authority/HealthControllerTest.java +++ b/authority/src/test/java/dev/knoe/auth/HealthControllerTest.java @@ -1,4 +1,4 @@ -package dev.knoe.authority; +package dev.knoe.auth; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; diff --git a/authority/src/test/java/dev/knoe/authority/admin/AdminControllerTest.java b/authority/src/test/java/dev/knoe/auth/admin/AdminControllerTest.java similarity index 98% rename from authority/src/test/java/dev/knoe/authority/admin/AdminControllerTest.java rename to authority/src/test/java/dev/knoe/auth/admin/AdminControllerTest.java index 75c8d45..ebc8f5f 100644 --- a/authority/src/test/java/dev/knoe/authority/admin/AdminControllerTest.java +++ b/authority/src/test/java/dev/knoe/auth/admin/AdminControllerTest.java @@ -1,6 +1,6 @@ -package dev.knoe.authority.admin; +package dev.knoe.auth.admin; -import dev.knoe.authority.enroll.InviteService; +import dev.knoe.auth.enroll.InviteService; import org.junit.jupiter.api.Test; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc; diff --git a/authority/src/test/java/dev/knoe/authority/enroll/EnrollValueTypesTest.java b/authority/src/test/java/dev/knoe/auth/enroll/EnrollValueTypesTest.java similarity index 98% rename from authority/src/test/java/dev/knoe/authority/enroll/EnrollValueTypesTest.java rename to authority/src/test/java/dev/knoe/auth/enroll/EnrollValueTypesTest.java index 94891d1..295f937 100644 --- a/authority/src/test/java/dev/knoe/authority/enroll/EnrollValueTypesTest.java +++ b/authority/src/test/java/dev/knoe/auth/enroll/EnrollValueTypesTest.java @@ -1,6 +1,6 @@ -package dev.knoe.authority.enroll; +package dev.knoe.auth.enroll; -import dev.knoe.authority.admin.AdminController; +import dev.knoe.auth.admin.AdminController; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; diff --git a/authority/src/test/java/dev/knoe/authority/enroll/EnrollmentControllerTest.java b/authority/src/test/java/dev/knoe/auth/enroll/EnrollmentControllerTest.java similarity index 99% rename from authority/src/test/java/dev/knoe/authority/enroll/EnrollmentControllerTest.java rename to authority/src/test/java/dev/knoe/auth/enroll/EnrollmentControllerTest.java index fcf0faf..4e56d63 100644 --- a/authority/src/test/java/dev/knoe/authority/enroll/EnrollmentControllerTest.java +++ b/authority/src/test/java/dev/knoe/auth/enroll/EnrollmentControllerTest.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.enroll; +package dev.knoe.auth.enroll; import org.junit.jupiter.api.Test; import org.springframework.beans.factory.annotation.Autowired; diff --git a/authority/src/test/java/dev/knoe/authority/enroll/TotpServiceTest.java b/authority/src/test/java/dev/knoe/auth/enroll/TotpServiceTest.java similarity index 98% rename from authority/src/test/java/dev/knoe/authority/enroll/TotpServiceTest.java rename to authority/src/test/java/dev/knoe/auth/enroll/TotpServiceTest.java index 4710d40..11a1d2a 100644 --- a/authority/src/test/java/dev/knoe/authority/enroll/TotpServiceTest.java +++ b/authority/src/test/java/dev/knoe/auth/enroll/TotpServiceTest.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.enroll; +package dev.knoe.auth.enroll; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; diff --git a/authority/src/test/java/dev/knoe/authority/kerberos/KadminClientTest.java b/authority/src/test/java/dev/knoe/auth/kerberos/KadminClientTest.java similarity index 98% rename from authority/src/test/java/dev/knoe/authority/kerberos/KadminClientTest.java rename to authority/src/test/java/dev/knoe/auth/kerberos/KadminClientTest.java index 49116bb..22389ac 100644 --- a/authority/src/test/java/dev/knoe/authority/kerberos/KadminClientTest.java +++ b/authority/src/test/java/dev/knoe/auth/kerberos/KadminClientTest.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.kerberos; +package dev.knoe.auth.kerberos; import org.junit.jupiter.api.Test; import java.lang.reflect.Method; diff --git a/authority/src/test/java/dev/knoe/authority/kerberos/KerberosSpnegoResultTest.java b/authority/src/test/java/dev/knoe/auth/kerberos/KerberosSpnegoResultTest.java similarity index 95% rename from authority/src/test/java/dev/knoe/authority/kerberos/KerberosSpnegoResultTest.java rename to authority/src/test/java/dev/knoe/auth/kerberos/KerberosSpnegoResultTest.java index e9e3b3e..3ba0db2 100644 --- a/authority/src/test/java/dev/knoe/authority/kerberos/KerberosSpnegoResultTest.java +++ b/authority/src/test/java/dev/knoe/auth/kerberos/KerberosSpnegoResultTest.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.kerberos; +package dev.knoe.auth.kerberos; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; diff --git a/authority/src/test/java/dev/knoe/authority/regression/IdentityRegressionTest.java b/authority/src/test/java/dev/knoe/auth/regression/IdentityRegressionTest.java similarity index 98% rename from authority/src/test/java/dev/knoe/authority/regression/IdentityRegressionTest.java rename to authority/src/test/java/dev/knoe/auth/regression/IdentityRegressionTest.java index 00623a8..9799dc1 100644 --- a/authority/src/test/java/dev/knoe/authority/regression/IdentityRegressionTest.java +++ b/authority/src/test/java/dev/knoe/auth/regression/IdentityRegressionTest.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.regression; +package dev.knoe.auth.regression; import org.junit.jupiter.api.Test; import java.io.File; diff --git a/authority/src/test/java/dev/knoe/authority/session/OidcCodeServiceTest.java b/authority/src/test/java/dev/knoe/auth/session/OidcCodeServiceTest.java similarity index 98% rename from authority/src/test/java/dev/knoe/authority/session/OidcCodeServiceTest.java rename to authority/src/test/java/dev/knoe/auth/session/OidcCodeServiceTest.java index 294883f..5bc4142 100644 --- a/authority/src/test/java/dev/knoe/authority/session/OidcCodeServiceTest.java +++ b/authority/src/test/java/dev/knoe/auth/session/OidcCodeServiceTest.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.session; +package dev.knoe.auth.session; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; diff --git a/authority/src/test/java/dev/knoe/authority/session/SessionServiceTest.java b/authority/src/test/java/dev/knoe/auth/session/SessionServiceTest.java similarity index 96% rename from authority/src/test/java/dev/knoe/authority/session/SessionServiceTest.java rename to authority/src/test/java/dev/knoe/auth/session/SessionServiceTest.java index f2714a9..9d01c31 100644 --- a/authority/src/test/java/dev/knoe/authority/session/SessionServiceTest.java +++ b/authority/src/test/java/dev/knoe/auth/session/SessionServiceTest.java @@ -1,7 +1,7 @@ -package dev.knoe.authority.session; +package dev.knoe.auth.session; import com.fasterxml.jackson.databind.ObjectMapper; -import dev.knoe.authority.config.AuthProperties; +import dev.knoe.auth.config.AuthProperties; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.springframework.mock.web.MockHttpServletRequest; diff --git a/authority/src/test/java/dev/knoe/authority/session/SessionTokenServiceTest.java b/authority/src/test/java/dev/knoe/auth/session/SessionTokenServiceTest.java similarity index 98% rename from authority/src/test/java/dev/knoe/authority/session/SessionTokenServiceTest.java rename to authority/src/test/java/dev/knoe/auth/session/SessionTokenServiceTest.java index ae28bf3..8631f65 100644 --- a/authority/src/test/java/dev/knoe/authority/session/SessionTokenServiceTest.java +++ b/authority/src/test/java/dev/knoe/auth/session/SessionTokenServiceTest.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.session; +package dev.knoe.auth.session; import com.fasterxml.jackson.databind.ObjectMapper; import java.time.Clock; diff --git a/authority/src/test/java/dev/knoe/authority/user/PrincipalNormalizerTest.java b/authority/src/test/java/dev/knoe/auth/user/PrincipalNormalizerTest.java similarity index 98% rename from authority/src/test/java/dev/knoe/authority/user/PrincipalNormalizerTest.java rename to authority/src/test/java/dev/knoe/auth/user/PrincipalNormalizerTest.java index 0dbe480..12949d7 100644 --- a/authority/src/test/java/dev/knoe/authority/user/PrincipalNormalizerTest.java +++ b/authority/src/test/java/dev/knoe/auth/user/PrincipalNormalizerTest.java @@ -1,4 +1,4 @@ -package dev.knoe.authority.user; +package dev.knoe.auth.user; import org.junit.jupiter.api.Test; import java.util.Optional; diff --git a/authority/src/test/java/dev/knoe/authority/web/LoginControllerTest.java b/authority/src/test/java/dev/knoe/auth/web/LoginControllerTest.java similarity index 97% rename from authority/src/test/java/dev/knoe/authority/web/LoginControllerTest.java rename to authority/src/test/java/dev/knoe/auth/web/LoginControllerTest.java index 22fd735..e56e88b 100644 --- a/authority/src/test/java/dev/knoe/authority/web/LoginControllerTest.java +++ b/authority/src/test/java/dev/knoe/auth/web/LoginControllerTest.java @@ -1,8 +1,8 @@ -package dev.knoe.authority.web; +package dev.knoe.auth.web; -import dev.knoe.authority.enroll.GoogleOAuthService; -import dev.knoe.authority.kerberos.KerberosPasswordService; -import dev.knoe.authority.kerberos.KerberosSpnegoService; +import dev.knoe.auth.enroll.GoogleOAuthService; +import dev.knoe.auth.kerberos.KerberosPasswordService; +import dev.knoe.auth.kerberos.KerberosSpnegoService; import org.junit.jupiter.api.Test; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc; diff --git a/authority/src/test/java/dev/knoe/authority/web/VerifyControllerTest.java b/authority/src/test/java/dev/knoe/auth/web/VerifyControllerTest.java similarity index 91% rename from authority/src/test/java/dev/knoe/authority/web/VerifyControllerTest.java rename to authority/src/test/java/dev/knoe/auth/web/VerifyControllerTest.java index 6ef9414..1300114 100644 --- a/authority/src/test/java/dev/knoe/authority/web/VerifyControllerTest.java +++ b/authority/src/test/java/dev/knoe/auth/web/VerifyControllerTest.java @@ -1,9 +1,9 @@ -package dev.knoe.authority.web; +package dev.knoe.auth.web; import java.time.Duration; import org.junit.jupiter.api.Test; -import dev.knoe.authority.session.SessionTokenService; -import dev.knoe.authority.session.SessionUser; +import dev.knoe.auth.session.SessionTokenService; +import dev.knoe.auth.session.SessionUser; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc; import org.springframework.boot.test.context.SpringBootTest; diff --git a/deploy/gcp/gke/cnpg-prometheus-ilb.yaml b/deploy/gcp/gke/cnpg-prometheus-ilb.yaml new file mode 100644 index 0000000..7311959 --- /dev/null +++ b/deploy/gcp/gke/cnpg-prometheus-ilb.yaml @@ -0,0 +1,40 @@ +# Internal LoadBalancer exposing the DB-cluster Prometheus to the app cluster. +# +# Both knoe-dev-0 (app) and knoe-dev-cnpg-0 (DB) sit on the same VPC subnet +# (us-west3 default). An internal-LB IP from that subnet is reachable from +# pods in the app cluster, so the app-cluster Grafana can use this URL as a +# datasource without the metric stream ever leaving Google's network. +# +# Why an ILB instead of cluster-local DNS: cluster-local DNS (.svc.cluster.local) +# is NOT cross-cluster. The two GKE clusters have separate kube-dns; the app +# cluster can't resolve services in the DB cluster's DNS namespace. The ILB +# gives a stable VPC-routable IP that crosses the cluster boundary. +# +# Apply (DB cluster): +# kubectl --context=$DB_CTX apply -f deploy/gcp/gke/cnpg-prometheus-ilb.yaml +--- +apiVersion: v1 +kind: Service +metadata: + name: prometheus-cnpg-ilb + namespace: monitoring + labels: + app.kubernetes.io/name: prometheus + app.kubernetes.io/instance: kps-kube-prometheus-stack-prometheus + app.kubernetes.io/managed-by: knoe-installer + annotations: + # GKE internal LB (RFC1918 IP from the VPC default subnet). + networking.gke.io/load-balancer-type: "Internal" + # Don't expand allow-list to other VPCs/regions; same-region/same-VPC only. + networking.gke.io/internal-load-balancer-allow-global-access: "false" +spec: + type: LoadBalancer + externalTrafficPolicy: Cluster # OK for an ILB; source-IP preservation not needed + selector: + app.kubernetes.io/name: prometheus + app.kubernetes.io/instance: kps-kube-prometheus-stack-prometheus + ports: + - name: http + port: 9090 + targetPort: 9090 + protocol: TCP diff --git a/deploy/gcp/gke/grafana-google-oidc-secret.example.yaml b/deploy/gcp/gke/grafana-google-oidc-secret.example.yaml new file mode 100644 index 0000000..7157515 --- /dev/null +++ b/deploy/gcp/gke/grafana-google-oidc-secret.example.yaml @@ -0,0 +1,30 @@ +apiVersion: v1 +kind: Secret +metadata: + name: grafana-google-oidc + namespace: monitoring + labels: + app: grafana + app.kubernetes.io/managed-by: knoe-installer +# Google OAuth 2.0 client credentials for Grafana's native Google sign-in +# (Grafana auth.google plugin, NOT oauth2-proxy). Mounted into kps-grafana +# via the chart's `envFromSecret: grafana-google-oidc` value. Grafana reads +# `GF_AUTH_GOOGLE_CLIENT_ID` / `GF_AUTH_GOOGLE_CLIENT_SECRET` env vars and +# uses them as `auth.google.client_id` / `client_secret`. +# +# Create the OAuth client at: https://console.cloud.google.com/apis/credentials +# - Application type: Web application +# - Name: knoe.dev Grafana +# - Authorized JavaScript origins: https://svc.knoe.dev +# - Authorized redirect URIs: https://svc.knoe.dev/grafana/login/google +# - OAuth consent screen: Internal (knoey.com Workspace) +# - Scopes: openid, email, profile (default for grafana auth.google) +# +# Why a separate client (not reusing knoe.dev Studio's): independent rotation +# surface — if Grafana access is compromised, rotating one client doesn't +# disrupt Studio sign-in (and vice versa). Same convention as gitlab-google-oidc +# and oauth2-proxy-google-oidc. +type: Opaque +stringData: + GF_AUTH_GOOGLE_CLIENT_ID: "${GRAFANA_GOOGLE_CLIENT_ID}" + GF_AUTH_GOOGLE_CLIENT_SECRET: "${GRAFANA_GOOGLE_CLIENT_SECRET}" diff --git a/deploy/gcp/gke/knoe-auth-deployment.yaml b/deploy/gcp/gke/knoe-auth-deployment.yaml index 7cc55ba..7b2d765 100644 --- a/deploy/gcp/gke/knoe-auth-deployment.yaml +++ b/deploy/gcp/gke/knoe-auth-deployment.yaml @@ -1,7 +1,7 @@ --- # knoe-auth-deployment.yaml # Deploys the knoe-auth pod (Spring Boot + KDC sidecar) to knoe-dev-0 / knoe-system. -# Based on deploy/opentofu/k3s/manifests/knoe/prole-auth-deployment.yaml. +# Based on deploy/opentofu/k3s/manifests/knoe/knoe-auth-deployment.yaml. # Realm: KNOE.DEV (not PROLE.LOCAL) # Image: pulled from Artifact Registry — built by knoe-auth CI pipeline. # diff --git a/deploy/gcp/gke/knoe-db-external-lb.yaml b/deploy/gcp/gke/knoe-db-external-lb.yaml new file mode 100644 index 0000000..d40542c --- /dev/null +++ b/deploy/gcp/gke/knoe-db-external-lb.yaml @@ -0,0 +1,58 @@ +# External TCP LoadBalancer for direct psql access to the CNPG primary. +# +# Hostname: pg.0.knoe.dev (DNS A record pointed manually at Namecheap to the +# IP allocated by this Service). +# +# Auth model: +# - Phase 1 (current): per-engineer postgres roles with strong passwords + +# VALID UNTIL 30d. SCRAM-SHA-256 over TLS (hostssl in pg_hba). +# - Phase 2 (queued): PostgreSQL 18 OAUTHBEARER with a custom validator +# library (knoe_oauth.so) that maps Google email -> postgres role. +# Connection target stays the same; only the credential issuance changes. +# +# TLS: CNPG issues the server cert and auto-rotates it. The cluster spec's +# `certificates.serverAltDnsNames` is extended with `pg.0.knoe.dev` so the +# auto-issued cert validates externally. Engineers download the CNPG-issued +# CA cert (kubectl get secret knoe-db-ca -n knoe-db-0 -o jsonpath='{.data.ca\.crt}' | base64 -d) +# and use `sslmode=verify-full sslrootcert=`. +# +# Source-IP allowlist: open today (loadBalancerSourceRanges: 0.0.0.0/0). Tighten +# to known engineer/CI IP ranges in a follow-up; SCRAM-SHA-256 is brute-force- +# resistant so the open window is acceptable for the Phase 1 horizon (~30d). +# +# Phase 1 verification (after DNS A record is up): +# psql "host=pg.0.knoe.dev port=5432 user=chrisfu dbname=postgres sslmode=verify-full sslrootcert=$HOME/.knoe/knoe-db-ca.crt" +--- +apiVersion: v1 +kind: Service +metadata: + name: knoe-db-rw-elb + namespace: knoe-db-0 + labels: + app.kubernetes.io/name: knoe-db + app.kubernetes.io/component: postgres-external + app.kubernetes.io/managed-by: knoe-installer + annotations: + # GKE L4 Regional Backend Service (newer LB type; required for SCTP and + # for some traffic-policy combinations; allocates faster than the legacy + # network LB on this cluster). + cloud.google.com/l4-rbs: "enabled" +spec: + type: LoadBalancer + # `Local` preserves the client's real source IP at the postgres backend + # (vs `Cluster` which SNATs to the receiving node IP, making external + # clients look like they live in the cluster's 10.0.0.0/8 range and bypass + # the `hostnossl reject` rule in pg_hba). + externalTrafficPolicy: Local + loadBalancerSourceRanges: + # Phase 1: open. Tighten in a follow-up once we have an engineer IP allowlist + # or a corp VPN egress range. + - 0.0.0.0/0 + selector: + cnpg.io/cluster: knoe-db + cnpg.io/instanceRole: primary + ports: + - name: psql + port: 5432 + targetPort: 5432 + protocol: TCP diff --git a/deploy/gcp/gke/knoe-db.yaml b/deploy/gcp/gke/knoe-db.yaml index c00eb77..183154e 100644 --- a/deploy/gcp/gke/knoe-db.yaml +++ b/deploy/gcp/gke/knoe-db.yaml @@ -40,12 +40,36 @@ spec: - pg_stat_statements - pg_tde pg_hba: + # Local Unix-socket connections (CNPG default + knoe role) - local all postgres trust - local all knoe scram-sha-256 - - host all postgres all scram-sha-256 - - host knoe knoe-db all scram-sha-256 - - host all all all scram-sha-256 - - hostssl knoe knoe-db all scram-sha-256 + + # postgres / knoe-db / knoe roles: cluster-internal (RFC1918) only. + # Cluster pod CIDRs: db cluster 10.24.0.0/14, app cluster 10.84.0.0/14; + # node subnet 10.180.0.0/16. 10.0.0.0/8 covers all of those. + - host all postgres 10.0.0.0/8 scram-sha-256 + - host knoe knoe-db 10.0.0.0/8 scram-sha-256 + - hostssl knoe knoe-db 10.0.0.0/8 scram-sha-256 + + # PHASE 1 EXTERNAL ACCESS — any member of `knoe_developer`, over TLS+SCRAM. + # `+rolename` in pg_hba matches role membership (not just literal name), + # so `etc/onboard_engineer.sh` adds new engineers via `GRANT knoe_developer + # TO ` without ever editing pg_hba — that's the reusable property. + # Phase 2 (queued for Junie) replaces this with libpq OAUTHBEARER: + # hostssl all all 0.0.0.0/0 oauth issuer=https://accounts.google.com validator=knoe_oauth scope="openid email" + - hostssl all +knoe_developer all scram-sha-256 + + # Internal cluster (RFC1918) — all roles, SCRAM (allows the supabase + # services in app cluster knoe-dev-0 to reach the DB cluster). + - host all all 10.0.0.0/8 scram-sha-256 + - hostssl all all 10.0.0.0/8 scram-sha-256 + + # Block any plaintext from external (TLS required for the public LB) + - hostnossl all all 0.0.0.0/0 reject + + # Catch-all reject for anything not matched above + - host all all 0.0.0.0/0 reject + - hostssl all all 0.0.0.0/0 reject bootstrap: initdb: @@ -56,7 +80,14 @@ spec: secret: name: knoe-db-user postInitTemplateSQL: - - CREATE EXTENSION IF NOT EXISTS pg_stat_statements; + # Supabase convention: relocatable extensions live in `extensions`, not + # `public`. Studio's Database Advisor flags `public.pg_stat_statements` + # as a Security warning the moment a user opens the dashboard. Without + # an explicit SCHEMA clause `CREATE EXTENSION` lands the relocatable + # extension in the first writable schema in the connecting role's + # search_path, which for `postgres` is `public`. + - CREATE SCHEMA IF NOT EXISTS extensions; + - CREATE EXTENSION IF NOT EXISTS pg_stat_statements SCHEMA extensions; postInitSQL: - DO $do$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'knoe') THEN CREATE ROLE knoe LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT; END IF; END $do$; - DO $do$ DECLARE owner_password text; BEGIN SELECT rolpassword INTO owner_password FROM pg_authid WHERE rolname = 'knoe'; IF owner_password IS NOT NULL THEN EXECUTE format('ALTER ROLE knoe PASSWORD %L', owner_password); END IF; END $do$; @@ -119,13 +150,34 @@ spec: resources: requests: cpu: "100m" - memory: "128Mi" + # 512Mi (was 128Mi) — postgres baseline working set is ~290Mi on the + # primary (shared_buffers + wal_buffers + per-backend memory + a small + # OS page cache visible to cgroups), so 128Mi caused the cnpg-grafana + # "Resource Pressure" tile to flag Memory at working_set / request ≈ 2x, + # which the dashboard maps to a red "Data Loss" label (>0.98 ratio). + # Right-sizing to 512Mi puts the steady-state ratio in the green + # "Healthy" zone (<0.8) and gives the scheduler an accurate signal for + # spreading replicas across nodes. Pods still have plenty of headroom: + # 2Gi limit is unchanged. + memory: "512Mi" limits: cpu: "500m" - memory: "512Mi" + # 2Gi (was 512Mi) — barman-cloud-backup is single-threaded gzip + GCS + # upload buffering and the throughput tops out at the memory ceiling. + # 2Gi cuts a 9 GB DB backup from 30–90 min down to 5–10 min. + memory: "2Gi" enableSuperuserAccess: true + # CNPG-issued server cert is auto-rotated by the operator. Listing + # pg.0.knoe.dev as an alt DNS name lets engineers connect with + # `sslmode=verify-full` after fetching the CNPG-issued CA cert from the + # `knoe-db-ca` Secret. Phase 1 of the per-engineer psql access plan; replaced + # by libpq OAUTHBEARER + Let's Encrypt in Phase 2. + certificates: + serverAltDNSNames: + - pg.0.knoe.dev + storage: size: 50Gi pvcTemplate: @@ -147,4 +199,15 @@ spec: storageClassName: premium-rwo # pd-ssd; restore to premium-rwo after quota increase (matches PGDATA above) monitoring: - enablePodMonitor: true # kube-prometheus-stack (Prometheus Operator) is installed before CNPG — PodMonitor CRD is present + # NOTE: enablePodMonitor + podMonitorRelabelings are CNPG-deprecated and + # will be removed in a future release. Migration path: manually manage + # the PodMonitor resource. Tracked in docs/TODO.md as a follow-up. + enablePodMonitor: true # kube-prometheus-stack is installed before CNPG → PodMonitor CRD present + # Add a `cluster` label to every scraped sample, sourced from the + # cnpg.io/cluster pod label. The cnpg-grafana dashboards filter every + # panel by `cluster="$cluster"`, so without this relabel only 2 of 85 + # CNPG metrics (the operator-collector ones) render. + podMonitorRelabelings: + - sourceLabels: ["__meta_kubernetes_pod_label_cnpg_io_cluster"] + targetLabel: cluster + action: replace diff --git a/deploy/gcp/gke/knoe-onboard.yaml b/deploy/gcp/gke/knoe-onboard.yaml new file mode 100644 index 0000000..5b26522 --- /dev/null +++ b/deploy/gcp/gke/knoe-onboard.yaml @@ -0,0 +1,347 @@ +# knoe-onboard: tiny static-content service that renders an engineer's +# psql onboarding details from a URL fragment. +# +# How it fits: +# etc/onboard_engineer.sh +# -> CREATE ROLE + temp password (24h) +# -> Builds URL: https://db.0.knoe.dev/onboard.html#user=...&pw=...&exp=... +# -> Outputs URL + QR-code rendering for chrisfu to email or screenshare +# Engineer clicks URL or scans QR +# -> oauth2-proxy `--skip-auth-route` matches /onboard.* (page itself is +# ungated; URL secrecy + 24h expiry + immediate rotation = the security +# envelope, since URL fragments don't survive an OAuth redirect anyway) +# -> Kong routes /onboard/* to this nginx pod +# -> Browser loads onboard.html; vanilla JS reads window.location.hash, +# decodes the password, displays it with [Copy] buttons + a ready-to-paste +# psql connection string. No backend calls. No state. +# +# Phase 2 (Junie's queue): when libpq OAUTHBEARER lands, the script stops +# generating passwords; the onboard page becomes "you're already authorized +# (Google), here's your connection string with oauth_issuer=...". The +# Deployment + Service + Kong route here stay; only the HTML changes. +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: knoe-onboard-html + namespace: supabase + labels: + app: knoe-onboard + app.kubernetes.io/managed-by: knoe-installer +data: + onboard.html: | + + + + + + + knoe-db onboarding + + + +
+ +
+ + + +--- +apiVersion: v1 +kind: Service +metadata: + name: knoe-onboard + namespace: supabase + labels: + app: knoe-onboard + app.kubernetes.io/managed-by: knoe-installer +spec: + type: ClusterIP + selector: + app: knoe-onboard + ports: + - name: http + port: 80 + targetPort: 80 + protocol: TCP +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: knoe-onboard + namespace: supabase + labels: + app: knoe-onboard + app.kubernetes.io/managed-by: knoe-installer +spec: + replicas: 1 + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 + selector: + matchLabels: + app: knoe-onboard + template: + metadata: + labels: + app: knoe-onboard + spec: + securityContext: + runAsNonRoot: true + runAsUser: 101 # nginx in alpine image + runAsGroup: 101 + seccompProfile: + type: RuntimeDefault + containers: + - name: nginx + image: nginx:1.27-alpine + imagePullPolicy: IfNotPresent + ports: + - name: http + containerPort: 80 + protocol: TCP + volumeMounts: + # ConfigMap mounted directly at the nginx html root so we don't + # need a writable rootfs (readOnlyRootFilesystem: true). + - name: html + mountPath: /usr/share/nginx/html + readOnly: true + - name: nginx-cache + mountPath: /var/cache/nginx + - name: nginx-run + mountPath: /var/run + livenessProbe: + httpGet: + path: /onboard.html + port: http + initialDelaySeconds: 5 + periodSeconds: 10 + timeoutSeconds: 3 + readinessProbe: + httpGet: + path: /onboard.html + port: http + initialDelaySeconds: 2 + periodSeconds: 5 + timeoutSeconds: 3 + resources: + requests: + cpu: 10m + memory: 16Mi + limits: + cpu: 50m + memory: 32Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + readOnlyRootFilesystem: true + volumes: + - name: html + configMap: + name: knoe-onboard-html + items: + # Two paths under the same volume: nginx serves /onboard.html + # directly, and / falls back to index.html (same content) for + # convenience if the trailing .html ever gets dropped. + - key: onboard.html + path: onboard.html + - key: onboard.html + path: index.html + - name: nginx-cache + emptyDir: {} + - name: nginx-run + emptyDir: {} diff --git a/deploy/gcp/gke/oauth2-proxy-deployment.yaml b/deploy/gcp/gke/oauth2-proxy-deployment.yaml new file mode 100644 index 0000000..6a6a0db --- /dev/null +++ b/deploy/gcp/gke/oauth2-proxy-deployment.yaml @@ -0,0 +1,193 @@ +# oauth2-proxy gate in front of Supabase Studio at db.0.knoe.dev. +# +# Sits between the GCE Ingress and supabase-kong; redirects unauthenticated +# requests to Google Workspace, allows any @knoey.com identity through, then +# proxies the request upstream to supabase-kong:8000 (which serves Studio at +# the / route). Outside-domain users are rejected at this layer. +# +# Bootstrap: ./etc/init_oauth2_proxy.sh +# Cluster: knoe-dev-0 (GKE app cluster) +# Namespace: supabase (alongside supabase-* workloads + the existing Ingress) +# +# After this Deployment is Ready, the supabase-kong Ingress is patched to +# route db.0.knoe.dev through oauth2-proxy:80, and the basic-auth plugin on +# the Kong dashboard route is removed (oauth2-proxy is the gate now). +# +# Unified host model (Studio + SDK on the same hostname): +# the `--skip-auth-route` args below let SDK requests on /auth/v1, /rest/v1, +# /realtime/v1, /storage/v1, /functions/v1, /graphql/v1, /pg/* bypass the +# Google sign-in flow and pass straight through to supabase-kong, where +# Kong's `key-auth` plugin enforces the existing anon/service-role API keys. +# The Studio UI at / remains Google-gated (no skip rule). Net effect: one +# URL (db.0.knoe.dev) covers humans-in-Studio AND robots-via-supabase-py, +# with the same security envelope as before — Kong's key-auth is the +# enforcer on API paths regardless of which hostname they came in on. +# +# When knoe-auth Round 1 ships an OIDC OP at https://api.knoe.dev/auth, swap +# the args below from `--provider=google` to `--provider=oidc` + +# `--oidc-issuer-url=https://api.knoe.dev/auth` and reapply — no other +# manifest changes needed. +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: oauth2-proxy + namespace: supabase + labels: + app: oauth2-proxy + app.kubernetes.io/managed-by: knoe-installer +--- +apiVersion: cloud.google.com/v1 +kind: BackendConfig +metadata: + name: oauth2-proxy-backend-config + namespace: supabase + labels: + app: oauth2-proxy + app.kubernetes.io/managed-by: knoe-installer +spec: + healthCheck: + type: HTTP + requestPath: /ping + port: 4180 + timeoutSec: 60 +--- +apiVersion: v1 +kind: Service +metadata: + name: oauth2-proxy + namespace: supabase + labels: + app: oauth2-proxy + app.kubernetes.io/managed-by: knoe-installer + annotations: + cloud.google.com/backend-config: '{"default": "oauth2-proxy-backend-config"}' +spec: + type: ClusterIP + selector: + app: oauth2-proxy + ports: + - name: http + port: 80 + targetPort: 4180 + protocol: TCP +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: oauth2-proxy + namespace: supabase + labels: + app: oauth2-proxy + app.kubernetes.io/managed-by: knoe-installer +spec: + replicas: 1 + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 + selector: + matchLabels: + app: oauth2-proxy + template: + metadata: + labels: + app: oauth2-proxy + spec: + serviceAccountName: oauth2-proxy + securityContext: + runAsNonRoot: true + runAsUser: 65532 + runAsGroup: 65532 + seccompProfile: + type: RuntimeDefault + containers: + - name: oauth2-proxy + image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.2 + imagePullPolicy: IfNotPresent + args: + - --provider=google + - --email-domain=knoey.com + - --upstream=http://supabase-kong:8000 + - --http-address=0.0.0.0:4180 + - --reverse-proxy=true + - --cookie-secure=true + - --cookie-name=_knoe_studio_oauth2 + - --cookie-domain=db.0.knoe.dev + - --whitelist-domain=db.0.knoe.dev + - --redirect-url=https://db.0.knoe.dev/oauth2/callback + - --skip-provider-button=true + - --scope=openid email profile + # NB: do NOT set --pass-authorization-header / --set-authorization-header. + # Those flags rewrite (or strip) the request's Authorization header, + # which clobbers the supabase-py / Studio Authorization: Bearer + # value that the storage and other Supabase services expect to parse + # against JWT_SECRET. When per-user identity passthrough lands (post + # knoe-auth Round 1), use --pass-user-headers / --set-xauthrequest + # instead — those use X-Forwarded-* headers and don't collide with + # Supabase's own auth. + # + # Bypass Google auth for Supabase API surfaces — Kong's key-auth + # is the enforcer on these paths (anon / service-role keys). + - --skip-auth-route=^/(auth|rest|realtime|storage|functions|graphql)/v1/.* + - --skip-auth-route=^/pg/.* + # Bypass Google auth for the onboarding reveal page. URL fragments + # don't survive an OAuth redirect, so we can't gate this with + # Google sign-in; instead, the engineer's URL is single-shot + # delivery (email / QR), the temp password expires in 24h, and + # the page recommends immediate rotation via `\password`. + - --skip-auth-route=^/onboard\.html$ + # Public support endpoint — 302s to mailto:support@knoe.dev. Skipping + # auth so an unauthenticated user looking for help isn't bounced into + # a confusing Google sign-in detour. + - --skip-auth-route=^/support$ + - --request-logging=true + - --auth-logging=true + - --standard-logging=true + env: + - name: OAUTH2_PROXY_CLIENT_ID + valueFrom: + secretKeyRef: + name: oauth2-proxy-google-oidc + key: client_id + - name: OAUTH2_PROXY_CLIENT_SECRET + valueFrom: + secretKeyRef: + name: oauth2-proxy-google-oidc + key: client_secret + - name: OAUTH2_PROXY_COOKIE_SECRET + valueFrom: + secretKeyRef: + name: oauth2-proxy-google-oidc + key: cookie_secret + ports: + - name: http + containerPort: 4180 + protocol: TCP + livenessProbe: + httpGet: + path: /ping + port: http + initialDelaySeconds: 5 + periodSeconds: 10 + timeoutSeconds: 3 + readinessProbe: + httpGet: + path: /ping + port: http + initialDelaySeconds: 5 + periodSeconds: 5 + timeoutSeconds: 3 + resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 100m + memory: 128Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + readOnlyRootFilesystem: true diff --git a/deploy/gcp/gke/oauth2-proxy-google-oidc-secret.example.yaml b/deploy/gcp/gke/oauth2-proxy-google-oidc-secret.example.yaml new file mode 100644 index 0000000..807065a --- /dev/null +++ b/deploy/gcp/gke/oauth2-proxy-google-oidc-secret.example.yaml @@ -0,0 +1,31 @@ +apiVersion: v1 +kind: Secret +metadata: + name: oauth2-proxy-google-oidc + namespace: supabase + labels: + app: oauth2-proxy + app.kubernetes.io/managed-by: knoe-installer +# Google OAuth 2.0 client credentials + cookie secret for the oauth2-proxy +# gating Supabase Studio at db.0.knoe.dev. Applied by etc/init_oauth2_proxy.sh +# via envsubst from etc/secrets/oauth2-proxy-{client-id,client-secret,cookie-secret}. +# +# Create the OAuth client at: https://console.cloud.google.com/apis/credentials +# - Application type: Web application +# - Name: knoe.dev Studio +# - Authorized JavaScript origins: https://db.0.knoe.dev +# - Authorized redirect URIs: https://db.0.knoe.dev/oauth2/callback +# - OAuth consent screen: Internal (knoey.com Workspace) +# - Scopes: openid, email, profile (default for oauth2-proxy) +# +# cookie_secret: must be 32 bytes (24-byte base64-encoded). Generate with: +# openssl rand -base64 32 +# +# Why a separate client (not reusing knoe.dev GitLab): independent rotation +# surface — if Studio access is compromised, rotating one client doesn't +# disrupt the GitLab login flow (and vice versa). +type: Opaque +stringData: + client_id: "${OAUTH2_PROXY_CLIENT_ID}" + client_secret: "${OAUTH2_PROXY_CLIENT_SECRET}" + cookie_secret: "${OAUTH2_PROXY_COOKIE_SECRET}" diff --git a/deploy/opentofu/k3s/manifests/knoe/prole-auth-deployment.yaml b/deploy/opentofu/k3s/manifests/knoe/knoe-auth-deployment.yaml similarity index 100% rename from deploy/opentofu/k3s/manifests/knoe/prole-auth-deployment.yaml rename to deploy/opentofu/k3s/manifests/knoe/knoe-auth-deployment.yaml diff --git a/deploy/opentofu/k3s/manifests/knoe/prole-auth-kerberos-configmap.yaml b/deploy/opentofu/k3s/manifests/knoe/knoe-auth-kerberos-configmap.yaml similarity index 100% rename from deploy/opentofu/k3s/manifests/knoe/prole-auth-kerberos-configmap.yaml rename to deploy/opentofu/k3s/manifests/knoe/knoe-auth-kerberos-configmap.yaml diff --git a/deploy/opentofu/k3s/manifests/knoe/prole-auth-secrets.example.yaml b/deploy/opentofu/k3s/manifests/knoe/knoe-auth-secrets.example.yaml similarity index 100% rename from deploy/opentofu/k3s/manifests/knoe/prole-auth-secrets.example.yaml rename to deploy/opentofu/k3s/manifests/knoe/knoe-auth-secrets.example.yaml diff --git a/deploy/opentofu/k3s/manifests/knoe/prole-auth-service.yaml b/deploy/opentofu/k3s/manifests/knoe/knoe-auth-service.yaml similarity index 100% rename from deploy/opentofu/k3s/manifests/knoe/prole-auth-service.yaml rename to deploy/opentofu/k3s/manifests/knoe/knoe-auth-service.yaml diff --git a/deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml b/deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml index 5c9e8cc..189dce0 100644 --- a/deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml +++ b/deploy/opentofu/k3s/manifests/knoe/knoe-db.yaml @@ -57,7 +57,11 @@ spec: secret: name: knoe-db-user postInitTemplateSQL: - - CREATE EXTENSION IF NOT EXISTS pg_stat_statements; + # Supabase convention: relocatable extensions live in `extensions`, + # not `public`. Studio's Database Advisor flags `public.pg_stat_statements` + # as a Security warning on the dashboard. + - CREATE SCHEMA IF NOT EXISTS extensions; + - CREATE EXTENSION IF NOT EXISTS pg_stat_statements SCHEMA extensions; postInitSQL: - CREATE EXTENSION IF NOT EXISTS pg_tde; - CREATE EXTENSION IF NOT EXISTS pgcrypto; diff --git a/deploy/opentofu/k3s/manifests/knoe/kong-configmap.yaml b/deploy/opentofu/k3s/manifests/knoe/kong-configmap.yaml index 0d11216..7874117 100644 --- a/deploy/opentofu/k3s/manifests/knoe/kong-configmap.yaml +++ b/deploy/opentofu/k3s/manifests/knoe/kong-configmap.yaml @@ -10,34 +10,79 @@ data: _transform: true services: + # Root-path redirect: bare https://svc.knoe.dev/ -> /grafana so users + # land somewhere useful instead of getting Kong's "no Route matched" + # error. Other unmatched paths (typos, probes) still return that error + # by design — surfacing 404s is better than silently swallowing them. + - name: root-redirect + url: http://knoe.redirect.invalid/ + routes: + - name: root-redirect-to-grafana + hosts: + - svc.knoe.dev + paths: + - ~/$ + strip_path: false + plugins: + - name: request-termination + config: + status_code: 302 + message: redirecting + - name: response-transformer + config: + add: + headers: + - 'Location: /grafana' + + # Dedicated health endpoint for the GCE LB BackendConfig. request-termination + # returns 200 synchronously with no upstream call. URL is RFC-2606 reserved. + - name: healthz + url: http://knoe.healthz.invalid/ + routes: + - name: healthz + paths: + - /healthz + strip_path: true + plugins: + - name: request-termination + config: + status_code: 200 + message: ok + - name: knoe-service url: http://knoe-svc.knoe-system.svc.cluster.local:8080 routes: - name: knoe-k3s-kubeconfig hosts: - - svc.knoe.org + - svc.knoe.dev paths: - /k3s/kube_config.sh strip_path: false - name: db-manager + # NOTE: live GKE cluster runs CNPG in `knoe-db-0` namespace; k3s mode + # uses `knoe-db`. Per-mode delta — keep the k3s name here. url: http://knoe-db-manager.knoe-db.svc.cluster.local:80 routes: - name: backup-route hosts: - - svc.knoe.org + - svc.knoe.dev paths: - /backup strip_path: false + # Grafana: scoped to /grafana so other routes on svc.knoe.dev (Prometheus, + # Alertmanager, future UIs) can land alongside it. Grafana's grafana.ini + # is configured with `serve_from_sub_path = true` so it expects the + # /grafana prefix; strip_path=false forwards the prefix unchanged. - name: grafana url: http://kps-grafana.monitoring.svc.cluster.local:80 routes: - - name: grafana-root + - name: grafana-all hosts: - - svc.knoe.org + - svc.knoe.dev paths: - - / + - /grafana strip_path: false - name: knoe-auth @@ -45,7 +90,7 @@ data: routes: - name: knoe-auth-root hosts: - - api.knoe.org + - api.knoe.dev paths: - / strip_path: false @@ -55,7 +100,7 @@ data: routes: - name: gitea-root hosts: - - git.knoe.org + - git.knoe.dev paths: - / strip_path: false diff --git a/docs/db-access.md b/docs/db-access.md new file mode 100644 index 0000000..5d3738a --- /dev/null +++ b/docs/db-access.md @@ -0,0 +1,180 @@ +# Direct database access — psql + Rust apps + +Per-engineer access to the `knoe-db` Postgres cluster (PG18 / Percona) at +`pg.0.knoe.dev:5432`. + +> **Phase 1 (current)**: per-engineer postgres roles with strong passwords +> (30-day rotation), TLS required, named-roles-only ACL on the public LB. +> **Phase 2 (queued for Junie)**: replaces the password mechanism with +> PG18 native OAUTHBEARER (libpq drives a Google Device Flow) + a small +> validator library bundled into our `knoe-db` image. Connection target stays +> the same; engineers' `psql` invocation only changes from `-W` to setting +> `oauth_issuer` / `oauth_client_id` in the connection string. + +--- + +## First-day flow (engineer-facing) + +See **[`docs/onboarding.md`](onboarding.md)** — the canonical engineer +checklist. Two-line summary: + +1. chrisfu runs `etc/onboard_engineer.sh `. Script outputs + a one-time URL (and QR-code rendering of it) plus a 24h temp password. +2. Engineer opens URL on phone or laptop → page at `db.0.knoe.dev/onboard.html` + shows their password + connection string with [Copy] buttons. Engineer + pastes into psql, immediately rotates via `\password`, saves their new + password to **personal** 1Password (no shared vault required). + +The CA cert lives at [`etc/knoe-db-ca.crt`](../etc/knoe-db-ca.crt) in this +repo — engineers reference it with `sslrootcert=etc/knoe-db-ca.crt` after +cloning. It's a public CA root, not a secret. Auto-rotates with CNPG; we +re-export and commit when it does. + +Connection string (engineers substitute their own role): + +```bash +psql "host=pg.0.knoe.dev port=5432 user=$USER dbname=postgres \ + sslmode=verify-full sslrootcert=etc/knoe-db-ca.crt" +``` + +--- + +## Rust apps (sqlx / tokio-postgres / deadpool-postgres) + +Connection string with the same TLS configuration: + +```rust +let url = format!( + "postgres://{user}:{password}@pg.0.knoe.dev:5432/postgres\ + ?sslmode=verify-full&sslrootcert={ca_path}", + user = std::env::var("KNOE_PG_USER")?, + password = std::env::var("KNOE_PG_PASSWORD")?, + ca_path = format!("{}/.knoe/knoe-db-ca.crt", std::env::var("HOME")?), +); + +let pool = sqlx::PgPool::connect(&url).await?; +``` + +For `tokio-postgres` directly, configure the `MakeTlsConnector` with the same +CA bundle (use `postgres-native-tls` or `tokio-postgres-rustls`). + +**Service apps on GKE** (Phase 2.5 territory): the same connection string, but +`KNOE_PG_USER` / `KNOE_PG_PASSWORD` come from a per-app k8s Secret rather than +1Password. Phase 2 swaps this to Workload-Identity-issued OAUTHBEARER tokens — +the app code change is one line. + +--- + +## What can a `knoe_developer` do? + +The per-engineer roles (`chrisfu`, `ron`) are members of the `knoe_developer` +group. Grants: + +| Schema | Permission | Notes | +|--------------|---------------------------------------|-------| +| `knoe` | `SELECT, INSERT, UPDATE, DELETE` | Application schema; ALTER DEFAULT PRIVILEGES propagates to new tables | +| `public` | `SELECT, INSERT, UPDATE, DELETE` | Same | +| `auth` | `SELECT` only | Read-only on Supabase GoTrue tables (debugging / inspection) | +| `storage` | `SELECT` only | Read-only on Supabase storage tables | +| `extensions` | `USAGE` | Lets queries reach `extensions.pg_stat_statements` etc. | + +What you can NOT do: +- Drop / truncate `auth.*` or `storage.*` (read-only) +- `ALTER ROLE`, `CREATE ROLE`, `CREATE EXTENSION` (those are postgres-only) +- Connect as `postgres`, `supabase_admin`, `authenticator` from outside the cluster +- Connect at all without TLS (pg_hba `hostnossl ... reject`) + +Need broader access? Ask chrisfu — easier to extend the `knoe_developer` group +than to mint role-specific grants per engineer. + +--- + +## Onboarding a new engineer (admin-facing) + +```bash +./etc/onboard_engineer.sh +``` + +Outputs: +- A one-time URL: `https://db.0.knoe.dev/onboard.html#user=...&pw=...&exp=...` +- A QR-code rendering of that URL (in your terminal, via `qrencode`) +- The plaintext temp password (fallback) + +Deliver the URL to the new engineer via Gmail (to their `@knoey.com` address) +or by showing the QR on screenshare for them to scan with their phone. The +landing page handles all the engineer-side UX (copy buttons, connection +string, rotation guidance). They save the password to their **personal** +1Password — no shared vault required. See [`docs/onboarding.md`](onboarding.md). + +The script idempotently CREATEs / ALTERs the role: +- LOGIN INHERIT VALID UNTIL `` +- PASSWORD set to a fresh 192-bit random +- GRANT knoe_developer + +The pg_hba.conf `hostssl all +knoe_developer all scram-sha-256` rule means +the new role is reachable via `pg.0.knoe.dev` immediately — no pg_hba edits +needed per onboard. + +Other actions: +```bash +./etc/onboard_engineer.sh --rotate # fresh 24h temp password +./etc/onboard_engineer.sh --revoke # DROP ROLE +``` + +--- + +## Rotate your password + +The engineer-side path is `\password` inside psql — they pick a new password +locally, postgres updates the SCRAM verifier, the password never leaves their +machine. No coordination with chrisfu needed. + +If chrisfu needs to issue a fresh **temp** password (engineer locked out, +forgot to rotate before `VALID UNTIL` expired): + +```bash +./etc/onboard_engineer.sh --rotate +``` + +Same delivery (URL/QR/email) as initial onboarding. + +--- + +## Troubleshooting + +- `pg_hba.conf rejects connection for host ... no encryption` — you set + `sslmode=disable`. Use `verify-full` (requires the CA cert) or `require` + (encrypted but not verified). + +- `pg_hba.conf rejects connection for host , user , no encryption` — + you're on a network that's stripping TLS, OR your psql client is too old. + Update libpq to ≥17. + +- `password authentication failed for user ""` — typically wrong password, + but also fires for roles not in our allow list (postgres, knoe-db, etc.) + when connecting from outside the cluster. Check you're using your own role. + +- `FATAL: role "" is not permitted to log in` — your `VALID UNTIL` lapsed. + Ask for rotation. + +--- + +## Related + +- **Phase 2 plan (queued, owned by Junie)**: install/deploy.sh switches to + `pg_oauth` as the default until knoey-system/authority auth is live. + Includes building a validator library against PG18's `OAuthValidatorCallbacks` + API and bundling into the knoe-db image. See `docs/TODO.md` follow-up for + the active reference. +- **k3d build**: same Service + cluster cert pattern in k3d mode. Owned by + Junie alongside the install.sh true-up. +- **knoe-auth Round 1**: the longer-term identity story (Kerberos KDC + Java + authority). Becomes the Phase 2 issuer when it deploys, but Phase 2 doesn't + block on it — Google's issuer is sufficient for now. + +## Files + +- [`deploy/gcp/gke/knoe-db.yaml`](../deploy/gcp/gke/knoe-db.yaml) — CNPG + cluster spec; pg_hba block, server cert SAN list. +- [`deploy/gcp/gke/knoe-db-external-lb.yaml`](../deploy/gcp/gke/knoe-db-external-lb.yaml) — + external TCP LoadBalancer Service. diff --git a/docs/onboarding.md b/docs/onboarding.md new file mode 100644 index 0000000..230b240 --- /dev/null +++ b/docs/onboarding.md @@ -0,0 +1,158 @@ +# Engineer onboarding — first-day checklist + +Welcome. This doc gets you from "I just got an onboarding link" to "psql is open +and I'm productive". Should take ~5 minutes. + +> **Phase 1 (current).** Each engineer has a per-name postgres role with a +> temp password (24h) delivered via a one-time link. **Phase 2 (queued)** +> replaces the password mechanism with libpq OAUTHBEARER (Google sign-in). +> Your role name persists; only the connection string changes when Phase 2 lands. + +--- + +## What you should have + +1. An onboarding link that looks like + `https://db.0.knoe.dev/onboard.html#user=...&pw=...&exp=...`, + delivered by chrisfu via either: + - A Gmail message to your `@knoey.com` mailbox, or + - A QR code shown on screenshare during a Signal call. +2. A clone of the [`knoe-db` repo](https://git.knoe.dev/knoey.dev/knoe-db). + +If you don't have either, ping chrisfu. + +--- + +## Step 1 — Open the link / scan the QR + +Click the URL (or scan the QR with your phone camera). The page at +`db.0.knoe.dev/onboard.html` decodes the URL fragment **client-side** and +displays: + +- Your temporary password — with a **[Copy]** button +- Your psql connection string — pre-filled with your role name, also **[Copy]** +- A bootstrap one-liner that fetches the CA cert and connects you +- Inline `\password` rotation instructions + +Save the password to your **personal** 1Password (or whichever password +manager you use). The page strips the password from your browser history on +first render, but DO NOT close it before saving — the page is one-shot. + +--- + +## Step 2 — Connect and rotate + +Clone the repo if you haven't, then connect: + +```bash +git clone git@git.knoe.dev:knoey.dev/knoe-db.git +cd knoe-db +psql "host=pg.0.knoe.dev port=5432 user=$YOUR_USERNAME dbname=postgres \ + sslmode=verify-full sslrootcert=etc/knoe-db-ca.crt" +``` + +Paste the temporary password when prompted. + +**Rotate immediately:** + +```sql +postgres=> \password +Enter new password: +Enter it again: +postgres=> -- save the new password to your personal 1Password. +postgres=> -- everything from here uses YOUR password. +``` + +Why immediately? The temp `VALID UNTIL` is 24h; if you don't rotate before +then, you'll be locked out (chrisfu can rerun the onboard script to issue a +fresh temp). + +--- + +## Step 3 — What you can do as `knoe_developer` + +Your role is a member of the `knoe_developer` group. Grants: + +| Schema | Permission | Notes | +|--------------|----------------------------------|-------| +| `knoe` | `SELECT, INSERT, UPDATE, DELETE` | Application schema | +| `public` | `SELECT, INSERT, UPDATE, DELETE` | Same | +| `auth` | `SELECT` only | Read-only on Supabase GoTrue tables | +| `storage` | `SELECT` only | Read-only on Supabase storage tables | +| `extensions` | `USAGE` | Lets you reach `extensions.pg_stat_statements` etc. | + +What you can't do: +- Drop / truncate `auth.*` or `storage.*` (read-only on those by design). +- `CREATE ROLE`, `CREATE EXTENSION` (postgres superuser only). +- Connect from outside the cluster as `postgres`, `supabase_admin`, etc. + (those roles are pinned to RFC1918 by `pg_hba.conf`). + +Need broader access? File an issue or ping chrisfu — easier to extend the +group than mint per-engineer overrides. + +--- + +## Step 4 — Rust apps (if applicable) + +Same connection string in `sqlx` / `tokio-postgres` / `deadpool-postgres`: + +```rust +let url = format!( + "postgres://{user}:{password}@pg.0.knoe.dev:5432/postgres\ + ?sslmode=verify-full&sslrootcert={ca}", + user = std::env::var("KNOE_PG_USER")?, + password = std::env::var("KNOE_PG_PASSWORD")?, + ca = std::env::current_dir()?.join("etc/knoe-db-ca.crt").display(), +); +let pool = sqlx::PgPool::connect(&url).await?; +``` + +For deployed services (not your laptop): Phase 2 swaps the user/password env +vars for an OIDC token. The connection string skeleton stays the same. + +--- + +## Step 5 — Studio access (web UI, optional) + +Open in any browser, sign in with your `@knoey.com` +Google account. Studio is the same database; you can use it for ad-hoc +queries, table inspection, RLS rule editing. Most of what you need is in the +SQL editor. + +--- + +## Troubleshooting + +- **`pg_hba.conf rejects connection ... no encryption`** — you set + `sslmode=disable`. Use `verify-full` (with `sslrootcert=etc/knoe-db-ca.crt`) + or at minimum `require`. + +- **`password authentication failed`** — wrong password (typo when copy-paste, + or temp expired). Run psql with `\password` once you're in, OR ping chrisfu + to rerun the onboard script. + +- **`role "" is not permitted to log in`** — your `VALID UNTIL` lapsed. + Ask for rotation. + +- **Connection times out** — DNS for `pg.0.knoe.dev` may not have propagated + on your network yet. Try `dig pg.0.knoe.dev`. Should resolve to + `34.106.156.196` (or whatever the current LB IP is). + +- **`server certificate for "..." does not match host name "pg.0.knoe.dev"`** + — you're using `verify-full` but the CA cert in your repo is stale (CNPG + rotated the CA). `git pull` and try again, or fetch the fresh cert from + chrisfu. + +--- + +## Related + +- [`docs/db-access.md`](db-access.md) — the underlying access architecture + (LB, pg_hba, role grants, Phase 2 plan). +- The plan file (live, not in git) at + `~/.claude/plans/we-re-continuing-work-on-happy-toast.md` for chrisfu — has + the design rationale for why this delivery channel was chosen over a shared + vault. +- [`etc/onboard_engineer.sh`](../etc/onboard_engineer.sh) — the script chrisfu + runs to provision your role and emit your URL. Same script handles + `--rotate` and `--revoke`. diff --git a/docs/pipeline-phases.md b/docs/pipeline-phases.md index a03318d..fac5833 100644 --- a/docs/pipeline-phases.md +++ b/docs/pipeline-phases.md @@ -85,15 +85,41 @@ Before picking up work on Phase 1, verify: --- +## Off-pipeline work since Phase 0 + +These commits and live-cluster operations landed between Phase 0 and Phase 1. They aren't part of the autobuild pipeline itself — they're cluster-recovery and Maven build-readiness work that interrupted the Phase 1 sequence. Captured here so future readers see the actual chronology. + +### 2026-04-28 → 2026-04-29: repo cleanup + Maven build readiness + cluster recovery + +| Commit / event | Scope | Notes | +|---|---|---| +| [b355855](#) `refactor(auth): rename to dev.knoe.auth; archive tmp/ post-CNPG cleanup` | Phase A repo cleanup + Phase C Maven build readiness | tmp/ → archive/2026-04-cnpg-and-gitlab-cleanup/ (87 ad-hoc shell scripts); authority/ groupId `org.knoe` → `dev.knoe`, artifactId → `auth`, source tree moved `org/prole/authority/` → `dev/knoe/auth/`; build is now green via `make build-auth` (`mvn -f authority/pom.xml package` → `authority/target/knoe-auth.jar`); 4 k3s manifests renamed `prole-auth-*.yaml` → `knoe-auth-*.yaml`. | +| Phase B live cluster recovery (uncommitted; live-cluster operations) | Cluster recovery | `knoe-db-1` had been crashlooping for 14 days (full WAL volume) — fixed via PVC delete + CNPG re-bootstrap; cluster failover-cycled all 3 pods to refresh stale plugin-barman-cloud sidecars; **knoe-dev-cnpg-0 is now 3/3 healthy**, primary `knoe-db-2`. | +| Garage cluster recovery (uncommitted; live-cluster operations) | Backup target recovery | Garage on `knoe-dev-cnpg-0` was 100% full (`LMDB: No space left on device`) starting 2026-04-28 14:00 UTC; PVC expanded 29 → 80 GiB, layout capacity 10 → 60 GB, bucket cleared (8382 stale objects + 12 orphan multipart uploads). | +| GCS migration completed (uncommitted; live-cluster operations) | Backup target standardization | ObjectStore now `gs://knoe-0-backups/` with `googleCredentials.gkeEnvironment: true`; WI auth initially via auto-generated `knoe-db` SA annotation; 2 verified `data.tar.gz` backups in catalog; ScheduledBackup unsuspended. | + +**Suspected root cause of the 14:00 UTC outage:** an `install.sh --mode k3d` run while shell `kubectx` was set to a GKE context overwrote the GCS-backed ObjectStore with a Garage-backed one, then Garage filled up. Env-contamination guard tracked in [`docs/TODO.md`](TODO.md). + +### 2026-04-29 (later same day): CNPG operator + backup stabilization + +After 5 consecutive scheduled backups completed cleanly to GCS, did a small wave to remove the remaining drift: + +| Event | Scope | Notes | +|---|---|---| +| Operators restarted | Refresh stale state | `barman-cloud` (plugin operator) and `cnpg-controller-manager` rolled — both had been running 15-19 days. Plugin re-discovery confirmed in logs; stale Garage retention-worker endpoint cache cleared. | +| **CNPG operator upgraded v1.28.1 → v1.29.0** | Operator upgrade | Applied upstream release manifest (`releases/cnpg-1.29.0.yaml`). v1.29.0 added `spec.serviceAccountName` field (mutually exclusive with `serviceAccountTemplate`), required to land the SA migration below. Cluster healthy throughout. | +| **Cluster pod SA migration: knoe-db → cnpg-backup-sa** | Canonical pattern alignment | `cluster.spec.serviceAccountName=cnpg-backup-sa` applied. CNPG supervised rolling restart cycled all 3 pods. **First pod (db-3) failed RBAC check** — CNPG only auto-binds roles when it auto-creates the SA; for a pre-existing SA we had to add `cnpg-backup-sa` as a subject to the existing `knoe-db` and `knoe-db-barman-cloud` RoleBindings. After the patch all pods rolled cleanly. 23:03 UTC scheduled backup completed end-to-end with the new SA — verification gate passed. | +| **Garage removed from `knoe-dev-cnpg-0`** | DB-cluster cleanup | StatefulSet, both services (`garage` ClusterIP and `garage-s3-ilb` LoadBalancer), configmap, PVC, and `garage-hdd` StorageClass all deleted. ObjectStore untouched (still `gs://knoe-0-backups/`); backups continued through the Garage tear-down. | + +Reality-vs-intent gaps surfaced by this work are captured in [`docs/TODO.md`](TODO.md) and as `> **Reality TODO:**` blocks in [`CLAUDE.md`](../CLAUDE.md) and [`AGENTS.md`](../AGENTS.md). Several gaps closed in this session (Garage on DB cluster, SA alignment); new ones added (redundant `[knoe-db-0/knoe-db]` IAM binding, init script + patch script need updates to reflect the new canonical end state). + +--- + ## Phase 1 — `min` Mode Pipeline -**Status:** ✅ Complete -**Depends on:** Phase 0 complete ✅ -**Commit Summary:** - -``` -3728889 Phase 1: OIDC provider integration and GKE auth deployment -``` +**Status:** 🔲 Not started — ready to begin (deferred during the 2026-04-28 cluster-recovery work above) +**Depends on:** Phase 0 complete ✅ +**First task:** fix `knoe/core/ops/cloudnative_pg.py:1372` f-string syntax (30 collection errors, see Phase 0 known issues) ### What this phase does @@ -410,7 +436,7 @@ push to main only: | `GCP_SA_KEY` | GCP service account JSON with `container.viewer` + `storage.objectViewer` on both clusters | ✅ | ✅ | | `GKE_REGION` | `us-west3` | | | | `GKE_APP_CLUSTER` | `knoe-dev-0` | | | -| `GKE_DB_CLUSTER` | `knoe-cnpg-0` | | | +| `GKE_DB_CLUSTER` | `knoe-dev-cnpg-0` | | | ### `.gitlab-ci.yml` (full replacement) @@ -545,10 +571,10 @@ deploy-service: | Test | What it proves | |---|---| | `test_gke_app_cluster_reachable` | kubectl can reach `knoe-dev-0` | -| `test_gke_db_cluster_reachable` | kubectl can reach `knoe-cnpg-0` | +| `test_gke_db_cluster_reachable` | kubectl can reach `knoe-dev-cnpg-0` | | `test_cnpg_cluster_knoe_db_0_running` | `knoe-db` cluster in `knoe-db-0` namespace has 3 Ready instances | | `test_barman_backup_schedule_exists` | ScheduledBackup CR exists in `knoe-db-0` | -| `test_garage_only_in_app_cluster` | No Garage pods in `knoe-cnpg-0` namespace | +| `test_garage_only_in_app_cluster` | No Garage pods in `knoe-dev-cnpg-0` namespace | ### Commit Summary diff --git a/docs/plans/README.md b/docs/plans/README.md index 5a43e19..51229eb 100644 --- a/docs/plans/README.md +++ b/docs/plans/README.md @@ -35,7 +35,9 @@ A plan in this directory has one of three statuses, declared at the top: ## Customer deploys -The knoe.dev platform supports per-customer deployments — currently the most active is `prole.org`. The convention is **branches in this repo**, not separate forks. Customer-specific divergence (config, branding, on-prem manifests, kubeconfig handling) lives on a branch named after the customer; platform changes always land on `main` and customer branches rebase or merge from main on a regular cadence. See the customer's own `docs/plans/` for resync notes — for example, `~/dev/prole/docs/plans/customer-deploy-resync.md` if you have that working tree checked out. +The knoe.dev platform is designed to support per-customer deployments via **branches in this repo**, not separate forks. Customer-specific divergence (config, branding, on-prem manifests, kubeconfig handling) would live on a branch named after the customer; platform changes always land on `main` and customer branches rebase or merge from main on a regular cadence. + +Currently no customer branch is active. The original plan to converge `~/dev/prole` into a `customer/prole.org` branch is captured in [`customer-deploy-resync.md`](customer-deploy-resync.md) and is **dormant** — the prole→knoe rebrand has merged into `main` itself, and there is no separate `~/dev/prole` working tree under development. `origin` and `knoe` remotes both point at upstream `git@git-ssh.knoe.dev:knoe.dev/knoe-db.git`. ## How to propose a new plan diff --git a/docs/plans/customer-deploy-resync.md b/docs/plans/customer-deploy-resync.md index d72e1f1..0a9482d 100644 --- a/docs/plans/customer-deploy-resync.md +++ b/docs/plans/customer-deploy-resync.md @@ -1,6 +1,6 @@ # Resync `prole` as a Customer Deploy of `knoe-db` -**Status:** Active plan. Not yet executed. +**Status:** **Dormant.** The plan was written when `~/dev/prole` and `~/dev/knoe-db` were two separate working trees diverging on GitHub vs the knoe Gitea. As of 2026-04, the prole→knoe rebrand has merged into `knoe-db/main` directly, there is no separate `~/dev/prole` working tree under active development, and both `origin` and `knoe` remotes in this repo point at upstream `git@git-ssh.knoe.dev:knoe.dev/knoe-db.git`. The customer-branch workflow described below is the **target shape** for any future per-customer separation; it has not been executed and has no current customer driving it. Sections below are preserved as the design reference if the workflow is reactivated. **Owner:** chrisfu **Audience:** Jr/mid engineer joining the project. No prior knowledge of the prole↔knoe-db split assumed. diff --git a/docs/plans/deployment-modes.md b/docs/plans/deployment-modes.md index 9a70640..a56b322 100644 --- a/docs/plans/deployment-modes.md +++ b/docs/plans/deployment-modes.md @@ -19,7 +19,7 @@ Earlier installer revisions assumed the engineer already knew which mode they wa | **min** | A single `knoe-db` container under `containerd` — no Kubernetes, no Docker. The Spring Boot app runs locally on the host. | Engineers running a Spring app on a laptop and wanting a CNPG-compatible Postgres without a full cluster. | Homebrew, 1Password | | **k3d** | A Docker-based local k3s cluster. Full CNPG + optional Supabase / ArgoCD / Gitea / GitLab. | Local development with the full platform. | Docker, Homebrew, 1Password | | **k3s** | A multi-node k3s cluster on real hardware. Full Kerberos auth stack, Garage S3, monitoring. Mirrors the GKE shape. | Homelab contributors with their own hardware (e.g. a 3-node Pi cluster). | k3sup, Homebrew, 1Password | -| **gke** | The production dual-cluster on Google Cloud (`knoe-dev-0` + `knoe-cnpg-0` in `us-west3`). | Production. | gcloud, kubectl, 1Password | +| **gke** | The production dual-cluster on Google Cloud (`knoe-dev-0` + `knoe-dev-cnpg-0` in `us-west3`). | Production. | gcloud, kubectl, 1Password | The architectural commitment matters: **all four modes are scaled-down mirrors of the GKE shape.** Not parallel implementations of the same idea, not divergent forks — the same components composed at different scales. CNPG runs in all four. The Spring Boot `authority` service runs in all four. `knoe.user` lives in the same schema in all four. The only thing that changes is what platform the components run on and which optional services are enabled. @@ -173,7 +173,7 @@ Run after any change touching the wizard or mode logic. **k3d** — A wrapper that runs k3s clusters inside Docker containers. Useful for local dev because it gives you a "real" Kubernetes cluster without having to provision VMs. -**GKE (Google Kubernetes Engine)** — Google Cloud's managed Kubernetes. We use Standard for `knoe-dev-0` and `knoe-cnpg-0`. See `CLAUDE.md` for cluster topology. +**GKE (Google Kubernetes Engine)** — Google Cloud's managed Kubernetes. We use Standard for `knoe-dev-0` and `knoe-dev-cnpg-0`. See `CLAUDE.md` for cluster topology. **containerd** — A container runtime. Speaks the OCI image and runtime specs. Both Docker and Kubernetes use containerd under the hood. In `min` mode we talk to it directly via `nerdctl`, skipping Docker entirely. diff --git a/docs/plans/knoe-auth-round-1.md b/docs/plans/knoe-auth-round-1.md index 2eae620..32e9446 100644 --- a/docs/plans/knoe-auth-round-1.md +++ b/docs/plans/knoe-auth-round-1.md @@ -53,32 +53,32 @@ The actual files that implement Round 1. Verify with `git ls-files` before assum ### Java application — `authority/` -The Spring Boot service that implements the enrollment flow, the admin API, and SPNEGO-protected endpoints. Multi-module Maven build (`authority/pom.xml`); the application package is `dev.knoe.authority` (kept as-is across the prole→knoe rebrand for compatibility). +The Spring Boot service that implements the enrollment flow, the admin API, and SPNEGO-protected endpoints. Two-pom Maven build (root `pom.xml` for `knoe-db` controller, `authority/pom.xml` for `knoe-auth`); the application package is `dev.knoe.auth` and the Maven coordinate is `dev.knoe:auth`. | File | Responsibility | | --- | --- | -| `authority/src/main/java/dev/knoe/authority/KnoeAuthApplication.java` | `@SpringBootApplication` entry point. | -| `authority/src/main/java/dev/knoe/authority/HealthController.java` | `/health` endpoint. | -| `authority/src/main/java/dev/knoe/authority/web/LoginController.java` | Form-login + SPNEGO challenge for browsers without a ticket. | -| `authority/src/main/java/dev/knoe/authority/web/VerifyController.java` | Token-verify endpoint for downstream services. | -| `authority/src/main/java/dev/knoe/authority/session/SessionTokenService.java` | Issues HMAC-SHA256 JWT cookies after successful auth. | -| `authority/src/main/java/dev/knoe/authority/session/SessionUser.java` | Authenticated principal carried in the security context. | -| `authority/src/main/java/dev/knoe/authority/user/PrincipalNormalizer.java` | Strips realm/instance from a Kerberos principal (`alice/admin@KNOE.DEV` → `alice`). | -| `authority/src/main/java/dev/knoe/authority/kerberos/KerberosSpnegoService.java` | SPNEGO challenge/response handling. | -| `authority/src/main/java/dev/knoe/authority/kerberos/KerberosPasswordService.java` | Password-style auth fallback for browsers that can't do SPNEGO. | -| `authority/src/main/java/dev/knoe/authority/kerberos/KadminClient.java` | Shells out to `kadmin.local` (in the KDC sidecar) to `addprinc` and `cpw`. **Sanitizes input.** | -| `authority/src/main/java/dev/knoe/authority/enroll/EnrollmentController.java` | Web endpoints: `GET /auth/enroll`, `POST /auth/enroll/verify-otp`, `POST /auth/enroll/identity/start`, `GET /auth/enroll/google-callback`, `GET /auth/enroll/totp`, `POST /auth/enroll/totp/verify`, `POST /auth/enroll/complete`. | -| `authority/src/main/java/dev/knoe/authority/enroll/InviteService.java` | CRUD + validation against `knoe.invitation`. OTP hashing (bcrypt) and rate limiting (3 attempts). | -| `authority/src/main/java/dev/knoe/authority/enroll/GoogleOAuthService.java` | Exchange OAuth2 code → ID token, validate `email_verified`, return a `GoogleIdentity` record. **No `hd` allowlist.** | -| `authority/src/main/java/dev/knoe/authority/enroll/TotpService.java` | Generate TOTP secret, produce `otpauth://` URI, verify codes. | -| `authority/src/main/java/dev/knoe/authority/enroll/UserProvisioningService.java` | Transactional orchestrator: inserts user/identity/totp rows, calls `KadminClient`, queues provisioning jobs. | -| `authority/src/main/java/dev/knoe/authority/admin/AdminController.java` | `POST /auth/admin/invites`, `GET /auth/admin/users`, `POST /auth/admin/grants`. SPNEGO + admin-role gated. | -| `authority/src/main/java/dev/knoe/authority/admin/KnobjectService.java` | CRUD on `knoe.knobject` and `knoe.access_grant`; enqueues `provisioning_job` rows. | -| `authority/src/main/java/dev/knoe/authority/provisioning/ProvisioningWorker.java` | `@Scheduled` poller for `knoe.provisioning_job WHERE status = 'pending'`. Dispatches to GitLab/Gitea/CNPG. | -| `authority/src/main/java/dev/knoe/authority/config/AuthProperties.java` | Typed binding for `knoe.auth.*` keys. | -| `authority/src/main/java/dev/knoe/authority/config/KerberosProperties.java` | Typed binding for `knoe.auth.kerberos.*` keys. | +| `authority/src/main/java/dev/knoe/auth/KnoeAuthApplication.java` | `@SpringBootApplication` entry point. | +| `authority/src/main/java/dev/knoe/auth/HealthController.java` | `/health` endpoint. | +| `authority/src/main/java/dev/knoe/auth/web/LoginController.java` | Form-login + SPNEGO challenge for browsers without a ticket. | +| `authority/src/main/java/dev/knoe/auth/web/VerifyController.java` | Token-verify endpoint for downstream services. | +| `authority/src/main/java/dev/knoe/auth/session/SessionTokenService.java` | Issues HMAC-SHA256 JWT cookies after successful auth. | +| `authority/src/main/java/dev/knoe/auth/session/SessionUser.java` | Authenticated principal carried in the security context. | +| `authority/src/main/java/dev/knoe/auth/user/PrincipalNormalizer.java` | Strips realm/instance from a Kerberos principal (`alice/admin@KNOE.DEV` → `alice`). | +| `authority/src/main/java/dev/knoe/auth/kerberos/KerberosSpnegoService.java` | SPNEGO challenge/response handling. | +| `authority/src/main/java/dev/knoe/auth/kerberos/KerberosPasswordService.java` | Password-style auth fallback for browsers that can't do SPNEGO. | +| `authority/src/main/java/dev/knoe/auth/kerberos/KadminClient.java` | Shells out to `kadmin.local` (in the KDC sidecar) to `addprinc` and `cpw`. **Sanitizes input.** | +| `authority/src/main/java/dev/knoe/auth/enroll/EnrollmentController.java` | Web endpoints: `GET /auth/enroll`, `POST /auth/enroll/verify-otp`, `POST /auth/enroll/identity/start`, `GET /auth/enroll/google-callback`, `GET /auth/enroll/totp`, `POST /auth/enroll/totp/verify`, `POST /auth/enroll/complete`. | +| `authority/src/main/java/dev/knoe/auth/enroll/InviteService.java` | CRUD + validation against `knoe.invitation`. OTP hashing (bcrypt) and rate limiting (3 attempts). | +| `authority/src/main/java/dev/knoe/auth/enroll/GoogleOAuthService.java` | Exchange OAuth2 code → ID token, validate `email_verified`, return a `GoogleIdentity` record. **No `hd` allowlist.** | +| `authority/src/main/java/dev/knoe/auth/enroll/TotpService.java` | Generate TOTP secret, produce `otpauth://` URI, verify codes. | +| `authority/src/main/java/dev/knoe/auth/enroll/UserProvisioningService.java` | Transactional orchestrator: inserts user/identity/totp rows, calls `KadminClient`, queues provisioning jobs. | +| `authority/src/main/java/dev/knoe/auth/admin/AdminController.java` | `POST /auth/admin/invites`, `GET /auth/admin/users`, `POST /auth/admin/grants`. SPNEGO + admin-role gated. | +| `authority/src/main/java/dev/knoe/auth/admin/KnobjectService.java` | CRUD on `knoe.knobject` and `knoe.access_grant`; enqueues `provisioning_job` rows. | +| `authority/src/main/java/dev/knoe/auth/provisioning/ProvisioningWorker.java` | `@Scheduled` poller for `knoe.provisioning_job WHERE status = 'pending'`. Dispatches to GitLab/Gitea/CNPG. | +| `authority/src/main/java/dev/knoe/auth/config/AuthProperties.java` | Typed binding for `knoe.auth.*` keys. | +| `authority/src/main/java/dev/knoe/auth/config/KerberosProperties.java` | Typed binding for `knoe.auth.kerberos.*` keys. | -Tests for the above live under `authority/src/test/java/dev/knoe/authority/` — notably `web/VerifyControllerTest.java` and `session/SessionTokenServiceTest.java`. +Tests for the above live under `authority/src/test/java/dev/knoe/auth/` — notably `web/VerifyControllerTest.java` and `session/SessionTokenServiceTest.java`. ### Kubernetes manifests — GKE @@ -94,10 +94,10 @@ Tests for the above live under `authority/src/test/java/dev/knoe/authority/` — | File | Purpose | | --- | --- | -| `deploy/opentofu/k3s/manifests/prole/prole-kdc-configmap.yaml` | The `PROLE.LOCAL` realm KDC for the homelab cluster. The pattern the GKE configmap was modeled on. | -| `deploy/opentofu/k3s/manifests/prole/prole-kdc-secrets.example.yaml` | Templated secrets for the same. | -| `deploy/opentofu/k3s/manifests/prole/prole-auth-deployment.yaml` | KDC + Spring Boot for the homelab. | -| `deploy/opentofu/k3s/manifests/prole/prole-auth-kerberos-configmap.yaml` | `krb5.conf` for the auth pod's Kerberos client. | +| `deploy/opentofu/k3s/manifests/knoe/prole-kdc-configmap.yaml` | The `PROLE.LOCAL` realm KDC for the homelab cluster. The pattern the GKE configmap was modeled on. | +| `deploy/opentofu/k3s/manifests/knoe/prole-kdc-secrets.example.yaml` | Templated secrets for the same. | +| `deploy/opentofu/k3s/manifests/knoe/knoe-auth-deployment.yaml` | KDC + Spring Boot for the homelab. | +| `deploy/opentofu/k3s/manifests/knoe/knoe-auth-kerberos-configmap.yaml` | `krb5.conf` for the auth pod's Kerberos client. | ### Bootstrap scripts — `etc/` @@ -135,7 +135,7 @@ https://auth.knoe.dev/enroll?token= ### Cluster topology -Round 1 lives in the GKE app cluster (`knoe-dev-0`) in the `knoe-system` namespace. The database stays where it already is — the dedicated CNPG cluster `knoe-cnpg-0`. See `CLAUDE.md` for the cluster layout and storage-quota rules. +Round 1 lives in the GKE app cluster (`knoe-dev-0`) in the `knoe-system` namespace. The database stays where it already is — the dedicated CNPG cluster `knoe-dev-cnpg-0`. See `CLAUDE.md` for the cluster layout and storage-quota rules. ``` knoe-dev-0 / knoe-system namespace: @@ -154,7 +154,7 @@ knoe-dev-0 / knoe-system namespace: └────────────────────────────────────────────────────────────────────┘ │ ▼ - ┌─ CNPG / knoe-db (in knoe-cnpg-0) ─────────────────────────────────┐ + ┌─ CNPG / knoe-db (in knoe-dev-cnpg-0) ─────────────────────────────────┐ │ knoe.user (base table) │ │ knoe.user_role (base table) │ │ knoe.invitation (Round 1) │ @@ -172,7 +172,7 @@ The KDC runs as a sidecar in the same pod as the Spring Boot app. They share the ## 4. Schema -The `knoe.*` schema lives in CNPG (`knoe-cnpg-0` namespace `knoe-db-0`). Base tables (`knoe.user`, `knoe.user_role`) are created by `etc/init_knoe_users.sh` lines 482–510. Round 1 added the six tables below; they are created by the same script later in its run. +The `knoe.*` schema lives in CNPG (`knoe-dev-cnpg-0` namespace `knoe-db-0`). Base tables (`knoe.user`, `knoe.user_role`) are created by `etc/init_knoe_users.sh` lines 482–510. Round 1 added the six tables below; they are created by the same script later in its run. ```sql -- ──────────────────────────────────────────────────────────────────── @@ -447,6 +447,7 @@ Real, named items the team has discussed. They are **not** in Round 1 — when t - A "knobject inspector" admin UI. Right now the admin API is JSON-only. - Phone/SMS-based OTP delivery. Round 1 covers email; the `contact_type` column is already present on `knoe.invitation` so adding SMS later is additive. - Self-service password rotation, recovery flows, and deactivation. Admin-only for Round 1. +- **Round 1.5 — OpenBao transit encryption for secrets at rest.** `UserProvisioningService` currently writes the per-user TOTP secret and any other long-lived per-user secrets to `knoe.*` tables in the clear (relying only on PG-level encryption-at-rest). Round 1.5 wraps writes/reads with an OpenBao transit-key envelope so the database can't yield plaintext secrets even if it leaks. Tracked by the `// TODO Round 1.5` marker in `authority/src/main/java/dev/knoe/auth/enroll/UserProvisioningService.java`. --- @@ -476,7 +477,7 @@ Real, named items the team has discussed. They are **not** in Round 1 — when t **Workload Identity** — GKE feature that binds a Kubernetes ServiceAccount to a Google Cloud IAM service account. Lets pods talk to GCP APIs (e.g. KMS for our master key) without long-lived JSON keyfiles. -**CNPG** — CloudNativePG. The Postgres operator we use for `knoe-db`. Runs in `knoe-cnpg-0`. See `CLAUDE.md` for cluster topology. +**CNPG** — CloudNativePG. The Postgres operator we use for `knoe-db`. Runs in `knoe-dev-cnpg-0`. See `CLAUDE.md` for cluster topology. **knobject** — A platform-managed resource that an admin can hand out to a user — a Gitea repo, a Postgres role, an OpenBao policy. The name is a portmanteau of "knoe" + "object". Modeled by `knoe.knobject`; granted via `knoe.access_grant`. diff --git a/docs/plans/onboarding-tdd-phase-a.md b/docs/plans/onboarding-tdd-phase-a.md new file mode 100644 index 0000000..f15df4c --- /dev/null +++ b/docs/plans/onboarding-tdd-phase-a.md @@ -0,0 +1,322 @@ +# Phase A — TDD design brief for the onboarding system + +> **For Junie in Sonnet mode.** This is the design phase. You produce a +> design doc (`docs/plans/onboarding-tdd.md`) plus one hard test fixture. +> Phase B (Codex) consumes the design and writes the bulk of the test bodies. +> **Do not write production-quality test files in Phase A** — write the spec +> they'll be written from. + +--- + +## 1. Why we're slowing down + +Five commits landed in rapid succession over the last sitting, none with +automated test coverage: + +| Commit | Surface | +|---|---| +| [`f805404`](#) | `oauth2-proxy --skip-auth-route` for Supabase API surfaces (unifies `db.0.knoe.dev` for Studio + SDK) | +| [`79a3f76`](#) | `GRANT anon, authenticated, service_role TO supabase_storage_admin`; oauth2-proxy `--pass/set-authorization-header` removed; `/support` 302→mailto stop-gap | +| [`8c2956b`](#) | `pg_stat_statements` moved from `public` to `extensions` schema (durable across all 4 deploy modes) | +| [`317b20b`](#) | TCP LB `pg.0.knoe.dev` (34.106.156.196), CNPG cert SAN, `knoe_developer` role, `chrisfu` + `ron` engineer roles, pg_hba tightened | +| [`313ee88`](#) | Reusable engineer onboarding: `etc/onboard_engineer.sh`, knoe-onboard nginx pod, `/onboard.html` Kong route, membership-based pg_hba (`+knoe_developer`), committed CA cert | + +About to onboard the second human engineer (`ron`) against this stack. +Standing decision was: **don't add a user, even of one, before we have tests +covering the surfaces a user touches**. This brief is that decision being +acted on. + +--- + +## 2. What you produce in Phase A + +Three artifacts, in order: + +1. **The f-string fix at `knoe/core/ops/cloudnative_pg.py:1372`** committed + first. 30 test-collection errors block the Phase 1 pipeline; until that + lands, no test we write has anywhere to run. See top of + [`docs/TODO.md`](../TODO.md). Trivial: extract `field.replace('.', '\\.')` + to a variable before the f-string. **One commit, message starts + `fix(test):`**. + +2. **`docs/plans/onboarding-tdd.md`** — the design doc. This is the main + deliverable. It must be detailed enough that Codex (Sonnet's successor in + Phase B, no shared memory) can write every test body from it without + needing to make architectural calls. Sections required: + - Test taxonomy (Tier 1 / Tier 2 / Tier 3) with one-line acceptance per + test — not bash code, but a precise spec of what's being asserted. + - Fixture architecture decision (k3d-per-run vs. shared persistent vs. + hybrid) with the tradeoff analysis recorded. + - Browser-test framework decision (Playwright vs. Puppeteer vs. ...) with + why. + - Directory layout under `tests/` and how it slots into the existing + pytest structure (read [`tests/test_render_supabase_hostname.py`](../../tests/test_render_supabase_hostname.py) + and [`tests/test_repair_update_and_supabase_flags.py`](../../tests/test_repair_update_and_supabase_flags.py) + for the conventions to match). + - CI integration — these tests slot into Phase 1 of + [`docs/pipeline-phases.md`](../pipeline-phases.md). Document the new run + config(s), and whether the integration tier needs its own marker (e.g. + `pytest -m integration`) so unit tests stay fast for local dev. + - Definition of done per tier — what passes count as the boundary covered + vs. work-in-progress. + +3. **The first hard fixture, working.** Specifically: a deterministic way to + simulate "external client connecting to the cluster" inside CI. This is + the single hardest design call in Phase A; everything in Tier 1 hangs off + it. Get it functional and verified — even if it's just a 30-line + `conftest.py` helper — so Codex can rely on it from day one. **One commit, + message starts `test(infra):`**. + +You do **not** write test bodies in Phase A. You write the spec and the +hardest fixture. Resist the temptation to land a few "easy" tests — Codex +gets a much cleaner working environment if Phase A's output is purely +design + fixture. + +--- + +## 3. Test taxonomy (the spec to flesh out) + +This is what the conversation that produced this brief settled on. Use it as +the skeleton for `docs/plans/onboarding-tdd.md` Section "Test taxonomy", but +you're expected to refine it: add edge cases, kill duplicates, surface gaps. + +### Tier 1 — Security boundaries (must-have before more onboards) + +The wall protecting the database from the public internet. **No test = no +proof.** Hard-gating on the team's deploy pipeline. + +1. **pg_hba rejection matrix.** For each axis cross-product: + - **Source**: external IP, internal cluster IP (10.0.0.0/8 simulated) + - **Encryption**: TLS, plaintext + - **Role**: `chrisfu`, `ron`, a fresh-onboarded `knoe_developer` member, + `postgres`, `supabase_admin`, `anon`, a non-member of `knoe_developer` + with valid LOGIN + password + - The matrix has 36 cells minimum. Acceptance: each cell asserts the + expected `connect / reject / auth-fail` outcome. + +2. **`+knoe_developer` membership semantics is load-bearing.** A role granted + `knoe_developer` is reachable externally; revoking the grant immediately + removes external reachability. A role with LOGIN + password but NOT in the + group cannot connect externally even with valid creds. (This is the + reusability lynchpin from commit `313ee88`; a regression here means + "everyone with a postgres password is reachable from the public internet".) + +3. **oauth2-proxy `--skip-auth-route` regex tightness.** For each declared + route (`/auth/v1/.*`, `/rest/v1/.*`, `/realtime/v1/.*`, `/storage/v1/.*`, + `/functions/v1/.*`, `/graphql/v1/.*`, `/pg/.*`, `/onboard.html`, `/support`): + - The exact path skips Google sign-in. + - Variant paths that *should* be gated (e.g. `/api/profile`, + `/Auth/v1/foo`, double-slash, URL-encoded traversal probes) **do** get + redirected to Google. + - Each declared route is exercised with at least one negative test that + would fail if the regex is loosened by accident. + +4. **`hostnossl reject` precedence.** A property test: any pg_hba rule order + that places a permissive rule (e.g. `host all all 10.0.0.0/8`) before a + `hostnossl ... reject` line, and would let an external plaintext + connection match the permissive rule, must FAIL the test. This is the + regression detector for the `externalTrafficPolicy: Cluster -> Local` bug + we hit in Phase 1 — preserve the lesson. + +### Tier 2 — Onboarding flow correctness + +CI-gating (block merges) but not deploy-gating. + +5. **`onboard_engineer.sh` is idempotent.** Run twice in a row → one role + exists with the second run's password. `--rotate` on a non-existent user + → clear error, no partial state. `--revoke` on a connected session → role + dropped, sessions terminated cleanly. + +6. **`onboard_engineer.sh` input validation.** Username regex; email is + `*@knoey.com`; rejection of shell-injection / SQL-injection probes; clear + error messages. + +7. **Reveal page renders correctly.** Headless-browser tests: + - `onboard.html#user=test&pw=Zm9v&exp=2099-01-01T00:00:00Z` → password + decoded and shown. + - `exp` in the past → expired-state UI. + - Missing required fields → error UI. + - Copy buttons populate clipboard. + - On render, `history.replaceState` strips the fragment from the URL. + - The `pw` is base64-decoded correctly even with URL-safe `-_` chars. + +8. **CA cert in repo matches live CA.** SHA256 fingerprint of + `etc/knoe-db-ca.crt` matches `kubectl get secret knoe-db-ca`. When CNPG + rotates, this test fails loudly instead of engineers silently 502'ing. + +### Tier 3 — Adjacent surfaces we touched + +Regression coverage. CI-gating. + +9. **Studio internal paths still gate via Google.** `/`, `/api/profile`, + `/api/database/*` → 302 to `accounts.google.com`. +10. **`db.0.knoe.dev/storage/v1/bucket` reachable as service-role.** The + GRANT we landed in `79a3f76` makes this work; pin it. +11. **`pg_stat_statements` lives in `extensions` post-bootstrap.** PG18 + docker fixture for k3d/k3s/min; live-cluster check for gke. Database + advisor's `public.pg_stat_statements` warning stays cleared. +12. **`api.0.knoe.dev` alias still works** until [`docs/TODO.md`](../TODO.md) + follow-up #14 explicitly retires it. + +--- + +## 4. Decisions you must make in Phase A + +These are the architectural calls Codex needs nailed down before writing +test bodies. Sonnet's reasoning bandwidth is concentrated here. + +### 4a. Fixture architecture for "external client" + +Tier 1 needs to assert "an external client cannot connect with X". Options: + +| Option | Pros | Cons | +|---|---|---| +| **k3d cluster + sidecar pod with `externalTrafficPolicy: Local` LB** | Closest to prod (real LB IP, real pg_hba) | Slow to spin up; LB IP allocation in k3d is non-trivial | +| **k3d cluster + iptables rule simulating "external" via a labeled namespace's pod IP space** | Faster; deterministic | Diverges from prod; risk that test passes but real LB doesn't | +| **Live cluster (gke `knoe-dev-cnpg-0`) + ephemeral test roles** | Highest fidelity | Pollutes prod with test roles; risk of leaving them around; can't run in CI without GCP creds | +| **Hybrid**: k3d for most assertions; one nightly job against live cluster for the fidelity-sensitive few | Best of both | Two test rigs to maintain | + +You decide. Document the tradeoff. Note that CLAUDE.md establishes k3d as +the canonical local mode, so the bias should lean k3d unless you find a +real reason otherwise. + +### 4b. Browser-test framework + +Tier 2.7 needs headless-browser support. Candidates: Playwright (Node), +Puppeteer (Node), Selenium (multi-language), Cypress (Node, e2e-focused). + +Read [`tests/`](../../tests/) — current Python pytest-only. Adding a Node +toolchain just for browser tests is a real cost. Consider: + +- Are there ~5 browser-tested behaviors total? Then a dedicated framework + may be overkill — a pytest test that uses `selenium` (Python) + a headless + Chrome may be enough. +- Are there ~25? Dedicated framework worth the cost. + +### 4c. k3d-per-run vs. shared persistent local cluster + +| | Per-run | Shared persistent | +|---|---|---| +| **Speed** | Slow (60-90s spinup) | Fast (~0s) | +| **Determinism** | High | Low (tests can pollute each other) | +| **Local dev UX** | Friction | Zero friction | +| **CI** | Required for parallel runs | Doesn't work for parallel | + +You decide. Docs/pipeline-phases.md would benefit from your call being +recorded explicitly, since Phase 2 of that pipeline will face the same +question. + +### 4d. Acceptance criteria per tier + +What counts as "Tier 1 done"? Specifically: +- Coverage threshold (e.g. all 36 pg_hba cells; all 9+ skip-auth-route + cases)? +- Mutation testing pass rate? +- Just "the spec items are all green"? + +Document the answer per tier. **Tier 1's bar should be the highest** — it's +the security wall. + +### 4e. Retroactive coverage policy + +The five commits (`f805404` through `313ee88`) landed without tests. We can: + +- **Cover them all in this round.** The Tier 1 + 2 + 3 spec already covers + most of the surface area, but not all (e.g. `pg_stat_statements` schema + move's behavior on a NEW deploy isn't directly tested by the spec; the + storage GRANT's effect on bucket listing isn't directly tested). +- **Cover only the onboarding system in this round; queue the rest for + follow-up.** + +I lean toward "cover them all" — the test debt only grows otherwise — but +the call is yours given you can see the actual scope better than I drafted. + +--- + +## 5. Files to read first + +In order: + +1. [`docs/plans/README.md`](README.md) — the team's plan-doc conventions. +2. [`docs/pipeline-phases.md`](../pipeline-phases.md) — where these tests + slot into the autobuild pipeline. +3. [`docs/TODO.md`](../TODO.md) — current queue and the f-string blocker + you'll fix first. +4. [`tests/test_render_supabase_hostname.py`](../../tests/test_render_supabase_hostname.py) + and [`tests/test_repair_update_and_supabase_flags.py`](../../tests/test_repair_update_and_supabase_flags.py) + — existing test patterns to follow. +5. [`docs/db-access.md`](../db-access.md) and [`docs/onboarding.md`](../onboarding.md) + — what the onboarding system actually does. +6. [`etc/onboard_engineer.sh`](../../etc/onboard_engineer.sh) — the script + under test. +7. [`deploy/gcp/gke/knoe-onboard.yaml`](../../deploy/gcp/gke/knoe-onboard.yaml) + — the static page (HTML embedded in the ConfigMap). +8. [`deploy/gcp/gke/knoe-db.yaml`](../../deploy/gcp/gke/knoe-db.yaml) — + pg_hba block (the surface most needing test coverage). +9. [`deploy/gcp/gke/oauth2-proxy-deployment.yaml`](../../deploy/gcp/gke/oauth2-proxy-deployment.yaml) + — `--skip-auth-route` args. +10. [`CLAUDE.md`](../../CLAUDE.md) §"install.sh / deploy.sh pre-flight" — + env-contamination warning is a category-mate of the boundary tests + you're designing. + +--- + +## 6. Definition of done for Phase A + +When all four are true: + +- [ ] `knoe/core/ops/cloudnative_pg.py:1372` f-string fix committed; pytest + collection succeeds repo-wide (`pytest --collect-only` returns clean). +- [ ] `docs/plans/onboarding-tdd.md` exists, addresses every section in + Sec. 2.2 above, and is reviewable by chrisfu. +- [ ] The first hard fixture (Sec. 4a's choice, implemented) is committed + with at least one Tier 1 test that uses it green. +- [ ] You have **not** committed bodies for any other tests. Codex's job. + +--- + +## 7. Out of scope for Phase A (Codex picks up) + +- Writing test bodies for Tiers 1, 2, 3 (except the one Tier 1 test that + validates the fixture from Sec. 4a is functional). +- CI wiring beyond what's needed for the f-string fix. +- Mutation-testing setup, if your acceptance criteria call for it. +- Infrastructure for live-cluster integration tests (only relevant if your + Sec. 4a decision calls for one). +- Updating engineer-facing docs to mention the test suite. Codex updates + `docs/onboarding.md` and `docs/db-access.md` after Phase B lands. + +--- + +## 8. Handoff format to Phase B + +Codex's brief (separate doc, will be drafted by chrisfu after this lands) +will reference: + +- `docs/plans/onboarding-tdd.md` — the spec Codex implements. +- The fixture you committed in Sec. 4a — Codex's tests use it. +- Your acceptance criteria per tier — Codex knows when each tier is done. +- The directory layout you defined — Codex's PRs land in the right places. + +So `docs/plans/onboarding-tdd.md` needs to be **prescriptive**, not +"considerations" or "options". Codex isn't going to make architectural +calls; it'll execute against what you wrote. Land the design. + +--- + +## 9. Working notes + +- **You can ask chrisfu questions.** If you hit a decision that genuinely + needs human input (e.g. "should we let CI provision GCP credentials" — + policy call, not technical), surface it rather than guessing. But the + questions in Sec. 4 are yours to answer; don't punt them upward. +- **The plan in this brief is also wrong somewhere.** The taxonomy is my + best draft from a conversation; you may find that some Tier 1 items are + redundant with Tier 2, or that there's a Tier 0 ("does the cluster even + start?") missing. Calling that out IN the design doc is part of the + deliverable. +- **Slow is smooth.** This brief deliberately scopes Phase A to design + + one fixture so the design isn't rushed. Resist scope creep into Phase B. + +You're starting fresh in a Sonnet context. Read the files in Sec. 5, then +the existing plans in `docs/plans/`, then come back here. Good design lands. diff --git a/etc/init_grafana_oauth.sh b/etc/init_grafana_oauth.sh new file mode 100755 index 0000000..a7f5b2b --- /dev/null +++ b/etc/init_grafana_oauth.sh @@ -0,0 +1,99 @@ +#!/usr/bin/env bash +# init_grafana_oauth.sh +# +# Bootstrap the Google-OAuth secret that kps-grafana mounts as env vars +# (GF_AUTH_GOOGLE_CLIENT_ID / GF_AUTH_GOOGLE_CLIENT_SECRET) for its native +# auth.google sign-in flow. Companion to: +# - deploy/gcp/gke/grafana-google-oidc-secret.example.yaml (envsubst template) +# - monitoring/kps-values-gke.yaml (Helm overrides for grafana subchart) +# +# Auth model: +# - Anyone in the @knoey.com Workspace can sign in (auth.google.allowed_domains). +# - chrisfu@knoey.com + ron@knoey.com get Admin (role_attribute_path JMESPath). +# - Everyone else @knoey.com gets Editor. +# +# Usage: +# ./etc/init_grafana_oauth.sh +# +# Env vars (resolved from etc/secrets/* if not set in the shell): +# GRAFANA_GOOGLE_CLIENT_ID ← from etc/secrets/grafana-google-oidc-client-id +# GRAFANA_GOOGLE_CLIENT_SECRET ← from etc/secrets/grafana-google-oidc-client-secret +# +# Optional: +# APP_CLUSTER_KUBECONTEXT (default: $KUBECONTEXT then ambient) +# NAMESPACE (default: monitoring) +# +# Pre-reqs: +# - OAuth 2.0 client created at GCP Console (see the secret template +# deploy/gcp/gke/grafana-google-oidc-secret.example.yaml for the +# exact authorized redirect URI + consent screen settings). +# - Two values saved into etc/secrets/grafana-google-oidc-client-{id,secret} +# (chmod 0600 each; etc/secrets/ is gitignored except for .keep). +# +# After this script runs and the Secret is in place, the next `helm upgrade` +# (or kubectl-apply of the chart's rendered manifest) of kps-grafana picks +# up the Secret via `envFromSecret: grafana-google-oidc`. Verify with: +# kubectl --context=$APP_CLUSTER_KUBECONTEXT -n monitoring exec -it kps-grafana-0 -c grafana -- \ +# env | grep GF_AUTH_GOOGLE_ + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +GKE_MANIFEST_DIR="$REPO_ROOT/deploy/gcp/gke" + +NAMESPACE="${NAMESPACE:-monitoring}" +KCTX="${APP_CLUSTER_KUBECONTEXT:-${KUBECONTEXT:-}}" + +if [[ -n "$KCTX" ]]; then + KCTX_FLAG=(--context="$KCTX") +else + KCTX_FLAG=() +fi + +log() { printf "[%s] %s\n" "$(date +%H:%M:%S)" "$*"; } +die() { log "ERROR: $*" >&2; exit 1; } + +resolve_secret() { + local var="$1" file="$2" val="${!1:-}" + if [[ -z "$val" && -f "$REPO_ROOT/etc/secrets/$file" ]]; then + val="$(cat "$REPO_ROOT/etc/secrets/$file")" + fi + if [[ -z "$val" ]]; then + die "missing $var (set the env var, or save the value into etc/secrets/$file)" + fi + printf '%s' "$val" +} + +for tool in kubectl envsubst; do + command -v "$tool" >/dev/null 2>&1 || die "required tool not found: $tool" +done + +GRAFANA_GOOGLE_CLIENT_ID="$(resolve_secret GRAFANA_GOOGLE_CLIENT_ID grafana-google-oidc-client-id)" +GRAFANA_GOOGLE_CLIENT_SECRET="$(resolve_secret GRAFANA_GOOGLE_CLIENT_SECRET grafana-google-oidc-client-secret)" +export GRAFANA_GOOGLE_CLIENT_ID GRAFANA_GOOGLE_CLIENT_SECRET + +SECRET_TMPL="$GKE_MANIFEST_DIR/grafana-google-oidc-secret.example.yaml" +[[ -f "$SECRET_TMPL" ]] || die "missing manifest: $SECRET_TMPL" + +log "==> grafana google-oauth bootstrap" +log " namespace : $NAMESPACE" +log " kubectx : ${KCTX:-}" + +log "Applying grafana-google-oidc Secret ..." +envsubst '${GRAFANA_GOOGLE_CLIENT_ID} ${GRAFANA_GOOGLE_CLIENT_SECRET}' \ + < "$SECRET_TMPL" \ + | kubectl "${KCTX_FLAG[@]}" -n "$NAMESPACE" apply -f - + +log "==> grafana google-oauth secret applied." +echo "" +echo " Next steps:" +echo " 1. Apply the Helm values override at monitoring/kps-values-gke.yaml:" +echo " helm upgrade --reuse-values kps prometheus-community/kube-prometheus-stack \\" +echo " --namespace $NAMESPACE \\" +echo " -f $REPO_ROOT/monitoring/kps-values-gke.yaml" +echo " 2. Patch knoe-svc-kong-config to add the /grafana route (see" +echo " deploy/opentofu/k3s/manifests/knoe/kong-configmap.yaml for the canonical source)." +echo " 3. Restart Kong: kubectl rollout restart deployment/knoe-svc-kong -n knoe-system" +echo " 4. Browser-test: https://svc.knoe.dev/grafana → Google sign-in → Grafana" +echo "" diff --git a/etc/init_monitoring.sh b/etc/init_monitoring.sh index 4dab40c..c11c554 100755 --- a/etc/init_monitoring.sh +++ b/etc/init_monitoring.sh @@ -701,7 +701,18 @@ apply_grafana_dashboard() { local tmp tmp=$(mktemp) - sed "s/\\\${DS_PROMETHEUS}/${prom_uid}/g" "$dashboard_path" > "$tmp" + # Pass the dashboard JSON through unchanged. The source JSON templates the + # datasource UID via the dashboard's own `DS_PROMETHEUS` variable (resolved + # at render time by Grafana, configurable per-user via the Datasource + # dropdown). The earlier sed-substitution to a fixed `prom_uid` baked + # `uid: prometheus` into every variable + panel definition, defeating that + # template — the dashboard then queried only the default Prometheus + # datasource regardless of what the user selected. With cross-cluster + # CNPG metrics on `cnpg-prometheus` (commit 09f2c1a), passing through + # unchanged is required for variable dropdowns + panels to follow the + # Datasource selector. + cp "$dashboard_path" "$tmp" + : "$prom_uid" # silence unused-var warning local cm_yaml cm_yaml=$(mktemp) @@ -715,35 +726,19 @@ apply_grafana_dashboard() { } apply_grafana_datasource() { - local ns="$1" - local prom_uid="${GRAFANA_PROMETHEUS_DATASOURCE_UID:-prometheus}" - local prom_url="${GRAFANA_PROMETHEUS_DATASOURCE_URL:-http://kps-kube-prometheus-stack-prometheus.${ns}.svc.cluster.local:9090}" - - local cm_yaml - cm_yaml=$(mktemp) - cat > "$cm_yaml" </dev/null - rm -f "$cm_yaml" + # No-op: kube-prometheus-stack already provisions the Prometheus + Alertmanager + # datasources via its own ConfigMap (`-grafana-datasource`). The + # earlier extra `knoe-grafana-datasource` ConfigMap created here was a + # duplicate of that — it registered the same `uid: prometheus`, which collided + # with the chart's datasource and caused Grafana's provisioning reload to + # error out (HTTP 500 → datasources never refreshed). + # + # Additional datasources (e.g. the cross-cluster `cnpg-prometheus` pointing + # at the DB-cluster Prometheus) are wired through the chart values at + # `monitoring/kps-values-gke.yaml` `grafana.additionalDataSources`, NOT + # through this script. Keeping this function as a no-op so call sites don't + # need to change. + return 0 } helm_release_status() { diff --git a/etc/init_oauth2_proxy.sh b/etc/init_oauth2_proxy.sh new file mode 100755 index 0000000..4d5e433 --- /dev/null +++ b/etc/init_oauth2_proxy.sh @@ -0,0 +1,127 @@ +#!/usr/bin/env bash +# init_oauth2_proxy.sh +# +# Bootstrap the oauth2-proxy gate in front of Supabase Studio at +# db.0.knoe.dev. Gates access via Google Workspace OIDC (knoey.com) so any +# @knoey.com identity (chrisfu, ron) can sign in and share the Studio +# session. Outside-domain users are rejected at this layer. +# +# Usage: +# ./etc/init_oauth2_proxy.sh +# +# Env vars (resolved from etc/secrets/* if not set in the shell): +# OAUTH2_PROXY_CLIENT_ID ← from etc/secrets/oauth2-proxy-client-id +# OAUTH2_PROXY_CLIENT_SECRET ← from etc/secrets/oauth2-proxy-client-secret +# OAUTH2_PROXY_COOKIE_SECRET ← from etc/secrets/oauth2-proxy-cookie-secret +# +# Optional: +# APP_CLUSTER_KUBECONTEXT (default: $KUBECONTEXT then ambient) +# NAMESPACE (default: supabase) +# +# Pre-reqs: +# - OAuth 2.0 client created at GCP Console (see the secret template +# deploy/gcp/gke/oauth2-proxy-google-oidc-secret.example.yaml for the +# exact authorized redirect URI + consent screen settings). +# - cookie_secret generated with: openssl rand -base64 32 +# - Three values saved into etc/secrets/oauth2-proxy-{client-id,client-secret,cookie-secret} +# (chmod 0600 each; etc/secrets/ is gitignored except for .keep). +# +# After this script runs and the oauth2-proxy Deployment is Ready, two +# manual steps complete the wiring (NOT done by this script — see the plan +# in docs/plans/ for the full sequence): +# +# 1. Patch the supabase-kong Ingress to route db.0.knoe.dev through +# oauth2-proxy:80 instead of supabase-kong:8000: +# +# kubectl --context=$APP_CLUSTER_KUBECONTEXT -n supabase patch ingress \ +# supabase-kong --type=json -p '[ +# {"op": "replace", +# "path": "/spec/rules/1/http/paths/0/backend/service/name", +# "value": "oauth2-proxy"}, +# {"op": "replace", +# "path": "/spec/rules/1/http/paths/0/backend/service/port/number", +# "value": 80} +# ]' +# (verify the index by checking which rule has host=db.0.knoe.dev first; +# index may shift on future Helm reconciles) +# +# 2. Remove the basic-auth plugin from the dashboard route in the +# supabase-kong configmap (oauth2-proxy is the gate now; double-auth is +# friction). Then rollout-restart supabase-kong. +# +# When knoe-auth Round 1 ships an OIDC OP at https://api.knoe.dev/auth, change +# `--provider=google` to `--provider=oidc --oidc-issuer-url=https://api.knoe.dev/auth` +# in deploy/gcp/gke/oauth2-proxy-deployment.yaml and re-run this script. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +GKE_MANIFEST_DIR="$REPO_ROOT/deploy/gcp/gke" + +NAMESPACE="${NAMESPACE:-supabase}" +KCTX="${APP_CLUSTER_KUBECONTEXT:-${KUBECONTEXT:-}}" + +if [[ -n "$KCTX" ]]; then + KCTX_FLAG=(--context="$KCTX") +else + KCTX_FLAG=() +fi + +log() { printf "[%s] %s\n" "$(date +%H:%M:%S)" "$*"; } +die() { log "ERROR: $*" >&2; exit 1; } + +resolve_secret() { + # Resolve a value from env (preferred) or etc/secrets/. + local var="$1" file="$2" val="${!1:-}" + if [[ -z "$val" && -f "$REPO_ROOT/etc/secrets/$file" ]]; then + val="$(cat "$REPO_ROOT/etc/secrets/$file")" + fi + if [[ -z "$val" ]]; then + die "missing $var (set the env var, or save the value into etc/secrets/$file)" + fi + printf '%s' "$val" +} + +for tool in kubectl envsubst; do + command -v "$tool" >/dev/null 2>&1 || die "required tool not found: $tool" +done + +OAUTH2_PROXY_CLIENT_ID="$(resolve_secret OAUTH2_PROXY_CLIENT_ID oauth2-proxy-client-id)" +OAUTH2_PROXY_CLIENT_SECRET="$(resolve_secret OAUTH2_PROXY_CLIENT_SECRET oauth2-proxy-client-secret)" +OAUTH2_PROXY_COOKIE_SECRET="$(resolve_secret OAUTH2_PROXY_COOKIE_SECRET oauth2-proxy-cookie-secret)" +export OAUTH2_PROXY_CLIENT_ID OAUTH2_PROXY_CLIENT_SECRET OAUTH2_PROXY_COOKIE_SECRET + +SECRET_TMPL="$GKE_MANIFEST_DIR/oauth2-proxy-google-oidc-secret.example.yaml" +DEPLOY_MANIFEST="$GKE_MANIFEST_DIR/oauth2-proxy-deployment.yaml" + +[[ -f "$SECRET_TMPL" ]] || die "missing manifest: $SECRET_TMPL" +[[ -f "$DEPLOY_MANIFEST" ]] || die "missing manifest: $DEPLOY_MANIFEST" + +log "==> oauth2-proxy bootstrap" +log " namespace : $NAMESPACE" +log " kubectx : ${KCTX:-}" + +log "Applying oauth2-proxy-google-oidc secret ..." +envsubst '${OAUTH2_PROXY_CLIENT_ID} ${OAUTH2_PROXY_CLIENT_SECRET} ${OAUTH2_PROXY_COOKIE_SECRET}' \ + < "$SECRET_TMPL" \ + | kubectl "${KCTX_FLAG[@]}" -n "$NAMESPACE" apply -f - + +log "Applying oauth2-proxy ServiceAccount + BackendConfig + Service + Deployment ..." +kubectl "${KCTX_FLAG[@]}" -n "$NAMESPACE" apply -f "$DEPLOY_MANIFEST" + +log "Waiting for oauth2-proxy Deployment to become Ready (timeout 180s) ..." +kubectl "${KCTX_FLAG[@]}" -n "$NAMESPACE" rollout status deployment/oauth2-proxy --timeout=180s + +log "==> oauth2-proxy bootstrap complete." +echo "" +echo " Next steps (NOT performed by this script):" +echo " 1. Patch the supabase-kong Ingress so db.0.knoe.dev routes to" +echo " oauth2-proxy:80 instead of supabase-kong:8000." +echo " 2. Remove the basic-auth plugin from the dashboard route in the" +echo " supabase-kong configmap, then rollout-restart supabase-kong." +echo " 3. In a browser, sign in to https://db.0.knoe.dev/ with a" +echo " @knoey.com Google account. Try a non-knoey account too — should" +echo " receive 403 from oauth2-proxy." +echo "" +echo " See the active plan in ~/.claude/plans/ for the exact patch commands." diff --git a/etc/knoe-db-ca.crt b/etc/knoe-db-ca.crt new file mode 100644 index 0000000..0ae0ad3 --- /dev/null +++ b/etc/knoe-db-ca.crt @@ -0,0 +1,11 @@ +-----BEGIN CERTIFICATE----- +MIIBiTCCAS+gAwIBAgIUF+swDNndu9C+lNgcFIjTTHfF9VcwCgYIKoZIzj0EAwIw +GjEYMBYGA1UEAwwPa25vZS1kYiBDTlBHIENBMB4XDTI2MDQxMDEzNDEwMVoXDTI3 +MDQxMDEzNDEwMVowGjEYMBYGA1UEAwwPa25vZS1kYiBDTlBHIENBMFkwEwYHKoZI +zj0CAQYIKoZIzj0DAQcDQgAEzIAAV0RyJydcIsGGwEzgC/BIISmDowBray5pD7LG +9bUwtGv6m5XaXHNGIv/shAW4E5D2ioHIWFBJtjwfS/pe4qNTMFEwHQYDVR0OBBYE +FOzKPWn1lbqXOrCfo/fDTHV+aDSkMB8GA1UdIwQYMBaAFOzKPWn1lbqXOrCfo/fD +THV+aDSkMA8GA1UdEwEB/wQFMAMBAf8wCgYIKoZIzj0EAwIDSAAwRQIhAPrg+W+T +kRX5jGPDwFQZJHtaI/H1qhzEYUwWVsjnDFdoAiAB29iKZOFyIjaEPbYdixNRqU8p +u8MAWBsjGNpS/SaLJA== +-----END CERTIFICATE----- diff --git a/etc/onboard_engineer.sh b/etc/onboard_engineer.sh new file mode 100755 index 0000000..2d2052c --- /dev/null +++ b/etc/onboard_engineer.sh @@ -0,0 +1,237 @@ +#!/usr/bin/env bash +# onboard_engineer.sh — provision a per-engineer postgres role + 24h temp +# password, then emit a one-time onboarding URL (and QR-code rendering of it) +# that the new engineer redeems at https://db.0.knoe.dev/onboard.html. +# +# This is the Phase 1 bridge while Junie's Phase 2 (libpq OAUTHBEARER) is in +# flight. When OAUTHBEARER lands, the temp-password mechanism goes away and +# this script's role-creation step is replaced by an INSERT into +# `knoe.oauth_role_map`. The engineer-facing URL stays the same (the page just +# stops showing a password and instead displays the OAUTH connection string). +# +# Usage: +# ./etc/onboard_engineer.sh +# ./etc/onboard_engineer.sh --revoke +# +# Env (resolved from knoe_cfg.sh / KUBECONTEXT etc): +# DB_CLUSTER_KUBECONTEXT — the CNPG cluster context (knoe-dev-cnpg-0) +# DB_NAMESPACE — defaults to knoe-db-0 +# DB_PRIMARY_POD — auto-discovered if unset +# ONBOARD_HOST — defaults to db.0.knoe.dev +# ONBOARD_TTL_HOURS — defaults to 24 + +set -euo pipefail + +SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO_ROOT=$(cd "$SCRIPT_DIR/.." && pwd) + +DB_CTX="${DB_CLUSTER_KUBECONTEXT:-${KUBECONTEXT:-gke_plenary-truck-485623-p7_us-west3_knoe-dev-cnpg-0}}" +DB_NS="${DB_NAMESPACE:-knoe-db-0}" +ONBOARD_HOST="${ONBOARD_HOST:-db.0.knoe.dev}" +ONBOARD_TTL_HOURS="${ONBOARD_TTL_HOURS:-24}" + +log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*"; } +warn() { printf '\033[33m[!]\033[0m %s\n' "$*" >&2; } +err() { printf '\033[31m[ERROR]\033[0m %s\n' "$*" >&2; exit 1; } + +usage() { + cat < # provision new engineer + $(basename "$0") --revoke # drop role + cancel access + $(basename "$0") --rotate # generate a fresh 24h temp password + +Examples: + $(basename "$0") ron ron@knoey.com + $(basename "$0") --revoke test-user +EOF + exit 1 +} + +primary_pod() { + if [[ -n "${DB_PRIMARY_POD:-}" ]]; then printf '%s' "$DB_PRIMARY_POD"; return; fi + kubectl --context="$DB_CTX" -n "$DB_NS" get pod \ + -l cnpg.io/cluster=knoe-db,cnpg.io/instanceRole=primary \ + -o jsonpath='{.items[0].metadata.name}' +} + +run_psql() { + local sql="$1" + local pod + pod=$(primary_pod) + printf '%s' "$sql" | kubectl --context="$DB_CTX" -n "$DB_NS" exec -i "$pod" -c postgres -- \ + psql -U postgres -d postgres -v ON_ERROR_STOP=1 -X --quiet 2>&1 +} + +ensure_dependencies() { + command -v kubectl >/dev/null 2>&1 || err "kubectl not found" + command -v openssl >/dev/null 2>&1 || err "openssl not found" + command -v base64 >/dev/null 2>&1 || err "base64 not found" + if ! command -v qrencode >/dev/null 2>&1; then + warn "qrencode not installed — output will be URL-only (brew install qrencode for QR rendering)" + fi +} + +current_iso8601() { + date -u "+%Y-%m-%dT%H:%M:%SZ" +} + +ttl_iso8601() { + # macOS date and GNU date have different flag syntax; both ISO 8601 a + # given offset-from-now expressed in hours. Use python as a portable fallback. + python3 - < provisioning postgres role: $user (email: $email, valid for $ttl_hint)" + + # SQL is idempotent: works for fresh CREATE and re-onboarding. + local sql + sql=$(cat < onboarding link" + printf '\n %s\n\n' "$url" + + if command -v qrencode >/dev/null 2>&1; then + log "==> QR code (scan with engineer's phone camera)" + printf '\n' + qrencode -t ANSI256 -m 1 "$url" + printf '\n' + fi + + log "==> engineer's plaintext password (fallback if URL is unusable)" + printf '\n %s\n\n' "$pw" + + cat < \\password +================================================================== + +EOF +} + +action_revoke() { + local user="$1" + validate_username "$user" + log "==> revoking role: $user" + + local sql + sql=$(cat < revocation complete" +} + +action_rotate() { + local user="$1" + validate_username "$user" + + local pw exp pw_url url + pw=$(openssl rand -base64 24) + exp=$(ttl_iso8601) + + log "==> rotating temp password for $user (valid ${ONBOARD_TTL_HOURS}h)" + local sql + sql=$(cat </dev/null 2>&1; then + qrencode -t ANSI256 -m 1 "$url"; printf '\n' + fi + printf '\n plaintext: %s\n\n' "$pw" +} + +# --- main --- +ensure_dependencies + +case "${1:-}" in + ""|"-h"|"--help") usage ;; + "--revoke") [[ $# -eq 2 ]] || usage; action_revoke "$2" ;; + "--rotate") [[ $# -eq 2 ]] || usage; action_rotate "$2" ;; + *) [[ $# -eq 2 ]] || usage; action_provision "$1" "$2" ;; +esac diff --git a/etc/sync_cnpg_grafana_dashboard.py b/etc/sync_cnpg_grafana_dashboard.py new file mode 100755 index 0000000..5fb7609 --- /dev/null +++ b/etc/sync_cnpg_grafana_dashboard.py @@ -0,0 +1,418 @@ +#!/usr/bin/env python3 +""" +Self-updating sync tool for the cloudnative-pg Grafana dashboard. + +Designed for unattended / background-agent runs: fetches the upstream +dashboard JSON, applies the transformations declared in +`monitoring/cnpg-dashboard-transforms.yaml`, writes the result to +`knoe-db/grafana-dashboard.json` (the canonical source the +`knoe-db-grafana-dashboard` ConfigMap is built from), and optionally +applies the ConfigMap to a live cluster + reloads Grafana provisioning. + +Usage: + + # Default: fetch upstream, apply transforms, write to disk if changed. + # Exit 0 = no change, 1 = updated, 2 = error. + ./etc/sync_cnpg_grafana_dashboard.py + + # Dry run — print what would change, don't write anything. + ./etc/sync_cnpg_grafana_dashboard.py --dry-run + + # Drift check — exit 0 if our committed copy matches the upstream-after- + # transforms, 1 if it has drifted (newer upstream waiting to be merged + # OR a local hand-edit that the transforms don't cover). Doesn't write. + ./etc/sync_cnpg_grafana_dashboard.py --check + + # Apply to live cluster after writing (rolls Grafana provisioning). + ./etc/sync_cnpg_grafana_dashboard.py --apply --context $APP_CLUSTER_KUBECONTEXT + +Exit codes: + 0 no change (output matches transformed upstream) + 1 output updated (or would-be-updated in --dry-run / --check) + 2 error (network, parse, transform mismatch, kubectl failure, etc.) + +Background-agent run pattern (cron / k8s CronJob): + 1. Run with `--check` once. If exit 1, work to do. + 2. Run with `--apply --context=` to update + roll out. + 3. Optionally `git commit -m 'sync(cnpg): ...'` to persist the file change. + +Transformation philosophy: edits live in +`monitoring/cnpg-dashboard-transforms.yaml`, not in this tool. Adding a +new transformation type means extending `Transformer.apply()` here AND +documenting it in the transforms YAML's preamble. Today there's just one +type, `regex_replace`. +""" + +from __future__ import annotations + +import argparse +import base64 +import difflib +import hashlib +import json +import logging +import os +import re +import subprocess +import sys +import tempfile +import urllib.error +import urllib.request +from pathlib import Path + +try: + import yaml +except ImportError: + print( + "PyYAML required. Install via: pip install pyyaml (or apt install python3-yaml)", + file=sys.stderr, + ) + sys.exit(2) + +# -------------------------------------------------------------------------- +# Constants + +REPO_ROOT = Path(__file__).resolve().parent.parent +DEFAULT_TRANSFORMS = REPO_ROOT / "monitoring" / "cnpg-dashboard-transforms.yaml" +DEFAULT_OUTPUT = REPO_ROOT / "knoe-db" / "grafana-dashboard.json" +DEFAULT_CM_NAME = "knoe-db-grafana-dashboard" +DEFAULT_GRAFANA_STS = "kps-grafana" +DEFAULT_GRAFANA_SECRET = "kps-grafana" + +EXIT_NOCHANGE = 0 +EXIT_CHANGED = 1 +EXIT_ERROR = 2 + +logger = logging.getLogger("cnpg-dashboard-sync") + + +# -------------------------------------------------------------------------- +# Helpers + +def sha256_short(s: str) -> str: + return hashlib.sha256(s.encode("utf-8")).hexdigest()[:12] + + +def fetch_upstream(url: str, timeout: int = 30) -> str: + """Download the upstream dashboard JSON. Raises urllib.error on HTTP failure.""" + logger.info("fetching upstream: %s", url) + req = urllib.request.Request(url, headers={"User-Agent": "knoe-cnpg-dashboard-sync/1"}) + with urllib.request.urlopen(req, timeout=timeout) as r: + return r.read().decode("utf-8") + + +def load_transforms(path: Path) -> dict: + if not path.exists(): + raise FileNotFoundError(f"transforms file not found: {path}") + with path.open() as f: + data = yaml.safe_load(f) + if not isinstance(data, dict) or "source" not in data: + raise ValueError(f"transforms file missing required `source` block: {path}") + return data + + +# -------------------------------------------------------------------------- +# Transformer + +class TransformError(Exception): + pass + + +class Transformer: + """Applies an ordered list of transformations to a JSON string.""" + + def __init__(self, transforms: list[dict]): + self.transforms = transforms + + def apply(self, content: str) -> tuple[str, list[dict]]: + """Apply all transforms; return (transformed_content, summaries). + + Each summary is a dict {name, type, matches, ...} for logging / drift + detection. + """ + summaries: list[dict] = [] + for t in self.transforms: + name = t.get("name", "") + ttype = t.get("type") + try: + if ttype == "regex_replace": + content, summary = self._apply_regex(content, t) + elif ttype == "set_template_variable_default": + content, summary = self._apply_set_template_default(content, t) + else: + raise TransformError(f"unknown transformation type: {ttype!r}") + except TransformError: + raise + except Exception as e: + raise TransformError(f"transform {name!r} failed: {e}") from e + summary["name"] = name + summary["type"] = ttype + summaries.append(summary) + return content, summaries + + @staticmethod + def _apply_regex(content: str, t: dict) -> tuple[str, dict]: + pattern = t["pattern"] + replacement = t["replacement"] + compiled = re.compile(pattern) + new_content, n = compiled.subn(replacement, content) + + emin = t.get("expected_min") + emax = t.get("expected_max") + warn = None + if emin is not None and n < emin: + warn = f"matched {n} < expected_min {emin} — upstream may have moved on" + elif emax is not None and n > emax: + warn = f"matched {n} > expected_max {emax} — upstream changed shape" + + return new_content, {"matches": n, "warning": warn} + + @staticmethod + def _apply_set_template_default(content: str, t: dict) -> tuple[str, dict]: + """Set the `current` value of a template variable. + + Useful for pinning DS_PROMETHEUS (or any other variable) to a specific + value so the dashboard renders correctly on first load without the user + needing to override via URL parameters or the variable picker. + + Spec: + type: set_template_variable_default + variable: DS_PROMETHEUS # required, name of variable + value: cnpg-prometheus # required, current.value + text: cnpg-prometheus # optional, defaults to value + selected: true # optional, defaults to true + """ + var_name = t["variable"] + value = t["value"] + text = t.get("text", value) + selected = t.get("selected", True) + + data = json.loads(content) + templating = data.get("templating", {}).get("list", []) + found = False + for v in templating: + if v.get("name") == var_name: + v["current"] = {"selected": selected, "text": text, "value": value} + found = True + break + if not found: + warn = f"variable {var_name!r} not found in dashboard.templating.list" + return content, {"matches": 0, "warning": warn} + + # Re-serialize with the same format the upstream JSON uses (2-space indent, + # which is what `json.dumps(..., indent=2)` produces; matches what we get + # from the upstream so diffs stay clean). + return json.dumps(data, indent=2), {"matches": 1, "warning": None} + + +# -------------------------------------------------------------------------- +# Cluster apply + +def kubectl(args: list[str], context: str, namespace: str | None = None, + stdin: bytes | None = None, capture: bool = False) -> bytes: + """Run kubectl with explicit context (no ambient context drift).""" + cmd = ["kubectl", "--context", context] + if namespace: + cmd += ["-n", namespace] + cmd += args + logger.debug("running: %s", " ".join(cmd)) + if capture: + return subprocess.check_output(cmd, input=stdin) + subprocess.run(cmd, input=stdin, check=True) + return b"" + + +def apply_to_cluster(json_content: str, context: str, namespace: str = "monitoring", + cm_name: str = DEFAULT_CM_NAME) -> None: + """Apply the dashboard JSON as a ConfigMap + reload Grafana provisioning. + + Uses --server-side apply because the JSON is large enough to hit the 256KB + last-applied-configuration annotation limit on client-side apply. + """ + if not context: + raise ValueError("--apply requires --context (or APP_CLUSTER_KUBECONTEXT env)") + + logger.info("applying ConfigMap %s/%s in context %s", namespace, cm_name, context) + + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f: + f.write(json_content) + tmp = f.name + try: + # Render the cm yaml via `kubectl create --dry-run` + cm_yaml = kubectl( + ["create", "configmap", cm_name, + f"--from-file=knoe-db.json={tmp}", + "--dry-run=client", "-o", "yaml"], + context=context, namespace=namespace, capture=True, + ) + # Apply server-side + kubectl( + ["apply", "--server-side", "--force-conflicts", + "--field-manager=cnpg-dashboard-sync", "-f", "-"], + context=context, namespace=namespace, stdin=cm_yaml, + ) + # Sidecar discovers grafana_dashboard=1 labelled cms; ensure label is present + kubectl( + ["label", "configmap", cm_name, "grafana_dashboard=1", "--overwrite"], + context=context, namespace=namespace, + ) + finally: + os.unlink(tmp) + + # Reload Grafana provisioning so the new dashboard JSON is picked up + # without waiting for the sidecar's poll interval. + logger.info("reloading Grafana provisioning") + admin_pw_b64 = kubectl( + ["get", "secret", DEFAULT_GRAFANA_SECRET, + "-o", "jsonpath={.data.admin-password}"], + context=context, namespace=namespace, capture=True, + ) + admin_pw = base64.b64decode(admin_pw_b64).decode() + kubectl( + ["exec", f"sts/{DEFAULT_GRAFANA_STS}", "-c", "grafana", "--", + "wget", "-qO-", "--post-data", "", + f"http://admin:{admin_pw}@localhost:3000/api/admin/provisioning/dashboards/reload"], + context=context, namespace=namespace, capture=True, + ) + logger.info("grafana provisioning reload complete") + + +# -------------------------------------------------------------------------- +# Main + +def render_diff(old: str, new: str, max_lines: int = 60) -> str: + diff = list(difflib.unified_diff( + old.splitlines(keepends=True), + new.splitlines(keepends=True), + fromfile="committed", tofile="upstream-transformed", n=2, + )) + if len(diff) > max_lines: + diff = diff[:max_lines] + [f"... ({len(diff) - max_lines} more lines elided)\n"] + return "".join(diff) + + +def main(argv: list[str] | None = None) -> int: + p = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + p.add_argument("--transforms", type=Path, default=DEFAULT_TRANSFORMS, + help="Path to the transforms YAML (default: %(default)s)") + p.add_argument("--output", type=Path, default=DEFAULT_OUTPUT, + help="Path to write the transformed JSON (default: %(default)s)") + mode = p.add_mutually_exclusive_group() + mode.add_argument("--dry-run", action="store_true", + help="Show what would change; don't write or apply.") + mode.add_argument("--check", action="store_true", + help="Exit 1 if output is drifted from upstream-transformed; don't write.") + p.add_argument("--apply", action="store_true", + help="After writing, apply the ConfigMap to the live cluster and reload Grafana.") + p.add_argument("--context", default=os.environ.get("APP_CLUSTER_KUBECONTEXT", ""), + help="kubectl context for --apply (default: $APP_CLUSTER_KUBECONTEXT)") + p.add_argument("--namespace", default="monitoring", + help="ConfigMap namespace for --apply (default: %(default)s)") + p.add_argument("--show-diff", action="store_true", + help="Print a unified diff of committed-vs-transformed.") + p.add_argument("-v", "--verbose", action="store_true", + help="Verbose logging.") + args = p.parse_args(argv) + + logging.basicConfig( + level=logging.DEBUG if args.verbose else logging.INFO, + format="%(asctime)s %(levelname)s %(name)s: %(message)s", + datefmt="%H:%M:%S", + ) + + try: + config = load_transforms(args.transforms) + except (FileNotFoundError, ValueError) as e: + logger.error("failed to load transforms: %s", e) + return EXIT_ERROR + + src = config["source"] + upstream_url = src["url"] + expected_uid = src.get("expected_uid") + transforms = config.get("transformations", []) + + # 1. Fetch + try: + upstream = fetch_upstream(upstream_url) + except urllib.error.URLError as e: + logger.error("upstream fetch failed: %s", e) + return EXIT_ERROR + logger.info("upstream: sha256=%s, %d bytes", sha256_short(upstream), len(upstream)) + + # 2. Validate UID (catch silent renames) + if expected_uid: + try: + uid = json.loads(upstream).get("uid") + except json.JSONDecodeError as e: + logger.error("upstream is not valid JSON: %s", e) + return EXIT_ERROR + if uid != expected_uid: + logger.error("upstream uid %r != expected %r — refusing to proceed; " + "dashboard URLs would silently break. Update transforms YAML if intentional.", + uid, expected_uid) + return EXIT_ERROR + + # 3. Transform + try: + transformed, summaries = Transformer(transforms).apply(upstream) + except TransformError as e: + logger.error("transform failed: %s", e) + return EXIT_ERROR + + logger.info("applied %d transformation(s):", len(summaries)) + saw_warning = False + for s in summaries: + msg = f" - {s['name']} ({s['type']}): {s.get('matches', '?')} match(es)" + if s.get("warning"): + msg += f" [WARNING: {s['warning']}]" + saw_warning = True + logger.info(msg) + logger.info("transformed: sha256=%s, %d bytes", sha256_short(transformed), len(transformed)) + + # 4. Compare to existing + existing = args.output.read_text() if args.output.exists() else "" + drifted = (existing != transformed) + + if args.show_diff and drifted: + logger.info("DIFF (committed vs transformed-upstream):") + sys.stderr.write(render_diff(existing, transformed)) + + # 5. Act per mode + if args.check: + if drifted: + logger.warning("DRIFT: %s differs from upstream after transforms (run without --check to update)", args.output) + return EXIT_CHANGED + logger.info("OK: %s matches upstream after transforms", args.output) + return EXIT_ERROR if saw_warning else EXIT_NOCHANGE + + if args.dry_run: + if drifted: + logger.info("WOULD UPDATE: %s (run without --dry-run to apply)", args.output) + return EXIT_CHANGED + logger.info("NO CHANGE: %s", args.output) + return EXIT_ERROR if saw_warning else EXIT_NOCHANGE + + # Default mode: write if drifted, optionally apply + if drifted: + args.output.write_text(transformed) + logger.info("wrote %s", args.output) + rc = EXIT_CHANGED + else: + logger.info("no change to %s", args.output) + rc = EXIT_NOCHANGE + + if args.apply: + try: + apply_to_cluster(transformed, context=args.context, namespace=args.namespace) + except (subprocess.CalledProcessError, ValueError) as e: + logger.error("cluster apply failed: %s", e) + return EXIT_ERROR + + return EXIT_ERROR if saw_warning else rc + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/k8s/knoe/knoe-db-barman-objectstore-gcs.yaml b/k8s/knoe/knoe-db-barman-objectstore-gcs.yaml index b81b9ff..efb3895 100644 --- a/k8s/knoe/knoe-db-barman-objectstore-gcs.yaml +++ b/k8s/knoe/knoe-db-barman-objectstore-gcs.yaml @@ -2,17 +2,24 @@ apiVersion: barmancloud.cnpg.io/v1 kind: ObjectStore metadata: name: knoe-db-barman-objectstore - # Applied to knoe-cnpg-0 in namespace knoe-db-0. - # Auth: Workload Identity — cnpg-backup-sa (in knoe-db-0) is annotated with - # iam.gke.io/gcp-service-account: cnpg-backup@.iam.gserviceaccount.com - # No static credentials secret required. + # Applied to knoe-dev-cnpg-0 in namespace knoe-db-0. + # + # Auth: Workload Identity. Cluster pods run as cnpg-backup-sa, set via + # cluster.spec.serviceAccountName (requires CNPG >= v1.29.0). The SA is + # annotated with iam.gke.io/gcp-service-account= + # cnpg-backup@.iam.gserviceaccount.com. RoleBindings knoe-db and + # knoe-db-barman-cloud include cnpg-backup-sa as a subject so the pod has + # the RBAC needed to read the Cluster CR and the Backup CRs. + # + # The googleCredentials.gkeEnvironment: true block is REQUIRED by + # plugin-barman-cloud v0.12.0 — without it the plugin rejects the + # ObjectStore with "no credentials defined". spec: retentionPolicy: 30d configuration: destinationPath: gs://knoe-0-backups/ - # No googleCredentials block — barman-cloud uses the pod's Workload Identity - # token automatically when the ServiceAccount (cnpg-backup-sa in knoe-db-0) - # carries the iam.gke.io/gcp-service-account annotation set by init_cnpg_gke.sh. + googleCredentials: + gkeEnvironment: true wal: compression: gzip data: diff --git a/k8s/knoe/knoe-db.yaml b/k8s/knoe/knoe-db.yaml index ea68c5c..87226f8 100644 --- a/k8s/knoe/knoe-db.yaml +++ b/k8s/knoe/knoe-db.yaml @@ -57,7 +57,11 @@ spec: secret: name: knoe-db-user postInitTemplateSQL: - - CREATE EXTENSION IF NOT EXISTS pg_stat_statements; + # Supabase convention: relocatable extensions live in `extensions`, + # not `public`. Studio's Database Advisor flags `public.pg_stat_statements` + # as a Security warning on the dashboard. + - CREATE SCHEMA IF NOT EXISTS extensions; + - CREATE EXTENSION IF NOT EXISTS pg_stat_statements SCHEMA extensions; postInitSQL: - DO $$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'knoe') THEN CREATE ROLE knoe LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT; END IF; END $$; - DO $$ DECLARE owner_password text; BEGIN SELECT rolpassword INTO owner_password FROM pg_authid WHERE rolname = 'knoe'; IF owner_password IS NOT NULL THEN EXECUTE format('ALTER ROLE knoe PASSWORD %L', owner_password); END IF; END $$; diff --git a/knoe-db/grafana-dashboard.json b/knoe-db/grafana-dashboard.json index 3791d44..555e65c 100644 --- a/knoe-db/grafana-dashboard.json +++ b/knoe-db/grafana-dashboard.json @@ -276,14 +276,14 @@ { "options": { "0": { - "color": "red", - "index": 2, - "text": "None" - }, - "1": { - "color": "orange", + "color": "green", "index": 1, - "text": "Degraded" + "text": "Healthy" + }, + "-1": { + "color": "red", + "index": 0, + "text": "None" } }, "type": "value" @@ -292,9 +292,9 @@ "options": { "from": 2, "result": { - "color": "green", - "index": 0, - "text": "Healthy" + "color": "orange", + "index": 2, + "text": "Degraded" }, "to": 999 }, @@ -345,7 +345,7 @@ "uid": "${DS_PROMETHEUS}" }, "editorMode": "code", - "expr": "max(cnpg_pg_replication_streaming_replicas{namespace=~\"$namespace\", pod=~\"$instances\"} - cnpg_pg_replication_is_wal_receiver_up{namespace=~\"$namespace\", pod=~\"$instances\"})", + "expr": "(max(cnpg_pg_replication_streaming_replicas{namespace=~\"$namespace\", pod=~\"$instances\"}) - sum(cnpg_pg_replication_is_wal_receiver_up{namespace=~\"$namespace\", pod=~\"$instances\"})) + (clamp_max(max(cnpg_pg_replication_streaming_replicas{namespace=~\"$namespace\", pod=~\"$instances\"}), 1) - 1)", "legendFormat": "Replication", "range": true, "refId": "A" @@ -757,7 +757,7 @@ }, "editorMode": "code", "exemplar": true, - "expr": "sum(rate(cnpg_pg_stat_database_xact_commit{namespace=~\"$namespace\",pod=~\"$instances\"}[$__interval])) + sum(rate(cnpg_pg_stat_database_xact_rollback{namespace=~\"$namespace\",pod=~\"$instances\"}[$__interval]))", + "expr": "sum(rate(cnpg_pg_stat_database_xact_commit{namespace=~\"$namespace\",pod=~\"$instances\"}[$__rate_interval])) + sum(rate(cnpg_pg_stat_database_xact_rollback{namespace=~\"$namespace\",pod=~\"$instances\"}[$__rate_interval]))", "interval": "", "legendFormat": "TPS", "range": true, @@ -1311,7 +1311,7 @@ }, "editorMode": "code", "exemplar": false, - "expr": "-(time() - max(cnpg_collector_last_available_backup_timestamp{namespace=\"$namespace\",pod=~\"$instances\"}))", + "expr": "-(time() - max({__name__=~\"cnpg_collector_last_available_backup_timestamp|barman_cloud_cloudnative_pg_io_last_available_backup_timestamp\",namespace=\"$namespace\",pod=~\"$instances\"}))", "instant": true, "legendFormat": "__auto", "range": false, @@ -1960,7 +1960,7 @@ "uid": "${DS_PROMETHEUS}" }, "editorMode": "code", - "expr": "time() - max(cnpg_collector_last_available_backup_timestamp{namespace=\"$namespace\", pod=~\"$instances\"})", + "expr": "time() - max({__name__=~\"cnpg_collector_last_available_backup_timestamp|barman_cloud_cloudnative_pg_io_last_available_backup_timestamp\",namespace=\"$namespace\", pod=~\"$instances\"})", "legendFormat": "Backups", "range": true, "refId": "BACKUPS" @@ -1973,7 +1973,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "Base Backups are considered healthy when there has been at least one base backup in the last 24 hours.", + "description": "WAL is considered Healthy when the last WAL is 0min to 6min old, Delayed when it is less than 15min and Unsynced for >15min.", "fieldConfig": { "defaults": { "color": { @@ -2235,11 +2235,11 @@ "uid": "${DS_PROMETHEUS}" }, "editorMode": "code", - "expr": "sum by (label_app_kubernetes_io_name) (kube_pod_status_ready{namespace=\"$operatorNamespace\"} * on (pod) group_left( label_app_kubernetes_io_name ) kube_pod_labels{label_app_kubernetes_io_name=~\"cloudnative-pg\"})", + "expr": "sum(kube_pod_status_ready{namespace=\"$operatorNamespace\", pod=~\"cloudnative-pg.+|cnpg-controller-manager.+\", condition=\"true\"})", "hide": false, - "instant": false, + "instant": true, "legendFormat": "Operator Status", - "range": true, + "range": false, "refId": "A" } ], @@ -2409,7 +2409,7 @@ "uid": "${DS_PROMETHEUS}" }, "editorMode": "code", - "expr": "clamp_max(max(controller_runtime_reconcile_total{namespace=~\"$operatorNamespace\", result=\"error\", controller=\"scheduledbackup\"}), 1)", + "expr": "clamp_max(max(controller_runtime_reconcile_total{namespace=~\"$operatorNamespace\", result=\"error\", controller=~\"scheduledbackup|scheduled-backup\"}), 1)", "hide": true, "legendFormat": "__auto", "range": true, @@ -2511,7 +2511,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "Excluding cache", + "description": "Container memory working set", "fieldConfig": { "defaults": { "color": { @@ -2760,7 +2760,7 @@ }, "editorMode": "code", "exemplar": false, - "expr": "max(cnpg_collector_first_recoverability_point{namespace=~\"$namespace\",pod=~\"$instances\"})*1000", + "expr": "max({__name__=~\"cnpg_collector_first_recoverability_point|barman_cloud_cloudnative_pg_io_first_recoverability_point\",namespace=~\"$namespace\",pod=~\"$instances\"})*1000", "format": "time_series", "instant": true, "interval": "", @@ -5218,7 +5218,7 @@ "step": 10 } ], - "title": "Memory Usage (w/o cache)", + "title": "Memory Usage (container memory working set)", "type": "timeseries" }, { @@ -7374,7 +7374,7 @@ "color": "rgba(255,0,255,0.7)" }, "filterValues": { - "le": 1e-9 + "le": 1e-09 }, "legend": { "show": false @@ -7630,7 +7630,7 @@ "uid": "${DS_PROMETHEUS}" }, "exemplar": true, - "expr": "cnpg_collector_first_recoverability_point{namespace=~\"$namespace\",pod=~\"$instances\"}*1000 > 0", + "expr": "{__name__=~\"cnpg_collector_first_recoverability_point|barman_cloud_cloudnative_pg_io_first_recoverability_point\",namespace=~\"$namespace\",pod=~\"$instances\"}*1000 > 0", "format": "time_series", "interval": "", "legendFormat": "{{pod}}", @@ -7753,7 +7753,7 @@ "uid": "${DS_PROMETHEUS}" }, "exemplar": true, - "expr": "cnpg_pg_stat_bgwriter_checkpoints_req{namespace=~\"$namespace\",pod=~\"$instances\"}", + "expr": "{__name__=~\"cnpg_pg_stat_(bgwriter|checkpointer)_checkpoints_req\",namespace=~\"$namespace\",pod=~\"$instances\"}", "format": "time_series", "hide": false, "instant": false, @@ -7768,7 +7768,7 @@ "uid": "${DS_PROMETHEUS}" }, "exemplar": true, - "expr": "cnpg_pg_stat_bgwriter_checkpoints_timed{namespace=~\"$namespace\",pod=~\"$instances\"}", + "expr": "{__name__=~\"cnpg_pg_stat_(bgwriter|checkpointer)_checkpoints_timed\",namespace=~\"$namespace\",pod=~\"$instances\"}", "format": "time_series", "interval": "", "intervalFactor": 1, @@ -7867,7 +7867,7 @@ "uid": "${DS_PROMETHEUS}" }, "exemplar": true, - "expr": "cnpg_pg_stat_bgwriter_checkpoint_write_time{namespace=~\"$namespace\",pod=~\"$instances\"}", + "expr": "{__name__=~\"cnpg_pg_stat_(bgwriter_checkpoint|checkpointer)_write_time\",namespace=~\"$namespace\",pod=~\"$instances\"}", "format": "time_series", "hide": false, "instant": false, @@ -7882,7 +7882,7 @@ "uid": "${DS_PROMETHEUS}" }, "exemplar": true, - "expr": "cnpg_pg_stat_bgwriter_checkpoint_sync_time{namespace=~\"$namespace\",pod=~\"$instances\"}", + "expr": "{__name__=~\"cnpg_pg_stat_(bgwriter_checkpoint|checkpointer)_sync_time\",namespace=~\"$namespace\",pod=~\"$instances\"}", "format": "time_series", "interval": "", "intervalFactor": 1, @@ -7913,6 +7913,164 @@ "x": 0, "y": 57 }, + "id": 794, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "Show the installed extensions and their versions", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto", + "wrapText": false + }, + "filterable": false, + "inspect": false + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Update Available" + }, + "properties": [ + { + "id": "unit", + "value": "bool" + }, + { + "id": "mappings", + "value": [ + { + "options": { + "0": { + "color": "transparent", + "index": 1 + }, + "1": { + "color": "red", + "index": 0 + } + }, + "type": "value" + } + ] + }, + { + "id": "custom.cellOptions", + "value": { + "applyToRow": true, + "mode": "gradient", + "type": "color-background" + } + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 81 + }, + "id": 792, + "options": { + "cellHeight": "sm", + "footer": { + "countRows": false, + "fields": "", + "reducer": [ + "sum" + ], + "show": false + }, + "showHeader": true, + "sortBy": [ + { + "desc": false, + "displayName": "Value" + } + ] + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "disableTextWrap": false, + "exemplar": false, + "expr": "max(cnpg_pg_extensions_update_available{pod=~\"$instances\", namespace=~\"$namespace\"}) by (datname, extname, default_version, installed_version)", + "format": "table", + "fullMetaSearch": false, + "includeNullMetadata": true, + "instant": true, + "interval": "", + "legendFormat": "__auto", + "range": false, + "refId": "A", + "useBackend": false + } + ], + "title": "Installed extensions", + "transformations": [ + { + "id": "sortBy", + "options": { + "fields": {}, + "sort": [ + { + "field": "extname" + } + ] + } + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true + }, + "indexByName": { + "Time": 0, + "extname": 1, + "datname": 2, + "default_version": 3, + "installed_version": 4, + "Value": 5 + }, + "renameByName": { + "default_version": "Default Version", + "datname": "Database", + "extname": "Extension", + "installed_version": "Installed Version", + "Value": "Update Available" + }, + "includeByName": {} + } + } + ], + "type": "table" + } + ], + "title": "Extensions", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 58 + }, "id": 696, "panels": [ { @@ -7995,7 +8153,7 @@ }, "editorMode": "code", "exemplar": false, - "expr": "sum(kube_pod_status_ready{namespace=\"$operatorNamespace\"} * on (pod) group_left( label_app_kubernetes_io_name ) kube_pod_labels{label_app_kubernetes_io_name=~\"cloudnative-pg\"})", + "expr": "sum(kube_pod_status_ready{namespace=\"$operatorNamespace\", pod=~\"cloudnative-pg.+|cnpg-controller-manager.+\", condition=\"true\"})", "hide": false, "instant": true, "legendFormat": "Ready Operator Pods", @@ -8298,7 +8456,7 @@ }, "editorMode": "code", "exemplar": false, - "expr": "max(controller_runtime_reconcile_total{namespace=~\"$operatorNamespace\", result=\"error\", controller=\"scheduledbackup\"})", + "expr": "max(controller_runtime_reconcile_total{namespace=~\"$operatorNamespace\", result=\"error\", controller=~\"scheduledbackup|scheduled-backup\"})", "hide": false, "instant": true, "legendFormat": "Scheduled Backup Reconcile Errors", @@ -8516,7 +8674,7 @@ "uid": "${DS_PROMETHEUS}" }, "editorMode": "code", - "expr": "sum(kube_pod_status_ready{namespace=\"$operatorNamespace\"} * on (pod) group_left( label_app_kubernetes_io_name ) kube_pod_labels{label_app_kubernetes_io_name=~\"cloudnative-pg\"})", + "expr": "sum(kube_pod_status_ready{namespace=\"$operatorNamespace\", pod=~\"cloudnative-pg.+|cnpg-controller-manager.+\", condition=\"true\"})", "hide": false, "instant": false, "legendFormat": "Ready Operator Pods", @@ -8902,7 +9060,7 @@ }, "editorMode": "code", "exemplar": false, - "expr": "max(controller_runtime_reconcile_total{namespace=~\"$operatorNamespace\", result=\"error\", controller=\"scheduledbackup\"})", + "expr": "max(controller_runtime_reconcile_total{namespace=~\"$operatorNamespace\", result=\"error\", controller=~\"scheduledbackup|scheduled-backup\"})", "hide": false, "instant": false, "legendFormat": "Scheduled Backup Reconcile Errors", @@ -9056,9 +9214,9 @@ "list": [ { "current": { - "selected": false, - "text": "Prometheus", - "value": "prometheus" + "selected": true, + "text": "cnpg-prometheus", + "value": "P5531627C358300FE" }, "hide": 0, "includeAll": false, @@ -9079,7 +9237,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "definition": "label_values(controller_runtime_active_workers,namespace)", + "definition": "label_values(controller_runtime_webhook_requests_total{webhook=\"/mutate-postgresql-cnpg-io-v1-cluster\"},namespace)", "description": "Namespace where the CNPG operator is located", "hide": 0, "includeAll": false, @@ -9089,7 +9247,7 @@ "options": [], "query": { "qryType": 1, - "query": "label_values(controller_runtime_active_workers,namespace)", + "query": "label_values(controller_runtime_webhook_requests_total{webhook=\"/mutate-postgresql-cnpg-io-v1-cluster\"},namespace)", "refId": "PrometheusVariableQueryEditor-VariableQuery" }, "refresh": 2, @@ -9186,4 +9344,4 @@ "uid": "cloudnative-pg", "version": 2, "weekStart": "" -} +} \ No newline at end of file diff --git a/knoe/core/actions.py b/knoe/core/actions.py index edba3f5..b5b1001 100644 --- a/knoe/core/actions.py +++ b/knoe/core/actions.py @@ -118,9 +118,9 @@ from typing import Callable, Sequence DEFAULT_APP_CLUSTER_NAME = "knoe-dev-0" DEFAULT_APP_CLUSTER_MODE = "standard" -DEFAULT_APP_CLUSTER_MACHINE_TYPE = "e2-small" +DEFAULT_APP_CLUSTER_MACHINE_TYPE = "e2-standard-2" DEFAULT_APP_CLUSTER_NODE_COUNT = 3 -DEFAULT_DB_CLUSTER_NAME = "knoe-cnpg-0" +DEFAULT_DB_CLUSTER_NAME = "knoe-dev-cnpg-0" DEFAULT_DB_CLUSTER_MODE = "standard" DEFAULT_DB_CLUSTER_NODE_COUNT = 3 DEFAULT_DB_CLUSTER_MACHINE_TYPE = "e2-standard-2" diff --git a/knoe/ui/screens/deploy.py b/knoe/ui/screens/deploy.py index f33eeed..5307a56 100644 --- a/knoe/ui/screens/deploy.py +++ b/knoe/ui/screens/deploy.py @@ -823,14 +823,14 @@ kerberos.kdc={self.kerberos_kdc.get()} def install_launchagent_port_forwards(self) -> bool: """Create/update the user LaunchAgent and helper script to manage kubectl port-forwards. - Helper script: ~/Library/Application Support/Knoe/bin/knoe-kpf.sh - - LaunchAgent: ~/Library/LaunchAgents/org.knoe.knoe-db.kpf-dev.plist + - LaunchAgent: ~/Library/LaunchAgents/dev.knoe.knoe-db.kpf-dev.plist """ try: home = Path.home() bin_dir = home / "Library" / "Application Support" / "Knoe" / "bin" run_dir = home / "Library" / "Application Support" / "Knoe" / "run" plist_path = ( - home / "Library" / "LaunchAgents" / "org.knoe.knoe-db.kpf-dev.plist" + home / "Library" / "LaunchAgents" / "dev.knoe.knoe-db.kpf-dev.plist" ) script_path = bin_dir / "knoe-kpf.sh" @@ -841,7 +841,7 @@ kerberos.kdc={self.kerberos_kdc.get()} helper_script = """#!/bin/sh set -eu -LABEL="org.knoe.knoe-db.kpf-dev" +LABEL="dev.knoe.knoe-db.kpf-dev" PLIST="$HOME/Library/LaunchAgents/$LABEL.plist" RUNDIR="$HOME/Library/Application Support/Knoe/run" PIDFILE="$RUNDIR/kpf.pids" @@ -925,7 +925,7 @@ esac from plistlib import dumps as plist_dumps plist_dict = { - "Label": "org.knoe.knoe-db.kpf-dev", + "Label": "dev.knoe.knoe-db.kpf-dev", "RunAtLoad": True, "KeepAlive": True, # Ensure PATH has common locations for kubectl @@ -933,10 +933,10 @@ esac "PATH": os.environ.get("PATH", "/usr/local/bin:/usr/bin:/bin") }, "StandardOutPath": str( - home / "Library" / "Logs" / "org.knoe.knoe-db.kpf-dev.out.log" + home / "Library" / "Logs" / "dev.knoe.knoe-db.kpf-dev.out.log" ), "StandardErrorPath": str( - home / "Library" / "Logs" / "org.knoe.knoe-db.kpf-dev.err.log" + home / "Library" / "Logs" / "dev.knoe.knoe-db.kpf-dev.err.log" ), "ProgramArguments": [str(script_path), "start"], "KnoeCommands": default_cmds, @@ -953,7 +953,7 @@ esac "launchctl", "kickstart", "-k", - f"gui/{uid}/org.knoe.knoe-db.kpf-dev", + f"gui/{uid}/dev.knoe.knoe-db.kpf-dev", ], capture_output=True, ) @@ -963,7 +963,7 @@ esac "launchctl", "bootout", f"gui/{uid}", - f"gui/{uid}/org.knoe.knoe-db.kpf-dev", + f"gui/{uid}/dev.knoe.knoe-db.kpf-dev", ], capture_output=True, ) @@ -975,7 +975,7 @@ esac "launchctl", "kickstart", "-k", - f"gui/{uid}/org.knoe.knoe-db.kpf-dev", + f"gui/{uid}/dev.knoe.knoe-db.kpf-dev", ], check=True, ) diff --git a/monitoring/cnpg-dashboard-transforms.yaml b/monitoring/cnpg-dashboard-transforms.yaml new file mode 100644 index 0000000..8867e58 --- /dev/null +++ b/monitoring/cnpg-dashboard-transforms.yaml @@ -0,0 +1,101 @@ +# Transform spec for the upstream cloudnative-pg Grafana dashboard. +# +# Re-applied on every run of `etc/sync_cnpg_grafana_dashboard.py`. The script +# fetches the upstream JSON, walks `transformations` in order, writes the +# result to `knoe-db/grafana-dashboard.json` (which is what +# `etc/init_monitoring.sh` packages into the +# `knoe-db-grafana-dashboard` ConfigMap on the app cluster). +# +# When upstream releases a new version of the dashboard, the workflow is: +# +# 1. Re-run the sync tool (any branch — runs in CI or as a background +# agent on its own schedule). +# 2. The tool fetches the new upstream, applies these transforms, and +# either writes the result + opens a diff/PR (default), OR applies +# to the live cluster directly (--apply). +# 3. If a transformation no longer matches (e.g. upstream renamed a key +# or removed a panel that we were patching), the tool emits a warning +# with the affected transformation's `name`. Update or remove that +# transformation entry here, re-run. +# +# Adding a new transformation: append to `transformations`. Each entry has +# a `name`, `description`, `type`, and per-type parameters. Currently +# supported types: +# +# - `regex_replace`: sed-style on the raw JSON string. +# pattern: # use named groups for capture refs +# replacement: > +# expected_min: # tool warns if fewer matches than this +# expected_max: # tool warns if more matches than this +# +# Future types we may add: +# +# - `jsonpatch`: RFC 6902 operations against the parsed JSON tree +# - `panel_query_rewrite`: navigate `panels[*].targets[*].expr` specifically +# +# The current set is deliberately tiny — most of the cross-cluster CNPG +# observability stack is fixed via cluster-side config (Cluster.spec.monitoring +# podMonitorRelabelings, kps grafana.ini settings) rather than dashboard +# edits. Only edits that MUST live in the dashboard JSON go here. + +source: + # Pinned to `main` because the upstream repo's dashboard versioning is loose. + # If reproducibility against a specific revision is needed, swap `main` for + # a commit SHA or release tag. + url: https://raw.githubusercontent.com/cloudnative-pg/grafana-dashboards/main/charts/cluster/grafana-dashboard.json + + # The expected dashboard `uid` field. The tool will refuse to proceed if the + # upstream has changed UID (would silently break dashboard URLs). + expected_uid: cloudnative-pg + +transformations: + + - name: rate_interval_swap + description: | + Swap `[$__interval]` for `[$__rate_interval]` inside `rate()` and + `irate()` calls. Grafana's $__interval auto-shrinks below the scrape + interval at small dashboard time ranges (e.g. a 5-min view yields + ~1-2s rate windows; our 15s scrape interval means 0 samples in any 2s + window, so rate() returns nothing). $__rate_interval is the rate-safe + equivalent that Grafana auto-computes from min step + scrape interval. + As of upstream commit at sync time, this affects 2 panels. + type: regex_replace + pattern: '(?Prate|irate)\((?P[^()]*)\[\$__interval\]\)' + replacement: '\g(\g[$__rate_interval])' + expected_min: 1 # at least one match — if zero, upstream may have switched away from $__interval + expected_max: 50 # if more than this, something's odd; flag for review + + - name: ds_prometheus_default_to_cnpg + description: | + Pin the dashboard's DS_PROMETHEUS template variable to our + `cnpg-prometheus` datasource so panels render correctly on first load + without users needing to override via URL params or the variable picker. + + Why: upstream ships the dashboard with `current.value: "prometheus"` + (the upstream-default Prometheus UID). On a kps install, the standard + Prometheus datasource has uid=prometheus (default), and our DB-cluster + Prometheus is at uid=cnpg-prometheus (a stable name we set in + monitoring/kps-values-gke.yaml's `additionalDataSources`). Without + this transform, the dashboard initially queries the LOCAL prometheus + (which has no CNPG metrics — they're only on the DB cluster), and + Grafana's panel-cache + datasource-variable-rerun quirks make it + hard to recover even after the user changes the picker. + type: set_template_variable_default + variable: DS_PROMETHEUS + # CAVEAT: this UID is the auto-generated one from the FIRST helm install + # of kps. We pinned `uid: cnpg-prometheus` in monitoring/kps-values-gke.yaml + # so a fresh helm install would get the stable name, but Grafana refuses + # to update the UID of an already-provisioned datasource (read-only via + # API; even rollout-restart doesn't pick it up). On the next clean kps + # re-install (or after deleting the auto-uid datasource via direct DB + # manipulation), update both this `value` and `text` to `cnpg-prometheus`. + # Tracked in docs/TODO.md as a follow-up. + value: P5531627C358300FE + text: cnpg-prometheus + selected: true + +# (No further transformations needed at time of writing. Dashboard +# correctness for our cross-cluster setup comes mostly from server-side +# config: see `monitoring/kps-cnpg-values.yaml`, +# `monitoring/kps-values-gke.yaml`, and the +# `spec.monitoring.podMonitorRelabelings` in `deploy/gcp/gke/knoe-db.yaml`.) diff --git a/monitoring/kps-cnpg-values.yaml b/monitoring/kps-cnpg-values.yaml new file mode 100644 index 0000000..890cc96 --- /dev/null +++ b/monitoring/kps-cnpg-values.yaml @@ -0,0 +1,96 @@ +# kube-prometheus-stack (kps) Helm values — DB cluster (knoe-dev-cnpg-0). +# +# This file scopes a SECOND kube-prometheus-stack install — companion to +# `monitoring/kps-values-gke.yaml` (the app-cluster install on knoe-dev-0). +# +# Why two stacks: Kubernetes service discovery is cluster-local. The app- +# cluster Prometheus can't see CNPG pods (they live on knoe-dev-cnpg-0), so +# every cnpg-grafana dashboard rendered "No data". This stack runs Prometheus +# locally on the DB cluster, auto-discovers CNPG's PodMonitor, and is queried +# directly by the app-cluster Grafana via an internal-LB datasource. +# +# Topology: +# knoe-dev-cnpg-0 -[this stack]-> Prometheus + node-exporter + ksm +# | +# v +# ILB (prometheus-cnpg-ilb @ 10.180.x.x:9090) +# ^ +# | (datasource: cnpg-prometheus) +# knoe-dev-0 --------------- Grafana (single canonical Grafana) +# +# Apply: +# helm --kube-context=$DB_CLUSTER_KUBECONTEXT upgrade --install kps \ +# prometheus-community/kube-prometheus-stack --version 84.3.0 \ +# --namespace monitoring --create-namespace \ +# -f monitoring/kps-cnpg-values.yaml --wait --timeout 5m +# +# Storage: every PVC pinned to `standard-hdd` (pd-standard). The cluster also +# has `standard-rwo` (pd-balanced, partial-SSD) and `premium-rwo` (pd-ssd) — +# both would consume the SSD_TOTAL_GB regional quota that's shared with CNPG. +# Honor the standing rule: monitoring is HDD-only. +--- + +# One Grafana stays canonical on the app cluster (svc.knoe.dev/grafana). +# This stack is a metrics-only sidecar. +grafana: + enabled: false + +# Alerting routes through the app-cluster Alertmanager (follow-up: federation). +# For now, no AM here, no rules either (avoid scrape-rule noise into a void). +alertmanager: + enabled: false +defaultRules: + create: false + +# DB-cluster object metrics (pods/PVCs/Jobs etc.) for kube-state dashboards. +kube-state-metrics: + enabled: true + +# Per-DB-node host metrics (CPU/mem/disk/network). +prometheus-node-exporter: + enabled: true + +prometheus: + prometheusSpec: + # Short retention — this Prometheus is mostly a query-target. App-cluster + # Grafana queries it on demand; we don't archive long-form. + retention: 7d + + # CRITICAL: select ALL PodMonitors / ServiceMonitors / PrometheusRules, + # not just helm-release-labelled ones. The CNPG operator-emitted + # PodMonitor at knoe-db-0/knoe-db doesn't carry our `release=kps` label, + # but we want to scrape it. Selector with empty `{}` matches everything. + podMonitorSelectorNilUsesHelmValues: false + serviceMonitorSelectorNilUsesHelmValues: false + ruleSelectorNilUsesHelmValues: false + podMonitorSelector: {} + serviceMonitorSelector: {} + + # Tight resource shape — small cluster, low metric cardinality (3 PG + # pods + ~50 metrics each + 15s scrape ≈ 30k samples/day ≈ 250 MB/day). + resources: + requests: + cpu: 100m + memory: 512Mi + limits: + cpu: 500m + memory: 1Gi + + storageSpec: + volumeClaimTemplate: + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 20Gi + storageClassName: standard-hdd # explicit HDD (do not let default fall through) + +prometheusOperator: + resources: + requests: + cpu: 50m + memory: 128Mi + limits: + cpu: 200m + memory: 256Mi diff --git a/monitoring/kps-values-gke.yaml b/monitoring/kps-values-gke.yaml new file mode 100644 index 0000000..d116308 --- /dev/null +++ b/monitoring/kps-values-gke.yaml @@ -0,0 +1,291 @@ +# kube-prometheus-stack (kps) Helm values — GKE deploy mode. +# +# This file is the canonical source of the values applied to the `kps` +# release in namespace `monitoring` on knoe-dev-0. It captures the existing +# inline values that were `helm install`'d 2026-04-28 PLUS today's grafana +# subpath + Google OAuth additions. +# +# Apply with: +# helm upgrade kps prometheus-community/kube-prometheus-stack \ +# --namespace monitoring \ +# -f monitoring/kps-values-gke.yaml +# +# NOTE on prole.org/node-role nodeSelectors below: this is a legacy label +# from the prole-era cluster — knoe-dev-0 nodes still carry it for +# compatibility. Once the cluster is fully relabelled to knoe.dev/* a +# follow-up will rename these. Tracked in docs/TODO.md (Reality TODO list). +--- +alertmanager: + alertmanagerSpec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: prole.org/node-role + operator: In + values: + - general + nodeSelector: + prole.org/node-role: general + storage: + volumeClaimTemplate: + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 5Gi + storageClassName: standard-hdd + +grafana: + enabled: true + # adminPassword is the local break-glass; primary auth is Google OAuth (below). + # Rotate this whenever a person who once knew it leaves the team. Stored in + # 1Password (admin scope, separate from per-engineer entries). + adminPassword: admin + + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: prole.org/node-role + operator: In + values: + - general + nodeSelector: + prole.org/node-role: general + + initChownData: + enabled: false + + persistence: + accessModes: + - ReadWriteOnce + enabled: true + size: 10Gi + storageClassName: standard-hdd + type: sts + + service: + port: 80 + targetPort: 3000 + + sidecar: + dashboards: + enabled: true + label: grafana_dashboard + labelValue: "1" + datasources: + enabled: true + label: grafana_datasource + labelValue: "1" + + # Mount Google OAuth client credentials from the grafana-google-oidc Secret. + # Created by etc/init_grafana_oauth.sh from etc/secrets/grafana-google-oidc-*. + # Provides GF_AUTH_GOOGLE_CLIENT_ID and GF_AUTH_GOOGLE_CLIENT_SECRET env vars. + envFromSecret: grafana-google-oidc + + # Additional Prometheus datasource pointing at the DB-cluster (knoe-dev-cnpg-0) + # Prometheus install (monitoring/kps-cnpg-values.yaml). The cnpg-grafana + # dashboards' DS_PROMETHEUS template variable can switch to this datasource + # to render CNPG metrics — Kubernetes service discovery is cluster-local, so + # the local app-cluster Prometheus can't see CNPG pods on its own. + # Reachable across clusters via the internal-LB IP allocated to + # `prometheus-cnpg-ilb` in monitoring ns of the DB cluster (same VPC subnet). + additionalDataSources: + - name: cnpg-prometheus + # Stable UID so the cnpg-grafana dashboard's DS_PROMETHEUS current.value + # (set by the dashboard transform spec) can refer to this datasource by + # name instead of an auto-generated random UID. Also makes the datasource + # identity-preserving across kps re-installs (which would otherwise + # regenerate the auto UID and silently break the dashboard). + uid: cnpg-prometheus + type: prometheus + url: http://10.180.15.216:9090 + access: proxy + isDefault: false + editable: false + jsonData: + timeInterval: 15s + manageAlerts: false + prometheusType: Prometheus + + # grafana.ini — appended to the chart's defaults. + # Subpath: served at https://svc.knoe.dev/grafana via the knoe-svc-kong route. + # Auth: Google OAuth restricted to @knoey.com Workspace; chrisfu + ron mapped + # to Admin via JMESPath, everyone else in the Workspace gets Editor. + grafana.ini: + server: + domain: svc.knoe.dev + root_url: "https://svc.knoe.dev/grafana" + serve_from_sub_path: true + security: + # Behind Kong + GCE LB on HTTPS — issue cookies with the Secure flag set + # so browsers send them on every request. Without this, Grafana 13's + # session-token rotation logic fights with the proxy chain and every + # API call returns 401 with `[session.token.rotate] token needs to be + # rotated`, breaking panel data fetches in a loop. + cookie_secure: true + cookie_samesite: lax + # Grafana 10+ enforces a same-origin CSRF check on every state-changing + # method (POST/PUT/PATCH/DELETE), comparing Origin/Referer to root_url. + # Behind Kong → grafana the upstream Host header is the cluster-internal + # service name (kps-grafana.monitoring.svc.cluster.local), not + # svc.knoe.dev, so the CSRF middleware rejects POSTs from the public + # origin with 403. Symptoms: every /api/ds/query → 403 (panels render + # empty); Share → Copy Link → "origin not allowed" toast. + # Trust the public hostname explicitly. csrf_trusted_origins takes + # space-separated bare hostnames (no scheme); csrf_additional_headers + # tells Grafana to also accept X-Forwarded-Host (which Kong sets + # correctly) as a valid origin source — this is the documented pairing + # for proxied installs. + csrf_trusted_origins: svc.knoe.dev + csrf_additional_headers: X-Forwarded-Host + "live": + # WebSocket origin check (Grafana live streaming, dashboard refresh). + # The default rejects any Origin not exactly matching root_url, which + # surfaces in the UI as a "origin not allowed" toast popup. Allow the + # public hostname explicitly. Multiple origins comma-separated if needed. + allowed_origins: "https://svc.knoe.dev" + auth: + # Keep the local login form available as a break-glass for adminPassword. + disable_login_form: false + # Increase the session-token rotation interval so the rotation race + # condition is rare enough not to break panel queries. Default was + # 10 minutes; bumped to 24 hours. The "right" fix is figuring out why + # rotation fails through the Kong proxy at all (TODO follow-up); this + # is the pragmatic mitigation for tonight. + token_rotation_interval_minutes: 1440 + "auth.google": + enabled: true + # client_id / client_secret arrive via env (GF_AUTH_GOOGLE_CLIENT_ID etc.) + # from the grafana-google-oidc Secret. Don't duplicate here. + allowed_domains: knoey.com + scopes: "openid email profile" + auth_url: https://accounts.google.com/o/oauth2/v2/auth + token_url: https://oauth2.googleapis.com/token + api_url: https://openidconnect.googleapis.com/v1/userinfo + # JMESPath: chrisfu + ron get Admin; every other knoey.com user gets Editor. + # auto_assign_org_role below is the fallback if role_attribute_path produces + # an empty result. + role_attribute_path: "contains(['chrisfu@knoey.com', 'ron@knoey.com'], email) && 'Admin' || 'Editor'" + # Re-evaluate role on each login so a dropped engineer immediately loses + # the elevated bit; if you want manual elevation in Studio to survive, + # set this to true. + skip_org_role_sync: false + users: + auto_assign_org_role: Editor + +kube-state-metrics: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: prole.org/node-role + operator: In + values: + - general + nodeSelector: + prole.org/node-role: general + +prometheus: + prometheusSpec: + additionalScrapeConfigs: + - job_name: kubernetes-pods + kubernetes_sd_configs: + - role: pod + relabel_configs: + - action: keep + regex: true + source_labels: + - __meta_kubernetes_pod_annotation_prometheus_io_scrape + - action: replace + regex: (.+) + source_labels: + - __meta_kubernetes_pod_annotation_prometheus_io_path + target_label: __metrics_path__ + - action: replace + regex: (.*?):\d+;(\d+) + replacement: $1:$2 + source_labels: + - __address__ + - __meta_kubernetes_pod_annotation_prometheus_io_port + target_label: __address__ + - job_name: cnpg-metrics + kubernetes_sd_configs: + - role: pod + relabel_configs: + - action: keep + regex: .+ + source_labels: + - __meta_kubernetes_pod_label_cnpg_io_cluster + - action: keep + regex: Running + source_labels: + - __meta_kubernetes_pod_phase + - action: replace + replacement: $1:9187 + source_labels: + - __meta_kubernetes_pod_ip + target_label: __address__ + - action: replace + source_labels: + - __meta_kubernetes_namespace + target_label: namespace + - action: replace + source_labels: + - __meta_kubernetes_pod_name + target_label: pod + - action: replace + source_labels: + - __meta_kubernetes_pod_label_cnpg_io_cluster + target_label: cluster + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: prole.org/node-role + operator: In + values: + - general + nodeSelector: + prole.org/node-role: general + storageSpec: + volumeClaimTemplate: + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 30Gi + storageClassName: standard-hdd + +prometheus-node-exporter: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: NotIn + values: + - pi.prole.org + nodeSelector: + prole.org/node-role: general + +prometheusOperator: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: prole.org/node-role + operator: In + values: + - general + nodeSelector: + prole.org/node-role: general diff --git a/pom.xml b/pom.xml index 7c05069..0c62b43 100644 --- a/pom.xml +++ b/pom.xml @@ -10,7 +10,7 @@ - org.knoe + dev.knoe knoe-db 0.0.1-SNAPSHOT knoe-db-ctrl @@ -20,9 +20,6 @@ 24 1.0.0 false - - true @@ -224,23 +221,6 @@ - - authority - verify - - exec - - - ${skip.authority} - mvn - - -f - ${project.basedir}/authority/pom.xml - clean - test - - - diff --git a/supabase/deploy.sh b/supabase/deploy.sh index 0037d26..f382cad 100755 --- a/supabase/deploy.sh +++ b/supabase/deploy.sh @@ -2108,6 +2108,14 @@ setup_knoe_db_for_supabase() { GRANT service_role TO authenticator; GRANT supabase_admin TO authenticator; + -- supabase-storage runs `SET LOCAL role = 'service_role'` on every request + -- (and 'authenticated' / 'anon' depending on the JWT). Without these grants + -- the SET fails with 42501 inside guc.c:call_string_check_hook, which the + -- service surfaces as a misleading "new row violates row-level security + -- policy" error -- regardless of whether RLS is even involved. Mirrors what + -- the upstream supabase/postgres image's bootstrap does for `authenticator`. + GRANT anon, authenticated, service_role TO supabase_storage_admin; + -- Schemas on postgres database CREATE SCHEMA IF NOT EXISTS knoe AUTHORIZATION knoe; CREATE SCHEMA IF NOT EXISTS auth AUTHORIZATION supabase_auth_admin; @@ -2129,6 +2137,13 @@ setup_knoe_db_for_supabase() { GRANT USAGE ON SCHEMA auth TO anon, authenticated, service_role; GRANT USAGE ON SCHEMA storage TO anon, authenticated, service_role; GRANT USAGE ON SCHEMA graphql_public TO anon, authenticated, service_role; + -- The `extensions` schema is created by the CNPG postInitTemplateSQL block + -- (see deploy/gcp/gke/knoe-db.yaml) so relocatable extensions like + -- pg_stat_statements live there instead of `public` (Supabase's Database + -- Advisor flags extensions in public as a Security warning). The supabase + -- roles need USAGE so PostgREST/Studio queries that walk the schema graph + -- (e.g. the advisor itself) don't 42501 on it. + GRANT USAGE ON SCHEMA extensions TO anon, authenticated, service_role; -- Keep extension objects out of public schema to satisfy database linter checks. DO \$\$ diff --git a/supabase/helm/knoe-supabase/templates/kong/config.yaml b/supabase/helm/knoe-supabase/templates/kong/config.yaml index 237b661..13b9a3e 100644 --- a/supabase/helm/knoe-supabase/templates/kong/config.yaml +++ b/supabase/helm/knoe-supabase/templates/kong/config.yaml @@ -76,6 +76,31 @@ data: config: status_code: 200 message: ok + # Stop-gap support endpoint -- 302s to mailto:support@knoe.dev so any user + # who lands at https://db.0.knoe.dev/support gets a working escape hatch. + # The in-Studio "Report a problem" / "Send feedback" buttons in upstream + # Studio are hardcoded to supabase.com endpoints and are not yet rewired + # (would require forking the Studio image -- tracked in docs/TODO.md). + # In the meantime we tell users: "for help, go to db.0.knoe.dev/support". + # request-termination synthesizes a 302 status; response-transformer + # injects the Location header (request-termination alone can't set it). + - name: support + url: http://knoe.support.invalid/ + routes: + - name: support-redirect + strip_path: true + paths: + - /support + plugins: + - name: request-termination + config: + status_code: 302 + message: "Redirecting to support@knoe.dev" + - name: response-transformer + config: + add: + headers: + - "Location:mailto:support@knoe.dev" {{- if .Values.deployment.auth.enabled }} - name: auth-v1-open url: http://{{ include "supabase.auth.fullname" . }}:{{ .Values.service.auth.port }}/verify @@ -273,18 +298,22 @@ data: allow: - admin {{- end }} - {{- if .Values.deployment.oauth2proxy.enabled }} - - name: dashboard - _comment: 'Studio via oauth2-proxy: /* -> http://{{ include "supabase.oauth2proxy.fullname" . }}:{{ .Values.deployment.oauth2proxy.port }}/*' - url: http://{{ include "supabase.oauth2proxy.fullname" . }}:{{ .Values.deployment.oauth2proxy.port }}/ + # Onboarding reveal page (knoe-onboard nginx pod). Listed BEFORE the + # /-wildcard dashboard route so Kong's longest-prefix match picks + # /onboard.html over the dashboard. The page itself is also exempted + # from oauth2-proxy gating (`--skip-auth-route=^/onboard\.html$`) since + # URL fragments don't survive an OAuth redirect; URL secrecy + 24h + # expiry + immediate-rotation is the security envelope. + - name: onboard + _comment: 'Onboarding reveal: /onboard.html -> knoe-onboard nginx pod' + url: http://knoe-onboard:80/ routes: - - name: dashboard-all - strip_path: true + - name: onboard-html + strip_path: false paths: - - / + - /onboard.html plugins: - name: cors - {{- else }} - name: dashboard _comment: 'Studio: /* -> http://{{ include "supabase.studio.fullname" . }}:{{ .Values.service.studio.port }}/*' url: http://{{ include "supabase.studio.fullname" . }}:{{ .Values.service.studio.port }}/ @@ -293,12 +322,6 @@ data: strip_path: true paths: - / - {{- if .Values.secret.dashboard }} plugins: - name: cors - - name: basic-auth - config: - hide_credentials: true - {{- end }} - {{- end }} {{- end }} diff --git a/supabase/helm/knoe-supabase/values.yaml b/supabase/helm/knoe-supabase/values.yaml index ec7b6a6..ee1ddd6 100644 --- a/supabase/helm/knoe-supabase/values.yaml +++ b/supabase/helm/knoe-supabase/values.yaml @@ -623,7 +623,10 @@ environment: # (only listed plugins can be referenced in the declarative config), so # without this entry the Kong pod crashlooped with # init_by_lua error: plugin 'request-termination' not enabled - KONG_PLUGINS: request-transformer,cors,key-auth,acl,basic-auth,request-termination + # `response-transformer` added for the /support route, where it injects + # the `Location: mailto:support@knoe.dev` header onto the 302 synthesized + # by request-termination (request-termination alone can't set Location). + KONG_PLUGINS: request-transformer,response-transformer,cors,key-auth,acl,basic-auth,request-termination KONG_NGINX_PROXY_PROXY_BUFFER_SIZE: 160k KONG_NGINX_PROXY_PROXY_BUFFERS: 64 160k KONG_LOG_LEVEL: warn