Enables iterating on GitLab reconciles without a full deploy.sh run and
unblocks the end-to-end login flow:
- etc/init_gitlab.sh
* Pre-scan $@ for --config before sourcing prole_cfg.sh so
PROLE_DEPLOY_CFG is set for standalone invocations; previously
prole_cfg.sh auto-picked conf/k3d.cfg and leaked KUBECONTEXT=dev,
failing with `error: context "dev" does not exist`.
* _init_gitlab_resolve_secretref: bash mirror of
knoe/core/actions.py:_resolve_secretref_value so
secretref://google-oidc-client-* resolves from etc/secrets/ files
when run outside deploy.sh Python env-injection.
* Garage bucket array: drop -storage suffix from uploads/artifacts/
lfs/packages/dependency-proxy. Chart defaults for these have no
suffix, and the object_store block in the CR does not override
per-object bucket names; the prior mismatch caused first-login 500s
(NoSuchBucket on avatar PUT).
- deploy/gcp/gke/gitlab-google-oidc-secret.example.yaml
* Add discovery: true so omniauth-openid_connect fetches Google
.well-known/openid-configuration; fixes "Could not authenticate
from OpenIDConnect: No host info" on the callback.
- conf/gke.cfg
* GITLAB_WEBSERVICE_LIMITS_MEMORY 1800M -> 3Gi, REQUESTS_MEMORY
900M -> 2Gi, REQUESTS_CPU 200m -> 500m. Live pod was sitting at
1706Mi/1800M (99%) in OOMKilled loop.
- conf/port-mapping.cfg: add supabase + gitea forwards, fix postgres
namespace knoe-db -> knoe-db-0 for split-cluster CNPG layout.
Adds read-only diagnostics used to chase the above:
- etc/diag_gitlab_boot.sh, diag_gitlab_webservice_oom.sh, diag_gke_storage.sh
- etc/ensure_default_storage_class.sh + k8s/prole/storageclass-gcp-standard-hdd.yaml
(preflight + HDD-default SC manifest for SSD-quota-constrained GKE projects).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Documented step-by-step procedure for integrating GitLab with Google Workspace via OIDC (Path A).
- Outlined secret management patterns, GCP OAuth client setup, and deployment instructions.
- Included detailed remediation steps for secret leakage and Git history cleanup.
- Added verification and troubleshooting guides for pre- and post-deployment scenarios.
test: add helm monitoring tests for stale lock recovery and upgrade handling
- Introduced tests to validate Prometheus update flow with stale pending locks, including rollback and upgrade scenarios.
- Ensured robust handling for edge cases like missing deployed revisions and context mismatches.
- Added pre-apply validation for ManagedCertificate and FrontendConfig objects to ensure existence before ingress application.
- Improved GKE ingress diagnostics with annotations and resource state checks for pre- and post-apply phases.
- Introduced patch-only handling for annotation drifts to avoid unnecessary ingress replacements.
- Enhanced HTTPS probing logic with GCLB convergence checks for managed certificate states.
- Updated `deploy.sh`, `init_kong.sh`, and `init_gitlab.sh` with detailed ingress dependency validation and reconciliation steps.
- Introduced dependency validation for ManagedCertificate and FrontendConfig annotations in Supabase ingresses.
- Enhanced `deploy.sh` with public ingress TLS resource application and improved error diagnostics.
- Updated `render_supabase.py` to generate GKE TLS manifests and log rendered resources.
- Refined Supabase deployment flow to include detailed ingress dependency validation and reconciliation.
- Improved handling of GKE preload TLS annotations by ignoring `SERVICE_PRE_SHARED_CERT` in k8s mode with managed TLS resources.
- Added drift detection for ingress class, ManagedCertificate, and FrontendConfig annotations, with forced replacement for GCLB binding consistency.
- Enhanced TLS diagnostics with detailed path state, failure reasons, and HTTPS probe handling for GKE configurations.
- Updated tests and scripts to validate GKE-specific behavior and ingress rendering.
- Updated `deploy.sh` and `init_kong.sh` to support GKE ManagedCertificate and FrontendConfig reconciliation in k8s mode with GCE ingress class.
- Enhanced TLS path diagnostics to distinguish between missing, attached-but-not-serving, and actively serving configurations.
- Added HTTPS probing for GKE-managed public ingress paths and validation for managed certificate statuses.
- Updated `render_supabase.py` to generate and attach TLS annotations for Supabase API and Studio ingresses in k8s mode.
- Added tests to validate TLS path handling, probing, and manifest generation for GKE deployments.
- Enhanced `deploy.sh` to include detailed storage class diagnostics for blocking PVCs (e.g., provisioner, type, binding mode).
- Added `validate_supabase_rendered_pvc_storage` to ensure rendered PVC configurations align with expected storage classes.
- Improved event parsing logic to prioritize recent PVC errors and resolve storage class details.
- Updated deployment flow to handle Supabase Studio ingress reconciliation errors if deployment is missing or replicas are zero.
- Introduced `GITLAB_FALLBACK_INGRESS_NAME` for better flexibility in managing fallback ingress names.
- Added logic to clean up both current and legacy fallback ingresses (`gitlab-frontdoor-ingress`, `gitlab-kong-ingress`).
- Enhanced diagnostics for ingress host conflicts and TLS path validation.
- Updated `deploy.sh` to improve runtime configuration resolution and host diagnostics.
- Refined tests to ensure correct fallback ingress handling and conflict resolution.
- Updated `deploy.sh` to resolve project IDs from multiple configuration fallbacks and environment contexts for retained disk cleanup.
- Added support for `supabase_cfg_first_nonempty_value` to prioritize configuration over defaults.
- Enhanced logging with explicit reasons for skipped disk cleanup actions (e.g., missing gcloud, unresolved project).
- Updated tests to validate fallback logic and skip reason reporting.
- Updated `deploy.sh` to resolve project IDs from multiple configuration fallbacks and environment contexts for retained disk cleanup.
- Added support for `supabase_cfg_first_nonempty_value` to prioritize configuration over defaults.
- Enhanced logging with explicit reasons for skipped disk cleanup actions (e.g., missing gcloud, unresolved project).
- Updated tests to validate fallback logic and skip reason reporting.
- Changed GitLab ingress default ownership in GKE/k8s mode from `operator` to `fallback` for explicit GCE ingress path anchoring.
- Added logic to clean up stale operator-managed ingresses when fallback ownership is selected.
- Simplified fallback ingress creation workflow by removing redundant skip logic.
- Updated relevant tests to reflect ownership changes and ingress handling enhancements.
- Introduced `_recover_stale_prometheus_helm_lock` to detect and recover from stale Helm release locks during Prometheus deployment.
- Enhanced Prometheus deployment flow with stability verification and rollback to the last deployed state.
- Implemented `gitlab_split_cluster_ownership_diagnostics` to enforce and log split-cluster ownership policies between app and DB contexts.
- Updated workload convergence logic to prioritize readiness detection over historical restart counts for webservice pods.
- Enhanced deployment utilities with additional diagnostics for pod readiness states and replica discrepancies.
- Introduced tests to validate split-cluster enforcement and refined convergence behavior.
- Added logic to set and validate `GITLAB_TRUSTED_PROXIES` for forwarded headers behind ingress/load balancers.
- Improved `gitlab_old_replicaset_live_summary` to handle empty summaries gracefully by returning zero.
- Enhanced logging to confirm trusted proxies configuration.
- Updated tests to validate trusted proxies rendering and summary handling behavior.
- Introduced `gitlab_rendered_replica_source_fields_from_cr` to extract and log replica fields from rendered CR values.
- Enhanced logging for replica source-of-truth mismatches, including detailed diagnostics of rendered CR fields.
- Updated registry-specific key sanitization to align with chart 9+ authoritative fields.
- Added and extended tests to validate replica field rendering, sanitization, and logging enhancements.
- Introduced utilities to identify and strip deprecated top-level replica keys for chart v9 and newer.
- Added preflight validation to block CR apply if removed keys remain.
- Enhanced logging to detail stripped keys and remaining deprecated fields.
- Updated tests to ensure proper handling of top-level and hpa replica keys in rendered CR values.
- Added utilities for diagnosing and cleaning up StatefulSet template and PVC storage class mismatches in Gitaly.
- Improved logging for storage class fields in GitLab CR rendering and live StatefulSet diagnostics.
- Introduced `cleanup_gitlab_wrong_gitaly_template_storage` for automated destructive repair of misconfigured storage templates.
- Added tests to ensure authoritative Gitaly storage class enforcement and error handling for mismatches.
- Replaced `is_gke_standard_storage_equivalent` with `gitlab_storage_class_matches_expected` for stricter logic and better readability.
- Added utilities for diagnosing and cleaning up PVC/PV and StatefulSet template storage class mismatches.
- Introduced `enforce_gitlab_storage_class_target` to automate configuration enforcement and diagnostics.
- Enhanced logging and failure remediation for storage class mismatches in GitLab deployments.
- Introduced `gitlab_verify_replica_source_of_truth` and `gitlab_replica_source_of_truth_report` for replica drift validation and convergence.
- Enhanced logging and remediation logic to handle source-of-truth mismatches during deployment reconciliation.
- Added explicit `minReplicas` and `maxReplicas` handling for key components to prevent drift.
- Improved failure classification and reporting for post-remediation settle verification.
- Added `supabase_collect_stale_retained_pv_info` for clearer separation of logic and improved reusability.
- Enhanced zone detection for retained PVs using regex-based fallback.
- Updated cleanup logic to auto-enable `SUPABASE_AUTO_CLEAN_RETAINED_PVS` and streamline handling of GCE disk deletion.
- Introduced verification and retry mechanism to ensure stale PV artifacts are fully removed.
- Improved error handling for remaining GCE disk cleanup with detailed remediation instructions.
- Added detailed handling and cleanup for stale retained PVs to prevent quota exhaustion.
- Introduced `check_supabase_retained_pv_blocked` utility for preflight artifact detection and optional auto-cleanup.
- Improved ingress reconciliation for `supabase-kong` and `supabase-studio`.
- Enhanced retry logic with optional destructive namespace resets when the app cluster is unhealthy.
- Added `gitlab_selector_for_deployment` utility to dynamically resolve deployment selectors.
- Updated pod count and name retrieval to use resolved selectors, ensuring accuracy.
- Enhanced logging to include selector details for better debugging and remediation tracking.
- Addressed replica drift detection with CR-based reconciliation and scale-down logic.
- Added settle verification logic with configurable timeout and polling intervals for corrected deployments.
- Introduced new utility function `gitlab_non_terminal_pod_names_for_app` to retrieve non-terminal pod names.
- Improved logging for final deployment state and pod details after remediation actions.
- Added settle verification logic with configurable timeout and polling intervals for corrected deployments.
- Introduced new utility function `gitlab_non_terminal_pod_names_for_app` to retrieve non-terminal pod names.
- Improved logging for final deployment state and pod details after remediation actions.
- Added `_resolve_secretref_value` to handle `secretref://` values for `auth.clientId` and `auth.clientSecret`.
- Extended environment and file-based lookups for secretref resolution.
- Updated tests to validate new secretref resolution logic in config and GitOps workflows.
- Implemented `auth.clientId` and `auth.clientSecret` backfilling from Global section and environment variables.
- Added tests to verify correct hydration of OIDC inputs in configuration defaults.
- Introduced `gitlab_post_remediation_success` flag to streamline corrective action logging.
- Adjusted logic to skip unnecessary workload convergence gates when remediation is successful.
- Introduced `gitlab_post_remediation_success` flag to streamline corrective action logging.
- Adjusted logic to skip unnecessary workload convergence gates when remediation is successful.
Summarize CLAUDE.md updates:
- Document both live GKE clusters as e2-standard-2 x3 in us-west3 and note stale app default in code.
- Call out DB cluster naming mismatch between conf (knoe-dev-cnpg-0) and code default (knoe-cnpg-0).
- Add SSD quota guidance: 300GB fully consumed by CNPG, so non-CNPG PVCs must use standard/pd-standard.
- Refresh reset script reference and quota section details to match current runtime state.
Co-authored-by: Junie <junie@jetbrains.com>
- Added detailed logging for APP and DB contexts in `init_gitlab.sh` and `deploy.sh`.
- Enabled explicit passing of contexts and configuration paths to avoid fallbacks.
- Introduced strict environment resolution logic in `prole_cfg.sh` with fail-safe mechanisms.
- Added skipping logic for aged or bound PVC events in `deploy.sh` to reduce noise in diagnostics.
- Enforced replica target of 1 for specific GitLab workloads to ensure compliance with requirements.
- Added skipping logic for aged or bound PVC events in `deploy.sh` to reduce noise in diagnostics.
- Enforced replica target of 1 for specific GitLab workloads to ensure compliance with requirements.
- Added skipping logic for aged or bound PVC events in `deploy.sh` to reduce noise in diagnostics.
- Enforced replica target of 1 for specific GitLab workloads to ensure compliance with requirements.
- Improved PVC error diagnostics in `deploy.sh` with additional context.
- Added robust StorageClass reconciliation logic, including dynamic creation and parameter validation.
- Updated Helm templates to support advanced StorageClass attributes like `reclaimPolicy` and `volumeBindingMode`.
- Redirected error messages in `deploy.sh` and `render_supabase.py` to `stderr` for better logging clarity.
- Extended allowed prefixes for Supabase StorageClass to include `pd-standard`.
- Added hard validation for Supabase StorageClass configuration in `deploy.sh` and Helm templates.
- Implemented detailed reconciliation logic for GitLab workloads, addressing over-deployment scenarios.
- Enhanced `deploy.sh` with explicit namespace handling and PVC compliance checks.
- Added PVC quota error detection with detailed logging in `deploy.sh`.
- Enhanced ingress readiness checks with additional Supabase ingress resources.
- Updated Helm templates to standardize and always enable ingress for API and Studio.
