- Implemented `gitlab_split_cluster_ownership_diagnostics` to enforce and log split-cluster ownership policies between app and DB contexts.
- Updated workload convergence logic to prioritize readiness detection over historical restart counts for webservice pods.
- Enhanced deployment utilities with additional diagnostics for pod readiness states and replica discrepancies.
- Introduced tests to validate split-cluster enforcement and refined convergence behavior.
- Added logic to set and validate `GITLAB_TRUSTED_PROXIES` for forwarded headers behind ingress/load balancers.
- Improved `gitlab_old_replicaset_live_summary` to handle empty summaries gracefully by returning zero.
- Enhanced logging to confirm trusted proxies configuration.
- Updated tests to validate trusted proxies rendering and summary handling behavior.
- Introduced `gitlab_rendered_replica_source_fields_from_cr` to extract and log replica fields from rendered CR values.
- Enhanced logging for replica source-of-truth mismatches, including detailed diagnostics of rendered CR fields.
- Updated registry-specific key sanitization to align with chart 9+ authoritative fields.
- Added and extended tests to validate replica field rendering, sanitization, and logging enhancements.
- Introduced utilities to identify and strip deprecated top-level replica keys for chart v9 and newer.
- Added preflight validation to block CR apply if removed keys remain.
- Enhanced logging to detail stripped keys and remaining deprecated fields.
- Updated tests to ensure proper handling of top-level and hpa replica keys in rendered CR values.
- Added utilities for diagnosing and cleaning up StatefulSet template and PVC storage class mismatches in Gitaly.
- Improved logging for storage class fields in GitLab CR rendering and live StatefulSet diagnostics.
- Introduced `cleanup_gitlab_wrong_gitaly_template_storage` for automated destructive repair of misconfigured storage templates.
- Added tests to ensure authoritative Gitaly storage class enforcement and error handling for mismatches.
- Replaced `is_gke_standard_storage_equivalent` with `gitlab_storage_class_matches_expected` for stricter logic and better readability.
- Added utilities for diagnosing and cleaning up PVC/PV and StatefulSet template storage class mismatches.
- Introduced `enforce_gitlab_storage_class_target` to automate configuration enforcement and diagnostics.
- Enhanced logging and failure remediation for storage class mismatches in GitLab deployments.
- Introduced `gitlab_verify_replica_source_of_truth` and `gitlab_replica_source_of_truth_report` for replica drift validation and convergence.
- Enhanced logging and remediation logic to handle source-of-truth mismatches during deployment reconciliation.
- Added explicit `minReplicas` and `maxReplicas` handling for key components to prevent drift.
- Improved failure classification and reporting for post-remediation settle verification.
- Added `gitlab_selector_for_deployment` utility to dynamically resolve deployment selectors.
- Updated pod count and name retrieval to use resolved selectors, ensuring accuracy.
- Enhanced logging to include selector details for better debugging and remediation tracking.
- Addressed replica drift detection with CR-based reconciliation and scale-down logic.
- Added settle verification logic with configurable timeout and polling intervals for corrected deployments.
- Introduced new utility function `gitlab_non_terminal_pod_names_for_app` to retrieve non-terminal pod names.
- Improved logging for final deployment state and pod details after remediation actions.
- Added settle verification logic with configurable timeout and polling intervals for corrected deployments.
- Introduced new utility function `gitlab_non_terminal_pod_names_for_app` to retrieve non-terminal pod names.
- Improved logging for final deployment state and pod details after remediation actions.
- Introduced `gitlab_post_remediation_success` flag to streamline corrective action logging.
- Adjusted logic to skip unnecessary workload convergence gates when remediation is successful.
- Introduced `gitlab_post_remediation_success` flag to streamline corrective action logging.
- Adjusted logic to skip unnecessary workload convergence gates when remediation is successful.
Summarize CLAUDE.md updates:
- Document both live GKE clusters as e2-standard-2 x3 in us-west3 and note stale app default in code.
- Call out DB cluster naming mismatch between conf (knoe-dev-cnpg-0) and code default (knoe-cnpg-0).
- Add SSD quota guidance: 300GB fully consumed by CNPG, so non-CNPG PVCs must use standard/pd-standard.
- Refresh reset script reference and quota section details to match current runtime state.
Co-authored-by: Junie <junie@jetbrains.com>
- Added detailed logging for APP and DB contexts in `init_gitlab.sh` and `deploy.sh`.
- Enabled explicit passing of contexts and configuration paths to avoid fallbacks.
- Introduced strict environment resolution logic in `prole_cfg.sh` with fail-safe mechanisms.
- Added skipping logic for aged or bound PVC events in `deploy.sh` to reduce noise in diagnostics.
- Enforced replica target of 1 for specific GitLab workloads to ensure compliance with requirements.
- Added skipping logic for aged or bound PVC events in `deploy.sh` to reduce noise in diagnostics.
- Enforced replica target of 1 for specific GitLab workloads to ensure compliance with requirements.
- Added detailed validation of Sidekiq concurrency, resource requests, and limits.
- Introduced live state comparison for Sidekiq resource configuration.
- Improved logging to track desired and live Sidekiq states for better diagnostics.
- Added split-cluster detection with detailed app and DB context logging.
- Introduced robust DB connectivity checks for DNS resolution and TCP probes.
- Enabled automated cross-cluster DB ILB service creation and validation.
- Improved logging for DB migration failures with enhanced diagnostics and connectivity hints.
- Added split-cluster detection with detailed app and DB context logging.
- Introduced robust DB connectivity checks for DNS resolution and TCP probes.
- Enabled automated cross-cluster DB ILB service creation and validation.
- Improved logging for DB migration failures with enhanced diagnostics and connectivity hints.
- Introduced `get_gitlab_migrations_diagnostics` for detailed migrations pod diagnostics.
- Enhanced active migration failure detection with detailed logs and diagnostics.
- Improved job context logging with pod restart, status, and waiting reason details.
- Refined stale job repair logic by incorporating diagnostics and failure insights.
- Introduced `get_gitlab_migrations_diagnostics` for detailed migrations pod diagnostics.
- Enhanced active migration failure detection with detailed logs and diagnostics.
- Improved job context logging with pod restart, status, and waiting reason details.
- Refined stale job repair logic by incorporating diagnostics and failure insights.
- Added detailed context logging for GitLab blockers, including failed migrations and Gitaly resources.
- Introduced automatic repair for stalled migration jobs by deleting stale jobs and pods.
- Improved Gitaly resource validation with enhanced logging for PVCs and StatefulSet configurations.
- Extended PVC phase handling with detailed storageClass descriptions for better troubleshooting.
- Improved detection and logging of mismatched storageClass across CR, StatefulSet, and PVCs.
- Added auto-repair mechanisms for inconsistencies in Gitaly storageClass configuration.
- Enhanced state validation with detailed logs for better troubleshooting.
- Introduced a grace period and explicit replacement of legacy StatefulSet after expiry.
- Enhanced logging to provide clearer details on convergence progress and outcomes.
- Added handling for operator recreation and state validation during legacy repairs.
- Ensured robust detection and resolution of legacy nodeSelector and storageClass issues.
- Added logic to wait for Gitaly StatefulSet corrections after autoclean actions.
- Implemented live state validation for nodeSelector and storageClass updates.
- Enhanced logging and error handling for legacy storage repairs in k8s mode.
- Updated PV and PVC handling to address legacy resource cleanup.
- Added logging to show Kubernetes context and service inspection details for Redis.
- Improved validation by checking for service endpoints and readiness.
- Enabled IP address bypass for external host DNS checks.
- Added logging to show Kubernetes context and service inspection details for Redis.
- Improved validation by checking for service endpoints and readiness.
- Enabled IP address bypass for external host DNS checks.
- Improved `check_gitlab_pre_apply_blocked` to better detect invalid placeholders, legacy storage configurations, and DNS issues for Redis and Garage endpoints.
- Added auto-repair for Garage endpoint secrets and storage class mismatches in k8s mode.
- Introduced warnings and blocking logic for unsupported configurations like `gitlab-gitaly-static`.
- Refined storage class handling to dynamically align with `standard-rwo` in k8s/GKE.
- Introduced `check_gitlab_pre_apply_blocked` and `check_gitlab_post_apply_blocked` functions to detect misconfigurations and legacy setups during deployment.
- Added logic to validate DNS resolution for Redis and Garage endpoints, preventing invalid placeholders or inaccessible hosts.
- Enhanced storage checks to identify and optionally repair legacy storage configurations with `GITLAB_REPAIR_BLOCKED_AUTOCLEAN`.
- Updated deployment wait loop to validate readiness and detect blocking states dynamically.
- remove hardcoded physical host assumptions in UI/init flows and rely on config-driven selectors\n- optimize GitLab reruns to skip unnecessary operator upgrades/long reconcile waits when unchanged\n- add and update regression/shell tests for fast-path and k8s context/ingress/garage behaviors
Co-authored-by: Junie <junie@jetbrains.com>
- Refactored `init_gitlab.sh` and `init_gitea.sh` to use configurable node selector keys, removing hardcoded defaults like `gandalf.prole.org`.
- Enhanced scheduling logic to validate required fields and prevent stale node constraints during reruns.
- Added fast-path guards in GitLab init to skip redundant operations when no changes occur in operator or CR specifications.
- Updated Supabase deployment to clear stale topology spread constraints alongside node selectors and affinity.
- Added tests for configurable node selector keys, fast-path guards, and stale constraint cleanups.
- Added detection for "already exists" errors during bucket creation and reused existing buckets when applicable.
- Enhanced logging to provide context when reusing previously created buckets.
- Added support for validating existing ingress ownership using `gitlab_release` and `expected_backend_service`.
- Improved conflict detection by distinguishing between same-owner and foreign ingress claims.
- Introduced explicit handling for operator-managed ingress and fallback ingress creation logic.
- Enhanced logging and status return codes for better error reporting and decision-making in deployment scripts.
- Introduced `resolve_garage_admin_context` and `kubectl_garage_admin` for dynamic context resolution.
- Enhanced Garage setup with explicit logging of admin context and namespace details.
- Improved error handling and command execution reliability for multi-cluster environments.
- Added validation and explicit handling for Garage S3 endpoints in split APP/DB cluster setups.
- Improved dynamic role detection and authoritative context enforcement in initialization scripts.
- Updated Supabase and Garage configurations to support private cross-cluster endpoints, ensuring cluster interoperability and DNS constraints.
- Enhanced error handling and logging for invalid or missing configurations.
- Added base64 decoding and validation for admin token in `garage-secrets`.
- Implemented reusable `garage_exec` helper for consistent command execution with detailed error handling.
- Improved key creation, retrieval, and fallback mechanisms with enhanced output parsing and failure logging.
- Unified bucket creation and permission granting logic using the new helper function.
- Added validation for admin token existence in `garage-secrets`.
- Improved `garage key` operations with support for deterministic fallback keys and secret fetch retries.
- Unified access/secret key parsing with flexible format support (`sed` fallback).
- Added dynamic `GITALY_STORAGE_CLASS` logic for k8s and legacy modes.
- Introduced context-aware DB operations with `resolve_db_cluster_context` and `kubectl_db`.
- Improved object storage configuration with failure handling based on deployment requirements.
- Refined workload placement logic and node selector defaults for improved flexibility.
- Introduced jemalloc hostPath optimizations with configurable modes (`auto`, `off`, `force`).
- Integrated jemalloc setup with best-effort and forced validation flows for ensuring cluster compatibility.
- Enhanced monitoring storage class logic with mode-specific handling (`k3s`, `k3d`, `gke`) and improved validation of required classes.
- Added safeguards and detailed logging for unsupported configurations and failure scenarios.
