Commit Graph

729 Commits

Author SHA1 Message Date
chrisfu
a4f96d5365 chore: refine post-remediation checks in init_gitlab.sh for GitLab workload convergence
- Introduced `gitlab_post_remediation_success` flag to streamline corrective action logging.
- Adjusted logic to skip unnecessary workload convergence gates when remediation is successful.
2026-04-18 15:20:53 -07:00
chrisfu
b04ce825c9 chore: scale down over-replicated GitLab deployments to target replica count of 1 2026-04-18 15:05:55 -07:00
chrisfu
7a975e29b1 chore: scale down over-replicated GitLab deployments to target replica count of 1 2026-04-18 14:02:50 -07:00
chrisfu
3298d5cf2f chore: scale down over-replicated GitLab deployments to target replica count of 1 2026-04-18 13:32:18 -07:00
chrisfu
b8a55888be checkpoint: update GKE deployment docs and infra changes
Summarize CLAUDE.md updates:

- Document both live GKE clusters as e2-standard-2 x3 in us-west3 and note stale app default in code.

- Call out DB cluster naming mismatch between conf (knoe-dev-cnpg-0) and code default (knoe-cnpg-0).

- Add SSD quota guidance: 300GB fully consumed by CNPG, so non-CNPG PVCs must use standard/pd-standard.

- Refresh reset script reference and quota section details to match current runtime state.

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-18 07:18:10 -07:00
chrisfu
27d8a723f0 chore: scale down over-replicated GitLab deployments to target replica count of 1 2026-04-16 17:12:38 -07:00
chrisfu
b59a53f491 chore: redirect info logs to stderr in init_gitlab.sh 2026-04-16 16:41:25 -07:00
chrisfu
9a30a60a2b chore: enhance GitLab reconciliation with explicit context handling
- Added detailed logging for APP and DB contexts in `init_gitlab.sh` and `deploy.sh`.
- Enabled explicit passing of contexts and configuration paths to avoid fallbacks.
- Introduced strict environment resolution logic in `prole_cfg.sh` with fail-safe mechanisms.
2026-04-16 16:36:05 -07:00
chrisfu
20efd6de14 chore: remove unused variable in init_gitlab.sh 2026-04-16 16:04:24 -07:00
chrisfu
14edd3038c chore: improve PVC event handling and enforce GitLab workload replica targets
- Added skipping logic for aged or bound PVC events in `deploy.sh` to reduce noise in diagnostics.
- Enforced replica target of 1 for specific GitLab workloads to ensure compliance with requirements.
2026-04-16 15:42:39 -07:00
chrisfu
3651ebfb2d chore: improve PVC event handling and enforce GitLab workload replica targets
- Added skipping logic for aged or bound PVC events in `deploy.sh` to reduce noise in diagnostics.
- Enforced replica target of 1 for specific GitLab workloads to ensure compliance with requirements.
2026-04-16 01:33:32 -07:00
chrisfu
b9047cd2ad chore: improve PVC event handling and enforce GitLab workload replica targets
- Added skipping logic for aged or bound PVC events in `deploy.sh` to reduce noise in diagnostics.
- Enforced replica target of 1 for specific GitLab workloads to ensure compliance with requirements.
2026-04-16 01:16:10 -07:00
chrisfu
d14605ea23 chore: extend PVC diagnostics and enhance StorageClass reconciliation
- Improved PVC error diagnostics in `deploy.sh` with additional context.
- Added robust StorageClass reconciliation logic, including dynamic creation and parameter validation.
- Updated Helm templates to support advanced StorageClass attributes like `reclaimPolicy` and `volumeBindingMode`.
2026-04-16 00:35:15 -07:00
chrisfu
5ebe4d6c06 chore: improve error logging and extend StorageClass validation
- Redirected error messages in `deploy.sh` and `render_supabase.py` to `stderr` for better logging clarity.
- Extended allowed prefixes for Supabase StorageClass to include `pd-standard`.
2026-04-16 00:05:11 -07:00
chrisfu
0f68c4ddfe chore: enforce strict StorageClass validation and add GitLab reconciliation logic
- Added hard validation for Supabase StorageClass configuration in `deploy.sh` and Helm templates.
- Implemented detailed reconciliation logic for GitLab workloads, addressing over-deployment scenarios.
- Enhanced `deploy.sh` with explicit namespace handling and PVC compliance checks.
2026-04-15 23:40:58 -07:00
chrisfu
d0fc4af23c chore: add Supabase PVC quota diagnostics and improve ingress handling
- Added PVC quota error detection with detailed logging in `deploy.sh`.
- Enhanced ingress readiness checks with additional Supabase ingress resources.
- Updated Helm templates to standardize and always enable ingress for API and Studio.
2026-04-15 23:06:51 -07:00
chrisfu
bdeb0b62f3 chore: add Supabase PVC quota diagnostics and improve ingress handling
- Added PVC quota error detection with detailed logging in `deploy.sh`.
- Enhanced ingress readiness checks with additional Supabase ingress resources.
- Updated Helm templates to standardize and always enable ingress for API and Studio.
2026-04-15 22:51:57 -07:00
chrisfu
7c46b52e7e chore: refine deploy.sh and Supabase ingress timeout diagnostics
- Simplified public APP endpoint fallback logic in `deploy.sh`.
- Enhanced timeout handling with detailed Supabase ingress reconciliation diagnostics.
- Removed redundant ingress class override in Helm template for Supabase.
2026-04-15 22:32:41 -07:00
chrisfu
e3b1ed21c9 chore: disable PodDisruptionBudget for Knoe-DB across all deployments 2026-04-15 21:44:10 -07:00
chrisfu
a2de7213ff chore: disable PodDisruptionBudget for Knoe-DB across all deployments 2026-04-15 21:20:43 -07:00
chrisfu
1d45abb833 chore: rename Prole to Knoe.dev across deployment scripts and update ingress logic
- Standardized naming from "Prole" to "Knoe" in README, scripts, and deployment files for consistency.
- Refined ingress configuration to adjust class and annotations based on mode.
- Enhanced deploy.sh to handle public APP endpoint status for specific hosts.
2026-04-15 11:11:23 -07:00
chrisfu
e73f320740 chore: rename Prole to Knoe.dev across deployment scripts and update ingress logic
- Standardized naming from "Prole" to "Knoe" in README, scripts, and deployment files for consistency.
- Refined ingress configuration to adjust class and annotations based on mode.
- Enhanced deploy.sh to handle public APP endpoint status for specific hosts.
2026-04-15 10:55:00 -07:00
chrisfu
06862bab5c chore: fix DB-to-APP context mismatches and refine deployment logic
- Resolved context mismatches by updating DB references to APP where appropriate.
- Updated Supabase deployment to ensure proper handling of cross-cluster DB hosts.
- Simplified ingress logic by setting `split_frontdoor_to_db` to `false` for APP clusters lacking GCE Ingress support.
2026-04-15 10:10:46 -07:00
chrisfu
16873885f5 chore: fix DB-to-APP context mismatches and refine deployment logic
- Resolved context mismatches by updating DB references to APP where appropriate.
- Updated Supabase deployment to ensure proper handling of cross-cluster DB hosts.
- Simplified ingress logic by setting `split_frontdoor_to_db` to `false` for APP clusters lacking GCE Ingress support.
2026-04-15 02:05:49 -07:00
chrisfu
b6e96988fa Update GKE deployment: 5 public endpoints, cluster isolation, and IP resolution wait. - Added support for 5 public endpoints: api.knoe.dev, git.knoe.dev, db.0.knoe.dev, api.0.knoe.dev, svc.knoe.dev. - Explicitly defined APP_CLUSTER_KUBECONTEXT, DB_CLUSTER_KUBECONTEXT, and GRAFANA_HOSTNAME in conf/gke.cfg. - Updated etc/init_kong.sh to manage api.knoe.dev public ingress. - Fixed DB initialization cluster isolation by ensuring database operations use the DB cluster context in knoe/core/actions.py and knoe/core/milestones.py. - Enhanced deploy.sh with a public endpoints summary and a wait mechanism for DB cluster external IP resolution.
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-15 01:41:28 -07:00
chrisfu
4b8da412a4 chore: refine ingress logic and add post-deploy endpoint summary
- Updated `init_kong.sh` to handle `include_gitea_host` flag for improved flexibility in Gitea ingress setup.
- Added comprehensive post-deploy endpoint summary in `deploy.sh` to display public services and external IPs.
- Enhanced kubectl command handling with role-specific context logic for better split-cluster support.
- Improved DB context logging across CNPG operations and milestones.
- Updated GKE configuration with new contexts and Grafana hostname.
2026-04-15 01:28:58 -07:00
chrisfu
f2d4e2adf2 chore: improve deployment logic and readiness checks
- Added automatic persistence disabling when deployment is disabled.
- Enhanced readiness checks with support for selectors and detailed status logging.
- Improved retry logic to avoid unnecessary namespace resets for healthy app clusters.
- Refined helm deployment flow with clearer readiness and health diagnostics.
2026-04-15 00:43:27 -07:00
chrisfu
8d911b2739 chore: add MinIO diagnostics and pod security configuration
- Introduced diagnostics for MinIO failures due to file access issues, with suggested fixes for pod security settings.
- Added podSecurityContext and securityContext configurations to enforce non-root execution for MinIO.
- Updated resource requests/limits for MinIO pods to ensure optimal resource usage.
- Standardized file path references in manifest-summary.json to relative paths.
2026-04-15 00:13:16 -07:00
chrisfu
ebf08b9adb chore: add Sidekiq resource validation and logging enhancements
- Added detailed validation of Sidekiq concurrency, resource requests, and limits.
- Introduced live state comparison for Sidekiq resource configuration.
- Improved logging to track desired and live Sidekiq states for better diagnostics.
2026-04-14 23:19:06 -07:00
chrisfu
a7e03c344d chore: enhance DB connectivity diagnostics and cross-cluster support
- Added split-cluster detection with detailed app and DB context logging.
- Introduced robust DB connectivity checks for DNS resolution and TCP probes.
- Enabled automated cross-cluster DB ILB service creation and validation.
- Improved logging for DB migration failures with enhanced diagnostics and connectivity hints.
2026-04-14 22:33:18 -07:00
chrisfu
e6374e3e46 chore: enhance DB connectivity diagnostics and cross-cluster support
- Added split-cluster detection with detailed app and DB context logging.
- Introduced robust DB connectivity checks for DNS resolution and TCP probes.
- Enabled automated cross-cluster DB ILB service creation and validation.
- Improved logging for DB migration failures with enhanced diagnostics and connectivity hints.
2026-04-14 20:21:41 -07:00
chrisfu
ff5b73a435 chore: add migration diagnostics and improve failure handling
- Introduced `get_gitlab_migrations_diagnostics` for detailed migrations pod diagnostics.
- Enhanced active migration failure detection with detailed logs and diagnostics.
- Improved job context logging with pod restart, status, and waiting reason details.
- Refined stale job repair logic by incorporating diagnostics and failure insights.
2026-04-14 19:29:27 -07:00
chrisfu
e2827c97e6 chore: add migration diagnostics and improve failure handling
- Introduced `get_gitlab_migrations_diagnostics` for detailed migrations pod diagnostics.
- Enhanced active migration failure detection with detailed logs and diagnostics.
- Improved job context logging with pod restart, status, and waiting reason details.
- Refined stale job repair logic by incorporating diagnostics and failure insights.
2026-04-14 19:28:18 -07:00
chrisfu
ad351ba394 chore: enhance migration job handling and Gitaly resource validation
- Added detailed context logging for GitLab blockers, including failed migrations and Gitaly resources.
- Introduced automatic repair for stalled migration jobs by deleting stale jobs and pods.
- Improved Gitaly resource validation with enhanced logging for PVCs and StatefulSet configurations.
- Extended PVC phase handling with detailed storageClass descriptions for better troubleshooting.
2026-04-14 19:15:28 -07:00
chrisfu
444bb3926a chore: enhance Gitaly storageClass validation and repair logic
- Improved detection and logging of mismatched storageClass across CR, StatefulSet, and PVCs.
- Added auto-repair mechanisms for inconsistencies in Gitaly storageClass configuration.
- Enhanced state validation with detailed logs for better troubleshooting.
2026-04-14 18:36:30 -07:00
chrisfu
285d9fc185 chore: remove redundant local declarations in GitLab init script
- Cleaned up unnecessary `local` keyword usage for `_wait_start` and `_now` variables to improve clarity and alignment with the script's style.
2026-04-14 17:11:52 -07:00
chrisfu
7bae08db5d chore: remove redundant local declarations in GitLab init script
- Cleaned up unnecessary `local` keyword usage for `_wait_start` and `_now` variables to improve clarity and alignment with the script's style.
2026-04-14 16:56:41 -07:00
chrisfu
1502527d6f chore: improve Gitaly StatefulSet convergence and enhance legacy repair logic
- Introduced a grace period and explicit replacement of legacy StatefulSet after expiry.
- Enhanced logging to provide clearer details on convergence progress and outcomes.
- Added handling for operator recreation and state validation during legacy repairs.
- Ensured robust detection and resolution of legacy nodeSelector and storageClass issues.
2026-04-14 16:29:55 -07:00
chrisfu
5657ccb4e5 chore: refine Gitaly convergence checks and auto-repair legacy configs
- Added logic to wait for Gitaly StatefulSet corrections after autoclean actions.
- Implemented live state validation for nodeSelector and storageClass updates.
- Enhanced logging and error handling for legacy storage repairs in k8s mode.
- Updated PV and PVC handling to address legacy resource cleanup.
2026-04-14 16:14:30 -07:00
chrisfu
7999b2f81a chore: enhance Redis validation with Kubernetes context and endpoint readiness checks
- Added logging to show Kubernetes context and service inspection details for Redis.
- Improved validation by checking for service endpoints and readiness.
- Enabled IP address bypass for external host DNS checks.
2026-04-14 15:52:15 -07:00
chrisfu
9bf0758d7e chore: enhance Redis validation with Kubernetes context and endpoint readiness checks
- Added logging to show Kubernetes context and service inspection details for Redis.
- Improved validation by checking for service endpoints and readiness.
- Enabled IP address bypass for external host DNS checks.
2026-04-14 15:40:32 -07:00
chrisfu
a37015c887 chore: enhance Garage ILB endpoint resolution and persist config updates
- Added fallback logic to resolve Garage ILB hostname if IP is unavailable.
- Improved handling of `self.inputs` with prioritized merging for better automation.
- Updated config persistence to use `write_config` for milestones and repairs.
- Extended Garage ILB IP retry logic to 5 minutes for GKE provisioning delays.
2026-04-14 14:45:23 -07:00
chrisfu
5fabb94a19 chore: update Garage ILB logic for split-cluster GKE and GitLab object storage
- Changed ILB deployment context to DB cluster (`knoe-cnpg-0`) for GitLab object storage setup.
- Renamed service to `garage-s3-ilb` and updated related references.
- Enhanced logging for Garage ILB IP resolution with retry and error handling.
- Persisted resolved ILB endpoint to config files for improved reliability.
- Refined split-cluster Garage deployment workflows with clearer context management and debugging.
2026-04-14 14:17:41 -07:00
chrisfu
9ab6b3a1cd chore: refactor service_ns initialization in GitLab deployment pipeline
- Moved `service_ns` definition earlier in the initialization process for improved readability and consistency.
- Removed redundant `service_ns` reassignment, optimizing namespace resolution logic.
2026-04-14 14:04:06 -07:00
chrisfu
3db704bc52 chore: add support for resolving Garage ILB endpoint in split-cluster GKE setups
- Introduced `get_endpoint_ip` utility to fetch Garage ILB IPs across k3s, k3d, and k8s modes.
- Updated initialization to deploy Garage on the DB cluster when using split GKE clusters.
- Enhanced logic to persist resolved Garage ILB endpoints for GitLab object storage configurations.
- Added Redis master resolution and removed placeholder Garage endpoints during deployment.
- Updated post-init scripts to include Redis initialization alongside Kong and backup scripts.
2026-04-14 14:02:11 -07:00
chrisfu
8c4a965ded chore: enhance GitLab init to block misconfigurations and repair legacy setups
- Improved `check_gitlab_pre_apply_blocked` to better detect invalid placeholders, legacy storage configurations, and DNS issues for Redis and Garage endpoints.
- Added auto-repair for Garage endpoint secrets and storage class mismatches in k8s mode.
- Introduced warnings and blocking logic for unsupported configurations like `gitlab-gitaly-static`.
- Refined storage class handling to dynamically align with `standard-rwo` in k8s/GKE.
2026-04-14 13:35:11 -07:00
chrisfu
cf293aa751 chore: add pre/post-apply sanity checks for GitLab deployment blockers
- Introduced `check_gitlab_pre_apply_blocked` and `check_gitlab_post_apply_blocked` functions to detect misconfigurations and legacy setups during deployment.
- Added logic to validate DNS resolution for Redis and Garage endpoints, preventing invalid placeholders or inaccessible hosts.
- Enhanced storage checks to identify and optionally repair legacy storage configurations with `GITLAB_REPAIR_BLOCKED_AUTOCLEAN`.
- Updated deployment wait loop to validate readiness and detect blocking states dynamically.
2026-04-14 13:12:56 -07:00
chrisfu
174877201e chore: update GKE storage class and reconcile PVCs for Supabase deployment
- Changed default GKE non-DB PVCs to use CSI `pd-standard` with WFFC mode, deprecating legacy `standard`.
- Added reconciliation logic for APP PVCs to align with updated storage class.
- Enforced single replica for Supabase functions on GKE, disabling autoscaling.
- Updated tests to validate storage class changes, PVC reconciliation, and single-replica logic.
2026-04-14 11:13:14 -07:00
chrisfu
fddaf80823 checkpoint: make node placement config-driven and add GitLab rerun fast-path
- remove hardcoded physical host assumptions in UI/init flows and rely on config-driven selectors\n- optimize GitLab reruns to skip unnecessary operator upgrades/long reconcile waits when unchanged\n- add and update regression/shell tests for fast-path and k8s context/ingress/garage behaviors

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-14 06:43:37 -07:00
chrisfu
492caabf60 chore: make node selector keys configurable and improve scheduling logic
- Refactored `init_gitlab.sh` and `init_gitea.sh` to use configurable node selector keys, removing hardcoded defaults like `gandalf.prole.org`.
- Enhanced scheduling logic to validate required fields and prevent stale node constraints during reruns.
- Added fast-path guards in GitLab init to skip redundant operations when no changes occur in operator or CR specifications.
- Updated Supabase deployment to clear stale topology spread constraints alongside node selectors and affinity.
- Added tests for configurable node selector keys, fast-path guards, and stale constraint cleanups.
2026-04-14 06:01:50 -07:00