- Documented step-by-step procedure for integrating GitLab with Google Workspace via OIDC (Path A).
- Outlined secret management patterns, GCP OAuth client setup, and deployment instructions.
- Included detailed remediation steps for secret leakage and Git history cleanup.
- Added verification and troubleshooting guides for pre- and post-deployment scenarios.
test: add helm monitoring tests for stale lock recovery and upgrade handling
- Introduced tests to validate Prometheus update flow with stale pending locks, including rollback and upgrade scenarios.
- Ensured robust handling for edge cases like missing deployed revisions and context mismatches.
- Improved handling of GKE preload TLS annotations by ignoring `SERVICE_PRE_SHARED_CERT` in k8s mode with managed TLS resources.
- Added drift detection for ingress class, ManagedCertificate, and FrontendConfig annotations, with forced replacement for GCLB binding consistency.
- Enhanced TLS diagnostics with detailed path state, failure reasons, and HTTPS probe handling for GKE configurations.
- Updated tests and scripts to validate GKE-specific behavior and ingress rendering.
- Updated `deploy.sh` and `init_kong.sh` to support GKE ManagedCertificate and FrontendConfig reconciliation in k8s mode with GCE ingress class.
- Enhanced TLS path diagnostics to distinguish between missing, attached-but-not-serving, and actively serving configurations.
- Added HTTPS probing for GKE-managed public ingress paths and validation for managed certificate statuses.
- Updated `render_supabase.py` to generate and attach TLS annotations for Supabase API and Studio ingresses in k8s mode.
- Added tests to validate TLS path handling, probing, and manifest generation for GKE deployments.
- Introduced `GITLAB_FALLBACK_INGRESS_NAME` for better flexibility in managing fallback ingress names.
- Added logic to clean up both current and legacy fallback ingresses (`gitlab-frontdoor-ingress`, `gitlab-kong-ingress`).
- Enhanced diagnostics for ingress host conflicts and TLS path validation.
- Updated `deploy.sh` to improve runtime configuration resolution and host diagnostics.
- Refined tests to ensure correct fallback ingress handling and conflict resolution.
- Updated `deploy.sh` to resolve project IDs from multiple configuration fallbacks and environment contexts for retained disk cleanup.
- Added support for `supabase_cfg_first_nonempty_value` to prioritize configuration over defaults.
- Enhanced logging with explicit reasons for skipped disk cleanup actions (e.g., missing gcloud, unresolved project).
- Updated tests to validate fallback logic and skip reason reporting.
- Changed GitLab ingress default ownership in GKE/k8s mode from `operator` to `fallback` for explicit GCE ingress path anchoring.
- Added logic to clean up stale operator-managed ingresses when fallback ownership is selected.
- Simplified fallback ingress creation workflow by removing redundant skip logic.
- Updated relevant tests to reflect ownership changes and ingress handling enhancements.
- Implemented `gitlab_split_cluster_ownership_diagnostics` to enforce and log split-cluster ownership policies between app and DB contexts.
- Updated workload convergence logic to prioritize readiness detection over historical restart counts for webservice pods.
- Enhanced deployment utilities with additional diagnostics for pod readiness states and replica discrepancies.
- Introduced tests to validate split-cluster enforcement and refined convergence behavior.
- Added logic to set and validate `GITLAB_TRUSTED_PROXIES` for forwarded headers behind ingress/load balancers.
- Improved `gitlab_old_replicaset_live_summary` to handle empty summaries gracefully by returning zero.
- Enhanced logging to confirm trusted proxies configuration.
- Updated tests to validate trusted proxies rendering and summary handling behavior.
- Introduced `gitlab_rendered_replica_source_fields_from_cr` to extract and log replica fields from rendered CR values.
- Enhanced logging for replica source-of-truth mismatches, including detailed diagnostics of rendered CR fields.
- Updated registry-specific key sanitization to align with chart 9+ authoritative fields.
- Added and extended tests to validate replica field rendering, sanitization, and logging enhancements.
- Introduced utilities to identify and strip deprecated top-level replica keys for chart v9 and newer.
- Added preflight validation to block CR apply if removed keys remain.
- Enhanced logging to detail stripped keys and remaining deprecated fields.
- Updated tests to ensure proper handling of top-level and hpa replica keys in rendered CR values.
- Added utilities for diagnosing and cleaning up StatefulSet template and PVC storage class mismatches in Gitaly.
- Improved logging for storage class fields in GitLab CR rendering and live StatefulSet diagnostics.
- Introduced `cleanup_gitlab_wrong_gitaly_template_storage` for automated destructive repair of misconfigured storage templates.
- Added tests to ensure authoritative Gitaly storage class enforcement and error handling for mismatches.
- Added `_resolve_secretref_value` to handle `secretref://` values for `auth.clientId` and `auth.clientSecret`.
- Extended environment and file-based lookups for secretref resolution.
- Updated tests to validate new secretref resolution logic in config and GitOps workflows.
- Implemented `auth.clientId` and `auth.clientSecret` backfilling from Global section and environment variables.
- Added tests to verify correct hydration of OIDC inputs in configuration defaults.
- Introduced `gitlab_post_remediation_success` flag to streamline corrective action logging.
- Adjusted logic to skip unnecessary workload convergence gates when remediation is successful.
- Changed default GKE non-DB PVCs to use CSI `pd-standard` with WFFC mode, deprecating legacy `standard`.
- Added reconciliation logic for APP PVCs to align with updated storage class.
- Enforced single replica for Supabase functions on GKE, disabling autoscaling.
- Updated tests to validate storage class changes, PVC reconciliation, and single-replica logic.
- remove hardcoded physical host assumptions in UI/init flows and rely on config-driven selectors\n- optimize GitLab reruns to skip unnecessary operator upgrades/long reconcile waits when unchanged\n- add and update regression/shell tests for fast-path and k8s context/ingress/garage behaviors
Co-authored-by: Junie <junie@jetbrains.com>
- Refactored `init_gitlab.sh` and `init_gitea.sh` to use configurable node selector keys, removing hardcoded defaults like `gandalf.prole.org`.
- Enhanced scheduling logic to validate required fields and prevent stale node constraints during reruns.
- Added fast-path guards in GitLab init to skip redundant operations when no changes occur in operator or CR specifications.
- Updated Supabase deployment to clear stale topology spread constraints alongside node selectors and affinity.
- Added tests for configurable node selector keys, fast-path guards, and stale constraint cleanups.
- Changed default storage class for GKE to `standard` to reduce pressure on SSD/balanced quotas.
- Disabled `enforceGeneralNodeRole` scheduling for APP/DB values to support split cluster setups.
- Updated tests to validate storage class and scheduling logic changes.
- Introduced `SERVICE_INGRESS_TLS_ENABLED` for configurable svc ingress TLS rendering in `init_kong.sh`.
- Updated ingress templates to include or exclude TLS annotations and blocks dynamically.
- Improved PVC and pod readiness checks in `supabase/deploy.sh` with enriched logging and detailed blockers.
- Added support for configurable storage class resolution for Supabase DB frontdoor PVCs.
- Enhanced deployment workflows to handle split APP/DB cluster setups with context-aware readiness checks.
- Updated test cases to validate svc ingress TLS configuration and storage class logic.
- Introduced `enforceGeneralNodeRole` to make node affinity and selector enforcement configurable.
- Updated `_split_frontdoor_docs` to handle custom release labels for Kong services.
- Enhanced Helm templates to support dynamic scheduling configurations.
- Added `reconcile_db_frontdoor_studio_pvcs` to migrate PVCs to the target storage class.
- Updated tests to validate custom release labels and scheduling logic.
- Updated `_extract_k8s_docs` to use `yaml.safe_load_all` for improved parsing and reliability.
- Enhanced `_split_frontdoor_docs` with stricter validation of Deployment, Service, and Ingress specs.
- Added namespace enforcement for frontdoor resources during manifest splitting.
- Refactored and optimized test cases to cover new validation and splitting functionality.
- Updated `_extract_k8s_docs` to use `yaml.safe_load_all` for improved parsing and reliability.
- Enhanced `_split_frontdoor_docs` with stricter validation of Deployment, Service, and Ingress specs.
- Added namespace enforcement for frontdoor resources during manifest splitting.
- Refactored and optimized test cases to cover new validation and splitting functionality.
- Introduced jemalloc hostPath optimizations with configurable modes (`auto`, `off`, `force`).
- Integrated jemalloc setup with best-effort and forced validation flows for ensuring cluster compatibility.
- Enhanced monitoring storage class logic with mode-specific handling (`k3s`, `k3d`, `gke`) and improved validation of required classes.
- Added safeguards and detailed logging for unsupported configurations and failure scenarios.
- Added `gke-gcloud-auth-plugin` as a required dependency for `linux/apt` environments.
- Enhanced GKE context acquisition with writable kubeconfig target resolution.
- Improved failure reporting with detailed messages for GKE dependency and context issues.
- Expanded test coverage for scenarios involving missing auth plugins, kubeconfig errors, and GKE configurations.
- Replaced static dependency configuration with dynamic resolution using `get_required_dependencies` and `get_required_dependency_ids`.
- Streamlined runtime checks, fallback behaviors, and handling of missing/optional dependencies.
- Hardened GKE context acquisition and validation for Kubernetes clusters.
- Enhanced test coverage for dependency resolution under various deployment modes (`dev`, `prod`, `gke`).
- Replaced static dependency configuration with dynamic resolution using `get_required_dependencies` and `get_required_dependency_ids`.
- Streamlined runtime checks, fallback behaviors, and handling of missing/optional dependencies.
- Hardened GKE context acquisition and validation for Kubernetes clusters.
- Enhanced test coverage for dependency resolution under various deployment modes (`dev`, `prod`, `gke`).
- Updated default paths, namespaces, and cluster names for k3d/dev configuration.
- Added `skip_names` set to exclude `prod.cfg` and `gcp.cfg` from override processing.
- Refined init parameters for Supabase, Kubernetes, and database deployments to align with dev-specific settings.
- Enhanced test coverage for excluded config files (`prod.cfg`, `gcp.cfg`) within `test_prole_conf.py`.
- Updated default paths, namespaces, and cluster names for k3d/dev configuration.
- Added `skip_names` set to exclude `prod.cfg` and `gcp.cfg` from override processing.
- Refined init parameters for Supabase, Kubernetes, and database deployments to align with dev-specific settings.
- Enhanced test coverage for excluded config files (`prod.cfg`, `gcp.cfg`) within `test_prole_conf.py`.
- Updated default paths, namespaces, and cluster names for k3d/dev configuration.
- Added `skip_names` set to exclude `prod.cfg` and `gcp.cfg` from override processing.
- Refined init parameters for Supabase, Kubernetes, and database deployments to align with dev-specific settings.
- Enhanced test coverage for excluded config files (`prod.cfg`, `gcp.cfg`) within `test_prole_conf.py`.
Rename env config files from conf/*/prole.cfg to conf/k3d.cfg, conf/k3s.cfg, and conf/gke.cfg. Update shell/Python loaders and etc/deploy scripts to resolve named configs cleanly while keeping legacy fallback behavior. Align k3s Ansible tasks, docs, and regression coverage with the new configuration layout.
Co-authored-by: Junie <junie@jetbrains.com>
Switch production config to k8s/GKE contexts and align service naming. Add immutable StatefulSet update fallback for Garage across k3d/k3s/k8s. Harden CNPG deploy and backup bootstrap paths, and update installer coverage for CNPG webhook and Garage common ops.
Co-authored-by: Junie <junie@jetbrains.com>
- Updated default paths, namespaces, and cluster names for k3d/dev configuration.
- Added `skip_names` set to exclude `prod.cfg` and `gcp.cfg` from override processing.
- Refined init parameters for Supabase, Kubernetes, and database deployments to align with dev-specific settings.
- Enhanced test coverage for excluded config files (`prod.cfg`, `gcp.cfg`) within `test_prole_conf.py`.
- Introduced support for dynamic GitLab and Supabase domain and ingress configuration based on deployment mode (k8s vs local).
- Default GitLab domain and ingress class aligned with `prole.cfg` deployment settings.
- Expanded Supabase support:
- Added configurable API and Studio hostnames with default fallbacks.
- Introduced separate ingress class configuration for API and Studio.
- Standardized public URL normalization for both services.
- Updated tests to validate domain, ingress, and configuration behaviors across environments.
- separate default app and db cluster contexts (knoe-dev-0 and knoe-cnpg-0) across actions, ops flows, and installer config\n- update CNPG placement and GKE manifests to keep PostgreSQL on the db cluster while app services stay on the app cluster\n- add GCS-backed Barman ObjectStore manifest and cross-cluster Garage patch workflow\n- refresh UI cluster/database/service screens and monitoring wiring for dual-cluster operation\n- add reset/patch scripts, db context selection test coverage, and architecture/network documentation updates
Co-authored-by: Junie <junie@jetbrains.com>
- harden CNPG backup ObjectStore readiness fallback in statusless clusters by validating controller availability and expected spec
- improve backup/plugin reconciliation paths and related test coverage for barman-cloud flows
- include ongoing GKE/CNPG deployment and configuration updates from this checkpoint
Co-authored-by: Junie <junie@jetbrains.com>
UI screens
- database.py: fix mode detection to use env_key priority (prod→k8s, service→k3s) so stale DEPLOYMENT_MODE never overrides the user's chosen environment
- database.py: Registry status reads ARTIFACT_REGISTRY_AVAILABLE persisted by cluster screen; uses SERVICE_NAMESPACE for Artifact Registry repo name
- cluster.py: add Artifact Registry traffic light (amber→green/red) to prod section; _check_artifact_registry_async persists ARTIFACT_REGISTRY_AVAILABLE into Global cfg
- cluster.py: re-trigger Artifact Registry check after GKE cluster selection so the light re-evaluates once region is available from KUBECONTEXT
- cluster_nodes.py: fix TclError on Python 3.14 — pady=(2,0) tuple → pady=2 scalar
- __init__.py: seed knoe-system namespace when saved value is "default", not only when empty
- services.py: replace hardcoded "Prole DB" log string with dynamic cnpg_cluster name
Core ops
- cloudnative_pg.py: replace one-shot Barman plugin retry with 6-attempt loop; first cert-manager/x509 failure triggers rollout restart + 30 s CA propagation wait; subsequent failures back off up to 60 s per attempt
- cloudnative_pg.py: TLS CA CN now uses cluster_name instead of hardcoded "Prole CNPG CA"
- registry.py, garage_store.py: refactored into per-mode modules (k3d/k3s/k8s registry and garage store, shared _garage_common)
Deploy / config
- deploy/gcp/gke/knoe-db.yaml: GKE-specific CNPG cluster manifest (rw/ro/r on separate nodes with premium-rwo storage)
- etc/init_common_services.sh, modes/k8s/knoe-db/.version: updated for current deploy
- kong-deployment.yaml: updated manifest
Tests
- test_cluster_nodes_render_smoke.py: add pack/grid, winfo_children, winfo_reqheight, update_idletasks, grid_slaves to dummy widgets; monkeypatch tk.Label so CNPG placement render completes without a real Tkinter root
Co-authored-by: Junie <junie@jetbrains.com>
- update Kong/Supabase ingress and service manifests for k3s
- expand init scripts and runtime config/topology handling
- refresh KNØE UI screens and Supabase deployment/render logic
- include related env helper test adjustments
Co-authored-by: Junie <junie@jetbrains.com>
