Follow-up to 23f3f14. With the checksum annotation in place, `helm
upgrade` did start rolling Kong forward, but the new pod never became
Ready: `rollout status` timed out with
"1 old replicas are pending termination"
Kong's ConfigMap contains the /healthz route correctly, the BackendConfig
is HTTP /healthz, but the Kong process itself refuses to start cleanly
when one of its services has `url: http://127.0.0.1:8000/` -- Kong's own
proxy port. This self-reference apparently trips the declarative-config
parser (undocumented; symptomatic only).
Switch to a RFC-2606 `.invalid` placeholder. The request-termination
plugin short-circuits before any DNS lookup or upstream connection, so
a non-resolvable URL is functionally equivalent and Kong's parser has
nothing to complain about.
Same change in both places we define the /healthz route:
- supabase/helm/knoe-supabase/templates/kong/config.yaml (supabase-kong)
- etc/init_kong.sh inline kong.yml heredoc (knoe-svc-kong)
The knoe-svc-kong backend was already HEALTHY with the self-referential
URL -- different Kong instance, possibly different parser path -- but
keeping the two configs aligned so future edits don't drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to 391c4f5. Live deploy showed GCE's L7 BackendConfig CRD
hard-rejects type: TCP with:
Error syncing to GCP: error running backend syncing routine:
error ensuring health check:
Protocol "TCP" is not valid, must be one of [HTTP,HTTPS,HTTP2]
The sync never completes, so the LB has no healthy backend and TCP
connections to the public endpoint just close (ERR_CONNECTION_CLOSED).
Fix: switch all three BackendConfigs to type: HTTP with request paths
that return 200:
- supabase-kong & knoe-svc-kong: add a dedicated /healthz route to the
Kong declarative config via the request-termination plugin, which
returns 200 synchronously with no upstream call. Equivalent liveness
semantics to the TCP check we wanted (backend is alive as long as Kong
accepts connections) but over HTTP, which GCE actually accepts.
- supabase/helm/knoe-supabase/templates/kong/config.yaml
- etc/init_kong.sh (inline kong.yml heredoc)
- supabase-studio: Studio returns 301 on / (Next.js default) so we
point the probe at /favicon.ico -- Next.js serves it as a static asset
with 200 unconditionally. Not as clean as a real readiness endpoint
but Studio does not expose one that returns 200 without auth.
- supabase/helm/knoe-supabase/templates/studio/backendconfig.yaml
Verified locally via helm template -f values.generated.json: the
rendered BackendConfigs come out with the HTTP protocol + correct paths,
and the Kong ConfigMap has the healthz service block before the
auth-v1-open service.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GCE L7 Ingresses for knoe-svc-kong (svc.knoe.dev / api.knoe.dev),
supabase-kong (api.0.knoe.dev) and supabase-studio (db.0.knoe.dev) were
all stuck UNHEALTHY: the default GCE healthCheck is HTTP GET `/` on the
backend port, but Kong returns 404 on any unrouted path and Studio
returns a 301 redirect -- neither passes the default probe, so the LB
serves "Server Error" instead of reaching the pod.
Replicate the pattern already working for gitlab-webservice-default:
emit a BackendConfig CRD with a TCP healthCheck on the service port and
annotate the Service with cloud.google.com/backend-config so GCE picks
it up. TCP is sufficient for LB-level liveness -- the backend is "alive"
as long as the process is accepting connections.
- etc/init_kong.sh: new SVC_KNOE_BACKEND_CONFIG_NAME; apply BackendConfig
inside k8s/GCE branch; annotate Service post-apply.
- knoe-supabase chart: new kong/backendconfig.yaml + studio/backendconfig.yaml
(TCP on 8000 / 3000), gated on service.{kong,studio}.backendConfigName.
- knoe-supabase chart: kong/service.yaml + studio/service.yaml pick up
cloud.google.com/backend-config when backendConfigName is set.
- render_supabase.py: sets service.{kong,studio}.backendConfigName in k8s
mode so the above wires up automatically.
Separately, the chart-managed supabase-kong Ingress was being reaped
from the cluster seconds after helm install (manifest present in the
release, gone via `kubectl get`). Root cause TBD -- suspected
meta.helm.sh/* annotation ownership colliding with a GKE/Anthos audit
controller. Workaround: render_supabase.py now emits a standalone
public-ingress-kong.yaml (no helm metadata) that supabase/deploy.sh
applies alongside public-ingress-tls.yaml, and the chart template gains
an `ingress.externallyManaged` guard so it no-ops in k8s mode. Default
`false` keeps k3d/k3s behavior unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Discovered today after the SSH-ingress reconcile: init_gitlab.sh had a
destructive false-positive in its "legacy storage detection" at the
top of setup_gitlab_storage(). The bare check
if kubectl -n "$NAMESPACE" get pvc repo-data-gitlab-gitaly-0 >/dev/null 2>&1;
then has_legacy=1; fi
flagged has_legacy=1 whenever the PVC object simply existed — which is
always true after the first successful deploy. Combined with
GITLAB_REPAIR_BLOCKED_AUTOCLEAN=1, every init_gitlab.sh invocation
triggered the "repairing legacy Gitaly storage" branch, which scales
the Gitaly StatefulSet to 0 and deletes repo-data-gitlab-gitaly-0
immediately, wiping all git-repo data.
We caught it when the knoe-db repo (pushed via HTTPS earlier in the
day) disappeared after an unrelated init_gitlab.sh run for SSH ingress
wiring. The PVC age on inspection was ~26 min — newer than the push.
git ls-remote against the project now returns a garbled gzip body,
consistent with an empty gitaly backing an existing DB project.
Fix:
- etc/init_gitlab.sh:setup_gitlab_storage()
* Remove the bare `pvc exists` legacy check.
* Keep the real legacy markers: synology PV, gandalf.prole.org
nodeSelector, gitlab-gitaly-static storageClass on the STS.
* Add a new check that flags has_legacy=1 only when the PVC's actual
storageClassName is a known-legacy value (gitlab-gitaly-static or
synology-iscsi), never on mere existence.
- conf/gke.cfg
* GITLAB_REPAIR_BLOCKED_AUTOCLEAN: 1 -> 0. The script fix above makes
re-enabling safe, but the conservative default is to let the user
see repair_blocked and consent rather than silently wipe data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reserved 5 named external static IPs and threaded the
kubernetes.io/ingress.global-static-ip-name annotation through every
public GCE L7 ingress the installer renders, so ingress delete/recreate
stops churning DNS. Also turned on a dedicated regional L4
LoadBalancer for gitlab-shell (port 22) so SSH git workflows work.
New config surface in conf/gke.cfg:
GITLAB_GLOBAL_STATIC_IP_NAME = git-knoe-dev (34.102.141.87)
SVC_KNOE_GLOBAL_STATIC_IP_NAME = svc-knoe (34.111.197.33)
SUPABASE_API_GLOBAL_STATIC_IP_NAME = supabase-api (34.120.221.5)
SUPABASE_STUDIO_GLOBAL_STATIC_IP_NAME = supabase-studio (136.110.189.6)
GITLAB_SSH_HOST = git-ssh.knoe.dev
GITLAB_SHELL_LOADBALANCER_IP = 34.106.243.154 (regional us-west3)
GITLAB_SHELL_EXTERNAL_TRAFFIC_POLICY = Local (default)
Why SSH lives on a separate hostname+IP: Google-managed certs require
a GCE global L7 IP; port 22 needs a regional Network LB. Those can not
share an IP on GCP, so git.knoe.dev stays on HTTPS and git-ssh.knoe.dev
takes SSH.
Wiring per surface:
- etc/init_gitlab.sh
* Added GITLAB_SSH_HOST (default git-ssh.<root> in k8s mode),
GITLAB_SHELL_LOADBALANCER_IP, GITLAB_SHELL_EXTERNAL_TRAFFIC_POLICY,
GITLAB_GLOBAL_STATIC_IP_NAME config keys.
* CR global.hosts.ssh now reads ${GITLAB_SSH_HOST}.
* gitlab-shell block conditionally renders service: {type:
LoadBalancer, loadBalancerIP, externalTrafficPolicy} when the LB IP
is set.
* GITLAB_GCE_TLS_ANNOTATIONS_YAML gains
kubernetes.io/ingress.global-static-ip-name: "<name>" when
GITLAB_GLOBAL_STATIC_IP_NAME is set.
- etc/init_kong.sh
* Added SVC_KNOE_GLOBAL_STATIC_IP_NAME var near other SERVICE_TLS_*
defaults.
* gce_tls_annotations heredoc gets the static-IP line appended on the
same condition.
- supabase/helm/render_supabase.py
* Two new _first(env, cfg, default) extractions for
SUPABASE_API_GLOBAL_STATIC_IP_NAME and
SUPABASE_STUDIO_GLOBAL_STATIC_IP_NAME near the existing managed-cert
/ frontend-config vars.
* Two new dict-spread blocks in the Kong + Studio ingress annotations
that emit the static-IP annotation only when mode=k8s and the value
is non-empty.
All three ingress surfaces already use kubectl apply (merge-friendly);
re-running init_gitlab.sh / init_kong.sh / deploy.sh (supabase step) is
enough to pick up the new annotation. The GKE LB controller will swap
each ingress's forwarding rule from the auto-generated k8s2-fr-*
reservation to the named reservation, then release the old ephemeral.
Ingress IPs change; DNS records need updating (TTL <=300s while
iterating). Google-managed cert for gitlab-managed-cert will briefly
Provision again during the swap; the other three were already
Provisioning.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enables iterating on GitLab reconciles without a full deploy.sh run and
unblocks the end-to-end login flow:
- etc/init_gitlab.sh
* Pre-scan $@ for --config before sourcing prole_cfg.sh so
PROLE_DEPLOY_CFG is set for standalone invocations; previously
prole_cfg.sh auto-picked conf/k3d.cfg and leaked KUBECONTEXT=dev,
failing with `error: context "dev" does not exist`.
* _init_gitlab_resolve_secretref: bash mirror of
knoe/core/actions.py:_resolve_secretref_value so
secretref://google-oidc-client-* resolves from etc/secrets/ files
when run outside deploy.sh Python env-injection.
* Garage bucket array: drop -storage suffix from uploads/artifacts/
lfs/packages/dependency-proxy. Chart defaults for these have no
suffix, and the object_store block in the CR does not override
per-object bucket names; the prior mismatch caused first-login 500s
(NoSuchBucket on avatar PUT).
- deploy/gcp/gke/gitlab-google-oidc-secret.example.yaml
* Add discovery: true so omniauth-openid_connect fetches Google
.well-known/openid-configuration; fixes "Could not authenticate
from OpenIDConnect: No host info" on the callback.
- conf/gke.cfg
* GITLAB_WEBSERVICE_LIMITS_MEMORY 1800M -> 3Gi, REQUESTS_MEMORY
900M -> 2Gi, REQUESTS_CPU 200m -> 500m. Live pod was sitting at
1706Mi/1800M (99%) in OOMKilled loop.
- conf/port-mapping.cfg: add supabase + gitea forwards, fix postgres
namespace knoe-db -> knoe-db-0 for split-cluster CNPG layout.
Adds read-only diagnostics used to chase the above:
- etc/diag_gitlab_boot.sh, diag_gitlab_webservice_oom.sh, diag_gke_storage.sh
- etc/ensure_default_storage_class.sh + k8s/prole/storageclass-gcp-standard-hdd.yaml
(preflight + HDD-default SC manifest for SSD-quota-constrained GKE projects).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Added pre-apply validation for ManagedCertificate and FrontendConfig objects to ensure existence before ingress application.
- Improved GKE ingress diagnostics with annotations and resource state checks for pre- and post-apply phases.
- Introduced patch-only handling for annotation drifts to avoid unnecessary ingress replacements.
- Enhanced HTTPS probing logic with GCLB convergence checks for managed certificate states.
- Updated `deploy.sh`, `init_kong.sh`, and `init_gitlab.sh` with detailed ingress dependency validation and reconciliation steps.
- Improved handling of GKE preload TLS annotations by ignoring `SERVICE_PRE_SHARED_CERT` in k8s mode with managed TLS resources.
- Added drift detection for ingress class, ManagedCertificate, and FrontendConfig annotations, with forced replacement for GCLB binding consistency.
- Enhanced TLS diagnostics with detailed path state, failure reasons, and HTTPS probe handling for GKE configurations.
- Updated tests and scripts to validate GKE-specific behavior and ingress rendering.
- Updated `deploy.sh` and `init_kong.sh` to support GKE ManagedCertificate and FrontendConfig reconciliation in k8s mode with GCE ingress class.
- Enhanced TLS path diagnostics to distinguish between missing, attached-but-not-serving, and actively serving configurations.
- Added HTTPS probing for GKE-managed public ingress paths and validation for managed certificate statuses.
- Updated `render_supabase.py` to generate and attach TLS annotations for Supabase API and Studio ingresses in k8s mode.
- Added tests to validate TLS path handling, probing, and manifest generation for GKE deployments.
- Introduced `GITLAB_FALLBACK_INGRESS_NAME` for better flexibility in managing fallback ingress names.
- Added logic to clean up both current and legacy fallback ingresses (`gitlab-frontdoor-ingress`, `gitlab-kong-ingress`).
- Enhanced diagnostics for ingress host conflicts and TLS path validation.
- Updated `deploy.sh` to improve runtime configuration resolution and host diagnostics.
- Refined tests to ensure correct fallback ingress handling and conflict resolution.
- Changed GitLab ingress default ownership in GKE/k8s mode from `operator` to `fallback` for explicit GCE ingress path anchoring.
- Added logic to clean up stale operator-managed ingresses when fallback ownership is selected.
- Simplified fallback ingress creation workflow by removing redundant skip logic.
- Updated relevant tests to reflect ownership changes and ingress handling enhancements.
- Implemented `gitlab_split_cluster_ownership_diagnostics` to enforce and log split-cluster ownership policies between app and DB contexts.
- Updated workload convergence logic to prioritize readiness detection over historical restart counts for webservice pods.
- Enhanced deployment utilities with additional diagnostics for pod readiness states and replica discrepancies.
- Introduced tests to validate split-cluster enforcement and refined convergence behavior.
- Added logic to set and validate `GITLAB_TRUSTED_PROXIES` for forwarded headers behind ingress/load balancers.
- Improved `gitlab_old_replicaset_live_summary` to handle empty summaries gracefully by returning zero.
- Enhanced logging to confirm trusted proxies configuration.
- Updated tests to validate trusted proxies rendering and summary handling behavior.
- Introduced `gitlab_rendered_replica_source_fields_from_cr` to extract and log replica fields from rendered CR values.
- Enhanced logging for replica source-of-truth mismatches, including detailed diagnostics of rendered CR fields.
- Updated registry-specific key sanitization to align with chart 9+ authoritative fields.
- Added and extended tests to validate replica field rendering, sanitization, and logging enhancements.
- Introduced utilities to identify and strip deprecated top-level replica keys for chart v9 and newer.
- Added preflight validation to block CR apply if removed keys remain.
- Enhanced logging to detail stripped keys and remaining deprecated fields.
- Updated tests to ensure proper handling of top-level and hpa replica keys in rendered CR values.
- Added utilities for diagnosing and cleaning up StatefulSet template and PVC storage class mismatches in Gitaly.
- Improved logging for storage class fields in GitLab CR rendering and live StatefulSet diagnostics.
- Introduced `cleanup_gitlab_wrong_gitaly_template_storage` for automated destructive repair of misconfigured storage templates.
- Added tests to ensure authoritative Gitaly storage class enforcement and error handling for mismatches.
- Replaced `is_gke_standard_storage_equivalent` with `gitlab_storage_class_matches_expected` for stricter logic and better readability.
- Added utilities for diagnosing and cleaning up PVC/PV and StatefulSet template storage class mismatches.
- Introduced `enforce_gitlab_storage_class_target` to automate configuration enforcement and diagnostics.
- Enhanced logging and failure remediation for storage class mismatches in GitLab deployments.
- Introduced `gitlab_verify_replica_source_of_truth` and `gitlab_replica_source_of_truth_report` for replica drift validation and convergence.
- Enhanced logging and remediation logic to handle source-of-truth mismatches during deployment reconciliation.
- Added explicit `minReplicas` and `maxReplicas` handling for key components to prevent drift.
- Improved failure classification and reporting for post-remediation settle verification.
- Added `gitlab_selector_for_deployment` utility to dynamically resolve deployment selectors.
- Updated pod count and name retrieval to use resolved selectors, ensuring accuracy.
- Enhanced logging to include selector details for better debugging and remediation tracking.
- Addressed replica drift detection with CR-based reconciliation and scale-down logic.
- Added settle verification logic with configurable timeout and polling intervals for corrected deployments.
- Introduced new utility function `gitlab_non_terminal_pod_names_for_app` to retrieve non-terminal pod names.
- Improved logging for final deployment state and pod details after remediation actions.
- Added settle verification logic with configurable timeout and polling intervals for corrected deployments.
- Introduced new utility function `gitlab_non_terminal_pod_names_for_app` to retrieve non-terminal pod names.
- Improved logging for final deployment state and pod details after remediation actions.
- Introduced `gitlab_post_remediation_success` flag to streamline corrective action logging.
- Adjusted logic to skip unnecessary workload convergence gates when remediation is successful.
- Introduced `gitlab_post_remediation_success` flag to streamline corrective action logging.
- Adjusted logic to skip unnecessary workload convergence gates when remediation is successful.
Summarize CLAUDE.md updates:
- Document both live GKE clusters as e2-standard-2 x3 in us-west3 and note stale app default in code.
- Call out DB cluster naming mismatch between conf (knoe-dev-cnpg-0) and code default (knoe-cnpg-0).
- Add SSD quota guidance: 300GB fully consumed by CNPG, so non-CNPG PVCs must use standard/pd-standard.
- Refresh reset script reference and quota section details to match current runtime state.
Co-authored-by: Junie <junie@jetbrains.com>
- Added detailed logging for APP and DB contexts in `init_gitlab.sh` and `deploy.sh`.
- Enabled explicit passing of contexts and configuration paths to avoid fallbacks.
- Introduced strict environment resolution logic in `prole_cfg.sh` with fail-safe mechanisms.
- Added skipping logic for aged or bound PVC events in `deploy.sh` to reduce noise in diagnostics.
- Enforced replica target of 1 for specific GitLab workloads to ensure compliance with requirements.
- Added skipping logic for aged or bound PVC events in `deploy.sh` to reduce noise in diagnostics.
- Enforced replica target of 1 for specific GitLab workloads to ensure compliance with requirements.
- Standardized naming from "Prole" to "Knoe" in README, scripts, and deployment files for consistency.
- Refined ingress configuration to adjust class and annotations based on mode.
- Enhanced deploy.sh to handle public APP endpoint status for specific hosts.
- Updated `init_kong.sh` to handle `include_gitea_host` flag for improved flexibility in Gitea ingress setup.
- Added comprehensive post-deploy endpoint summary in `deploy.sh` to display public services and external IPs.
- Enhanced kubectl command handling with role-specific context logic for better split-cluster support.
- Improved DB context logging across CNPG operations and milestones.
- Updated GKE configuration with new contexts and Grafana hostname.
- Added detailed validation of Sidekiq concurrency, resource requests, and limits.
- Introduced live state comparison for Sidekiq resource configuration.
- Improved logging to track desired and live Sidekiq states for better diagnostics.
- Added split-cluster detection with detailed app and DB context logging.
- Introduced robust DB connectivity checks for DNS resolution and TCP probes.
- Enabled automated cross-cluster DB ILB service creation and validation.
- Improved logging for DB migration failures with enhanced diagnostics and connectivity hints.
- Added split-cluster detection with detailed app and DB context logging.
- Introduced robust DB connectivity checks for DNS resolution and TCP probes.
- Enabled automated cross-cluster DB ILB service creation and validation.
- Improved logging for DB migration failures with enhanced diagnostics and connectivity hints.
- Introduced `get_gitlab_migrations_diagnostics` for detailed migrations pod diagnostics.
- Enhanced active migration failure detection with detailed logs and diagnostics.
- Improved job context logging with pod restart, status, and waiting reason details.
- Refined stale job repair logic by incorporating diagnostics and failure insights.
- Introduced `get_gitlab_migrations_diagnostics` for detailed migrations pod diagnostics.
- Enhanced active migration failure detection with detailed logs and diagnostics.
- Improved job context logging with pod restart, status, and waiting reason details.
- Refined stale job repair logic by incorporating diagnostics and failure insights.
- Added detailed context logging for GitLab blockers, including failed migrations and Gitaly resources.
- Introduced automatic repair for stalled migration jobs by deleting stale jobs and pods.
- Improved Gitaly resource validation with enhanced logging for PVCs and StatefulSet configurations.
- Extended PVC phase handling with detailed storageClass descriptions for better troubleshooting.
- Improved detection and logging of mismatched storageClass across CR, StatefulSet, and PVCs.
- Added auto-repair mechanisms for inconsistencies in Gitaly storageClass configuration.
- Enhanced state validation with detailed logs for better troubleshooting.
- Introduced a grace period and explicit replacement of legacy StatefulSet after expiry.
- Enhanced logging to provide clearer details on convergence progress and outcomes.
- Added handling for operator recreation and state validation during legacy repairs.
- Ensured robust detection and resolution of legacy nodeSelector and storageClass issues.
- Added logic to wait for Gitaly StatefulSet corrections after autoclean actions.
- Implemented live state validation for nodeSelector and storageClass updates.
- Enhanced logging and error handling for legacy storage repairs in k8s mode.
- Updated PV and PVC handling to address legacy resource cleanup.
- Added logging to show Kubernetes context and service inspection details for Redis.
- Improved validation by checking for service endpoints and readiness.
- Enabled IP address bypass for external host DNS checks.
- Added logging to show Kubernetes context and service inspection details for Redis.
- Improved validation by checking for service endpoints and readiness.
- Enabled IP address bypass for external host DNS checks.
- Improved `check_gitlab_pre_apply_blocked` to better detect invalid placeholders, legacy storage configurations, and DNS issues for Redis and Garage endpoints.
- Added auto-repair for Garage endpoint secrets and storage class mismatches in k8s mode.
- Introduced warnings and blocking logic for unsupported configurations like `gitlab-gitaly-static`.
- Refined storage class handling to dynamically align with `standard-rwo` in k8s/GKE.
- Introduced `check_gitlab_pre_apply_blocked` and `check_gitlab_post_apply_blocked` functions to detect misconfigurations and legacy setups during deployment.
- Added logic to validate DNS resolution for Redis and Garage endpoints, preventing invalid placeholders or inaccessible hosts.
- Enhanced storage checks to identify and optionally repair legacy storage configurations with `GITLAB_REPAIR_BLOCKED_AUTOCLEAN`.
- Updated deployment wait loop to validate readiness and detect blocking states dynamically.
- remove hardcoded physical host assumptions in UI/init flows and rely on config-driven selectors\n- optimize GitLab reruns to skip unnecessary operator upgrades/long reconcile waits when unchanged\n- add and update regression/shell tests for fast-path and k8s context/ingress/garage behaviors
Co-authored-by: Junie <junie@jetbrains.com>
- Refactored `init_gitlab.sh` and `init_gitea.sh` to use configurable node selector keys, removing hardcoded defaults like `gandalf.prole.org`.
- Enhanced scheduling logic to validate required fields and prevent stale node constraints during reruns.
- Added fast-path guards in GitLab init to skip redundant operations when no changes occur in operator or CR specifications.
- Updated Supabase deployment to clear stale topology spread constraints alongside node selectors and affinity.
- Added tests for configurable node selector keys, fast-path guards, and stale constraint cleanups.
- Introduced `SERVICE_INGRESS_TLS_ENABLED` for configurable svc ingress TLS rendering in `init_kong.sh`.
- Updated ingress templates to include or exclude TLS annotations and blocks dynamically.
- Improved PVC and pod readiness checks in `supabase/deploy.sh` with enriched logging and detailed blockers.
- Added support for configurable storage class resolution for Supabase DB frontdoor PVCs.
- Enhanced deployment workflows to handle split APP/DB cluster setups with context-aware readiness checks.
- Updated test cases to validate svc ingress TLS configuration and storage class logic.
- Added detection for "already exists" errors during bucket creation and reused existing buckets when applicable.
- Enhanced logging to provide context when reusing previously created buckets.
- Added support for validating existing ingress ownership using `gitlab_release` and `expected_backend_service`.
- Improved conflict detection by distinguishing between same-owner and foreign ingress claims.
- Introduced explicit handling for operator-managed ingress and fallback ingress creation logic.
- Enhanced logging and status return codes for better error reporting and decision-making in deployment scripts.
- Introduced `resolve_garage_admin_context` and `kubectl_garage_admin` for dynamic context resolution.
- Enhanced Garage setup with explicit logging of admin context and namespace details.
- Improved error handling and command execution reliability for multi-cluster environments.
- Added validation and explicit handling for Garage S3 endpoints in split APP/DB cluster setups.
- Improved dynamic role detection and authoritative context enforcement in initialization scripts.
- Updated Supabase and Garage configurations to support private cross-cluster endpoints, ensuring cluster interoperability and DNS constraints.
- Enhanced error handling and logging for invalid or missing configurations.
- Added base64 decoding and validation for admin token in `garage-secrets`.
- Implemented reusable `garage_exec` helper for consistent command execution with detailed error handling.
- Improved key creation, retrieval, and fallback mechanisms with enhanced output parsing and failure logging.
- Unified bucket creation and permission granting logic using the new helper function.
- Added validation for admin token existence in `garage-secrets`.
- Improved `garage key` operations with support for deterministic fallback keys and secret fetch retries.
- Unified access/secret key parsing with flexible format support (`sed` fallback).
- Added dynamic `GITALY_STORAGE_CLASS` logic for k8s and legacy modes.
- Introduced context-aware DB operations with `resolve_db_cluster_context` and `kubectl_db`.
- Improved object storage configuration with failure handling based on deployment requirements.
- Refined workload placement logic and node selector defaults for improved flexibility.
- Added `PROLE_MODE` check to skip auxiliary host handling for Kubernetes (`k8s`) mode.
- Ensured conditional inclusion of `auth_host` and `gitea_host` based on deployment context.
- Improved clarity and flexibility in ingress target configuration logic.
- Added logic to derive and override APP cluster kubecontext based on `DEPLOYMENT_MODE` and `PROLE_MODE`.
- Enhanced support for legacy runtime context aliases with improved fallback behaviors.
- Unified context resolution for split-cluster environments to ensure role-consistent deployments.
- Introduced jemalloc hostPath optimizations with configurable modes (`auto`, `off`, `force`).
- Integrated jemalloc setup with best-effort and forced validation flows for ensuring cluster compatibility.
- Enhanced monitoring storage class logic with mode-specific handling (`k3s`, `k3d`, `gke`) and improved validation of required classes.
- Added safeguards and detailed logging for unsupported configurations and failure scenarios.
- Introduced jemalloc hostPath optimizations with configurable modes (`auto`, `off`, `force`).
- Integrated jemalloc setup with best-effort and forced validation flows for ensuring cluster compatibility.
- Enhanced monitoring storage class logic with mode-specific handling (`k3s`, `k3d`, `gke`) and improved validation of required classes.
- Added safeguards and detailed logging for unsupported configurations and failure scenarios.
Rename env config files from conf/*/prole.cfg to conf/k3d.cfg, conf/k3s.cfg, and conf/gke.cfg. Update shell/Python loaders and etc/deploy scripts to resolve named configs cleanly while keeping legacy fallback behavior. Align k3s Ansible tasks, docs, and regression coverage with the new configuration layout.
Co-authored-by: Junie <junie@jetbrains.com>
Switch production config to k8s/GKE contexts and align service naming. Add immutable StatefulSet update fallback for Garage across k3d/k3s/k8s. Harden CNPG deploy and backup bootstrap paths, and update installer coverage for CNPG webhook and Garage common ops.
Co-authored-by: Junie <junie@jetbrains.com>
- Introduced support for dynamic GitLab and Supabase domain and ingress configuration based on deployment mode (k8s vs local).
- Default GitLab domain and ingress class aligned with `prole.cfg` deployment settings.
- Expanded Supabase support:
- Added configurable API and Studio hostnames with default fallbacks.
- Introduced separate ingress class configuration for API and Studio.
- Standardized public URL normalization for both services.
- Updated tests to validate domain, ingress, and configuration behaviors across environments.
- Remove premature migrate_registry_images_to_gitlab() call (was running
before GitLab wait, against a not-yet-ready gitlab-registry)
- Add registry migration as the true final step of init_gitlab.sh, after
the GitLab CR is Available and Kong ingress is configured
- Delegates to init_registry.sh migrate so logic lives in one place
- SKIP_REGISTRY_MIGRATE=1 suppresses migration if needed
- Fix migrate_registry_to_gitlab() in init_registry.sh:
- dst_registry defaults to gitlab-registry.<gitlab_ns>.svc.cluster.local:5000
(in-cluster address, no port-forward needed)
- Checks registry:2 pod exists before attempting catalog fetch
- Counts ok/fail per-repo and reports retry command on partial failure
- Passes GITLAB_NAMESPACE from init_gitlab.sh caller context
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
KNOE_IMAGE_REGISTRY auto-detection in cloudnative_pg._apply_manifest():
- Explicit config wins; falls back to gitlab-registry when GitLab ns exists;
final fallback to registry:2 in knoe-system. No longer raises when unset.
conf/service/prole.cfg: point KNOE_IMAGE_REGISTRY at gitlab-registry since
GitLab is already deployed on this cluster.
etc/init_registry.sh: add 'migrate' action that copies all images from
registry:2 to gitlab-registry via skopeo (prints commands if skopeo absent).
Run after GitLab is up, before decommissioning registry:2.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- knoe/milestone.py: copy [Global] prole.cfg keys into _get_script_env()
env dict so CNPG and other Python-owned ops see KNOE_IMAGE_REGISTRY and
other config-declared vars (mirrors what prole_cfg.sh does for shell scripts)
- etc/init_redis.sh: skip helm upgrade when redis-master is already running
to avoid Docker Hub timeout on constrained networks; set REDIS_FORCE_UPGRADE=1
to override
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- harden CNPG backup ObjectStore readiness fallback in statusless clusters by validating controller availability and expected spec
- improve backup/plugin reconciliation paths and related test coverage for barman-cloud flows
- include ongoing GKE/CNPG deployment and configuration updates from this checkpoint
Co-authored-by: Junie <junie@jetbrains.com>
UI screens
- database.py: fix mode detection to use env_key priority (prod→k8s, service→k3s) so stale DEPLOYMENT_MODE never overrides the user's chosen environment
- database.py: Registry status reads ARTIFACT_REGISTRY_AVAILABLE persisted by cluster screen; uses SERVICE_NAMESPACE for Artifact Registry repo name
- cluster.py: add Artifact Registry traffic light (amber→green/red) to prod section; _check_artifact_registry_async persists ARTIFACT_REGISTRY_AVAILABLE into Global cfg
- cluster.py: re-trigger Artifact Registry check after GKE cluster selection so the light re-evaluates once region is available from KUBECONTEXT
- cluster_nodes.py: fix TclError on Python 3.14 — pady=(2,0) tuple → pady=2 scalar
- __init__.py: seed knoe-system namespace when saved value is "default", not only when empty
- services.py: replace hardcoded "Prole DB" log string with dynamic cnpg_cluster name
Core ops
- cloudnative_pg.py: replace one-shot Barman plugin retry with 6-attempt loop; first cert-manager/x509 failure triggers rollout restart + 30 s CA propagation wait; subsequent failures back off up to 60 s per attempt
- cloudnative_pg.py: TLS CA CN now uses cluster_name instead of hardcoded "Prole CNPG CA"
- registry.py, garage_store.py: refactored into per-mode modules (k3d/k3s/k8s registry and garage store, shared _garage_common)
Deploy / config
- deploy/gcp/gke/knoe-db.yaml: GKE-specific CNPG cluster manifest (rw/ro/r on separate nodes with premium-rwo storage)
- etc/init_common_services.sh, modes/k8s/knoe-db/.version: updated for current deploy
- kong-deployment.yaml: updated manifest
Tests
- test_cluster_nodes_render_smoke.py: add pack/grid, winfo_children, winfo_reqheight, update_idletasks, grid_slaves to dummy widgets; monkeypatch tk.Label so CNPG placement render completes without a real Tkinter root
Co-authored-by: Junie <junie@jetbrains.com>
## GCP / Cluster Environment Screen
- Auto-populate Cloud tab from conf/prod/gcp.cfg on screen open (org_id,
billing_account, billing_project, project_id)
- gcloud auth validity checked on screen startup; friendly modal dialog
streams gcloud auth login output live so user never leaves the app
- Live GKE cluster browser: fetches clusters via gcloud container clusters
list, displays with checkmark selector, auto-selects saved cluster
- Selecting a cluster runs get-credentials, sets KUBECONFIG/KUBECONTEXT,
and syncs the region dropdown to the selected cluster's location
- Region dropdown populated live from gcloud compute regions list with
checkmark on currently selected region; graceful fallback when offline
- New 'GCP Storage' tab with workload->StorageClass mapping (CNPG->premium-rwo,
Redis/Monitoring->standard-rwo, Garage->garage-hdd) and Fetch from Cluster
- Provider readonly field styled correctly (no solid-black on macOS)
- Stale prole.cfg/conf/prole.cfg symlinks removed; all config I/O now
resolves env-specific paths via prole_conf.entrypoint_path()
## GKE Autopilot Compatibility (Common Services)
- Synology iSCSI StorageClass and static PVs guarded behind PROLE_MODE!=k8s
in init_openbao.sh (GKE Autopilot forbids hostPath/iSCSI volumes)
- In-cluster Docker registry (hostPath) skipped in k8s mode; GCP Artifact
Registry used instead
- Kong renamed knoe-svc-kong in k8s mode; all health-check kubectl calls in
init_common_services.sh and status_common_services.sh updated accordingly
- DNS endpoints switched from *.prole.org to *.knoe.dev in k8s mode
(api.knoe.dev, git.knoe.dev, svc.knoe.dev); ingress uses gce class
- New GKE-clean Kong manifests under deploy/opentofu/k8s/manifests/prole/:
no k3s node affinity, explicit Autopilot resource requests/limits
## Garage S3 Store (GKE)
- New garage-statefulset-gcp.yaml targeting garage-hdd StorageClass
(pd-standard, avoids SSD_TOTAL_GB quota exhaustion in us-west3)
- New storageclass-gcp-hdd.yaml (pd-standard, Retain, WaitForFirstConsumer)
- GCP StorageClass manifests skipped on re-runs (Autopilot built-ins are
immutable; skip-if-exists guard added)
- PVC deletion guard extended to cover any storageClass (not just synology)
so stale claims are cleaned before StatefulSet recreation
## Topology (GKE Autopilot)
- DaemonSet collector skipped in prod mode (forbidden in kube-system by
GKE Warden); Kubernetes-only node facts path used instead
- All ready GKE nodes assumed cnpg-eligible and monitoring-eligible without
taint/synology-mount checks (skip_collector + assume_nodes_eligible flags)
## KUBECONFIG / kubectl (k8s mode)
- actions.py: new elif mode==k8s branch sets KUBECONFIG=~/.kube/config
and injects KUBECONTEXT from prole_cfg_data into script env
- _build_kubectl_cmd falls back to Global.KUBECONTEXT when
init_cluster.selected_kubectx is empty
- _activate_selected_gke_cluster persists KUBECONFIG/KUBECONTEXT to
prole_cfg_data and saves prole.cfg immediately after get-credentials
## Database Build Screen (GKE)
- Registry display shows correct Artifact Registry URL
(<region>-docker.pkg.dev/<project>/<namespace>/knoe-db) in green
- Build+push: gcloud auth configure-docker, auto-creates AR repository
named after SERVICE_NAMESPACE (e.g. knoe-system) if missing, then
docker tag + push; falls back to gcr.io if region unavailable
- GCP config loaded from conf/prod/gcp.cfg on every screen entry;
keys normalised to lowercase so project_id lookup is always consistent
## Config / Namespace persistence
- prole_conf.py activate_environment: symlink creation removed; sets
CLUSTER_ENV env-var so all subsequent calls resolve correct env directory
- knoe/ui/screens/__init__.py: startup config load uses entrypoint_path()
instead of hardcoded conf/prole.cfg; seeds SERVICE_NAMESPACE=knoe-system
for managed envs so Common Services never defaults to 'default'
- cfg.py _save_prole_cfg: saves to env-specific path via entrypoint_path()
- etc/prole_cfg.sh: removed all ln -snf symlink creation
Co-authored-by: Junie <junie@jetbrains.com>
The auth deployment in knoe-system uses label app=auth; init_knoe_users.sh
was searching for app=prole-auth causing the pod lookup to always fail.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Kerberos is now the single source of truth for all knoe-system accounts.
In-cluster KDC (PROLE.LOCAL):
- Rename in-cluster KDC realm PROLE.ORG → PROLE.LOCAL to avoid collision
with the myrddin.prole.org Samba AD DC which owns PROLE.ORG
- Add [capaths] cross-realm trust block: PROLE.ORG (Samba AD) ↔ PROLE.LOCAL
- Create admin@PROLE.LOCAL on KDC startup (master password) for UI login
- Create guest@PROLE.LOCAL on KDC startup (read-only, PROLE_KDC_GUEST_PASSWORD)
- Update prole-kdc-secrets.example.yaml with trust_shared_password and guest_password
- Update prole-auth-kerberos ConfigMap realm/servicePrincipal to PROLE.LOCAL
- Add PROLE_AUTH_ADMIN_PRINCIPALS and PROLE_KDC_GUEST_PASSWORD to prole-auth Deployment
Database (knoe-db — k3s and GKE):
- Add spec.managed.roles: admin (superuser), guest (login), developer (group)
- Enable GSS pg_hba rules for both PROLE.ORG and PROLE.LOCAL realms
- Add krb_server_keyfile parameter for postgres service principal keytab
- Mount knoe-db-pg-keytab Secret via spec.additionalVolumes
- Add demo schema in postInitSQL with guest read-only grants
prole-auth (Spring Boot):
- SessionUser: add List<String> groups field
- AuthProperties: add adminPrincipals list (env: PROLE_AUTH_ADMIN_PRINCIPALS)
- LoginController: stamp groups=["admin"] for configured admin principals at login
- SessionTokenService: carry groups through TokenPayload; @JsonIgnoreProperties
for backward compat with existing 2-field session cookies
- VerifyController: emit X-Prole-Groups header on /auth/verify
- grafana-proxy nginx: strip inbound X-Prole-Groups; capture and forward from auth
Services:
- ArgoCD (k3s + GKE): add g, admin, role:admin to argocd-rbac-cm policy.csv
etc/init_knoe_users.sh (new):
- Orchestrates full user provisioning: KDC principals, postgres keytab export,
CNPG managed.roles patch, demo schema SQL, ArgoCD RBAC patch,
Gitea and GitLab admin promotion via API
- Actions: initialize | status | cleanup
- Idempotent; sources prole_cfg.sh; follows existing init_*.sh style
Cross-realm trust activation (myrddin side):
samba-tool user add krbtgt_PROLELOCAL --random-password
samba-tool user setpassword krbtgt_PROLELOCAL --newpassword=<trust_shared_password>
samba-tool spn add "krbtgt/PROLE.LOCAL" krbtgt_PROLELOCAL -U administrator
samba-tool user setexpiry krbtgt_PROLELOCAL --noexpiry
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- etc/init_gitlab.sh: Add global.redis block (host/port/auth) to the GitLab CR so
the chart does not fail NOTES.txt validation when redis.install: false.
Drop the GITOPS_NAMESPACE config fallback in namespace resolution to prevent the
Gitea namespace from bleeding into GitLab deployments; GITLAB_NAMESPACE is now the
sole source of truth with a hard default of "gitlab".
- knoe/core/milestones.py: Fix GitOpsMilestone to route to init_gitlab.sh when
gitops.git_provider = GitLab (was hardcoded to init_gitea.sh). Namespace resolution
now prefers gitops.gitlab_namespace input key, then gitops.namespace, then "gitlab" —
never picks up a stale GITLAB_NAMESPACE from the OS environment.
- conf/service/prole.cfg: Switch gitops.git_provider / GITOPS_PROVIDER to GitLab.
Update accumulated runtime state from install runs.
- install.sh: Prefer the repo-local venv Python (PROLE_HOME/bin/python3) so that
PyYAML and other prole_requirements.txt deps are always available.
- .gitlab-ci.yml: New CI pipeline — on every push to main, run the silent install
(./install.sh -S -c conf/service/prole.cfg) to deploy a fresh CNPG ecosystem.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- update Kong/Supabase ingress and service manifests for k3s
- expand init scripts and runtime config/topology handling
- refresh KNØE UI screens and Supabase deployment/render logic
- include related env helper test adjustments
Co-authored-by: Junie <junie@jetbrains.com>
