Follow-up to the ansible excision: remove other artifacts that should
never have been tracked, and extend .gitignore so they do not come back.
Removed:
- ssl/prole/*.key, *.crt, *.srl -- prole.org-era private keys, registry
and DB CA material. References in knoe/core/env.py (auto-detect path
at :1488) are best-effort inside try/except, so deletion is safe.
- ssl/prole.ssl.tar -- tarball of the same material.
- deploy/opentofu/k3s/terraform.tfstate -- 114KB committed OpenTofu
state snapshot with live cluster UIDs. tfstate is never supposed to
live in git.
- docs/deploy_gke.log, docs/deploy_gke_done.log,
docs/compressed-history.log -- deploy-time log captures.
- end_time.txt, start_time.txt, start_time_final.txt -- ansible wall-
clock timing breadcrumbs.
- network_description.txt, network_prompt.txt -- local scratch prompts.
- sidekiq_logs.txt -- transient log capture.
- test_resolve.sh -- ad-hoc debug script.
.gitignore additions:
- *.tfstate, *.tfstate.backup
- /ssl/
- .ai/
- /end_time.txt, /start_time*.txt, /sidekiq_logs.txt,
/network_description.txt, /network_prompt.txt, /test_resolve.sh,
/docs/deploy_*.log, /docs/compressed-history.log
Note: the purged SSL keys remain in git history until the deferred
history-scrub pass (task #14). The repo will be created as Private in
git.knoe.dev so this is acceptable for now; it MUST be scrubbed before
any broader visibility change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switch production config to k8s/GKE contexts and align service naming. Add immutable StatefulSet update fallback for Garage across k3d/k3s/k8s. Harden CNPG deploy and backup bootstrap paths, and update installer coverage for CNPG webhook and Garage common ops.
Co-authored-by: Junie <junie@jetbrains.com>
Add [CNPG Clusters] section to prole.cfg as the authoritative source for
CNPG cluster definitions. The deploy pipeline reads this registry instead
of computing version-appended namespace names at runtime.
- conf/service/prole.cfg: add [CNPG Clusters] with knoe-db as identity
cluster; fix DATABASE_NAMESPACE=knoe-db (was knoe-db-18-140); add
KNOE_IMAGE_REGISTRY for the k3s in-cluster registry hostname
- deploy/opentofu/k3s/manifests/prole/knoe-db.yaml: replace hardcoded
registry.test-system image with templated KNOE_IMAGE_REGISTRY
- knoe/core/actions.py: add _cnpg_cluster_registry(), _cnpg_identity_cluster();
fix _initial_db_namespace() to use declared namespace without version suffix
- knoe/core/milestones.py: iterate over all declared clusters in Init and
Deploy milestones; per-cluster CNPG_IMAGE_NAME support
- knoe/core/ops/cloudnative_pg.py: resolve KNOE_IMAGE_REGISTRY and
CNPG_IMAGE_NAME overrides in _apply_manifest()
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Drop spec.additionalVolumes, krb_server_keyfile, and hostgssenc pg_hba
rules from the k3s Cluster manifest. The CNPG CRD on k3s rejects
additionalVolumes as an unknown field, the keytab secret does not exist,
and hostgssenc rules would prevent PostgreSQL from starting without a
mounted keytab. Kerberos is disabled in conf/service/prole.cfg and
will be wired up once k3s KDC and keytab provisioning are in place.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Kerberos is now the single source of truth for all knoe-system accounts.
In-cluster KDC (PROLE.LOCAL):
- Rename in-cluster KDC realm PROLE.ORG → PROLE.LOCAL to avoid collision
with the myrddin.prole.org Samba AD DC which owns PROLE.ORG
- Add [capaths] cross-realm trust block: PROLE.ORG (Samba AD) ↔ PROLE.LOCAL
- Create admin@PROLE.LOCAL on KDC startup (master password) for UI login
- Create guest@PROLE.LOCAL on KDC startup (read-only, PROLE_KDC_GUEST_PASSWORD)
- Update prole-kdc-secrets.example.yaml with trust_shared_password and guest_password
- Update prole-auth-kerberos ConfigMap realm/servicePrincipal to PROLE.LOCAL
- Add PROLE_AUTH_ADMIN_PRINCIPALS and PROLE_KDC_GUEST_PASSWORD to prole-auth Deployment
Database (knoe-db — k3s and GKE):
- Add spec.managed.roles: admin (superuser), guest (login), developer (group)
- Enable GSS pg_hba rules for both PROLE.ORG and PROLE.LOCAL realms
- Add krb_server_keyfile parameter for postgres service principal keytab
- Mount knoe-db-pg-keytab Secret via spec.additionalVolumes
- Add demo schema in postInitSQL with guest read-only grants
prole-auth (Spring Boot):
- SessionUser: add List<String> groups field
- AuthProperties: add adminPrincipals list (env: PROLE_AUTH_ADMIN_PRINCIPALS)
- LoginController: stamp groups=["admin"] for configured admin principals at login
- SessionTokenService: carry groups through TokenPayload; @JsonIgnoreProperties
for backward compat with existing 2-field session cookies
- VerifyController: emit X-Prole-Groups header on /auth/verify
- grafana-proxy nginx: strip inbound X-Prole-Groups; capture and forward from auth
Services:
- ArgoCD (k3s + GKE): add g, admin, role:admin to argocd-rbac-cm policy.csv
etc/init_knoe_users.sh (new):
- Orchestrates full user provisioning: KDC principals, postgres keytab export,
CNPG managed.roles patch, demo schema SQL, ArgoCD RBAC patch,
Gitea and GitLab admin promotion via API
- Actions: initialize | status | cleanup
- Idempotent; sources prole_cfg.sh; follows existing init_*.sh style
Cross-realm trust activation (myrddin side):
samba-tool user add krbtgt_PROLELOCAL --random-password
samba-tool user setpassword krbtgt_PROLELOCAL --newpassword=<trust_shared_password>
samba-tool spn add "krbtgt/PROLE.LOCAL" krbtgt_PROLELOCAL -U administrator
samba-tool user setexpiry krbtgt_PROLELOCAL --noexpiry
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- update Kong/Supabase ingress and service manifests for k3s
- expand init scripts and runtime config/topology handling
- refresh KNØE UI screens and Supabase deployment/render logic
- include related env helper test adjustments
Co-authored-by: Junie <junie@jetbrains.com>
STABLE BUILD — all 35+ pods Running, zero non-Running pods, CNPG ready=3.
## Architecture changes (Supabase slim-down)
- Removed local Supabase Kong, Storage, MinIO, Imgproxy from deployed stack
- Rewired API routing to shared Kong in kube-system
- Rewired object storage to shared Garage in knoe-system
- Supabase footprint reduced to control-plane only: auth, rest, meta,
analytics, realtime (studio port-forward retained)
- Supabase pods renamed: stripped redundant '-prole-supabase-' prefix via
per-component fullnameOverride (e.g. supabase-auth, supabase-rest, ...)
- Supabase workloads pinned to merlin.prole.org (memory-safe node)
## File changes
- conf/service/prole.cfg
* Replaced all hardcoded /Users/chrisfu/dev/prole paths with ${HOME}/dev/prole
so config is portable when run from myrddin.prole.org or any user home
* SUPABASE_ENABLED=True, SUPABASE_PRIMARY_NODE=merlin.prole.org in
both [Global] and [Optional Features]
* supabase_enabled=true in [Inputs]/init_cluster section
- supabase/helm/render_supabase.py
* Explicitly disabled: kong, storage, minio, imgproxy
* Active components pinned via nodeSelector to merlin.prole.org
* fullnameOverride per component to clean up pod names
* Default primary node changed from pi.prole.org -> merlin.prole.org
- supabase/deploy.sh
* Removed svc/kong port-forward; API via shared Kong in kube-system
* Removed imgproxy/storage from PV/PVC prep and helm --set persistence args
* Fixed kubectl exec missing -i flag for psql heredoc stdin in
setup_knoe_db_for_supabase
* Updated Studio access messaging to reference shared platform Kong
- k8s/prole/knoe-db.yaml
* instances: 3
* Added control-plane/master tolerations for myrddin scheduling
- deploy/opentofu/k3s/manifests/prole/knoe-db.yaml
* instances: 3, added control-plane tolerations (mirrors k8s/prole/)
- k8s/prole/iscsi-pvs.yaml
* Expanded d003-data/d003-wal nodeAffinity to [pi, merlin, myrddin]
to reflect post-OOM-recovery live state
- k8s/registry/deployment.yaml
* nodeSelector restored to myrddin.prole.org (hostPort 5000 owner)
* Added control-plane + master tolerations so pod schedules correctly
- modes/k3s/knoe-db/.version
* Aligned to 140 (image tag 18-140) matching repo-root version
- etc/init_common_services.sh, etc/init_registry.sh
* Minor fixes aligned with slimmed service layout and node targeting
- infrastructure/inventory/host_vars/pi.prole.org.yml
* Updated host vars to reflect post-recovery pi state
- knoe/core/topology.py, knoe/ui/screens/cluster.py,
knoe/ui/screens/cluster_nodes.py
* Topology and UI updates for 3-node cluster representation
- scan/network_description.txt
* Refreshed network scan output post-recovery
- conf/port-mapping.cfg
* Updated port mapping to reflect removal of local Kong/storage forwards
- scripts/cleanup_stale_rs.sh (new)
* Utility script to prune orphaned ReplicaSets after rollouts
## Verified stable state
- CNPG: Cluster in healthy state, ready=3
knoe-db-2 (merlin, primary), knoe-db-3 (pi), knoe-db-5 (myrddin)
- Supabase: 5 pods Running on merlin.prole.org
supabase-auth, supabase-rest, supabase-meta,
supabase-analytics, supabase-realtime
- Monitoring: all prometheus/grafana/alertmanager pods Running on merlin
- Registry: Running on myrddin.prole.org (hostPort 5000)
- Zero non-Running / non-Completed pods across all namespaces
Co-authored-by: Junie <junie@jetbrains.com>
- set global MODE/DEPLOYMENT_MODE to k3s and remove cross-environment/prod cluster entries from active dev/service config paths\n- switch k3s prole-auth image references to myrddin.prole.org:5000\n- strengthen CNPG image resolution and test coverage for rewriting k3d/localhost registries in k3s mode\n- include updated port mapping/network snapshot and current k3d mode version metadata
Co-authored-by: Junie <junie@jetbrains.com>
- Rename iSCSI storage class and PV/PVC selectors/labels from prole to synology across k8s and OpenTofu manifests\n- Update CNPG/OpenBao/Garage/monitoring init flows, render helpers, and mock scripts for synology-backed storage objects\n- Integrate related UI/core/service config/version updates and add supporting regression tests for CNPG storage/image behavior\n- Keep storage reconciliation tests aligned with current CNPG affinity output
Co-authored-by: Junie <junie@jetbrains.com>
- Introduce mode-scoped directories under `~/.prole` for build contexts, runtime data, and versioning.
- Update DB image version generation and Dockerfile creation to respect deployment modes (e.g., k3d, k3s, k8s).
- Adjust UI, core actions, and scripts to avoid cross-mode conflicts and ensure correct version isolation.
- Add tests for mode-aware behaviors and update existing tests for new paths and markers.
- Bump PostgreSQL and image versions to `18` and `138`, respectively.
- Update Kubernetes manifests to align with new image and version configuration.
- Rename host mount namespace from /prole/d00x to /synology/d00x
- Make LUN ownership explicit per host in inventory (myrddin=d001; merlin=d002,d004; pi=d003)
- Add safe migration cleanup for legacy /prole/d00x mounts and stale /etc/fstab entries
- Update k3s/ArgoCD/OpenTofu manifests and PV node affinities to match new mount paths
- Improve iSCSI role check-mode behavior and add quiesce/detach flow in ansible.sh
- Persist and export supabase_hostname for canonical external Supabase entrypoint (db.prole.org)
- Render Supabase Helm ingress host and public URLs (API_EXTERNAL_URL/GOTRUE_SITE_URL/SUPABASE_PUBLIC_URL) from config
- Align tracked Traefik ingress manifests to db.prole.org
- Generalize service hostname/TLS wiring in k3s automation and refresh Kong/monitoring behavior
- Make optional workload policy checks deterministic when kube context is absent; add render tests
- Pin Prometheus/Alertmanager + Grafana to merlin.prole.org in k3s to avoid PV node-affinity issues
- Taint myrddin.prole.org as control-plane-only and add CNPG toleration
- Prevent init_cloudnative_pg from deploying prole-svc-kong into the DB namespace
- GitOps-manage Kong declarative config via ConfigMap + add placement/init tests
Prefer infra-managed manifests for bootstrap; reconcile CNPG instances based on Ready+schedulable labeled db nodes; update manifests to use node-role affinity + anti-affinity; add policy/tests and config touch-ups (incl. prole.cfg).
- Rewrite stale foreign home prefixes when loading env defaults (macOS <-> Linux)
- Write /Users/chrisfu-relative values to env.sh; sanitize prole.cfg paths to use ${HOME}
- Remove remaining /Users/chrisfu defaults from helper scripts and OpenTofu kubeconfig path
- Make installer tests more reliable by forcing repo root to front of sys.path
Co-authored-by: Junie <junie@jetbrains.com>
- Reuse namespace from existing prole.cfg unless env overrides
- Support k3s_hosts children groups and prefer k3s_servers as default
- Add headless pytest tkinter stubs and blocked-cluster reconciliation tests
- Update registry mirror endpoint, port mappings, and prole-db manifests
We can now complete a full K3s deployment run and deliver a working CloudNativePG (CNPG) stack from the standalone installer.\n\nHighlights:\n- Installer scripts hardened for CNPG deploy + recovery templating + rollout control\n- OpenTofu/K3s manifests aligned with in-repo k8s templates for prole-db\n- Supporting init flows updated (OpenBao, monitoring, Kerberos, OpenTofu)
Co-authored-by: Junie <junie@jetbrains.com>
Summary:
Removed the prole-db-manager microservice and simplified deployment to use
prole-authority as the internal management and authorization point. Fixed two
blocking bugs that prevented silent install from completing on knoe-dev-cluster.
Removed: prole-db-manager
- Deleted db-manager-deployment.yaml and db-manager-service.yaml from opentofu manifests
- Deleted src/db-manager/ (Dockerfile, server.js, package.json, tests)
- Removed prole-db-manager port-forward mapping from installer/core/env.py
- Removed init_db_manager.sh from Initialization Scripts (milestones.py, actions.py)
- Removed init_certmgr.sh and init_db_manager.sh tabs from services screen (services.py)
- Removed live k8s Deployment/Service from knoe-dev-cluster
Fixed: PostgreSQL version downgrade error (pg17 -> pg18)
- Created conf/postgresql/.version with value 18
- Updated k8s/prole/prole-db.yaml and prole-db-recovery.yaml.tpl imageName to prole-db:18-089
- Fixed _init_database_options_state() to restore saved version_type from prole.cfg
so db_version_type defaults to v18 (pg18) instead of silently reverting to pg17
- Added database_options.* keys to _collect_input_snapshot() in cfg.py so
distribution, version_type, and all extension toggles persist to prole.cfg
Fixed: Cluster name inconsistency
- Removed stale prole-dev-cluster references; all scripts now use knoe-dev-cluster
- Added knoe-dev-cluster to mode-detection case in etc/prole_cfg.sh
Config: conf/prole.cfg
- Set kerberos_config.enabled = False, KERBEROS_AUTO_ENABLED = False
- Added database_options.distribution = percona, version_type = v18
- Added all 13 extension flags set to True (postgis, pgvector, pgcrypto, pgaudit,
pg_repack, pg_stat_statements, pg_buffercache, pg_freespacemap, pgrowlocks,
postgres_fdw, dblink, pg_stat_monitor, pgbadger)
Verification:
./install.py -s -l -v -c conf/prole.cfg completed successfully.
CNPG deployed prole-db:18-089 to knoe-dev-cluster; all milestones passed.
Co-authored-by: Junie <junie@jetbrains.com>
Align Kong k3s manifests with etc/init_kong.sh and svc-check Helm values (prole-svc-kong + prole-svc-kong-config) so rollout waits for the correct deployment.
- Deleted PostgREST Kubernetes deployment and service manifests.
- Removed `init_postgrest.sh` initialization script and PostgREST references across UI, installer, and config.
- Updated Kong API Gateway to exclude `/rest/*` routing to PostgREST.
- Simplified `supabase/deploy.sh` by removing references to PostgREST scaling and alias creation.
- Adjusted cluster configuration and UI to reflect the absence of PostgREST.
- Include new `prole.spec` for build configurations and dependencies.
- Add Terraform state handling for OpenTofu in `k3s` cluster.
- Provision multiple Kubernetes resources in `prole-db` namespace: namespace, services, ConfigMaps, StatefulSets, Ingress rules, and PersistentVolumes.
- Integrate deployment and configuration enhancements for `garage`, `prole`, and related components.
Kong API Gateway (replacing prole nginx):
- Add etc/init_kong.sh provisioning script (DB-less mode, prole-db namespace)
- Add kong-deployment.yaml and kong-service.yaml manifests
- Rewire ingress rules (svc/git/api.prole.org) to prole-db-kong:8000
- Update kustomization.yaml to reference kong manifests
PostgREST & DB Manager in prole-db namespace:
- Add etc/init_postgrest.sh and etc/init_db_manager.sh scripts
- Add postgrest/db-manager deployment and service manifests
- Add src/db-manager/ Node.js REST endpoint for backup triggers
- Default NAMESPACE changed to prole-db in both scripts
Docker image pre-load from PROLE_DATA/docker-import:
- Add _preload_docker_images() to init_common_services.sh
- Scan for .tar files exported by final_deployment.sh
- Import via k3d image import (k3d) or ctr (k3s) before deployments
- Increase rollout timeouts to 300s (configurable via ROLLOUT_TIMEOUT) in init_openbao.sh, init_opentofu.sh, init_garage_store.sh, init_registry.sh
OpenTofu password resolution fix:
- Add Kubernetes secret fallback in resolve_admin_password()
- Change hard exit 1 to graceful return 1 with warning
- Wrap call in if-guard so set -e doesn't abort the script chain
Milestone fix (init scripts not running):
- Add init_kong.sh, init_postgrest.sh, init_db_manager.sh to InitializationScriptsMilestone.execute() script list and arg branches
- Previously only actions.py had these; milestones.py was missing them
Installer integration:
- Add Kong/PostgREST/DB Manager to silent installer _step_init_scripts
- Add corresponding tabs and execution blocks in UI services.py
- Monitoring: Migrated from manual Grafana/Prometheus manifests to kube-prometheus-stack based setup in etc/init_monitoring.sh. Removed old manifest files from deploy/ and k8s/.
- Installer Core: Refactored installer with new modules for actions, environment handling, and UI screens. Enhanced Milestone logic to support advanced configuration (ArgoCD, Registry namespaces, Kerberos flags, etc.).
- Service & Init Scripts: Updated multiple initialization scripts (init_*.sh) for better integration with OpenBao, Kerberos, and the new monitoring stack. Added new scripts for Nginx Ingress, Ollama parsing, and K3D route fixes.
- Infrastructure: Enhanced Samba AD DC Ansible role with realm derivation, provisioning guidance, and group management. Updated K3s role tasks.
- Configuration: Refined default settings in conf/ to align with the new deployment architecture.
- App & Tools: Updated prole-app Swift code and prole.sh for improved environment variable handling and installation flow.
- Consolidated and split initialization scripts in etc/:
- Removed init_prole-db.sh and init_authority.sh.
- Added init_kdc.sh for in-cluster MIT Kerberos KDC (prole-authority).
- Added init_ollama.sh for Ollama AI service integration.
- Added init_service_layer.sh for high-level service orchestration.
- Added init_k3s_registry.sh for private registry management.
- Major updates to install.py:
- Support for new Ollama and KDC configuration.
- Improved prole.cfg rendering and namespace handling.
- Updated unattended install flags.
- Infrastructure and Deployment:
- Updated K3s Ansible role with private registry support (registries.yaml template).
- Added prole-authority Dockerfile.
- Updated OpenBao Kerberos ConfigMap and other K8s manifests.
- Configuration:
- Updated prole.cfg with new sections for Ollama and Monitoring.
- Refined environment variable exports in env.sh and prole_cfg.sh.
- Add k3s start/stop Ansible playbooks and roles.
- Implement OpenTofu initialization scripts and k8s manifests.
- Update ncurses installer with OpenTofu support and improved k3s integration.
- Add mode support (--mode) to etc/ initialization scripts.
- Update prole-db with recovery, barman objectstore, and SSH OpenBao support.
- Refine k8s manifests for OpenBao and prole-db.
