Commit Graph

47 Commits

Author SHA1 Message Date
chrisfu
fb7e8b7135 chore(k3s): rename prole-*.yaml manifests to knoe-* and align contents
Mirrors commit b355855 (auth-side rename). The kustomization at
deploy/opentofu/k3s/manifests/knoe/kustomization.yaml has been
referencing knoe-* names since the rebrand; kustomize was broken
on the k3s deploy mode until this lands.

Five git mv:
  prole-configmap.yaml           -> knoe-configmap.yaml
  prole-deployment.yaml          -> knoe-deployment.yaml
  prole-service.yaml             -> knoe-service.yaml
  prole-kdc-configmap.yaml       -> knoe-kdc-configmap.yaml
  prole-kdc-secrets.example.yaml -> knoe-kdc-secrets.example.yaml

Plus content sweep of knoe-kdc-configmap.yaml: replaced stale
PROLE.LOCAL/PROLE.ORG realm refs and PROLE_KDC_* env vars with
KNOE.DEV/KNOE_KDC_* to match the canonical GKE configmap.

Closes queue item #2 in docs/TODO.md (drift R5).

Co-authored-by: Junie <junie@jetbrains.com>
2026-05-02 02:58:03 -07:00
chrisfu
00f0ebec07 Merge claude/crazy-bose-fec256 into main
Bringing the long-running session-feature branch back into main in one
deliberate sweep. The branch carried the cluster work that's been live for
weeks (cross-cluster CNPG metrics, Grafana w/ Google OAuth, supabase
oauth2-proxy, cluster recovery, pg.0.knoe.dev + per-engineer onboarding,
GCS-backed CNPG backups via Workload Identity, the env-contamination
guard, the Junie brief queue, the cnpg-grafana CSRF + memory-request
fixes from today), while main accumulated Junie's parallel knoe-auth
Phase 2 OIDC work (full provider surface: discovery, authorize, token,
userinfo, JWKS, RS256 signing, code exchange, session services).

Key decision: the two branches did COMPETING rebrands off the same
starting point (5ba9b63, 2026-04-27):

  - claude branch (commit b355855, earlier): org.prole.authority.* →
                                              dev.knoe.auth.*
                                              (artifact renamed to
                                              knoe-auth.jar)
  - main (commit 9daa94b, recent): org.prole.authority.* →
                                    dev.knoe.authority.*
                                    (kept "authority" artifact name)

dev.knoe.auth wins: cluster runs from this name, the Maven artifact is
already knoe-auth.jar, and the broader rename is the documented
namespace direction (per ~/.claude/projects/-Users-chrisfu-dev-knoe-db/
memory/MEMORY.md). All of main's recent Phase 2 OIDC content was ported
from authority/src/.../dev/knoe/authority/ into
authority/src/.../dev/knoe/auth/ with package declarations rewritten.

== File-level resolution summary ==

Textual conflicts (4):

  authority/pom.xml
    - Took our artifactId="auth"
    - Took our branch's removal of spring-security-kerberos-client
      (verified: Junie's Phase 2 OIDC code does not import it; the dep
      was already-dead config)

  docs/pipeline-phases.md
    - Took our branch's "Phase 1 not started" status. Main had a
      misplaced " Complete" with a knoe-auth-Phase-1 commit ref
      in the autobuild Phase 1 section — different domain.

  docs/plans/knoe-auth-round-1.md
    - Took our branch's dev.knoe.auth file table (vs main's
      dev.knoe.authority listing). Pure rename mismatch.

  supabase/helm/knoe-supabase/templates/kong/config.yaml
    - Took our branch's onboard route + plain dashboard wiring.
      Main had an oauth2proxy.enabled toggle that put oauth2-proxy as
      a Kong upstream — but the deployed architecture (commit 25f1b2e)
      has oauth2-proxy in FRONT of Kong, not behind. Main's wrapper
      reflected an architecture that was never deployed.
    - Took our branch's removal of basic-auth from dashboard route
      (queue #15 brief still tracks the matching values.yaml /
      kong/deployment.yaml cleanup).

Java tree reconciliation (44 file-pairs):

  20 dual-path source files + 2 dual-path tests
    Body-identical between main's authority/ and our branch's auth/
    after stripping package decls — main's commit 9daa94b was a pure
    rebrand. Took our branch's auth/ version for all 22.

  8 main-only source files (Phase 2 OIDC), ported into auth/:
    web/JwksController.java
    web/OidcAuthorizeController.java
    web/OidcDiscoveryController.java
    web/OidcTokenController.java
    web/OidcUserInfoController.java
    session/OidcCodeService.java
    session/OidcTokenService.java
    session/SessionService.java

  12 main-only test files, ported into auth/:
    HealthControllerTest.java
    enroll/EnrollValueTypesTest.java
    enroll/EnrollmentControllerTest.java
    enroll/TotpServiceTest.java
    kerberos/KadminClientTest.java
    kerberos/KerberosSpnegoResultTest.java
    web/LoginControllerTest.java
    admin/AdminControllerTest.java
    user/PrincipalNormalizerTest.java
    regression/IdentityRegressionTest.java
    session/OidcCodeServiceTest.java
    session/SessionServiceTest.java

  Port mechanics: read main:authority/...<file> via git show, then sed
  rewrite `package dev.knoe.authority` → `package dev.knoe.auth` and
  `import dev.knoe.authority` → `import dev.knoe.auth`. Body content
  unchanged.

  authority/src/main/java/dev/knoe/authority/ — DELETED (duplicate)
  authority/src/test/java/dev/knoe/authority/  — DELETED (duplicate)

== Verification ==

- grep -rln '<<<<<<<' across .java/.md/.yaml/.yml/.sh/.xml/.tpl: clean
- find authority/src -path '*/dev/knoe/authority*': empty (subtree gone)
- grep 'package dev.knoe.authority' across repo: clean
- bash -n install.sh deploy.sh etc/preflight_kubecontext.sh: clean
- git ls-files -u | wc -l: 0 unmerged paths
- helm lint supabase/helm/knoe-supabase: pre-existing failure on
  studioIngress.enabled undefined in values.yaml (introduced by Junie
  on main; unrelated to this merge — flagging as follow-up).

== Followups (carried into TODO ranked queue or noted here) ==

  - helm lint failure: studioIngress block in values.yaml is missing
    enable flag; templates/studio/{ingress,oauth2proxy-deployment,
    oauth2proxy-service}.yaml all reference studioIngress.enabled with
    no default. Pre-existing on main; not introduced by this merge.
  - The five Junie briefs filed on this branch are now reachable from
    main at docs/plans/junie/{02,06,07,13,15}-*.md. Junie can pick them
    up in any order.
  - knoe-auth Phase 2 OIDC source (now at dev.knoe.auth.*) is not yet
    deployed to the cluster. Deployment is its own task.
  - The branch claude/crazy-bose-fec256 stays in place (worktree at
    .claude/worktrees/crazy-bose-fec256 may have ongoing context for
    Claude Code sessions). Safe to delete once next session starts
    cleanly from main.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 16:39:10 -07:00
chrisfu
d17270bbe2 feat(oauth): add Google OIDC/OAuth2 Proxy support for Studio and Grafana. Add Google OIDC configuration for Grafana in gke.cfg, introduce oauth2-proxy templates and configurations for Supabase Studio, update Helm values/templates for authenticated Studio access, and add ingress/service manifests for Postgres TCP and Prole services.
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-28 11:27:34 -07:00
chrisfu
eb8c952523 chore(deploy): cluster config and k3s manifest updates
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-27 14:47:27 -07:00
chrisfu
40ea30e4c3 feat(auth): init scripts and k3s/k8s auth manifests for knoe-auth
init_knoe_auth.sh: provisions KDC secrets via 1Password, applies GKE manifests
init_knoe_users.sh: creates Kerberos principals for initial contributors
kerberos-configmap.yaml: krb5.conf for OpenBao Kerberos auth (KNOE.DEV realm)
prole-auth-deployment.yaml: k3s auth + kdc sidecar deployment for homelab
prole-kdc-configmap.yaml: k3s KDC config for homelab

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-27 14:47:27 -07:00
chrisfu
55b6a6aff3 Complete rebranding from prole to knoe and fix macOS application identity. Bulk renamed 'prole' to 'knoe' across code, scripts, and manifests. Updated Makefile with 'knoe', 'build', and 'start' targets. Implemented macOS Application Bundle (.app) support for correct identity. Fixed macOS 'Python' process name to 'Knoe.DB Installer' via Objective-C bridge. Standardized application name to 'Knoe.DB Installer' across all interfaces.
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-22 15:08:35 -07:00
chrisfu
a2de7213ff chore: disable PodDisruptionBudget for Knoe-DB across all deployments 2026-04-15 21:20:43 -07:00
chrisfu
65369811f7 Harden ingress targeting and normalize HOME paths
- enforce app-cluster-only rendering for public ingress hosts with DB-cluster guardrails\n- fix Supabase/GitLab/authority ingress host ownership and ingress-class safety checks\n- normalize persisted home-directory paths to /Users/chrisfu and update gke config defaults\n- add/adjust regression tests for ingress placement/hostname and cfg path normalization

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-12 12:32:09 -07:00
chrisfu
b1755f4e7b Stabilize prod deployment flow and harden Garage/CNPG ops
Switch production config to k8s/GKE contexts and align service naming. Add immutable StatefulSet update fallback for Garage across k3d/k3s/k8s. Harden CNPG deploy and backup bootstrap paths, and update installer coverage for CNPG webhook and Garage common ops.

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-11 20:19:36 -07:00
chrisfu
8e5cf9c9bc feat: config-driven CNPG cluster registry
Add [CNPG Clusters] section to prole.cfg as the authoritative source for
CNPG cluster definitions. The deploy pipeline reads this registry instead
of computing version-appended namespace names at runtime.

- conf/service/prole.cfg: add [CNPG Clusters] with knoe-db as identity
  cluster; fix DATABASE_NAMESPACE=knoe-db (was knoe-db-18-140); add
  KNOE_IMAGE_REGISTRY for the k3s in-cluster registry hostname
- deploy/opentofu/k3s/manifests/prole/knoe-db.yaml: replace hardcoded
  registry.test-system image with templated KNOE_IMAGE_REGISTRY
- knoe/core/actions.py: add _cnpg_cluster_registry(), _cnpg_identity_cluster();
  fix _initial_db_namespace() to use declared namespace without version suffix
- knoe/core/milestones.py: iterate over all declared clusters in Init and
  Deploy milestones; per-cluster CNPG_IMAGE_NAME support
- knoe/core/ops/cloudnative_pg.py: resolve KNOE_IMAGE_REGISTRY and
  CNPG_IMAGE_NAME overrides in _apply_manifest()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 10:56:12 -07:00
chrisfu
5460c58eb9 fix: remove Kerberos fields from k3s CNPG manifest
Drop spec.additionalVolumes, krb_server_keyfile, and hostgssenc pg_hba
rules from the k3s Cluster manifest. The CNPG CRD on k3s rejects
additionalVolumes as an unknown field, the keytab secret does not exist,
and hostgssenc rules would prevent PostgreSQL from starting without a
mounted keytab. Kerberos is disabled in conf/service/prole.cfg and
will be wired up once k3s KDC and keytab provisioning are in place.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 09:54:51 -07:00
chrisfu
a14fa654e6 chore: checkpoint deployment and identity bootstrap updates
- switch default git provider to GitLab and add cluster kubecontext settings in prole config

- add knoe user/user_role schema grants in GKE and k3s database manifests

- extend init_knoe_users bootstrap to create schema tables and provision default users

- make deploy pipeline honor configured cluster env and gate GKE-only defaults

- update GitLab OIDC default frontdoor host and refresh generated manifest/db metadata

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-10 20:55:30 -07:00
chrisfu
eb9430df9d fix(gitlab,infra): ARM64 RPi service cluster – GitLab deploy in gitlab ns on gandalf
Namespace & routing
- milestones.py: GitOpsMilestone now resolves namespace from
  gitops.gitlab_namespace (new) → Global.GITLAB_NAMESPACE → 'gitlab'
  hardcoded; never falls through to gitops.namespace (was 'gitea')
- prole.cfg: add gitops.gitlab_namespace=gitlab + GITLAB_NAMESPACE=gitlab
- init_gitlab.sh: NAMESPACE defaults to gitlab, NODE_SELECTOR blanked so
  only gitaly+minio are node-pinned; GITOPS_NAMESPACE fallback removed

GitLab on ARM64 RPi (16 KB kernel pages)
- init_gitlab.sh: DaemonSet compiles jemalloc-5.3.0 with --with-lg-page=14
  (glibc/Ubuntu) on every node; LD_PRELOAD injected per Ruby component
- Minio: quay.io 2022 image (ARM64); configure init container replaced
  with ARM64 alpine that writes credential files; MINIO_ROOT_USER/PASSWORD
  injected directly into main container env via secretKeyRef
- Minio buckets auto-created post-deploy (registry, lfs, artifacts, etc.)
- webservice/sidekiq: replicaCount=1, reduced memory (1500M/800M),
  liveness probe initialDelaySeconds=3600 (Rails loads 25-40min on RPi)
- allowedHosts set as flat string list (chart 9.x default is list-of-maps
  which breaks URI initializer in 7_gitlab_http.rb)
- gitaly+minio always pinned to gandalf (local PV); other workloads spread

Storage
- Static PVs created for gitaly (50Gi) + minio (10Gi) on synology d005
- Synology dirs created before PVs; bucket creation idempotent

Redis (shared for GitLab KAS)
- init_redis.sh: persistence disabled (no dynamic provisioner); Redis used
  as pub/sub broker only

Infrastructure / pi.prole.org
- Removed pi.prole.org from [k3s_agents] – dedicated pihole node, OOM
- host_vars: k3s_enabled=false, k3s_state=absent (storage preserved)
- New playbook: infrastructure/playbooks/disable_pi_k3s.yml (drain + disable)
- monitoring.py: node-exporter DaemonSet excludes pi.prole.org
- init_monitoring.sh: pi.prole.org excluded from node-exporter affinity
- kong-deployment.yaml: affinity rule prevents scheduling on pi (pihole owns 80/443)

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-02 20:15:11 -07:00
chrisfu
872d715864 feat: Kerberos identity management — PROLE.LOCAL realm, cross-realm trust, user provisioning
Kerberos is now the single source of truth for all knoe-system accounts.

In-cluster KDC (PROLE.LOCAL):
- Rename in-cluster KDC realm PROLE.ORG → PROLE.LOCAL to avoid collision
  with the myrddin.prole.org Samba AD DC which owns PROLE.ORG
- Add [capaths] cross-realm trust block: PROLE.ORG (Samba AD) ↔ PROLE.LOCAL
- Create admin@PROLE.LOCAL on KDC startup (master password) for UI login
- Create guest@PROLE.LOCAL on KDC startup (read-only, PROLE_KDC_GUEST_PASSWORD)
- Update prole-kdc-secrets.example.yaml with trust_shared_password and guest_password
- Update prole-auth-kerberos ConfigMap realm/servicePrincipal to PROLE.LOCAL
- Add PROLE_AUTH_ADMIN_PRINCIPALS and PROLE_KDC_GUEST_PASSWORD to prole-auth Deployment

Database (knoe-db — k3s and GKE):
- Add spec.managed.roles: admin (superuser), guest (login), developer (group)
- Enable GSS pg_hba rules for both PROLE.ORG and PROLE.LOCAL realms
- Add krb_server_keyfile parameter for postgres service principal keytab
- Mount knoe-db-pg-keytab Secret via spec.additionalVolumes
- Add demo schema in postInitSQL with guest read-only grants

prole-auth (Spring Boot):
- SessionUser: add List<String> groups field
- AuthProperties: add adminPrincipals list (env: PROLE_AUTH_ADMIN_PRINCIPALS)
- LoginController: stamp groups=["admin"] for configured admin principals at login
- SessionTokenService: carry groups through TokenPayload; @JsonIgnoreProperties
  for backward compat with existing 2-field session cookies
- VerifyController: emit X-Prole-Groups header on /auth/verify
- grafana-proxy nginx: strip inbound X-Prole-Groups; capture and forward from auth

Services:
- ArgoCD (k3s + GKE): add g, admin, role:admin to argocd-rbac-cm policy.csv

etc/init_knoe_users.sh (new):
- Orchestrates full user provisioning: KDC principals, postgres keytab export,
  CNPG managed.roles patch, demo schema SQL, ArgoCD RBAC patch,
  Gitea and GitLab admin promotion via API
- Actions: initialize | status | cleanup
- Idempotent; sources prole_cfg.sh; follows existing init_*.sh style

Cross-realm trust activation (myrddin side):
  samba-tool user add krbtgt_PROLELOCAL --random-password
  samba-tool user setpassword krbtgt_PROLELOCAL --newpassword=<trust_shared_password>
  samba-tool spn add "krbtgt/PROLE.LOCAL" krbtgt_PROLELOCAL -U administrator
  samba-tool user setexpiry krbtgt_PROLELOCAL --noexpiry

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-04-01 20:38:57 -07:00
chrisfu
bb92ec3576 feat: service layer — GitLab, Redis, DB ArgoCD app, Kong, monitoring updates
- Add init_redis.sh for Redis deployment
- Extend init_gitlab.sh with Kerberos/OIDC prep and service wiring
- Add init_monitoring.sh improvements
- Update init_kong.sh and init_service_layer.sh
- Rename ArgoCD application-openbao → application-db; add DB kustomization
- Update kong-configmap, kustomization, port-mapping, service configs
- Bump knoe-db version; refresh network scan

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-04-01 20:38:57 -07:00
chrisfu
620bdd25eb feat: GKE production deployment — CNPG + GCS backups + Google Workspace SSO
Add complete GKE deployment stack:
- deploy/gcp/gke/: namespace, CloudNativePG cluster (premium-rwo, 100Gi),
  GCS barman-cloud backup with Workload Identity SA, ArgoCD OIDC ConfigMap,
  GitLab OmniAuth secret template, knoe-auth Google OIDC secret template
- etc/init_cnpg_gke.sh: provision GCS buckets, GCP SA, WI binding, CNPG operator
- etc/init_argocd.sh: apply argocd-oidc-cm.yaml when PLATFORM_DOMAIN/FRONTDOOR_HOST set
- etc/init_gitlab.sh: inject OmniAuth openid_connect block when FRONTDOOR_HOST set
- prole-auth-deployment.yaml: add GOOGLE_OIDC_ENABLED env vars (all optional)
- supabase/helm/prole-supabase/values-gke.yaml: GoTrue Google OAuth overlay
- knoe/core/prod_config.py: add backupProvider, workloadIdentitySA, googleClientId/Secret/HostedDomain

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 10:41:55 -07:00
chrisfu
93f7259722 chore: checkpoint k3s deployment and knoe updates
- update Kong/Supabase ingress and service manifests for k3s

- expand init scripts and runtime config/topology handling

- refresh KNØE UI screens and Supabase deployment/render logic

- include related env helper test adjustments

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-31 23:15:54 -07:00
chrisfu
65d03ed442 stable: slimmed Supabase + CNPG 3-node healthy build (2026-03-25)
STABLE BUILD — all 35+ pods Running, zero non-Running pods, CNPG ready=3.

## Architecture changes (Supabase slim-down)
- Removed local Supabase Kong, Storage, MinIO, Imgproxy from deployed stack
- Rewired API routing to shared Kong in kube-system
- Rewired object storage to shared Garage in knoe-system
- Supabase footprint reduced to control-plane only: auth, rest, meta,
  analytics, realtime (studio port-forward retained)
- Supabase pods renamed: stripped redundant '-prole-supabase-' prefix via
  per-component fullnameOverride (e.g. supabase-auth, supabase-rest, ...)
- Supabase workloads pinned to merlin.prole.org (memory-safe node)

## File changes
- conf/service/prole.cfg
    * Replaced all hardcoded /Users/chrisfu/dev/prole paths with ${HOME}/dev/prole
      so config is portable when run from myrddin.prole.org or any user home
    * SUPABASE_ENABLED=True, SUPABASE_PRIMARY_NODE=merlin.prole.org in
      both [Global] and [Optional Features]
    * supabase_enabled=true in [Inputs]/init_cluster section
- supabase/helm/render_supabase.py
    * Explicitly disabled: kong, storage, minio, imgproxy
    * Active components pinned via nodeSelector to merlin.prole.org
    * fullnameOverride per component to clean up pod names
    * Default primary node changed from pi.prole.org -> merlin.prole.org
- supabase/deploy.sh
    * Removed svc/kong port-forward; API via shared Kong in kube-system
    * Removed imgproxy/storage from PV/PVC prep and helm --set persistence args
    * Fixed kubectl exec missing -i flag for psql heredoc stdin in
      setup_knoe_db_for_supabase
    * Updated Studio access messaging to reference shared platform Kong
- k8s/prole/knoe-db.yaml
    * instances: 3
    * Added control-plane/master tolerations for myrddin scheduling
- deploy/opentofu/k3s/manifests/prole/knoe-db.yaml
    * instances: 3, added control-plane tolerations (mirrors k8s/prole/)
- k8s/prole/iscsi-pvs.yaml
    * Expanded d003-data/d003-wal nodeAffinity to [pi, merlin, myrddin]
      to reflect post-OOM-recovery live state
- k8s/registry/deployment.yaml
    * nodeSelector restored to myrddin.prole.org (hostPort 5000 owner)
    * Added control-plane + master tolerations so pod schedules correctly
- modes/k3s/knoe-db/.version
    * Aligned to 140 (image tag 18-140) matching repo-root version
- etc/init_common_services.sh, etc/init_registry.sh
    * Minor fixes aligned with slimmed service layout and node targeting
- infrastructure/inventory/host_vars/pi.prole.org.yml
    * Updated host vars to reflect post-recovery pi state
- knoe/core/topology.py, knoe/ui/screens/cluster.py,
  knoe/ui/screens/cluster_nodes.py
    * Topology and UI updates for 3-node cluster representation
- scan/network_description.txt
    * Refreshed network scan output post-recovery
- conf/port-mapping.cfg
    * Updated port mapping to reflect removal of local Kong/storage forwards
- scripts/cleanup_stale_rs.sh (new)
    * Utility script to prune orphaned ReplicaSets after rollouts

## Verified stable state
- CNPG: Cluster in healthy state, ready=3
    knoe-db-2 (merlin, primary), knoe-db-3 (pi), knoe-db-5 (myrddin)
- Supabase: 5 pods Running on merlin.prole.org
    supabase-auth, supabase-rest, supabase-meta,
    supabase-analytics, supabase-realtime
- Monitoring: all prometheus/grafana/alertmanager pods Running on merlin
- Registry: Running on myrddin.prole.org (hostPort 5000)
- Zero non-Running / non-Completed pods across all namespaces

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-25 11:56:40 -07:00
chrisfu
0e806a9233 checkpoint: tighten CNPG k3s storage flow and policy updates
- normalize K3S/CNPG/OpenTofu env keys across installer and configs; - harden CNPG storage validation/provisioning and fail fast in init script; - update DB manifests/recovery template and port/network mapping data; - expand cluster node policy UI coverage and shell/python tests; - add k3s mode version marker files

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-23 12:39:03 -07:00
chrisfu
9f7771adeb Separate k3s and k3d config defaults; normalize k3s image registry refs
- set global MODE/DEPLOYMENT_MODE to k3s and remove cross-environment/prod cluster entries from active dev/service config paths\n- switch k3s prole-auth image references to myrddin.prole.org:5000\n- strengthen CNPG image resolution and test coverage for rewriting k3d/localhost registries in k3s mode\n- include updated port mapping/network snapshot and current k3d mode version metadata

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-23 03:48:58 -07:00
chrisfu
a069989315 Rename prole-db to knoe-db, add knoe-auth as cluster-internal KDC
Itemized changes:

1. knoe-auth: New cluster-internal KDC and SSO gateway service
   - Created etc/init_knoe_auth.sh based on init_kdc.sh with knoe-auth naming
   - Namespace defaults to SERVICE_NAMESPACE (knoe-system)
   - ConfigMap: knoe-auth-kdc-config, Secret: knoe-auth-secrets
   - Legacy cleanup removes old auth/dog/authority deployments

2. Orchestration: knoe-auth initializes before CloudNativePG
   - Updated prole.sh to insert init_knoe_auth.sh as step 2 (before CNPG)
   - Renumbered all subsequent initialization steps

3. Kong routing: Updated init_kong.sh to route to knoe-auth in SERVICE_NAMESPACE

4. Comment/reference updates for knoe-auth
   - Updated init_common_services.sh, init_service_layer.sh, init_kerberos.sh

5. prole-db renamed to knoe-db across the entire codebase
   - Renamed prole-db/ directory to knoe-db/
   - Renamed all prole-db Kubernetes manifests (deploy/opentofu, k8s/)
   - Renamed scripts: docker-root-knoe-db.sh, docker-run-knoe-db.sh, test-cnpg-knoe-db.sh
   - Renamed etc/init_prole-db-reset.sh to etc/init_knoe-db-reset.sh
   - Renamed etc/prole-db-passwwd.sh to etc/knoe-db-passwwd.sh
   - Renamed mock_val counterparts accordingly
   - Renamed tests/etc/test_init_prole-db-reset.sh to test_init_knoe-db-reset.sh
   - Renamed docs/prole-db-documentation-mcp-architecture.md to knoe-db variant
   - Renamed modes/k3d/prole-db/ to modes/k3d/knoe-db/
   - Renamed prole-db.iml to knoe-db.iml

6. Configuration updates
   - Updated conf/dev, conf/prod, conf/test, conf/service prole.cfg files
   - Updated conf/port-mapping.cfg
   - Updated etc/prole_cfg.sh and mock_val/prole_cfg.sh
   - Updated service/prole.cfg

7. Kubernetes manifests and deploy configuration
   - Updated deploy/opentofu/k3s ArgoCD application YAMLs
   - Updated kong-configmap.yaml and kustomization.yaml
   - Updated k3s/kong-config.yml and prole-resources.yaml
   - Updated prole-mssql-db deployment YAMLs
   - Updated supabase helm render and deploy scripts

8. Infrastructure and GCP Terraform
   - Updated deploy/gcp/terraform: folders, groups, IAM, service-projects

9. Python/installer code updates
   - Updated knoe/core: actions, build_context, controller, env, milestones
   - Updated knoe/milestone.py
   - Updated knoe/ui/screens: cfg, database, database_options, deploy, docker,
     navigation, security, services, validate
   - Updated knoe.spec, status.py

10. Shell script updates
    - Updated etc/: build_db, init_cloudnative_pg, init_cnpg_backup,
      init_db_manager, init_forgejo, init_gitlab, init_monitoring, init_openbao,
      init_port_forwards, init_postgrest, init_supabase_ports, status
    - Updated mock_val/ counterparts for all above scripts
    - Updated prole-net/init-prole-dns.sh
    - Updated bin/prole-kpf.sh, gitea/deploy.sh, supabase/deploy.sh

11. Test updates
    - Updated tests/etc/: test_init_cloudnative_pg*, test_init_cnpg_backup*,
      test_init_kdc*, test_init_kerberos*, test_init_kong*, test_prole_cfg*
    - Updated tests/installer/: test_actions_helpers, test_cfg_save_kubecontext,
      test_controller, test_core_classes, test_milestones, test_milestones_extended,
      test_namespace_propagation
    - Updated tests/: test_database_options, test_navigation,
      test_render_supabase_hostname, test_docker_build_fix,
      test_all_prole_home_fixes, silent_install_test, final_test

12. Documentation updates
    - Updated docs/: DOCKER-BUILD-FIX, PROLE-CFG-SECRETS, PROLE-HOME-DIRECTORY,
      build-system, patent
    - Updated scan/network_description.txt
    - Updated pom.xml

13. Miscellaneous script updates
    - Updated root-level: _adopt_replica_pvcs, _fix_replica_merlin, _import_pi,
      _patch_cluster, _prebind_pvcs, _rebind_d002, _rebind_d002b, test_resolve
    - Updated scripts/generate_spec.py

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 22:16:21 -07:00
chrisfu
9b9d6fdc88 Rename Prole storage objects to Synology and sync deployment updates
- Rename iSCSI storage class and PV/PVC selectors/labels from prole to synology across k8s and OpenTofu manifests\n- Update CNPG/OpenBao/Garage/monitoring init flows, render helpers, and mock scripts for synology-backed storage objects\n- Integrate related UI/core/service config/version updates and add supporting regression tests for CNPG storage/image behavior\n- Keep storage reconciliation tests aligned with current CNPG affinity output

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 21:50:02 -07:00
chrisfu
0262bd4be2 Refactor to support mode-scoped DB build contexts and versioning
- Introduce mode-scoped directories under `~/.prole` for build contexts, runtime data, and versioning.
- Update DB image version generation and Dockerfile creation to respect deployment modes (e.g., k3d, k3s, k8s).
- Adjust UI, core actions, and scripts to avoid cross-mode conflicts and ensure correct version isolation.
- Add tests for mode-aware behaviors and update existing tests for new paths and markers.
- Bump PostgreSQL and image versions to `18` and `138`, respectively.
- Update Kubernetes manifests to align with new image and version configuration.
2026-03-22 05:47:50 -07:00
chrisfu
4ee2b259c9 Checkpoint: rename installer to knoe + harden db build context
- Add build-context helper to copy Docker context safely (ignore runtime data, keep symlinks)

- Update UI and core actions to use ~/.prole/build and shared copy helper

- Add/adjust tests and scripts; introduce knoe ops helpers and update manifests

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 01:45:21 -07:00
chrisfu
2244d6acd6 Refactor iSCSI Synology mounts to /synology/d00x
- Rename host mount namespace from /prole/d00x to /synology/d00x

- Make LUN ownership explicit per host in inventory (myrddin=d001; merlin=d002,d004; pi=d003)

- Add safe migration cleanup for legacy /prole/d00x mounts and stale /etc/fstab entries

- Update k3s/ArgoCD/OpenTofu manifests and PV node affinities to match new mount paths

- Improve iSCSI role check-mode behavior and add quiesce/detach flow in ansible.sh
2026-03-21 10:43:06 -07:00
chrisfu
ecacf01266 Remove legacy secrets, configs, and scripts; introduce prole-auth service and Grafana SSO proxy configuration. 2026-03-20 13:00:36 -07:00
chrisfu
b114099e17 Use prole.cfg hostnames for Supabase front-door
- Persist and export supabase_hostname for canonical external Supabase entrypoint (db.prole.org)

- Render Supabase Helm ingress host and public URLs (API_EXTERNAL_URL/GOTRUE_SITE_URL/SUPABASE_PUBLIC_URL) from config

- Align tracked Traefik ingress manifests to db.prole.org

- Generalize service hostname/TLS wiring in k3s automation and refresh Kong/monitoring behavior

- Make optional workload policy checks deterministic when kube context is absent; add render tests
2026-03-19 04:19:38 -07:00
chrisfu
16a3dce584 k3s: enforce node placement for monitoring/CNPG/Supabase
- Pin Prometheus/Alertmanager + Grafana to merlin.prole.org in k3s to avoid PV node-affinity issues

- Taint myrddin.prole.org as control-plane-only and add CNPG toleration

- Prevent init_cloudnative_pg from deploying prole-svc-kong into the DB namespace

- GitOps-manage Kong declarative config via ConfigMap + add placement/init tests
2026-03-18 23:16:06 -07:00
chrisfu
133b719479 Align CNPG bootstrap placement with Ansible policy
Prefer infra-managed manifests for bootstrap; reconcile CNPG instances based on Ready+schedulable labeled db nodes; update manifests to use node-role affinity + anti-affinity; add policy/tests and config touch-ups (incl. prole.cfg).
2026-03-18 22:06:41 -07:00
chrisfu
b7fd89b373 Fix instance count enforcement for CNPG to always default to 3 and update manifests/tests accordingly 2026-03-11 13:21:50 -07:00
chrisfu
396cea60e7 installer: stabilize retries + improve k3s inventory detection
- Reuse namespace from existing prole.cfg unless env overrides

- Support k3s_hosts children groups and prefer k3s_servers as default

- Add headless pytest tkinter stubs and blocked-cluster reconciliation tests

- Update registry mirror endpoint, port mappings, and prole-db manifests
2026-03-07 23:57:21 -08:00
chrisfu
a43aed7134 k3s: eliminate localhost registry + dedupe common-core
- Ensure k3s mode uses the k3s registry endpoint and avoid localhost/k3d image prefixes.

- Make ArgoCD repo-server cmp symlink creation idempotent.

- Normalize common-core provisioning to knoe-system and add repair-time dedupe of stray default-namespace installs.

- Add k3s MariaDB datastore/refresh playbooks and regression tests.
2026-03-05 14:31:33 -08:00
chrisfu
86cf99cf76 milestone(k3s+cnpg): end-to-end deploy run via standalone installer
We can now complete a full K3s deployment run and deliver a working CloudNativePG (CNPG) stack from the standalone installer.\n\nHighlights:\n- Installer scripts hardened for CNPG deploy + recovery templating + rollout control\n- OpenTofu/K3s manifests aligned with in-repo k8s templates for prole-db\n- Supporting init flows updated (OpenBao, monitoring, Kerberos, OpenTofu)

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-03 20:03:15 -08:00
chrisfu
5618b662dd Remove prole-db-manager; simplify deployment via prole-authority; fix pg18 downgrade & cluster name
Summary:
Removed the prole-db-manager microservice and simplified deployment to use
prole-authority as the internal management and authorization point. Fixed two
blocking bugs that prevented silent install from completing on knoe-dev-cluster.

Removed: prole-db-manager
- Deleted db-manager-deployment.yaml and db-manager-service.yaml from opentofu manifests
- Deleted src/db-manager/ (Dockerfile, server.js, package.json, tests)
- Removed prole-db-manager port-forward mapping from installer/core/env.py
- Removed init_db_manager.sh from Initialization Scripts (milestones.py, actions.py)
- Removed init_certmgr.sh and init_db_manager.sh tabs from services screen (services.py)
- Removed live k8s Deployment/Service from knoe-dev-cluster

Fixed: PostgreSQL version downgrade error (pg17 -> pg18)
- Created conf/postgresql/.version with value 18
- Updated k8s/prole/prole-db.yaml and prole-db-recovery.yaml.tpl imageName to prole-db:18-089
- Fixed _init_database_options_state() to restore saved version_type from prole.cfg
  so db_version_type defaults to v18 (pg18) instead of silently reverting to pg17
- Added database_options.* keys to _collect_input_snapshot() in cfg.py so
  distribution, version_type, and all extension toggles persist to prole.cfg

Fixed: Cluster name inconsistency
- Removed stale prole-dev-cluster references; all scripts now use knoe-dev-cluster
- Added knoe-dev-cluster to mode-detection case in etc/prole_cfg.sh

Config: conf/prole.cfg
- Set kerberos_config.enabled = False, KERBEROS_AUTO_ENABLED = False
- Added database_options.distribution = percona, version_type = v18
- Added all 13 extension flags set to True (postgis, pgvector, pgcrypto, pgaudit,
  pg_repack, pg_stat_statements, pg_buffercache, pg_freespacemap, pgrowlocks,
  postgres_fdw, dblink, pg_stat_monitor, pgbadger)

Verification:
./install.py -s -l -v -c conf/prole.cfg completed successfully.
CNPG deployed prole-db:18-089 to knoe-dev-cluster; all milestones passed.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-01 20:40:44 -08:00
chrisfu
e64bbfb07f Fix k3s Kong manifest names
Align Kong k3s manifests with etc/init_kong.sh and svc-check Helm values (prole-svc-kong + prole-svc-kong-config) so rollout waits for the correct deployment.
2026-02-27 14:59:08 -08:00
chrisfu
f589cbd397 Remove PostgREST deployment, service, and related configs from prole-db namespace
- Deleted PostgREST Kubernetes deployment and service manifests.
- Removed `init_postgrest.sh` initialization script and PostgREST references across UI, installer, and config.
- Updated Kong API Gateway to exclude `/rest/*` routing to PostgREST.
- Simplified `supabase/deploy.sh` by removing references to PostgREST scaling and alias creation.
- Adjusted cluster configuration and UI to reflect the absence of PostgREST.
2026-02-25 00:38:07 -08:00
chrisfu
1573bb59c2 Add prole.spec and extend OpenTofu k3s configuration
- Include new `prole.spec` for build configurations and dependencies.
- Add Terraform state handling for OpenTofu in `k3s` cluster.
- Provision multiple Kubernetes resources in `prole-db` namespace: namespace, services, ConfigMaps, StatefulSets, Ingress rules, and PersistentVolumes.
- Integrate deployment and configuration enhancements for `garage`, `prole`, and related components.
2026-02-24 21:47:02 -08:00
chrisfu
51c91595b9 Refactor infrastructure, installer, and deployment configs: expanded CloudNativePG and PostgREST init scripts, updated ArgoCD/OpenTofu/K8s manifests (ingress, kustomization, postgrest, prole-db), added installer config helpers and revised cluster UI, updated prole.cfg and prole.sh, simplified Makefile, added env helper tests, removed stale spec/log files, added prole/ directory 2026-02-23 21:19:19 -08:00
chrisfu
a98c6d9d1c Fix Kong OOM crash: increase memory limit to 1Gi, mount ConfigMap to subdirectory 2026-02-22 01:02:00 -08:00
chrisfu
b03efa8f69 Kong API gateway, docker-import preload, OpenTofu graceful fallback, milestone fix
Kong API Gateway (replacing prole nginx):
- Add etc/init_kong.sh provisioning script (DB-less mode, prole-db namespace)
- Add kong-deployment.yaml and kong-service.yaml manifests
- Rewire ingress rules (svc/git/api.prole.org) to prole-db-kong:8000
- Update kustomization.yaml to reference kong manifests

PostgREST & DB Manager in prole-db namespace:
- Add etc/init_postgrest.sh and etc/init_db_manager.sh scripts
- Add postgrest/db-manager deployment and service manifests
- Add src/db-manager/ Node.js REST endpoint for backup triggers
- Default NAMESPACE changed to prole-db in both scripts

Docker image pre-load from PROLE_DATA/docker-import:
- Add _preload_docker_images() to init_common_services.sh
- Scan for .tar files exported by final_deployment.sh
- Import via k3d image import (k3d) or ctr (k3s) before deployments
- Increase rollout timeouts to 300s (configurable via ROLLOUT_TIMEOUT) in init_openbao.sh, init_opentofu.sh, init_garage_store.sh, init_registry.sh

OpenTofu password resolution fix:
- Add Kubernetes secret fallback in resolve_admin_password()
- Change hard exit 1 to graceful return 1 with warning
- Wrap call in if-guard so set -e doesn't abort the script chain

Milestone fix (init scripts not running):
- Add init_kong.sh, init_postgrest.sh, init_db_manager.sh to InitializationScriptsMilestone.execute() script list and arg branches
- Previously only actions.py had these; milestones.py was missing them

Installer integration:
- Add Kong/PostgREST/DB Manager to silent installer _step_init_scripts
- Add corresponding tabs and execution blocks in UI services.py
2026-02-22 00:57:49 -08:00
chrisfu
6e2d3e9011 refactor: modernize installer and monitoring setup
- Monitoring: Migrated from manual Grafana/Prometheus manifests to kube-prometheus-stack based setup in etc/init_monitoring.sh. Removed old manifest files from deploy/ and k8s/.
- Installer Core: Refactored installer with new modules for actions, environment handling, and UI screens. Enhanced Milestone logic to support advanced configuration (ArgoCD, Registry namespaces, Kerberos flags, etc.).
- Service & Init Scripts: Updated multiple initialization scripts (init_*.sh) for better integration with OpenBao, Kerberos, and the new monitoring stack. Added new scripts for Nginx Ingress, Ollama parsing, and K3D route fixes.
- Infrastructure: Enhanced Samba AD DC Ansible role with realm derivation, provisioning guidance, and group management. Updated K3s role tasks.
- Configuration: Refined default settings in conf/ to align with the new deployment architecture.
- App & Tools: Updated prole-app Swift code and prole.sh for improved environment variable handling and installation flow.
2026-02-19 21:07:39 -08:00
chrisfu
9aa959f61d Milestone: Balanced iSCSI IO, updated prole.sh launcher with log/verbose/debug/coverage/silent options, and enhanced installer logging 2026-02-16 16:44:18 -08:00
chrisfu
23cf4ac33d Configure K3s for retropie and add Traefik Ingress. Updated K3s templates for node-ip/flannel-iface, configured retropie host vars, enabled Traefik, added Ingress resources for Grafana, Prole, Kong, and Supabase, and set Postgres to LoadBalancer. 2026-02-16 04:28:40 -08:00
chrisfu
96f594fd3c Refactor initialization scripts and add new service components
- Consolidated and split initialization scripts in etc/:
    - Removed init_prole-db.sh and init_authority.sh.
    - Added init_kdc.sh for in-cluster MIT Kerberos KDC (prole-authority).
    - Added init_ollama.sh for Ollama AI service integration.
    - Added init_service_layer.sh for high-level service orchestration.
    - Added init_k3s_registry.sh for private registry management.
- Major updates to install.py:
    - Support for new Ollama and KDC configuration.
    - Improved prole.cfg rendering and namespace handling.
    - Updated unattended install flags.
- Infrastructure and Deployment:
    - Updated K3s Ansible role with private registry support (registries.yaml template).
    - Added prole-authority Dockerfile.
    - Updated OpenBao Kerberos ConfigMap and other K8s manifests.
- Configuration:
    - Updated prole.cfg with new sections for Ollama and Monitoring.
    - Refined environment variable exports in env.sh and prole_cfg.sh.
2026-02-11 13:09:31 -08:00
chrisfu
87e2f5d385 Checkpoint: Refactor common services and monitoring initialization. Updates: etc/init_monitoring.sh (Grafana naming, legacy cleanup), etc/init_common_services.sh (migration/cleanup logic), added etc/init_registry.sh, updated etc/init_cloudnative_pg.sh (manifest filtering), updated configs, and enhanced tests. Fixed: excluded .ansible logs and added to .gitignore. 2026-02-09 18:50:44 -08:00
chrisfu
f47c18fef7 feat(infrastructure): enhance k3s automation and OpenTofu integration
- Infrastructure:
    - Updated k3s Ansible role with mountpoint preflight checks and better permission management.
    - Automated deployment of prole configuration and port-forwarding scripts to cluster hosts.
    - Added systemd service for managing port forwards on k3s nodes.
    - Added prole-installer service account token automation.
- K8s Manifests:
    - Renamed and added Persistent Volumes in iscsi-pvs.yaml (including OpenBao support).
    - Updated StatefulSets for garage and openbao.
    - Migrated prole-db to CloudNativePG-based configuration.
    - Added comprehensive OpenTofu manifests for cluster deployment.
- Configuration:
    - Added cluster-specific configurations (k3d, k3s-hosts).
    - Added PostgreSQL configuration templates.
    - Updated .gitignore to track the conf/ directory.
- Tools:
    - Updated install.py and port-forwarding scripts.
    - Added render_manifest.py for manifest generation.
2026-02-07 22:53:18 -08:00
chrisfu
da2f6600ba feat: infrastructure and installer updates for k3s, OpenTofu, and prole-db
- Add k3s start/stop Ansible playbooks and roles.

- Implement OpenTofu initialization scripts and k8s manifests.

- Update ncurses installer with OpenTofu support and improved k3s integration.

- Add mode support (--mode) to etc/ initialization scripts.

- Update prole-db with recovery, barman objectstore, and SSH OpenBao support.

- Refine k8s manifests for OpenBao and prole-db.
2026-02-05 21:27:18 -08:00