Commit Graph

138 Commits

Author SHA1 Message Date
chrisfu
5cece408bd ansible: switch kerberos_trust_setup to KNOE.LOCAL + vault wiring
The cross-realm trust playbook was written for the pre-rebrand realm name
PROLE.LOCAL, which no longer exists in the in-cluster KDC — the canonical
realm is now KNOE.LOCAL. The Samba AD trust was therefore never actually
established, leaving chrisfu@KNOE.LOCAL→chrisfu@PROLE.ORG service ticket
flows blocked (and blocking pg_oauth / db.prole.org Kerberos work).

Changes:
- Realm: PROLE.LOCAL → KNOE.LOCAL across all task names, vars, and the
  krb5.conf [realms] / [domain_realm] blocks added on myrddin.
- samba_admin_password now resolves from the Ansible vault var
  vault_samba_dns_admin_pass (group_vars/ad_dc/vault.yml) by default,
  with SAMBA_ADMIN_PASSWORD env and -e overrides preserved for CI.
- trust_shared_password auto-resolves from the in-cluster Secret
  knoe-system/knoe-kdc-secrets (key trust_shared_password) when not
  passed explicitly — same Secret init_kdc.sh writes.
- Added [domain_realm] mapping for *.knoe.local → KNOE.LOCAL so Samba
  can resolve in-cluster service principals.
- Added a final `samba-tool domain trust show` smoke step so a successful
  run prints the visible trust state for log review.

Run:
  ANSIBLE_VAULT_PASSWORD_FILE=$PWD/.vault_pass   ansible-playbook infrastructure/playbooks/kerberos_trust_setup.yml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10 00:26:31 -07:00
chrisfu
48def19637 fix(inventory): specify ssh key for zinfandel — non-default key name 2026-05-09 22:42:26 -07:00
chrisfu
9020eae91e fix(ansible): remove become from Windows play — connect as admin directly
win_regedit writes HKLM keys without privilege escalation when
ansible_user is already a local/domain Administrator. become:true
with --ask-become-pass was overriding the playbook's runas method.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 22:36:48 -07:00
chrisfu
67b483bcf2 fix(ansible): split workstation playbook into two plays for Unix vs Windows
A single play with become:true uses sudo by default, which Ansible's
Windows exec wrapper rejects immediately at gather_facts. Split into:
- Play 1: hosts workstations,!workstations_windows — macOS/Linux, sudo
- Play 2: hosts workstations_windows — Windows, become_method:runas

Also set morgana to ansible_connection=local (running Ansible from
morgana itself, so SSH self-lookup fails DNS resolution).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 22:33:51 -07:00
chrisfu
5cd22c53d9 feat(inventory): add all 4 workstations; add Windows SPNEGO policy support
Add morgana and zinfandel (macOS) and morgoth + fairyland (Windows)
to the workstations inventory.

Extend workstation_kerberos.yml to handle Windows targets:
- Chrome and Edge AuthServerAllowlist set via win_regedit under
  HKLM\SOFTWARE\Policies\{Google\Chrome,Microsoft\Edge}
- krb5.conf and plist tasks skip on Windows (uses built-in SSPI/AD)
- Windows hosts in [workstations_windows] group with WinRM/NTLM
  transport; switch to kerberos transport after domain join

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 22:21:21 -07:00
chrisfu
7b782afef4 chore: restore vault secrets added on myrddin 2026-05-09 21:48:54 -07:00
chrisfu
7b391ab5ed feat(onboarding): scalable Kerberos SSO onboarding — service account auth, Ansible workstation policy
Problem: 'defaults write' is not an onboarding path, it's a support
incident waiting to happen. Browser SPNEGO must be deployed via managed
policy at machine provisioning time, not per-user.

Changes:
  etc/init_knoe_users.sh
  - _op_ensure_auth(): tries OP_SERVICE_ACCOUNT_TOKEN env, k8s secret
    'op-service-account-token', existing interactive session, and op-run
    inheritance — in that order. Gives actionable instructions when none
    work, including the correct 'op run --' invocation documented in k3s.cfg.
  - gitea_ensure_password(): calls _op_ensure_auth() at entry; eliminates
    the silent 'op not authenticated' failure path.
  - Script header: documents 'op run -- bash etc/init_knoe_users.sh' as
    the intended invocation for admin's laptop.
  - 'Next steps' output: replaces 'defaults write' with reference to
    workstation_kerberos.yml Ansible playbook.

  infrastructure/playbooks/workstation_kerberos.yml (new)
  - Deploys /etc/krb5.conf (PROLE.ORG realm, myrddin KDC) to all managed
    endpoints.
  - Deploys Chrome + Edge managed policy (AuthServerAllowlist) on macOS
    and Linux — no per-user browser configuration ever required.
  - Idempotent; run during laptop provisioning or re-run at any time.
  - Targets 'workstations' Ansible group.

  infrastructure/inventory/hosts.ini
  - Adds [workstations] group with example entries and onboarding notes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 20:10:32 -07:00
chrisfu
1892e137c1 feat(infra): replace GitLab with Gitea on git.prole.org
- Remove GitLab hostname/storage config from k3s.cfg; GitLab is gone.
- Update GITEA_HOSTNAME git-internal → git.prole.org
- Fix MONITORING_STORAGE_CLASS local-path → merlin-local-iscsi-prometheus
- Kong: route git.prole.org → gitea-http.gitea:3000
- Add svc-knoe-ingress.yaml as source-of-truth for the prole ingress (was
  missing; live object named svc-knoe-ingress, routes git.prole.org through
  Traefik → Kong → Gitea)
- DNS: add git.prole.org A record → 73.15.20.166 (public front-door)
- monitoring/kps-values-k3s.yaml: fix storage class for Prometheus/Alertmanager

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 01:25:32 -04:00
chrisfu
52f4053718 fix(acme): replace fragile key-file regex selector with rejectattr
selectattr with 'match' regex was returning empty sequence on this Ansible
version. Since each install entry has exactly two items (fullchain.cer +
key), rejectattr the fullchain to get the key — no regex needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-06 15:57:57 -04:00
chrisfu
e0c96f47d4 fix(acme): use correct mixed-case Namecom_Username/Token variable names
The dns_namecom.sh plugin uses Namecom_Username and Namecom_Token (not
NAMECOM_*). Fix the environment dict, the SAVED_ keys written to
account.conf, and add a cleanup task to remove the stale all-caps entries
already present on hosts from previous runs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-06 15:35:20 -04:00
chrisfu
18f2ce2167 fix(acme): restore no_log and switch to command+environment for credential safety
Re-adds no_log: true removed in 5e4e17e and switches the issue task from
ansible.builtin.shell with inline VAR=value to ansible.builtin.command with
an environment: dict. This prevents credentials appearing in Ansible failure
output and avoids the inline env var being clobbered when acme.sh sources
account.conf internally before the dns_namecom plugin runs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-06 15:27:44 -04:00
chrisfu
5e4e17ec60 debug: temporarily remove no_log from issue task to see acme.sh output 2026-05-06 15:15:47 -04:00
chrisfu
372496eea2 fix(acme): use shell inline env vars for issue; bypass sudo env_reset
Ansible command module + become does not reliably pass environment:
vars through sudo env_reset. Use ansible.builtin.shell with POSIX
inline assignment (VAR=val cmd) so credentials are set inside the
sudo-spawned shell subprocess, never touching the sudo env barrier.
account.conf writing retained for cron renewal.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-06 15:14:12 -04:00
chrisfu
5ddb76930f fix(acme): use SAVED_ prefix for namecom creds in account.conf
_readaccountconf_mutable checks SAVED_NAMECOM_Username first; if that
key lookup exits 0 with empty output it short-circuits and never reads
the bare NAMECOM_Username key. Write the SAVED_ prefix that acme.sh
uses internally when persisting credentials.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-06 15:10:13 -04:00
chrisfu
76b641842e fix(acme): write namecom creds to account.conf instead of env vars
sudo env_reset strips the environment: key through become. Write
NAMECOM_Username and NAMECOM_Token to acme.sh's account.conf directly,
which dns_namecom reads as a fallback via _readaccountconf_mutable.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-06 14:57:28 -04:00
chrisfu
d268693b2a fix(acme): pin --server letsencrypt on issue to avoid ZeroSSL fallback
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-06 14:55:03 -04:00
chrisfu
86990af9c8 feat(certs): add acme.sh DNS-01 role and cert-manager playbook for LE
- New acme role: installs acme.sh, issues certs for myrddin/merlin/gandalf
  via DNS-01 (name.com API), installs cert files to /etc/ssl/certs/prole/,
  sets up renewal cron. rc=2 (already valid) treated as ok.
- New playbooks/acme.yml: runs acme role on myrddin (issues all 3 host certs)
- New playbooks/certmgr.yml: cert-manager and letsencrypt-prod ClusterIssuer
  (HTTP-01/Traefik) for external endpoints (db/svc/api.prole.org)
- Add vault references for name.com API creds to group_vars/all/vars.yml
- Add prole_ssl_src_dir and prole_ssl_files to myrddin/merlin/gandalf host_vars

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-06 14:09:28 -04:00
chrisfu
296bcd15a5 fix(dns,ssl): add gandalf A record; make prole_ssl resilient to missing certs
- Add gandalf.prole.org → 10.0.0.7 to prole_dns_records and
  prole_internal_a_records in dns.yml (was missing, causing nslookup
  to fail on myrddin)
- prole_ssl: stat-check each source file on the controller before
  copying; emit a warning and skip rather than hard-failing when certs
  have not yet been placed in ssl/prole/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-05 20:22:22 -04:00
chrisfu
e4478f20ab feat(tailscale): add Ansible role and playbook for merlin + gandalf
Installs Tailscale on k3s agents via the Debian stable apt repo using
the modern signed-by keyring approach. Auth key stored in vault.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-05 18:57:05 -04:00
chrisfu
bb71cf68ea feat(auth): dual IdP for db.prole.org — Google OAuth + Kerberos SPNEGO
knoe-auth (Spring Boot OIDC provider):
- AuthProperties: add google.workspaceDomain and kerberos.servicePrincipal fields
- GoogleOAuthService: validate hd (hosted domain) claim; restrict to configured workspace
- LoginController: /login/google endpoint + SPNEGO negotiation entry point
- PrincipalNormalizer: map Kerberos principal (user@REALM) to knoe-auth user
- application.yml: add spring.security.kerberos and oauth2.client stubs (values
  injected at runtime from env / Kubernetes Secrets)
- knoe-auth-deployment.yaml: mount HTTP keytab Secret; add GOOGLE_PROLE_CLIENT_ID /
  GOOGLE_PROLE_CLIENT_SECRET env from oauth2-proxy-prole-secret
- knoe-auth-http-keytab-secret.example.yaml: example Secret for HTTP/<host> keytab

Kong (init_kong.sh):
- Add db.prole.org route in k3s mode block via oauth2-proxy upstream
- Mode-gate: only registered for k3s, excluded for k3d/k8s

Supabase / oauth2-proxy:
- New supabase/helm/oauth2-proxy Helm chart: gates Supabase Studio at db.prole.org
  with Google OAuth (email-domain=prole.org) + cookie settings for .prole.org domain
- values-k3s.yaml: k3s-specific overrides (upstream service, TLS, cookie domain)
- secret-example.yaml: placeholder for oauth2-proxy-prole-secret

Ansible:
- infrastructure/playbooks/kerberos_trust_setup.yml: automates samba-tool domain
  trust create on myrddin.prole.org for PROLE.LOCAL ↔ PROLE.ORG cross-realm trust

Test:
- GoogleLoginProleOrgTest: verifies hd=prole.org tokens are accepted; hd=other.com rejected

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-03 20:03:09 -07:00
chrisfu
5ff46a0f3a infrastructure: restore ansible configuration and refactor samba_dns for internal zone - Restored the 'infrastructure' directory and root-level 'ansible.sh', 'ansible.cfg' scripts. - Refactored 'samba_dns' role to dynamically handle 'prole.org' and 'internal.prole.org' DNS zones. - Switched 'samba-tool' commands to use machine account authentication (-P) in 'samba_dns' and 'samba_reverse_dns'. - Updated AD DC inventory variables to use 127.0.0.1 and correct admin principal. - Added tags to 'samba_dns' tasks for better target execution. - Updated IDE project configuration for knoe-db.
Co-authored-by: Junie <junie@jetbrains.com>
2026-04-28 00:05:03 -07:00
chrisfu
b421f49af1 Migrate Ansible-based scripts and documentation to 1Password integration. Removed deprecated and redundant Ansible vault workflows, added 1Password CLI handling for secrets management, and updated tests to reflect the change. 2026-04-22 20:54:13 -07:00
chrisfu
47d906cc28 chore: restore vault_db_master.yml for update.sh master-pw rotation
The ansible excision in 5a769f2 was correct in scope -- no more host
inventory, no playbooks, no roles -- except it took out the one file
update.sh depends on for master DB password rotation:
infrastructure/inventory/group_vars/all/vault_db_master.yml

update.sh(1) reads this file (ansible-vault encrypted) at the start of
every run to decrypt the master password used to rotate k8s DB secrets
(knoe-db-user/superuser/knoe), ALTER ROLE on CNPG primary for the
prole/postgres/knoe/authenticator users, and reset Grafana admin creds.
With the vault file gone, update.sh failed with
    ERROR: Vault file not found: .../vault_db_master.yml
and took down the tail of deploy.sh.

Restored from the parent commit of 5a769f2 verbatim -- content is
already ansible-vault AES256 encrypted, so it's safe to carry in a
private repo. The `.vault_pass` file that decrypts it remains
gitignored (never committed). Broader `infrastructure/` stays gone.

TODO (deferred): migrate this one remaining ansible-vault secret to
either OpenBao or a k8s Secret so knoe-db doesn't retain a residual
ansible-vault dependency surface. Tracked in our pending work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 00:48:44 -07:00
chrisfu
5a769f273b chore: excise prole.org ansible infrastructure for knoe-db fork
knoe-db deploys cluster-native (mode k3d for local dev, k8s for the
shared company cluster, k3s still supported for individual homelab
collaborators via conf/k3s.cfg). The ansible-managed distributed cluster
of prole.org machines is no longer the target topology, and the Ansible
inventory in this tree targets only the original author's specific hosts
(merlin/gandalf/myrddin/pi/raspberry/retropie.prole.org). It is neither
portable nor useful for the new deployment shape.

Removed:
- infrastructure/inventory/  (hosts.ini + 7 *.prole.org host_vars)
- infrastructure/playbooks/  (11 playbooks, all hardcoded to *.prole.org)
- infrastructure/roles/      (20 roles: pihole, samba AD DC, mariadb,
                              netplan, iscsi, prole_ssl, etc. -- all
                              tied to the bare-metal prole.org topology)
- infrastructure/setup/      (cr_ansible_user.sh, cr_samba_family_users.sh,
                              filesystem.txt -- prole.org bootstrap)
- infrastructure/deployments/svc-check-helm/
                              (legacy; etc/init_kong.sh already tears
                               down svc-check resources on every run)
- infrastructure/ansible.cfg (references deleted inventory)
- ansible.cfg, ansible.sh, ansible_min.cfg, ansible_recovery.log  (root;
                              orphaned without inventory)
- vault_backup/              (ansible-vault encrypted prole.org secrets
                              backup: ad_dc_vault.yml, all_prole_vault.yml,
                              iscsi_vault.yml)
- tools/                     (dashboard.sh, ghcr_docker_login.sh,
                              k3s-cluster-rename.sh, manage-node.sh,
                              export_mariadb_synology.sh,
                              prepare_mariadb_usb.sh, and a dumped
                              synology.prole.org mariadb .sql file --
                              all prole.org-era deployment helpers)

Kept: scripts/ (cluster ops), bin/ (prole-env.sh, prole-kpf.sh dev
shims), conf/k3s.cfg / conf/k3d.cfg (still supported modes; any
*.prole.org defaults in those files are legitimate mode fallbacks for
collaborators who keep running their own k3s/k3d setups). Test
fixtures referencing *.prole.org hostnames are left as-is (regression
data).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 18:53:03 -07:00
chrisfu
b8a55888be checkpoint: update GKE deployment docs and infra changes
Summarize CLAUDE.md updates:

- Document both live GKE clusters as e2-standard-2 x3 in us-west3 and note stale app default in code.

- Call out DB cluster naming mismatch between conf (knoe-dev-cnpg-0) and code default (knoe-cnpg-0).

- Add SSD quota guidance: 300GB fully consumed by CNPG, so non-CNPG PVCs must use standard/pd-standard.

- Refresh reset script reference and quota section details to match current runtime state.

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-18 07:18:10 -07:00
chrisfu
e3c2e625f1 refactor(config): separate k3d k3s and gke config entrypoints
Rename env config files from conf/*/prole.cfg to conf/k3d.cfg, conf/k3s.cfg, and conf/gke.cfg. Update shell/Python loaders and etc/deploy scripts to resolve named configs cleanly while keeping legacy fallback behavior. Align k3s Ansible tasks, docs, and regression coverage with the new configuration layout.

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-11 22:20:45 -07:00
chrisfu
b4fc5641db checkpoint: sanitize prole config for host portability
- fix conf/prod/prole.cfg by replacing hardcoded /Users/chrisfu paths with /Users/chrisfu

- remove injected [update.sh] log lines and stray password artifacts so config is executable

- include latest pending updates across deployment config, UI/core flow, vault/network artifacts, and helper scripts
2026-04-10 05:58:40 -07:00
chrisfu
c94c62e1bb Harden prod deploy namespace/context routing and vault password handling
- persist and load DB master password via Ansible Vault bootstrap flow

- enforce knoe-system service namespace and explicit app/db kubecontext targeting

- improve OpenBao/CNPG deploy reliability and logging; add retries/readiness diagnostics

- tighten reset/delete cluster behavior and expand installer/deploy pipeline test coverage

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-10 00:43:07 -07:00
chrisfu
5529fdea77 feat: full GKE/prod deployment pipeline from UI to Artifact Registry
## GCP / Cluster Environment Screen
- Auto-populate Cloud tab from conf/prod/gcp.cfg on screen open (org_id,
  billing_account, billing_project, project_id)
- gcloud auth validity checked on screen startup; friendly modal dialog
  streams gcloud auth login output live so user never leaves the app
- Live GKE cluster browser: fetches clusters via gcloud container clusters
  list, displays with checkmark selector, auto-selects saved cluster
- Selecting a cluster runs get-credentials, sets KUBECONFIG/KUBECONTEXT,
  and syncs the region dropdown to the selected cluster's location
- Region dropdown populated live from gcloud compute regions list with
  checkmark on currently selected region; graceful fallback when offline
- New 'GCP Storage' tab with workload->StorageClass mapping (CNPG->premium-rwo,
  Redis/Monitoring->standard-rwo, Garage->garage-hdd) and Fetch from Cluster
- Provider readonly field styled correctly (no solid-black on macOS)
- Stale prole.cfg/conf/prole.cfg symlinks removed; all config I/O now
  resolves env-specific paths via prole_conf.entrypoint_path()

## GKE Autopilot Compatibility (Common Services)
- Synology iSCSI StorageClass and static PVs guarded behind PROLE_MODE!=k8s
  in init_openbao.sh (GKE Autopilot forbids hostPath/iSCSI volumes)
- In-cluster Docker registry (hostPath) skipped in k8s mode; GCP Artifact
  Registry used instead
- Kong renamed knoe-svc-kong in k8s mode; all health-check kubectl calls in
  init_common_services.sh and status_common_services.sh updated accordingly
- DNS endpoints switched from *.prole.org to *.knoe.dev in k8s mode
  (api.knoe.dev, git.knoe.dev, svc.knoe.dev); ingress uses gce class
- New GKE-clean Kong manifests under deploy/opentofu/k8s/manifests/prole/:
  no k3s node affinity, explicit Autopilot resource requests/limits

## Garage S3 Store (GKE)
- New garage-statefulset-gcp.yaml targeting garage-hdd StorageClass
  (pd-standard, avoids SSD_TOTAL_GB quota exhaustion in us-west3)
- New storageclass-gcp-hdd.yaml (pd-standard, Retain, WaitForFirstConsumer)
- GCP StorageClass manifests skipped on re-runs (Autopilot built-ins are
  immutable; skip-if-exists guard added)
- PVC deletion guard extended to cover any storageClass (not just synology)
  so stale claims are cleaned before StatefulSet recreation

## Topology (GKE Autopilot)
- DaemonSet collector skipped in prod mode (forbidden in kube-system by
  GKE Warden); Kubernetes-only node facts path used instead
- All ready GKE nodes assumed cnpg-eligible and monitoring-eligible without
  taint/synology-mount checks (skip_collector + assume_nodes_eligible flags)

## KUBECONFIG / kubectl (k8s mode)
- actions.py: new elif mode==k8s branch sets KUBECONFIG=~/.kube/config
  and injects KUBECONTEXT from prole_cfg_data into script env
- _build_kubectl_cmd falls back to Global.KUBECONTEXT when
  init_cluster.selected_kubectx is empty
- _activate_selected_gke_cluster persists KUBECONFIG/KUBECONTEXT to
  prole_cfg_data and saves prole.cfg immediately after get-credentials

## Database Build Screen (GKE)
- Registry display shows correct Artifact Registry URL
  (<region>-docker.pkg.dev/<project>/<namespace>/knoe-db) in green
- Build+push: gcloud auth configure-docker, auto-creates AR repository
  named after SERVICE_NAMESPACE (e.g. knoe-system) if missing, then
  docker tag + push; falls back to gcr.io if region unavailable
- GCP config loaded from conf/prod/gcp.cfg on every screen entry;
  keys normalised to lowercase so project_id lookup is always consistent

## Config / Namespace persistence
- prole_conf.py activate_environment: symlink creation removed; sets
  CLUSTER_ENV env-var so all subsequent calls resolve correct env directory
- knoe/ui/screens/__init__.py: startup config load uses entrypoint_path()
  instead of hardcoded conf/prole.cfg; seeds SERVICE_NAMESPACE=knoe-system
  for managed envs so Common Services never defaults to 'default'
- cfg.py _save_prole_cfg: saves to env-specific path via entrypoint_path()
- etc/prole_cfg.sh: removed all ln -snf symlink creation

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-04 12:38:29 -07:00
chrisfu
eb9430df9d fix(gitlab,infra): ARM64 RPi service cluster – GitLab deploy in gitlab ns on gandalf
Namespace & routing
- milestones.py: GitOpsMilestone now resolves namespace from
  gitops.gitlab_namespace (new) → Global.GITLAB_NAMESPACE → 'gitlab'
  hardcoded; never falls through to gitops.namespace (was 'gitea')
- prole.cfg: add gitops.gitlab_namespace=gitlab + GITLAB_NAMESPACE=gitlab
- init_gitlab.sh: NAMESPACE defaults to gitlab, NODE_SELECTOR blanked so
  only gitaly+minio are node-pinned; GITOPS_NAMESPACE fallback removed

GitLab on ARM64 RPi (16 KB kernel pages)
- init_gitlab.sh: DaemonSet compiles jemalloc-5.3.0 with --with-lg-page=14
  (glibc/Ubuntu) on every node; LD_PRELOAD injected per Ruby component
- Minio: quay.io 2022 image (ARM64); configure init container replaced
  with ARM64 alpine that writes credential files; MINIO_ROOT_USER/PASSWORD
  injected directly into main container env via secretKeyRef
- Minio buckets auto-created post-deploy (registry, lfs, artifacts, etc.)
- webservice/sidekiq: replicaCount=1, reduced memory (1500M/800M),
  liveness probe initialDelaySeconds=3600 (Rails loads 25-40min on RPi)
- allowedHosts set as flat string list (chart 9.x default is list-of-maps
  which breaks URI initializer in 7_gitlab_http.rb)
- gitaly+minio always pinned to gandalf (local PV); other workloads spread

Storage
- Static PVs created for gitaly (50Gi) + minio (10Gi) on synology d005
- Synology dirs created before PVs; bucket creation idempotent

Redis (shared for GitLab KAS)
- init_redis.sh: persistence disabled (no dynamic provisioner); Redis used
  as pub/sub broker only

Infrastructure / pi.prole.org
- Removed pi.prole.org from [k3s_agents] – dedicated pihole node, OOM
- host_vars: k3s_enabled=false, k3s_state=absent (storage preserved)
- New playbook: infrastructure/playbooks/disable_pi_k3s.yml (drain + disable)
- monitoring.py: node-exporter DaemonSet excludes pi.prole.org
- init_monitoring.sh: pi.prole.org excluded from node-exporter affinity
- kong-deployment.yaml: affinity rule prevents scheduling on pi (pihole owns 80/443)

Co-authored-by: Junie <junie@jetbrains.com>
2026-04-02 20:15:11 -07:00
chrisfu
fdc1582bd0 Fix DB image build and k3s registry/CNPG robustness
- update Percona Dockerfiles for compatible extension/tooling install flow\n- add HTTP/HTTPS-aware k3s registry configuration path across scripts/Ansible\n- harden CNPG TLS bootstrap CN handling for long namespaces and add regression test\n- improve namespace reset pod-deletion wait behavior

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-29 19:35:37 -07:00
chrisfu
d02aadae89 Add node management workflow and installer config persistence updates
- add tools/manage-node.sh for k3s node add/remove automation with inventory reconciliation

- include kubectl drain flag compatibility and --ssh-host targeting for uninstall workflows

- persist installer cfg/env values and update deploy/database/cfg UI handling

- refresh related inventory/network artifacts and expand installer/database test coverage

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-29 11:22:33 -07:00
chrisfu
7b242e88f5 Persist k3s context rename and scrub service config paths
Update conf/service/prole.cfg to persist the k3s kube context as knoe.dev.prole.org and replace host-specific absolute paths with $HOME-based paths for portability.

Include all pending project changes: local_user role package and screenrc provisioning updates, plus tools/k3s-cluster-rename.sh and tests/etc/test_k3s_cluster_rename.sh.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-28 11:23:19 -07:00
chrisfu
fb7993de1d feat: add production config schema, validation, and API with tests
- Introduced `knoe.core.prod_config` with production configuration schema and helper methods.
- Added validation logic to enforce required fields and expected formats.
- Implemented in-process API (`ProdConfigApi`) for managing production configs.
- Added test coverage for default config behavior, validation, YAML generation, and API workflow.
- Updated cluster UI layout and tests to integrate production config with new navigation flows.
- Added host inventory and updated service flow to reflect production setup changes.
2026-03-28 10:43:25 -07:00
chrisfu
65d03ed442 stable: slimmed Supabase + CNPG 3-node healthy build (2026-03-25)
STABLE BUILD — all 35+ pods Running, zero non-Running pods, CNPG ready=3.

## Architecture changes (Supabase slim-down)
- Removed local Supabase Kong, Storage, MinIO, Imgproxy from deployed stack
- Rewired API routing to shared Kong in kube-system
- Rewired object storage to shared Garage in knoe-system
- Supabase footprint reduced to control-plane only: auth, rest, meta,
  analytics, realtime (studio port-forward retained)
- Supabase pods renamed: stripped redundant '-prole-supabase-' prefix via
  per-component fullnameOverride (e.g. supabase-auth, supabase-rest, ...)
- Supabase workloads pinned to merlin.prole.org (memory-safe node)

## File changes
- conf/service/prole.cfg
    * Replaced all hardcoded /Users/chrisfu/dev/prole paths with ${HOME}/dev/prole
      so config is portable when run from myrddin.prole.org or any user home
    * SUPABASE_ENABLED=True, SUPABASE_PRIMARY_NODE=merlin.prole.org in
      both [Global] and [Optional Features]
    * supabase_enabled=true in [Inputs]/init_cluster section
- supabase/helm/render_supabase.py
    * Explicitly disabled: kong, storage, minio, imgproxy
    * Active components pinned via nodeSelector to merlin.prole.org
    * fullnameOverride per component to clean up pod names
    * Default primary node changed from pi.prole.org -> merlin.prole.org
- supabase/deploy.sh
    * Removed svc/kong port-forward; API via shared Kong in kube-system
    * Removed imgproxy/storage from PV/PVC prep and helm --set persistence args
    * Fixed kubectl exec missing -i flag for psql heredoc stdin in
      setup_knoe_db_for_supabase
    * Updated Studio access messaging to reference shared platform Kong
- k8s/prole/knoe-db.yaml
    * instances: 3
    * Added control-plane/master tolerations for myrddin scheduling
- deploy/opentofu/k3s/manifests/prole/knoe-db.yaml
    * instances: 3, added control-plane tolerations (mirrors k8s/prole/)
- k8s/prole/iscsi-pvs.yaml
    * Expanded d003-data/d003-wal nodeAffinity to [pi, merlin, myrddin]
      to reflect post-OOM-recovery live state
- k8s/registry/deployment.yaml
    * nodeSelector restored to myrddin.prole.org (hostPort 5000 owner)
    * Added control-plane + master tolerations so pod schedules correctly
- modes/k3s/knoe-db/.version
    * Aligned to 140 (image tag 18-140) matching repo-root version
- etc/init_common_services.sh, etc/init_registry.sh
    * Minor fixes aligned with slimmed service layout and node targeting
- infrastructure/inventory/host_vars/pi.prole.org.yml
    * Updated host vars to reflect post-recovery pi state
- knoe/core/topology.py, knoe/ui/screens/cluster.py,
  knoe/ui/screens/cluster_nodes.py
    * Topology and UI updates for 3-node cluster representation
- scan/network_description.txt
    * Refreshed network scan output post-recovery
- conf/port-mapping.cfg
    * Updated port mapping to reflect removal of local Kong/storage forwards
- scripts/cleanup_stale_rs.sh (new)
    * Utility script to prune orphaned ReplicaSets after rollouts

## Verified stable state
- CNPG: Cluster in healthy state, ready=3
    knoe-db-2 (merlin, primary), knoe-db-3 (pi), knoe-db-5 (myrddin)
- Supabase: 5 pods Running on merlin.prole.org
    supabase-auth, supabase-rest, supabase-meta,
    supabase-analytics, supabase-realtime
- Monitoring: all prometheus/grafana/alertmanager pods Running on merlin
- Registry: Running on myrddin.prole.org (hostPort 5000)
- Zero non-Running / non-Completed pods across all namespaces

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-25 11:56:40 -07:00
chrisfu
a069989315 Rename prole-db to knoe-db, add knoe-auth as cluster-internal KDC
Itemized changes:

1. knoe-auth: New cluster-internal KDC and SSO gateway service
   - Created etc/init_knoe_auth.sh based on init_kdc.sh with knoe-auth naming
   - Namespace defaults to SERVICE_NAMESPACE (knoe-system)
   - ConfigMap: knoe-auth-kdc-config, Secret: knoe-auth-secrets
   - Legacy cleanup removes old auth/dog/authority deployments

2. Orchestration: knoe-auth initializes before CloudNativePG
   - Updated prole.sh to insert init_knoe_auth.sh as step 2 (before CNPG)
   - Renumbered all subsequent initialization steps

3. Kong routing: Updated init_kong.sh to route to knoe-auth in SERVICE_NAMESPACE

4. Comment/reference updates for knoe-auth
   - Updated init_common_services.sh, init_service_layer.sh, init_kerberos.sh

5. prole-db renamed to knoe-db across the entire codebase
   - Renamed prole-db/ directory to knoe-db/
   - Renamed all prole-db Kubernetes manifests (deploy/opentofu, k8s/)
   - Renamed scripts: docker-root-knoe-db.sh, docker-run-knoe-db.sh, test-cnpg-knoe-db.sh
   - Renamed etc/init_prole-db-reset.sh to etc/init_knoe-db-reset.sh
   - Renamed etc/prole-db-passwwd.sh to etc/knoe-db-passwwd.sh
   - Renamed mock_val counterparts accordingly
   - Renamed tests/etc/test_init_prole-db-reset.sh to test_init_knoe-db-reset.sh
   - Renamed docs/prole-db-documentation-mcp-architecture.md to knoe-db variant
   - Renamed modes/k3d/prole-db/ to modes/k3d/knoe-db/
   - Renamed prole-db.iml to knoe-db.iml

6. Configuration updates
   - Updated conf/dev, conf/prod, conf/test, conf/service prole.cfg files
   - Updated conf/port-mapping.cfg
   - Updated etc/prole_cfg.sh and mock_val/prole_cfg.sh
   - Updated service/prole.cfg

7. Kubernetes manifests and deploy configuration
   - Updated deploy/opentofu/k3s ArgoCD application YAMLs
   - Updated kong-configmap.yaml and kustomization.yaml
   - Updated k3s/kong-config.yml and prole-resources.yaml
   - Updated prole-mssql-db deployment YAMLs
   - Updated supabase helm render and deploy scripts

8. Infrastructure and GCP Terraform
   - Updated deploy/gcp/terraform: folders, groups, IAM, service-projects

9. Python/installer code updates
   - Updated knoe/core: actions, build_context, controller, env, milestones
   - Updated knoe/milestone.py
   - Updated knoe/ui/screens: cfg, database, database_options, deploy, docker,
     navigation, security, services, validate
   - Updated knoe.spec, status.py

10. Shell script updates
    - Updated etc/: build_db, init_cloudnative_pg, init_cnpg_backup,
      init_db_manager, init_forgejo, init_gitlab, init_monitoring, init_openbao,
      init_port_forwards, init_postgrest, init_supabase_ports, status
    - Updated mock_val/ counterparts for all above scripts
    - Updated prole-net/init-prole-dns.sh
    - Updated bin/prole-kpf.sh, gitea/deploy.sh, supabase/deploy.sh

11. Test updates
    - Updated tests/etc/: test_init_cloudnative_pg*, test_init_cnpg_backup*,
      test_init_kdc*, test_init_kerberos*, test_init_kong*, test_prole_cfg*
    - Updated tests/installer/: test_actions_helpers, test_cfg_save_kubecontext,
      test_controller, test_core_classes, test_milestones, test_milestones_extended,
      test_namespace_propagation
    - Updated tests/: test_database_options, test_navigation,
      test_render_supabase_hostname, test_docker_build_fix,
      test_all_prole_home_fixes, silent_install_test, final_test

12. Documentation updates
    - Updated docs/: DOCKER-BUILD-FIX, PROLE-CFG-SECRETS, PROLE-HOME-DIRECTORY,
      build-system, patent
    - Updated scan/network_description.txt
    - Updated pom.xml

13. Miscellaneous script updates
    - Updated root-level: _adopt_replica_pvcs, _fix_replica_merlin, _import_pi,
      _patch_cluster, _prebind_pvcs, _rebind_d002, _rebind_d002b, test_resolve
    - Updated scripts/generate_spec.py

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-22 22:16:21 -07:00
chrisfu
2244d6acd6 Refactor iSCSI Synology mounts to /synology/d00x
- Rename host mount namespace from /prole/d00x to /synology/d00x

- Make LUN ownership explicit per host in inventory (myrddin=d001; merlin=d002,d004; pi=d003)

- Add safe migration cleanup for legacy /prole/d00x mounts and stale /etc/fstab entries

- Update k3s/ArgoCD/OpenTofu manifests and PV node affinities to match new mount paths

- Improve iSCSI role check-mode behavior and add quiesce/detach flow in ansible.sh
2026-03-21 10:43:06 -07:00
chrisfu
75d54cdb28 k3s: gate startup on dedicated /var/lib/rancher mount
Co-authored-by: Junie <junie@jetbrains.com>
2026-03-20 01:38:14 -07:00
chrisfu
792329ddd2 Milestone: monitoring deployment + svc firewall persistence
- k3s role: persist svc.prole.org reachability with a systemd oneshot that inserts ACCEPT rules ahead of Tailscale filter rules (configurable ports/chain).

- init_monitoring: enable Grafana datasource sidecar and provision a stable Prometheus datasource (uid 'prometheus') so CloudNativePG dashboards bind correctly.

- tests: assert datasource sidecar config and datasource ConfigMap is applied.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-19 12:19:02 -07:00
chrisfu
b114099e17 Use prole.cfg hostnames for Supabase front-door
- Persist and export supabase_hostname for canonical external Supabase entrypoint (db.prole.org)

- Render Supabase Helm ingress host and public URLs (API_EXTERNAL_URL/GOTRUE_SITE_URL/SUPABASE_PUBLIC_URL) from config

- Align tracked Traefik ingress manifests to db.prole.org

- Generalize service hostname/TLS wiring in k3s automation and refresh Kong/monitoring behavior

- Make optional workload policy checks deterministic when kube context is absent; add render tests
2026-03-19 04:19:38 -07:00
chrisfu
16a3dce584 k3s: enforce node placement for monitoring/CNPG/Supabase
- Pin Prometheus/Alertmanager + Grafana to merlin.prole.org in k3s to avoid PV node-affinity issues

- Taint myrddin.prole.org as control-plane-only and add CNPG toleration

- Prevent init_cloudnative_pg from deploying prole-svc-kong into the DB namespace

- GitOps-manage Kong declarative config via ConfigMap + add placement/init tests
2026-03-18 23:16:06 -07:00
chrisfu
133b719479 Align CNPG bootstrap placement with Ansible policy
Prefer infra-managed manifests for bootstrap; reconcile CNPG instances based on Ready+schedulable labeled db nodes; update manifests to use node-role affinity + anti-affinity; add policy/tests and config touch-ups (incl. prole.cfg).
2026-03-18 22:06:41 -07:00
chrisfu
f051d1d42a iscsi/installer: provision merlin d004; fix etc script runtime
- Add merlin iSCSI config for PROLE-DATA-4 mounted at /prole/d004 (xfs, _netdev,noatime)

- Refine iscsi role login/mount flow (device resolution, mkfs-if-missing, UUID fstab, tags)

- Ensure installer run_script stages lib/shell into PROLE_HOME so etc scripts can source common libs; add regression test

- Add init scripts for Forgejo/GitLab; ignore generated conf/prole.cfg

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-16 21:30:52 -07:00
chrisfu
5dcd3b9581 ansible: treat /var/lib/rancher as local K3s state
Update inventory/roles to keep K3s state on local storage and prevent iSCSI from managing /var/lib/rancher.

Also add a single-host k3s install playbook, a systemd override template, and docs describing the installer ↔ Ansible boundary and slow-storage knobs.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-15 23:20:55 -07:00
chrisfu
2c6df07933 Harden monitoring init; improve k3s reset cleanup
- Add diagnostics/reset/retry logic for stuck monitoring Helm installs\n- Add optional cluster-side node/lease deletion during k3s reset via delegated control-plane\n- Bump prole-db image tag/version to 18-123\n- Update monitoring init unit test for reset/diagnostics path
2026-03-15 16:02:15 -07:00
chrisfu
56806984bc k3s: stabilize reset/sync and refresh controller kubeconfig
- k3s_sync: choose any server as sync source fallback; refresh kubeconfig and validate nodes\n- fetch_kubeconfig: save to prole-k3s.kubeconfig and etc/secrets/k3s.kubeconfig\n- k3s sync: sync server URL into config.yaml; restore TLS bundle only on servers\n- cert-manager/ACME: wait for CNI/webhook readiness; retry ClusterIssuer/Certificate apply\n- inventory: tune myrddin control-plane timing args; add render test\n- misc: update port mappings and prole-db manifests/version

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-15 12:24:35 -07:00
chrisfu
909e92109e Refactor shell layout; add shellspec + authority
- Introduce lib/shell helpers and keep etc/* scripts thin via compatibility shims

- Move Kerberos validation to scripts/validation/check_kerberos.sh and update callers

- Add deterministic shellspec unit tests under tests/shellspec/ and wire Maven to run them

- Add minimal Spring Boot authority module with startup + /health endpoint and Maven wiring

- Document the new layout in docs/layout.md

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-13 09:19:56 -07:00
chrisfu
502f039c0b Check for prole.cfg existence and conditionally load K3s registry defaults
- Add `ansible.builtin.stat` task to verify `prole.cfg` availability before loading defaults.
- Prevent errors when `prole.cfg` is missing by skipping `lookup` operations.
2026-03-13 02:42:10 -07:00
chrisfu
bd54755218 remove owner/group/mode enforcement for external mountpoint in Ansible tasks
- Update documentation to note `/external` may already be mounted before Ansible runs.
- Avoid redundant ownership and permissions enforcement in `mariadb_primary` role.
2026-03-13 02:15:05 -07:00
chrisfu
17cb827be5 test(dashboard): add unit tests for systemd unit and script validation
- Ensure dashboard service doesn't create an ordering cycle with `multi-user.target`.
- Verify service safely attaches to a virtual terminal (VT) and passes VT to Xorg.
- Ensure no `su` usage by default in the service.
- Validate script behavior for passing VT and `-keeptty` options to Xorg.

chore(dashboard): update systemd unit to avoid VT conflicts and improve logging

- Remove `After` and `Wants` directives for `multi-user.target` to prevent conflicts.
- Add `TTYPath`, `StandardInput`, and related options for VT attachment.
- Ensure `ExecStopPost` restores getty service for the dashboard VT.

fix(dashboard): improve Xorg resolution and clean up script logic

- Dynamically determine Xorg executable location in `dashboard.sh`.
- Pass VT, display, and `-keeptty` to Xorg to improve script reliability.

fix(mariadb): correct exFAT mount options for external storage

- Avoid `chown` failures by setting root ownership with group access for `mysql`.
- Update external mount options to `uid=0,gid=mysql,umask=0002`.

cleanup(mariadb): remove unnecessary owner/group permissions from migration marker task
2026-03-13 01:49:57 -07:00