Commit Graph

11 Commits

Author SHA1 Message Date
chrisfu
5dcd3b9581 ansible: treat /var/lib/rancher as local K3s state
Update inventory/roles to keep K3s state on local storage and prevent iSCSI from managing /var/lib/rancher.

Also add a single-host k3s install playbook, a systemd override template, and docs describing the installer ↔ Ansible boundary and slow-storage knobs.

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-15 23:20:55 -07:00
chrisfu
598eae4633 Enable TLS for k3s registry and deploy SSL certs
- Add ssl_hosts group and prole_ssl role to deploy registry cert/key to /etc/ssl/certs/prole

- Configure k8s registry deployment to serve HTTPS using mounted host certs

- Switch k3s/containerd registry mirror config to https:// and prefer HTTPS checks with HTTP fallback

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-10 00:58:08 -07:00
chrisfu
436c6214df checkpoint: k3s agent config + node labels + registry optimizations
- k3s: render server-only config keys only for servers; add regression test; make guardrail test non-sudo

- k3s/site: add post-provision node labeling tasks + playbook tier; extend inventory node label mapping

- registry: default namespace to common-services and skip redundant image pushes by probing registry manifests (installer + init scripts)
2026-03-07 03:14:26 -08:00
chrisfu
a43aed7134 k3s: eliminate localhost registry + dedupe common-core
- Ensure k3s mode uses the k3s registry endpoint and avoid localhost/k3d image prefixes.

- Make ArgoCD repo-server cmp symlink creation idempotent.

- Normalize common-core provisioning to knoe-system and add repair-time dedupe of stray default-namespace installs.

- Add k3s MariaDB datastore/refresh playbooks and regression tests.
2026-03-05 14:31:33 -08:00
chrisfu
60820b2b7c Checkpoint: ArgoCD IO load distribution and K3s configuration updates
ArgoCD IO load distribution: Updated k8s/argocd/install.yaml to use hostPath volumes for application-controller (/prole/d001), redis (/prole/d002), and repo-server (/prole/d003). Updated Ansible configure task to create hostPath directories on myrddin.prole.org.

K3s configuration and testing: Added test playbooks for kubeconfig rewriting and argument validation. Updated K3s install, configure, and image import tasks. Configured required mounts and host-specific variables for myrddin.prole.org.

Miscellaneous: Updated conf/prole.cfg with K3S_TOKEN. Bumped prole-db version and updated recovery manifest templates. Minor UI adjustments in cluster screen.
Co-authored-by: Junie <junie@jetbrains.com>
2026-03-04 12:23:15 -08:00
chrisfu
ec8a4e98af k3s: Fix installation hangs, CA mismatches, and arm64 networking
- Implement proactive CA hash verification and automated repair for mismatches

- Ensure agents prioritize discovered server tokens over stale vault values

- Fix K3s service hangs with explicit stop and killall before reinstall

- Add Retropie/Pi networking fixes (WiFi power save, wlan0 priority)

- Pin pre-staged images to stable, architecture-aware versions (arm64)

- Remove obsolete init-port-forwards and prole.cfg sync tasks

- Update k8s manifests and installer core logic with new tests
2026-02-16 23:30:27 -08:00
chrisfu
23cf4ac33d Configure K3s for retropie and add Traefik Ingress. Updated K3s templates for node-ip/flannel-iface, configured retropie host vars, enabled Traefik, added Ingress resources for Grafana, Prole, Kong, and Supabase, and set Postgres to LoadBalancer. 2026-02-16 04:28:40 -08:00
chrisfu
cad49cc0a6 Summary of recent repairs and infrastructure updates
Repairs and improvements:
- iSCSI: Added cleanup tasks to remove stale mounts and fstab entries. Improved robustness of iSCSI target management and added 'iscsi_absent_mounts' support.
- K3s:
    - Updated service start logic to accept 'activating' state, preventing premature failure during slow startups.
    - Improved service stop logic to safely handle missing or not-found services.
    - Ensured 'prole-installer' ServiceAccount and ClusterRoleBinding exist for K8s administration.
    - Added leader election and etcd tuning arguments (forgiving leases) to config.yaml.j2.
    - Removed deprecated 'prole-port-forwards' systemd service.
- Installer & Scripts:
    - Updated legacy_tk.py to support K3s mode, secret resolution for passwords, and better environment management (including ~/.prole/env.sh for service mode).
    - Updated init_ansible.sh to support PROLE_VAULT_PASS_FILE and ANSIBLE_VAULT_PASSWORD_FILE.
    - Improved directory and kubeconfig path resolution in prole_cfg.sh to support fallback to ~/.prole.
    - Enhanced Grafana password resolution in init_monitoring.sh.
    - Added automatic application of iSCSI StorageClass and PersistentVolumes in init_openbao.sh.
- General: Switched conf/prole.cfg to k3s deployment mode and updated vault_k3s.yml token.

New Ansible Tasks and Playbooks:
- infrastructure/playbooks/iscsi_cleanup.yml: Automates logout and removal of stale iSCSI node records.
- infrastructure/playbooks/prole_logs_migrate.yml: Orchestrates /prole/logs migration to iSCSI storage.
- infrastructure/playbooks/tmp_bao_dir.yml: Ensures host-level storage directories for OpenBao.
- infrastructure/playbooks/tmp_mount.yml: Utility to verify and enforce host-level mounts.
- infrastructure/playbooks/k3s_sync.yml: Added tasks to start K3s after sync and update local kubeconfig on the controller.
- Added 'Unmount stale iSCSI mounts' and 'Remove stale iSCSI fstab entries' to the iscsi role.
- Added 'Ensure prole-installer service account exists' to the k3s role.
2026-02-15 17:51:57 -08:00
chrisfu
96f594fd3c Refactor initialization scripts and add new service components
- Consolidated and split initialization scripts in etc/:
    - Removed init_prole-db.sh and init_authority.sh.
    - Added init_kdc.sh for in-cluster MIT Kerberos KDC (prole-authority).
    - Added init_ollama.sh for Ollama AI service integration.
    - Added init_service_layer.sh for high-level service orchestration.
    - Added init_k3s_registry.sh for private registry management.
- Major updates to install.py:
    - Support for new Ollama and KDC configuration.
    - Improved prole.cfg rendering and namespace handling.
    - Updated unattended install flags.
- Infrastructure and Deployment:
    - Updated K3s Ansible role with private registry support (registries.yaml template).
    - Added prole-authority Dockerfile.
    - Updated OpenBao Kerberos ConfigMap and other K8s manifests.
- Configuration:
    - Updated prole.cfg with new sections for Ollama and Monitoring.
    - Refined environment variable exports in env.sh and prole_cfg.sh.
2026-02-11 13:09:31 -08:00
chrisfu
f47c18fef7 feat(infrastructure): enhance k3s automation and OpenTofu integration
- Infrastructure:
    - Updated k3s Ansible role with mountpoint preflight checks and better permission management.
    - Automated deployment of prole configuration and port-forwarding scripts to cluster hosts.
    - Added systemd service for managing port forwards on k3s nodes.
    - Added prole-installer service account token automation.
- K8s Manifests:
    - Renamed and added Persistent Volumes in iscsi-pvs.yaml (including OpenBao support).
    - Updated StatefulSets for garage and openbao.
    - Migrated prole-db to CloudNativePG-based configuration.
    - Added comprehensive OpenTofu manifests for cluster deployment.
- Configuration:
    - Added cluster-specific configurations (k3d, k3s-hosts).
    - Added PostgreSQL configuration templates.
    - Updated .gitignore to track the conf/ directory.
- Tools:
    - Updated install.py and port-forwarding scripts.
    - Added render_manifest.py for manifest generation.
2026-02-07 22:53:18 -08:00
chrisfu
23624eb692 ansibles roles k3s, local_user 2026-01-28 13:05:28 -08:00