[INFO] Using Ansible Vault password file: /Users/chrisfu/dev/prole/.vault_pass [CONFIG] Using conf/prole.cfg ==> Dependencies [OK] Homebrew Homebrew 5.0.14 [OK] python Python 3.14.2 [OK] ansible ansible [core 2.20.2] [OK] kubectl Client Version: v1.35.0 [OK] kubectl-cnpg Build: {Version:1.28.0 Commit:a9696201f760013182c6cdba7c4ed3c236a6423b Date:2025-12-09T17:09:28Z} [OK] Docker Docker version 29.2.0, build 0b9d1985db [OK] k3d k3d version v5.8.3 [OK] Network scan already performed. KDC detected: 10.0.0.3 ==> Environment setup [OK] /Users/chrisfu/.prole/env.sh is up to date. ==> Database creation [RESET] OpenBao anchor detected without token; generated new DB password for initialization. [OK] /Users/chrisfu/.prole/env.sh is up to date. [SKIP] Local registry pre-pull only supported for k3d. ==> Environment Preparation (Secure Access) [SKIP] Secure access keys already exist at /Users/chrisfu/.ssh/id_prole_ed25519 [INFO] Deferring OpenBao initialization until cluster is ready. ==> Cluster setup Using local kubeconfig: /Users/chrisfu/dev/prole/etc/secrets/k3s.kubeconfig [RESET] Resetting k3s cluster (mode k3s)... [RESET] Removed local kubeconfig /Users/chrisfu/dev/prole/etc/secrets/k3s.kubeconfig Running: ./ansible.sh -p infrastructure/playbooks/k3s_delete.yml -v /Users/chrisfu/dev/prole/.vault_pass ANSIBLE_CONFIG=/Users/chrisfu/dev/prole/ansible.cfg LOGFILE=/opt/prole/logs/chrisfu/ansible/ansible-20260216-055810.log CMD: ansible-playbook infrastructure/playbooks/k3s_delete.yml --vault-password-file /Users/chrisfu/dev/prole/.vault_pass PLAY [Delete k3s nodes and remove cluster token from vault] ******************** TASK [Gathering Facts] ********************************************************* ok: [myrddin.prole.org] ok: [retropie.prole.org] ok: [pi.prole.org] TASK [k3s : Best-effort purge kube-system workloads before uninstall] ********** ok: [retropie.prole.org] ok: [pi.prole.org] ok: [myrddin.prole.org] TASK [k3s : Best-effort purge kube-system namespaces] ************************** ok: [myrddin.prole.org] ok: [retropie.prole.org] ok: [pi.prole.org] TASK [k3s : Force-remove kube-system finalizers if namespace is stuck] ********* ok: [myrddin.prole.org] ok: [retropie.prole.org] ok: [pi.prole.org] TASK [k3s : Stop k3s services if present] ************************************** changed: [myrddin.prole.org] => (item=k3s) ok: [retropie.prole.org] => (item=k3s) ok: [myrddin.prole.org] => (item=k3s-agent) ok: [pi.prole.org] => (item=k3s) changed: [retropie.prole.org] => (item=k3s-agent) changed: [pi.prole.org] => (item=k3s-agent) TASK [k3s : Check for k3s uninstall scripts] *********************************** ok: [myrddin.prole.org] => (item=/usr/local/bin/k3s-uninstall.sh) ok: [myrddin.prole.org] => (item=/usr/local/bin/k3s-agent-uninstall.sh) ok: [retropie.prole.org] => (item=/usr/local/bin/k3s-uninstall.sh) ok: [pi.prole.org] => (item=/usr/local/bin/k3s-uninstall.sh) ok: [retropie.prole.org] => (item=/usr/local/bin/k3s-agent-uninstall.sh) ok: [pi.prole.org] => (item=/usr/local/bin/k3s-agent-uninstall.sh) TASK [k3s : Run k3s uninstall scripts if present] ****************************** skipping: [pi.prole.org] => (item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/usr/local/bin/k3s-uninstall.sh', 'follow': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'get_selinux_context': False, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/usr/local/bin/k3s-uninstall.sh', 'ansible_loop_var': 'item'}) skipping: [retropie.prole.org] => (item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/usr/local/bin/k3s-uninstall.sh', 'follow': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'get_selinux_context': False, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/usr/local/bin/k3s-uninstall.sh', 'ansible_loop_var': 'item'}) changed: [myrddin.prole.org] => (item={'changed': False, 'stat': {'exists': True, 'path': '/usr/local/bin/k3s-uninstall.sh', 'mode': '0755', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 0, 'gid': 0, 'size': 2011, 'inode': 305262, 'dev': 45826, 'nlink': 1, 'atime': 1771248194.1921544, 'mtime': 1771248194.1921544, 'ctime': 1771248194.1961544, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': True, 'xgrp': True, 'woth': False, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'root', 'gr_name': 'root', 'checksum': 'c9554235f592bf87e98ffcd75871b6d749ea2729', 'mimetype': 'text/x-shellscript', 'charset': 'us-ascii', 'version': '3113320857', 'attributes': ['extents'], 'attr_flags': 'e'}, 'invocation': {'module_args': {'path': '/usr/local/bin/k3s-uninstall.sh', 'follow': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'get_selinux_context': False, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/usr/local/bin/k3s-uninstall.sh', 'ansible_loop_var': 'item'}) skipping: [myrddin.prole.org] => (item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/usr/local/bin/k3s-agent-uninstall.sh', 'follow': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'get_selinux_context': False, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/usr/local/bin/k3s-agent-uninstall.sh', 'ansible_loop_var': 'item'}) changed: [retropie.prole.org] => (item={'changed': False, 'stat': {'exists': True, 'path': '/usr/local/bin/k3s-agent-uninstall.sh', 'mode': '0755', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 0, 'gid': 0, 'size': 2047, 'inode': 4118, 'dev': 45826, 'nlink': 1, 'atime': 1771248199.8591769, 'mtime': 1771248199.8591769, 'ctime': 1771248199.8691769, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': True, 'xgrp': True, 'woth': False, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'root', 'gr_name': 'root', 'checksum': 'f3e6ce8bc198982270efc0a73bd0464113fafab7', 'mimetype': 'text/x-shellscript', 'charset': 'us-ascii', 'version': '2098092759', 'attributes': ['extents'], 'attr_flags': 'e'}, 'invocation': {'module_args': {'path': '/usr/local/bin/k3s-agent-uninstall.sh', 'follow': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'get_selinux_context': False, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/usr/local/bin/k3s-agent-uninstall.sh', 'ansible_loop_var': 'item'}) changed: [pi.prole.org] => (item={'changed': False, 'stat': {'exists': True, 'path': '/usr/local/bin/k3s-agent-uninstall.sh', 'mode': '0755', 'isdir': False, 'ischr': False, 'isblk': False, 'isreg': True, 'isfifo': False, 'islnk': False, 'issock': False, 'uid': 0, 'gid': 0, 'size': 2047, 'inode': 17206, 'dev': 45826, 'nlink': 1, 'atime': 1771248200.1626952, 'mtime': 1771248200.1626952, 'ctime': 1771248200.170695, 'wusr': True, 'rusr': True, 'xusr': True, 'wgrp': False, 'rgrp': True, 'xgrp': True, 'woth': False, 'roth': True, 'xoth': True, 'isuid': False, 'isgid': False, 'blocks': 8, 'block_size': 4096, 'device_type': 0, 'readable': True, 'writeable': True, 'executable': True, 'pw_name': 'root', 'gr_name': 'root', 'checksum': 'f3e6ce8bc198982270efc0a73bd0464113fafab7', 'mimetype': 'text/x-shellscript', 'charset': 'us-ascii', 'version': '181296699', 'attributes': ['extents'], 'attr_flags': 'e'}, 'invocation': {'module_args': {'path': '/usr/local/bin/k3s-agent-uninstall.sh', 'follow': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'get_selinux_context': False, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/usr/local/bin/k3s-agent-uninstall.sh', 'ansible_loop_var': 'item'}) TASK [k3s : Remove k3s systemd units and directories] ************************** changed: [myrddin.prole.org] => (item=/etc/rancher) ok: [myrddin.prole.org] => (item=/etc/systemd/system/k3s.service) changed: [retropie.prole.org] => (item=/etc/rancher) ok: [myrddin.prole.org] => (item=/etc/systemd/system/k3s-agent.service) ok: [myrddin.prole.org] => (item=/etc/systemd/system/k3s.service.d) changed: [pi.prole.org] => (item=/etc/rancher) ok: [retropie.prole.org] => (item=/etc/systemd/system/k3s.service) ok: [myrddin.prole.org] => (item=/etc/systemd/system/k3s-agent.service.d) ok: [myrddin.prole.org] => (item=/etc/rancher/node) ok: [myrddin.prole.org] => (item=/etc/rancher/k3s) ok: [retropie.prole.org] => (item=/etc/systemd/system/k3s-agent.service) ok: [myrddin.prole.org] => (item=/etc/cni) ok: [pi.prole.org] => (item=/etc/systemd/system/k3s.service) ok: [myrddin.prole.org] => (item=/opt/cni) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s) ok: [retropie.prole.org] => (item=/etc/systemd/system/k3s.service.d) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/server/tls) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/server/cred) ok: [retropie.prole.org] => (item=/etc/systemd/system/k3s-agent.service.d) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/server/token) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/server/node-token) ok: [pi.prole.org] => (item=/etc/systemd/system/k3s-agent.service) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/server/agent-token) ok: [retropie.prole.org] => (item=/etc/rancher/node) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/server/db) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/agent) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/data) ok: [retropie.prole.org] => (item=/etc/rancher/k3s) ok: [myrddin.prole.org] => (item=/var/lib/kubelet) ok: [pi.prole.org] => (item=/etc/systemd/system/k3s.service.d) ok: [myrddin.prole.org] => (item=/var/lib/etcd) ok: [myrddin.prole.org] => (item=/var/lib/cni) ok: [retropie.prole.org] => (item=/etc/cni) ok: [myrddin.prole.org] => (item=/var/lib/containerd) ok: [myrddin.prole.org] => (item=/run/k3s) ok: [pi.prole.org] => (item=/etc/systemd/system/k3s-agent.service.d) ok: [retropie.prole.org] => (item=/opt/cni) ok: [myrddin.prole.org] => (item=/run/flannel) ok: [myrddin.prole.org] => (item=/var/log/k3s) ok: [myrddin.prole.org] => (item=/usr/local/bin/k3s) ok: [retropie.prole.org] => (item=/var/lib/rancher/k3s) ok: [myrddin.prole.org] => (item=/usr/local/bin/k3s-killall.sh) ok: [myrddin.prole.org] => (item=/usr/local/bin/k3s-uninstall.sh) ok: [pi.prole.org] => (item=/etc/rancher/node) ok: [myrddin.prole.org] => (item=/usr/local/bin/k3s-agent-uninstall.sh) ok: [retropie.prole.org] => (item=/var/lib/rancher/k3s/server/tls) ok: [myrddin.prole.org] => (item=/usr/local/bin/kubectl) ok: [myrddin.prole.org] => (item=/usr/local/bin/crictl) ok: [myrddin.prole.org] => (item=/usr/local/bin/ctr) ok: [retropie.prole.org] => (item=/var/lib/rancher/k3s/server/cred) ok: [pi.prole.org] => (item=/etc/rancher/k3s) ok: [retropie.prole.org] => (item=/var/lib/rancher/k3s/server/token) ok: [retropie.prole.org] => (item=/var/lib/rancher/k3s/server/node-token) ok: [pi.prole.org] => (item=/etc/cni) ok: [retropie.prole.org] => (item=/var/lib/rancher/k3s/server/agent-token) ok: [pi.prole.org] => (item=/opt/cni) ok: [retropie.prole.org] => (item=/var/lib/rancher/k3s/server/db) ok: [retropie.prole.org] => (item=/var/lib/rancher/k3s/agent) ok: [pi.prole.org] => (item=/var/lib/rancher/k3s) ok: [retropie.prole.org] => (item=/var/lib/rancher/k3s/data) ok: [retropie.prole.org] => (item=/var/lib/kubelet) ok: [pi.prole.org] => (item=/var/lib/rancher/k3s/server/tls) ok: [retropie.prole.org] => (item=/var/lib/etcd) ok: [retropie.prole.org] => (item=/var/lib/cni) ok: [pi.prole.org] => (item=/var/lib/rancher/k3s/server/cred) ok: [retropie.prole.org] => (item=/var/lib/containerd) ok: [pi.prole.org] => (item=/var/lib/rancher/k3s/server/token) ok: [retropie.prole.org] => (item=/run/k3s) ok: [retropie.prole.org] => (item=/run/flannel) ok: [pi.prole.org] => (item=/var/lib/rancher/k3s/server/node-token) ok: [retropie.prole.org] => (item=/var/log/k3s) ok: [retropie.prole.org] => (item=/usr/local/bin/k3s) ok: [pi.prole.org] => (item=/var/lib/rancher/k3s/server/agent-token) ok: [retropie.prole.org] => (item=/usr/local/bin/k3s-killall.sh) ok: [retropie.prole.org] => (item=/usr/local/bin/k3s-uninstall.sh) ok: [pi.prole.org] => (item=/var/lib/rancher/k3s/server/db) ok: [retropie.prole.org] => (item=/usr/local/bin/k3s-agent-uninstall.sh) ok: [retropie.prole.org] => (item=/usr/local/bin/kubectl) ok: [pi.prole.org] => (item=/var/lib/rancher/k3s/agent) ok: [retropie.prole.org] => (item=/usr/local/bin/crictl) ok: [retropie.prole.org] => (item=/usr/local/bin/ctr) ok: [pi.prole.org] => (item=/var/lib/rancher/k3s/data) ok: [pi.prole.org] => (item=/var/lib/kubelet) ok: [pi.prole.org] => (item=/var/lib/etcd) ok: [pi.prole.org] => (item=/var/lib/cni) ok: [pi.prole.org] => (item=/var/lib/containerd) ok: [pi.prole.org] => (item=/run/k3s) ok: [pi.prole.org] => (item=/run/flannel) ok: [pi.prole.org] => (item=/var/log/k3s) ok: [pi.prole.org] => (item=/usr/local/bin/k3s) ok: [pi.prole.org] => (item=/usr/local/bin/k3s-killall.sh) ok: [pi.prole.org] => (item=/usr/local/bin/k3s-uninstall.sh) ok: [pi.prole.org] => (item=/usr/local/bin/k3s-agent-uninstall.sh) ok: [pi.prole.org] => (item=/usr/local/bin/kubectl) ok: [pi.prole.org] => (item=/usr/local/bin/crictl) ok: [pi.prole.org] => (item=/usr/local/bin/ctr) TASK [k3s : Reload systemd after k3s cleanup] ********************************** ok: [myrddin.prole.org] ok: [retropie.prole.org] ok: [pi.prole.org] PLAY [Remove k3s token from Ansible vault] ************************************* TASK [Require vault password file for token deletion] ************************** ok: [localhost] => { "changed": false, "msg": "All assertions passed" } TASK [Ensure vault file exists] ************************************************ ok: [localhost] TASK [Fail when vault file is missing] ***************************************** skipping: [localhost] TASK [Read vault_k3s.yml header] *********************************************** ok: [localhost] TASK [Mark vault_k3s.yml encryption state] ************************************* ok: [localhost] TASK [Decrypt vault_k3s.yml] *************************************************** skipping: [localhost] TASK [Remove vault_k3s_token entry] ******************************************** changed: [localhost] TASK [Encrypt vault_k3s.yml] *************************************************** skipping: [localhost] PLAY RECAP ********************************************************************* localhost : ok=5 changed=1 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 myrddin.prole.org : ok=9 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 pi.prole.org : ok=9 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 retropie.prole.org : ok=9 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 Running: ./ansible.sh -p infrastructure/playbooks/k3s_reset.yml -l myrddin.prole.org -v /Users/chrisfu/dev/prole/.vault_pass ANSIBLE_CONFIG=/Users/chrisfu/dev/prole/ansible.cfg LOGFILE=/opt/prole/logs/chrisfu/ansible/ansible-20260216-060531.log CMD: ansible-playbook infrastructure/playbooks/k3s_reset.yml --limit myrddin.prole.org --vault-password-file /Users/chrisfu/dev/prole/.vault_pass PLAY [Reset k3s nodes and reinstall with vault token] ************************** TASK [Gathering Facts] ********************************************************* ok: [myrddin.prole.org] TASK [cgroups : Find kernel cmdline file] ************************************** ok: [myrddin.prole.org] => (item=/boot/firmware/cmdline.txt) ok: [myrddin.prole.org] => (item=/boot/cmdline.txt) TASK [cgroups : Select kernel cmdline path] ************************************ ok: [myrddin.prole.org] TASK [cgroups : Select kernel cmdline update targets] ************************** ok: [myrddin.prole.org] TASK [cgroups : Fail when kernel cmdline file is missing] ********************** skipping: [myrddin.prole.org] TASK [cgroups : Remove literal backslash escapes from kernel cmdline files] **** ok: [myrddin.prole.org] => (item=/boot/firmware/cmdline.txt) TASK [cgroups : Remove stray backslashes from kernel cmdline files] ************ ok: [myrddin.prole.org] => (item=/boot/firmware/cmdline.txt) TASK [cgroups : Read kernel cmdline files] ************************************* ok: [myrddin.prole.org] => (item=/boot/firmware/cmdline.txt) TASK [cgroups : Read current /proc/cmdline] ************************************ ok: [myrddin.prole.org] TASK [cgroups : Check current cgroup params in /proc/cmdline] ****************** ok: [myrddin.prole.org] TASK [cgroups : Normalize kernel cmdline content] ****************************** ok: [myrddin.prole.org] => (item={'content': 'cmVib290PXcgY29oZXJlbnRfcG9vbD0xTSA4MjUwLm5yX3VhcnRzPTEgcGNpPXBjaWVfYnVzX3NhZmUgbnVtYV9wb2xpY3k9aW50ZXJsZWF2ZSBudm1lLm1heF9ob3N0X21lbV9zaXplX21iPTAgbnVtYT1mYWtlPTggc3lzdGVtX2hlYXAubWF4X29yZGVyPTAgaW9tbXVfZG1hX251bWFfcG9saWN5PWludGVybGVhdmUgc21zYzk1eHgubWFjYWRkcj0yQzpDRjo2Nzo4QzoyQjo0NyB2Y19tZW0ubWVtX2Jhc2U9MHgzZmMwMDAwMCB2Y19tZW0ubWVtX3NpemU9MHg0MDAwMDAwMCBjb25zb2xlPXR0eUFNQTEwLDExNTIwMCBjb25zb2xlPXR0eTEgcm9vdD1QQVJUVVVJRD1hMzhiMTY1OS0wMiByb290ZnN0eXBlPWV4dDQgZnNjay5yZXBhaXI9eWVzIHJvb3R3YWl0IGNmZzgwMjExLmllZWU4MDIxMV9yZWdkb209VVMgY2dyb3VwX21lbW9yeT0xIGNncm91cF9lbmFibGU9bWVtb3J5IA==', 'source': '/boot/firmware/cmdline.txt', 'encoding': 'base64', 'invocation': {'module_args': {'path': '/boot/firmware/cmdline.txt', 'src': '/boot/firmware/cmdline.txt'}}, 'failed': False, 'changed': False, 'item': '/boot/firmware/cmdline.txt', 'ansible_loop_var': 'item'}) TASK [cgroups : Select kernel cmdline line] ************************************ ok: [myrddin.prole.org] TASK [cgroups : Select cmdline source] ***************************************** ok: [myrddin.prole.org] TASK [cgroups : Sanitize kernel cmdline] *************************************** ok: [myrddin.prole.org] TASK [cgroups : Tokenize kernel cmdline] *************************************** ok: [myrddin.prole.org] TASK [cgroups : Remove conflicting cgroup params] ****************************** ok: [myrddin.prole.org] TASK [cgroups : Calculate missing cgroup params] ******************************* ok: [myrddin.prole.org] TASK [cgroups : Build updated kernel cmdline tokens] *************************** ok: [myrddin.prole.org] TASK [cgroups : Build updated kernel cmdline line] ***************************** ok: [myrddin.prole.org] TASK [cgroups : Sanitize updated kernel cmdline line] ************************** ok: [myrddin.prole.org] TASK [cgroups : Determine if kernel cmdline needs update] ********************** ok: [myrddin.prole.org] TASK [cgroups : Update kernel cmdline when params are missing] ***************** skipping: [myrddin.prole.org] => (item={'path': '/boot/firmware/cmdline.txt', 'raw_line': 'reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe numa_policy=interleave nvme.max_host_mem_size_mb=0 numa=fake=8 system_heap.max_order=0 iommu_dma_numa_policy=interleave smsc95xx.macaddr=2C:CF:67:8C:2B:47 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000 console=ttyAMA10,115200 console=tty1 root=PARTUUID=a38b1659-02 rootfstype=ext4 fsck.repair=yes rootwait cfg80211.ieee80211_regdom=US cgroup_memory=1 cgroup_enable=memory', 'sanitized': 'reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe numa_policy=interleave nvme.max_host_mem_size_mb=0 numa=fake=8 system_heap.max_order=0 iommu_dma_numa_policy=interleave smsc95xx.macaddr=2C:CF:67:8C:2B:47 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000 console=ttyAMA10,115200 console=tty1 root=PARTUUID=a38b1659-02 rootfstype=ext4 fsck.repair=yes rootwait cfg80211.ieee80211_regdom=US cgroup_memory=1 cgroup_enable=memory'}) skipping: [myrddin.prole.org] TASK [cgroups : Remove literal backslash escapes from updated cmdline files] *** skipping: [myrddin.prole.org] => (item=/boot/firmware/cmdline.txt) skipping: [myrddin.prole.org] TASK [cgroups : Normalize updated cmdline whitespace] ************************** skipping: [myrddin.prole.org] => (item=/boot/firmware/cmdline.txt) skipping: [myrddin.prole.org] TASK [cgroups : Determine if kernel cmdline was updated] *********************** ok: [myrddin.prole.org] TASK [cgroups : Determine if reboot is required] ******************************* ok: [myrddin.prole.org] TASK [cgroups : Reboot to apply cgroup settings] ******************************* skipping: [myrddin.prole.org] TASK [cgroups : Verify cgroup kernel parameters after reboot] ****************** ok: [myrddin.prole.org] TASK [cgroups : Read cgroup v2 controllers] ************************************ ok: [myrddin.prole.org] TASK [cgroups : Record cgroup memory controller presence] ********************** ok: [myrddin.prole.org] TASK [cgroups : Fail when cgroup params are missing (reboot required)] ********* skipping: [myrddin.prole.org] TASK [cgroups : Fail when conflicting cgroup params are present] *************** skipping: [myrddin.prole.org] TASK [k3s : Best-effort purge kube-system workloads before uninstall] ********** ok: [myrddin.prole.org] TASK [k3s : Best-effort purge kube-system namespaces] ************************** ok: [myrddin.prole.org] TASK [k3s : Force-remove kube-system finalizers if namespace is stuck] ********* ok: [myrddin.prole.org] TASK [k3s : Stop k3s services if present] ************************************** ok: [myrddin.prole.org] => (item=k3s) ok: [myrddin.prole.org] => (item=k3s-agent) TASK [k3s : Check for k3s uninstall scripts] *********************************** ok: [myrddin.prole.org] => (item=/usr/local/bin/k3s-uninstall.sh) ok: [myrddin.prole.org] => (item=/usr/local/bin/k3s-agent-uninstall.sh) TASK [k3s : Run k3s uninstall scripts if present] ****************************** skipping: [myrddin.prole.org] => (item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/usr/local/bin/k3s-uninstall.sh', 'follow': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'get_selinux_context': False, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/usr/local/bin/k3s-uninstall.sh', 'ansible_loop_var': 'item'}) skipping: [myrddin.prole.org] => (item={'changed': False, 'stat': {'exists': False}, 'invocation': {'module_args': {'path': '/usr/local/bin/k3s-agent-uninstall.sh', 'follow': False, 'get_checksum': True, 'get_mime': True, 'get_attributes': True, 'get_selinux_context': False, 'checksum_algorithm': 'sha1'}}, 'failed': False, 'item': '/usr/local/bin/k3s-agent-uninstall.sh', 'ansible_loop_var': 'item'}) skipping: [myrddin.prole.org] TASK [k3s : Remove k3s systemd units and directories] ************************** ok: [myrddin.prole.org] => (item=/etc/rancher) ok: [myrddin.prole.org] => (item=/etc/systemd/system/k3s.service) ok: [myrddin.prole.org] => (item=/etc/systemd/system/k3s-agent.service) ok: [myrddin.prole.org] => (item=/etc/systemd/system/k3s.service.d) ok: [myrddin.prole.org] => (item=/etc/systemd/system/k3s-agent.service.d) ok: [myrddin.prole.org] => (item=/etc/rancher/node) ok: [myrddin.prole.org] => (item=/etc/rancher/k3s) ok: [myrddin.prole.org] => (item=/etc/cni) ok: [myrddin.prole.org] => (item=/opt/cni) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/server/tls) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/server/cred) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/server/token) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/server/node-token) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/server/agent-token) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/server/db) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/agent) ok: [myrddin.prole.org] => (item=/var/lib/rancher/k3s/data) ok: [myrddin.prole.org] => (item=/var/lib/kubelet) ok: [myrddin.prole.org] => (item=/var/lib/etcd) ok: [myrddin.prole.org] => (item=/var/lib/cni) ok: [myrddin.prole.org] => (item=/var/lib/containerd) ok: [myrddin.prole.org] => (item=/run/k3s) ok: [myrddin.prole.org] => (item=/run/flannel) ok: [myrddin.prole.org] => (item=/var/log/k3s) ok: [myrddin.prole.org] => (item=/usr/local/bin/k3s) ok: [myrddin.prole.org] => (item=/usr/local/bin/k3s-killall.sh) ok: [myrddin.prole.org] => (item=/usr/local/bin/k3s-uninstall.sh) ok: [myrddin.prole.org] => (item=/usr/local/bin/k3s-agent-uninstall.sh) ok: [myrddin.prole.org] => (item=/usr/local/bin/kubectl) ok: [myrddin.prole.org] => (item=/usr/local/bin/crictl) ok: [myrddin.prole.org] => (item=/usr/local/bin/ctr) TASK [k3s : Reload systemd after k3s cleanup] ********************************** ok: [myrddin.prole.org] TASK [k3s : Skip k3s when state is absent] ************************************* skipping: [myrddin.prole.org] TASK [k3s : Set k3s service name] ********************************************** ok: [myrddin.prole.org] TASK [k3s : Check if k3s service is installed] ********************************* ok: [myrddin.prole.org] TASK [k3s : Check if k3s binary exists] **************************************** ok: [myrddin.prole.org] TASK [k3s : Determine if k3s needs installation] ******************************* ok: [myrddin.prole.org] TASK [k3s : Install k3s dependencies] ****************************************** ok: [myrddin.prole.org] TASK [k3s : Preflight - ensure rancher mountpoint is mounted] ****************** ok: [myrddin.prole.org] TASK [k3s : Preflight - ensure cgroup kernel params are present] *************** ok: [myrddin.prole.org] TASK [k3s : Fail if cgroup params are missing (reboot required)] *************** skipping: [myrddin.prole.org] TASK [k3s : Guardrail - ensure rancher mountpoint is not on SD/rootfs] ********* ok: [myrddin.prole.org] TASK [k3s : Fail if rancher mountpoint is on SD] ******************************* skipping: [myrddin.prole.org] TASK [k3s : Preflight - ensure required mountpoints are mounted] *************** ok: [myrddin.prole.org] => (item=/prole/d001) ok: [myrddin.prole.org] => (item=/prole/d002) ok: [myrddin.prole.org] => (item=/prole/d003) TASK [k3s : Guardrail - ensure required mountpoints are not on SD] ************* ok: [myrddin.prole.org] => (item=/prole/d001) ok: [myrddin.prole.org] => (item=/prole/d002) ok: [myrddin.prole.org] => (item=/prole/d003) TASK [k3s : Fail if required mountpoint is on SD] ****************************** skipping: [myrddin.prole.org] => (item={'changed': False, 'stdout': '/dev/sdc1', 'stderr': '', 'rc': 0, 'cmd': ['findmnt', '-n', '-o', 'SOURCE', '/prole/d001'], 'start': '2026-02-16 06:05:58.067260', 'end': '2026-02-16 06:05:59.072024', 'delta': '0:00:01.004764', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'findmnt -n -o SOURCE /prole/d001', '_uses_shell': False, 'expand_argument_vars': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'cmd': None, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['/dev/sdc1'], 'stderr_lines': [], 'failed': False, 'item': '/prole/d001', 'ansible_loop_var': 'item'}) skipping: [myrddin.prole.org] => (item={'changed': False, 'stdout': '/dev/sdd1', 'stderr': '', 'rc': 0, 'cmd': ['findmnt', '-n', '-o', 'SOURCE', '/prole/d002'], 'start': '2026-02-16 06:05:59.414480', 'end': '2026-02-16 06:05:59.418044', 'delta': '0:00:00.003564', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'findmnt -n -o SOURCE /prole/d002', '_uses_shell': False, 'expand_argument_vars': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'cmd': None, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['/dev/sdd1'], 'stderr_lines': [], 'failed': False, 'item': '/prole/d002', 'ansible_loop_var': 'item'}) skipping: [myrddin.prole.org] => (item={'changed': False, 'stdout': '/dev/sde1', 'stderr': '', 'rc': 0, 'cmd': ['findmnt', '-n', '-o', 'SOURCE', '/prole/d003'], 'start': '2026-02-16 06:05:59.730321', 'end': '2026-02-16 06:05:59.734126', 'delta': '0:00:00.003805', 'msg': '', 'invocation': {'module_args': {'_raw_params': 'findmnt -n -o SOURCE /prole/d003', '_uses_shell': False, 'expand_argument_vars': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'cmd': None, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['/dev/sde1'], 'stderr_lines': [], 'failed': False, 'item': '/prole/d003', 'ansible_loop_var': 'item'}) skipping: [myrddin.prole.org] TASK [k3s : Validate join configuration] *************************************** skipping: [myrddin.prole.org] TASK [k3s : Check for existing k3s server node token] ************************** ok: [myrddin.prole.org] TASK [k3s : Read existing k3s server node token] ******************************* skipping: [myrddin.prole.org] TASK [k3s : Fail when vault k3s token is missing for existing cluster] ********* skipping: [myrddin.prole.org] TASK [k3s : Fail on k3s token drift from live cluster] ************************* skipping: [myrddin.prole.org] TASK [k3s : Ensure k3s config directory exists] ******************************** changed: [myrddin.prole.org] TASK [k3s : Load registry defaults from prole.cfg] ***************************** ok: [myrddin.prole.org] TASK [k3s : Resolve k3s registry host] ***************************************** ok: [myrddin.prole.org] TASK [k3s : Normalize k3s registry host] *************************************** ok: [myrddin.prole.org] TASK [k3s : Resolve k3s registry namespace] ************************************ ok: [myrddin.prole.org] TASK [k3s : Validate k3s registry host] **************************************** ok: [myrddin.prole.org] => { "changed": false, "msg": "All assertions passed" } TASK [k3s : Render k3s registries config] ************************************** changed: [myrddin.prole.org] TASK [k3s : Ensure kubeconfig group exists] ************************************ ok: [myrddin.prole.org] TASK [k3s : Ensure kubeconfig users are in group (server)] ********************* ok: [myrddin.prole.org] => (item=pi) TASK [k3s : Ensure rancher directories are group-accessible] ******************* changed: [myrddin.prole.org] => (item=/etc/rancher) changed: [myrddin.prole.org] => (item=/etc/rancher/k3s) ok: [myrddin.prole.org] => (item=/var/lib/rancher) TASK [k3s : Render k3s config] ************************************************* changed: [myrddin.prole.org] TASK [k3s : Build k3s install environment] ************************************* ok: [myrddin.prole.org] TASK [k3s : Install k3s] ******************************************************* changed: [myrddin.prole.org] TASK [k3s : Ensure k3s service state] ****************************************** skipping: [myrddin.prole.org] TASK [k3s : Pre-stage k3s images] ********************************************** changed: [myrddin.prole.org] => (item=docker.io/library/registry:2) changed: [myrddin.prole.org] => (item=docker.io/prom/prometheus:latest) changed: [myrddin.prole.org] => (item=docker.io/prom/alertmanager:latest) changed: [myrddin.prole.org] => (item=quay.io/prometheus/node-exporter:latest) [ERROR]: Task failed: Module failed: non-zero return code Origin: /Users/chrisfu/dev/prole/infrastructure/roles/k3s/tasks/main.yml:238:3 236 when: k3s_service.stat.exists or k3s_bin.stat.exists 237 238 - name: Pre-stage k3s images ^ column 3 failed: [myrddin.prole.org] (item=quay.io/prometheus-operator/prometheus-operator:latest) => {"ansible_loop_var": "item", "changed": true, "cmd": "crictl pull quay.io/prometheus-operator/prometheus-operator:latest", "delta": "0:00:00.465932", "end": "2026-02-16 06:08:50.890945", "item": "quay.io/prometheus-operator/prometheus-operator:latest", "msg": "non-zero return code", "rc": 1, "start": "2026-02-16 06:08:50.425013", "stderr": "E0216 06:08:50.887164 38586 log.go:32] \"PullImage from image service failed\" err=\"rpc error: code = NotFound desc = failed to pull and unpack image \\\"quay.io/prometheus-operator/prometheus-operator:latest\\\": failed to resolve reference \\\"quay.io/prometheus-operator/prometheus-operator:latest\\\": quay.io/prometheus-operator/prometheus-operator:latest: not found\" image=\"quay.io/prometheus-operator/prometheus-operator:latest\"\ntime=\"2026-02-16T06:08:50-08:00\" level=fatal msg=\"pulling image: rpc error: code = NotFound desc = failed to pull and unpack image \\\"quay.io/prometheus-operator/prometheus-operator:latest\\\": failed to resolve reference \\\"quay.io/prometheus-operator/prometheus-operator:latest\\\": quay.io/prometheus-operator/prometheus-operator:latest: not found\"", "stderr_lines": ["E0216 06:08:50.887164 38586 log.go:32] \"PullImage from image service failed\" err=\"rpc error: code = NotFound desc = failed to pull and unpack image \\\"quay.io/prometheus-operator/prometheus-operator:latest\\\": failed to resolve reference \\\"quay.io/prometheus-operator/prometheus-operator:latest\\\": quay.io/prometheus-operator/prometheus-operator:latest: not found\" image=\"quay.io/prometheus-operator/prometheus-operator:latest\"", "time=\"2026-02-16T06:08:50-08:00\" level=fatal msg=\"pulling image: rpc error: code = NotFound desc = failed to pull and unpack image \\\"quay.io/prometheus-operator/prometheus-operator:latest\\\": failed to resolve reference \\\"quay.io/prometheus-operator/prometheus-operator:latest\\\": quay.io/prometheus-operator/prometheus-operator:latest: not found\""], "stdout": "", "stdout_lines": []} changed: [myrddin.prole.org] => (item=ghcr.io/openbao/openbao:latest) ...ignoring TASK [k3s : Ensure prole config directory exists] ****************************** ok: [myrddin.prole.org] TASK [k3s : Sync prole.cfg to k3s hosts] *************************************** ok: [myrddin.prole.org] TASK [k3s : Ensure prole home directory exists] ******************************** ok: [myrddin.prole.org] TASK [k3s : Ensure prole service directory exists] ***************************** ok: [myrddin.prole.org] TASK [k3s : Sync init_port_forwards.sh to k3s hosts] *************************** ok: [myrddin.prole.org] TASK [k3s : Sync prole_cfg.sh to k3s hosts] ************************************ ok: [myrddin.prole.org] TASK [k3s : Select k3s init server for port-forward token] ********************* skipping: [myrddin.prole.org] => (item=pi.prole.org) ok: [myrddin.prole.org -> localhost] => (item=myrddin.prole.org) skipping: [myrddin.prole.org] => (item=retropie.prole.org) TASK [k3s : Require k3s init server for port-forward token] ******************** ok: [myrddin.prole.org -> localhost] => { "changed": false, "msg": "All assertions passed" } TASK [k3s : Ensure prole-installer service account exists] ********************* changed: [myrddin.prole.org] TASK [k3s : Ensure prole-installer token secret exists] ************************ changed: [myrddin.prole.org] TASK [k3s : Read prole-installer token for port forwards] ********************** ok: [myrddin.prole.org] TASK [k3s : Store prole-installer token on controller] ************************* ok: [myrddin.prole.org -> localhost] TASK [k3s : Set prole port-forward token for host] ***************************** ok: [myrddin.prole.org] TASK [k3s : Render kubeconfig for port forwards] ******************************* changed: [myrddin.prole.org] TASK [k3s : Stop and disable prole port-forwards service] ********************** ok: [myrddin.prole.org] TASK [k3s : Remove prole port-forwards systemd unit] *************************** ok: [myrddin.prole.org] RUNNING HANDLER [k3s : Restart k3s] ********************************************