chrisfu/prole
1
0
Fork 0
mirror of https://github.com/dredx/prole.git synced 2026-09-23 10:13:58 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
dba8a2d1dc
prole/knoe/ui/screens/cfg.py
chrisfu dba8a2d1dc feat(installer): TDD stabilization for k3d install path; dual-cluster GKE TUI 
Junie's session targeted the prompt "stabilize ./install.py -c conf/k3d.cfg
using strict TDD" — broad installer-side work, not the k3d-mirror Phase 3
brief I had filed (which she didn't pick up; phase-3 brief stays open). All
750 installer tests pass post-change.

What Junie produced:

  install.py                          (NEW) Top-level CLI entry point. Was
                                            imagined by the prompt but didn't
                                            exist; this commit makes it real.
  knoe/deployment.py                  (NEW) `KnoeDeployment` orchestrator for
                                            the k3s service-mode deploy pipeline.
                                            Wraps Ansible kubeconfig fetch,
                                            opentofu apply, init_*.sh post-apply
                                            scripts, and (optionally) supabase/
                                            deploy.sh.
  knoe/ui/screens/cluster.py          Dual-cluster GKE kubecontext UI: prod env
  knoe/ui/screens/cfg.py              now shows separate "App Cluster:" and
                                       "DB Cluster:" dropdowns instead of a
                                       single "Kubernetes Context:" combo.
                                       New _app_kubectx_combo + _db_kubectx_combo
                                       widgets; new app/db_cluster_kubecontext
                                       tk.StringVars.
  knoe/core/{actions,env,milestones}.py
  knoe/core/ops/storage.py
  knoe/config.py, knoe/knoe_conf.py   Plumbing changes for the dual-cluster
                                       kubecontext flow + storage-class topology
                                       detection cleanup.
  knoe/tools/cleanup_cnpg_storage.py  (NEW) Stand-alone cleanup utility.
  tools/dashboard.sh                  (NEW) Dashboard helper.
  conf/knoe.cfg                       (NEW) Master cfg generated by knoe_conf.
  conf/dev/                           (NEW) Dev-mode cfg directory.
  conf/port-mapping.cfg               Port mapping tweaks for k3d.
  tests/installer/* (8 files)         New + extended tests for the dual-cluster
  tests/test_database_options.py      TUI, kubecontext save flow, storage ops,
                                       topology detection, deploy helpers,
                                       database-options screen.

Issues found in Junie's working state and fixed here:

  1. install.py was a 11-line import shim with no shebang, no `chmod +x`,
     no `if __name__ == '__main__'` block. `./install.py -c conf/k3d.cfg`
     returned `Permission denied` and `python install.py` did nothing.
     Added `#!/usr/bin/env python3`, `chmod +x`, and a __main__ block
     that delegates to `knoe.ui.screens.main()`. `./install.py --help`
     now prints the canonical argparse help.

  2. knoe/deployment.py had FIVE `subprocess.run()` call sites with no
     `timeout=` argument (`_run_script`, `_run_cmd`, the Ansible playbook
     fetch, `tofu init`, `tofu apply`). A hung child process — typical
     failure mode is a script waiting on stdin or a stalled network
     call — would lock up the installer indefinitely. Added timeouts:
       - Ansible kubeconfig fetch: 120s
       - tofu init: 300s
       - tofu apply, _run_script, _run_cmd: bounded by new module
         constant `_MILESTONE_TIMEOUT` (default 1800s = 30 min, override
         via `KNOE_MILESTONE_TIMEOUT_SECONDS` env var).
     `subprocess.TimeoutExpired` is caught explicitly; on timeout the
     run helpers return exit code 124 (conventional timeout code).

  3. `conf/k3d.cfg` was corrupted with MagicMock string-reprs on disk:
        KNOE_CONF = <MagicMock name='Canvas().tk.call().strip()' id='4743999712'>
        argocd.node_selector = <MagicMock name='mock.StringVar().get().strip()' id='...'>
     Likely path: Junie ran `./install.py -c conf/k3d.cfg` interactively
     in a non-Tk environment (or with a partially-mocked widget set) and
     the installer's "save current state" path wrote the mock-objects'
     `__repr__` strings into the cfg file. This commit reverts the cfg
     to its pre-Junie state. **Followup: harden the cfg save path
     against non-string widget values** — track separately.

  4. The corrupted cfg caused the installer to call `os.makedirs()` on
     the mock-string values, producing 10 directories on disk literally
     named `<MagicMock name='Canvas().tk.call().strip()' id='4733210304'>/`
     etc., with 5–86 files of install artifacts inside each. Removed.

The "final step is timing out" the user reported was almost certainly
issue #2 above: install.py walked the milestone pipeline, hit one of
the unbounded subprocess.run calls, and the wrapped command (probably
supabase/deploy.sh, which Junie was reading for context when her
session timed out) hung. With the timeouts in place that path now
exits cleanly with rc=124 instead of locking up.

Verification:
  - pytest tests/installer/ -q                                   750 passed in ~25s
  - python3 -c "import knoe.deployment"                          imports clean
  - ./install.py --help                                          prints argparse help
  - find . -maxdepth 1 -type d -name '<MagicMock*' | wc -l       0
  - head -7 conf/k3d.cfg                                          clean (no MagicMock)

Out of scope for this commit (followups):
  - The cfg save-path that wrote mock-objects-as-strings (issue #3 root cause).
    Reproducer: launch the installer in an env where Tk widget vars are
    `unittest.mock.MagicMock` instances. The cfg save code should refuse to
    serialize non-str values rather than calling `str()` on a MagicMock.
  - The k3d-mirror Phase 3 brief (`docs/plans/junie/k3d-knoe-auth-pod-deploy.md`)
    is still open — Junie picked a different prompt this round.

Co-authored-by: Junie <junie@jetbrains.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 23:51:14 -07:00

1194 lines
44 KiB
Python
Raw Blame History

"""knoe.cfg persistence, input snapshot collection and port-forward management."""
import configparser
import os
import socket
import subprocess
import tkinter as tk
from pathlib import Path
from tkinter import ttk, messagebox, filedialog
import platform
from knoe import knoe_conf
from knoe.config import (
_encrypt_cfg_secret,
_filter_cfg_values_for_persistence,
)
from knoe.core.env import (
DEFAULT_ACTION_FLAGS,
DEFAULT_OLLAMA_PORT,
PROJECT_ROOT,
_bool_str,
_build_required_port_forwards,
_cluster_env_radio_value,
_default_opentofu_pipeline_url,
_deployment_mode_from_env,
_deployment_target_label,
_detect_ansible_topology,
_format_ansible_topology_summary,
_normalize_k3s_token,
_pf_extract_id,
_pf_upsert_mapping,
_read_k3s_cfg,
_render_knoe_cfg,
)
class ConfigMixin:
"""knoe.cfg persistence, input snapshot collection and port-forward management."""
# Namespace typing can trigger many trace events; debounce propagation to avoid
# incremental substring rewrites across unrelated fields.
NAMESPACE_PROPAGATE_DEBOUNCE_MS = 250
def _save_knoe_cfg(self):
"""Generates knoe.cfg; master configuration file containing all values used in install process."""
try:
# Check port availability before saving
# try:
# port_val = int(self.db_host_port.get().strip())
# if not self._is_port_available(port_val):
# self._show_port_error(port_val)
# return
# except ValueError:
# messagebox.showerror("Invalid Port", "Please enter a valid numeric port.")
# return
# Determine conf dir; resolve the env-specific knoe.cfg path (no symlink needed).
_cfg_path_override = getattr(self, "_cfg_path_override", None)
if _cfg_path_override is not None:
cfg_path = _cfg_path_override
if cfg_path.is_dir():
conf_dir = cfg_path
else:
conf_dir = cfg_path.parent
else:
_knoe_conf_env = os.environ.get("KNOE_CONF")
if _knoe_conf_env:
conf_dir = Path(_knoe_conf_env).expanduser()
else:
conf_dir = PROJECT_ROOT / "conf"
os.environ.setdefault("KNOE_CONF", str(conf_dir))
conf_dir.mkdir(parents=True, exist_ok=True)
# Activate the environment (sets CLUSTER_ENV env var).
try:
knoe_conf.activate_environment(conf_dir, self.cluster_env.get())
except Exception:
pass
# Resolve the correct env-specific path only when no override is set.
if _cfg_path_override is None:
cfg_path = knoe_conf.entrypoint_path(conf_dir)
# Identify global candidates
mode = self._deployment_mode()
persistence_mode = mode or self.cluster_env.get()
target_label = _deployment_target_label(self.cluster_env.get())
# New policy: do not persist default filesystem paths into knoe.cfg.
# Persist KNOE_HOME only when it differs from the installer default.
default_home = str(Path(os.path.expanduser("~")) / "dev" / "knoe")
env_knoe_home = (os.environ.get("KNOE_HOME") or "").strip()
try:
env_knoe_home_norm = str(Path(env_knoe_home).expanduser()) if env_knoe_home else ""
except Exception:
env_knoe_home_norm = env_knoe_home
knoe_home_override = (
env_knoe_home_norm
if env_knoe_home_norm and env_knoe_home_norm != default_home
else ""
)
globals_to_save = {
"KNOE_DB_USER": self.db_username.get(),
"DB_PASSWORD": self._secret_cfg_value(
"Global", "DB_PASSWORD", self.db_password.get(), "db", "password"
),
"CLUSTER_ENV": self.cluster_env.get(),
"DEPLOYMENT_MODE": mode,
"DEPLOYMENT_TARGET": target_label,
"DATABASE_NAMESPACE": (self.db_namespace.get() or "").strip(),
"CLUSTER_NAME": (self.cnpg_cluster_name.get() or "").strip(),
"DB_HOST_PORT": (self.db_host_port.get() or "5432").strip(),
"SUPABASE_PV_NODE": (self.supabase_pv_node.get() or "").strip(),
"SUPABASE_PV_BASE": (self.supabase_pv_base_dir.get() or "").strip(),
"SUPABASE_PV_BASE_DIR": (self.supabase_pv_base_dir.get() or "").strip(),
"GITEA_NODE_SELECTOR": (self.gitops_node_selector.get() or "").strip(),
"ARGOCD_NODE_SELECTOR": (self.argocd_node_selector.get() or "").strip(),
"PROLE_OPENTOFU_URL": _default_opentofu_pipeline_url(),
}
if knoe_home_override:
globals_to_save["KNOE_HOME"] = knoe_home_override
# Persist the active kubectl context explicitly so shell helpers and scripts
# do not rely on implicit kubectl state.
try:
ctx = ""
if hasattr(self, "selected_kubectx"):
ctx = (self.selected_kubectx.get() or "").strip()
if ctx:
globals_to_save["KUBECONTEXT"] = ctx
except Exception:
pass
# Merge any other values already present in existing Global without overriding explicit UI values
existing_global = dict(self.knoe_cfg_data.get("Global", {}))
for k, v in existing_global.items():
if k not in globals_to_save or not str(globals_to_save.get(k, "")).strip():
globals_to_save[k] = v
# Ensure we never reintroduce default KNOE_HOME via existing config.
try:
if str(globals_to_save.get("KNOE_HOME", "")).strip() == default_home:
globals_to_save.pop("KNOE_HOME", None)
except Exception:
pass
globals_to_save.setdefault("DOCKER_PRELOAD", "false")
# Re-assert dynamic/derived globals to ensure they are not overwritten by existing values
globals_to_save["DB_PASSWORD"] = self._secret_cfg_value(
"Global", "DB_PASSWORD", self.db_password.get(), "db", "password"
)
globals_to_save["DEPLOYMENT_MODE"] = mode
globals_to_save["DEPLOYMENT_TARGET"] = target_label
if not (globals_to_save.get("CLUSTER_NAME") or "").strip():
globals_to_save["CLUSTER_NAME"] = "knoe-db"
globals_to_save["PROLE_K3S_SERVER"] = (self.k3s_server_url.get() or "").strip()
globals_to_save["PROLE_K3S_TOKEN"] = _encrypt_cfg_secret(
self.k3s_token.get() or ""
)
service_namespace = (self._get_service_namespace() or "").strip()
if service_namespace:
globals_to_save["SERVICE_NAMESPACE"] = service_namespace
globals_to_save.pop("NAMESPACE", None)
explicit_global_keys = {
"KNOE_DB_USER",
"DB_PASSWORD",
"CLUSTER_ENV",
"DEPLOYMENT_MODE",
"DEPLOYMENT_TARGET",
"DATABASE_NAMESPACE",
"CLUSTER_NAME",
"DB_HOST_PORT",
"PROLE_K3S_SERVER",
"PROLE_K3S_TOKEN",
"KUBECONTEXT",
}
if knoe_home_override:
explicit_global_keys.add("KNOE_HOME")
explicit_service_namespace = False
try:
explicit_service_namespace = bool(
(self.service_namespace.get() or "").strip()
)
except Exception:
explicit_service_namespace = False
if not explicit_service_namespace:
explicit_service_namespace = bool(
(existing_global.get("SERVICE_NAMESPACE") or "").strip()
)
if explicit_service_namespace:
explicit_global_keys.add("SERVICE_NAMESPACE")
globals_to_save = _filter_cfg_values_for_persistence(
"Global",
globals_to_save,
mode=persistence_mode,
explicit_keys=explicit_global_keys,
)
self._sync_port_forward_mappings()
# Inputs section (replayable UI inputs)
try:
inputs = self._collect_input_snapshot()
except Exception:
inputs = {}
sections = {
k: self.knoe_cfg_data.get(k, {})
for k in [
"Welcome",
"Dependencies",
"Network",
"System Environment",
"Monitoring",
"Kerberos Authentication",
"Ollama",
"Optional Features",
"Database Creation",
"Initialize Cluster",
"Docker Build",
"Initialization Scripts",
"GCP",
"Deployment",
"Install",
]
}
deployment_section = dict(sections.get("Deployment", {}))
if mode:
deployment_section.setdefault("MODE", mode)
if target_label:
deployment_section.setdefault("TARGET", target_label)
sections["Deployment"] = deployment_section
sections["Dev Cluster (k3d)"] = {
**self.knoe_cfg_data.get("Dev Cluster (k3d)", {}),
"MODE": "k3d",
"CLUSTER_ENV": "dev",
"DISPLAY_NAME": "knoe-dev-cluster",
"KUBECTL_CONTEXT": (self.selected_kubectx.get() or "").strip(),
}
sections["Service Cluster (k3s)"] = {
**self.knoe_cfg_data.get("Service Cluster (k3s)", {}),
"MODE": "k3s",
"CLUSTER_ENV": "knoe-service-cluster",
"DISPLAY_NAME": "knoe-service-cluster",
"K3S_SERVER_URL": (self.k3s_server_url.get() or "").strip(),
"K3S_TOKEN": _encrypt_cfg_secret(self.k3s_token.get() or ""),
"PIPELINE_URL": _default_opentofu_pipeline_url(),
}
sections["Prod Cluster (k8s)"] = {
**self.knoe_cfg_data.get("Prod Cluster (k8s)", {}),
"MODE": "k8s",
"CLUSTER_ENV": "knoe-prod-cluster",
"DISPLAY_NAME": "knoe-prod-cluster",
"ARTIFACTS_DIR": (self.prod_artifacts_path.get() or "").strip(),
"PIPELINE_URL": _default_opentofu_pipeline_url(),
}
sections = self._sanitize_sections_for_cfg(sections)
section_explicit_keys = {
"Initialize Cluster": {"ENVIRONMENT", "K3S_SERVER_URL", "K3S_TOKEN"},
"Service Cluster (k3s)": {"K3S_SERVER_URL", "K3S_TOKEN"},
"Dev Cluster (k3d)": {"KUBECTL_CONTEXT"},
}
sections = {
section_name: _filter_cfg_values_for_persistence(
section_name,
section_values,
mode=persistence_mode,
explicit_keys=section_explicit_keys.get(section_name, set()),
)
for section_name, section_values in sections.items()
}
# Sync to Ansible Knoe Vault
self._save_ansible_knoe_vault(self.db_password.get())
cfg_text = _render_knoe_cfg(inputs, globals_to_save, sections)
cfg_path.write_text(cfg_text)
print(f"[DEBUG] knoe.cfg saved to {cfg_path}")
except Exception as e:
print(f"[ERROR] Failed to save knoe.cfg: {e}")
def _collect_input_snapshot(self) -> dict:
"""Collect all possible user inputs for knoe.cfg replay."""
inputs: dict[str, str] = {}
def _set(key: str, val):
inputs[key] = "" if val is None else str(val)
def _set_bool(key: str, val: bool):
inputs[key] = _bool_str(bool(val))
def _get_var(var, default=""):
try:
return var.get()
except Exception:
return default
def _action(key: str, default: bool):
return bool(self._action_flags.get(key, default))
# Dependencies
_set_bool("dependencies.verify_all", _get_var(self.verify_mode, False))
_set_bool(
"dependencies.auto_install_missing",
_action(
"dependencies.auto_install_missing",
DEFAULT_ACTION_FLAGS.get("dependencies.auto_install_missing", True),
),
)
for dep in self.dependencies:
dep_key = f"dependencies.{dep['id']}.install"
_set_bool(dep_key, _action(dep_key, True))
# Network scan
_set_bool(
"network_scan.run",
_action(
"network_scan.run", DEFAULT_ACTION_FLAGS.get("network_scan.run", True)
),
)
# Environment setup values
env_vals: dict[str, str] = {}
defaults_raw: dict[str, str] = {}
try:
defaults_raw = dict(self._env_setup_defaults_raw())
env_vals.update(defaults_raw)
except Exception:
defaults_raw = {}
try:
env_vals.update(self._env_defaults())
except Exception:
pass
try:
env_vals.update(self._read_existing_env())
except Exception:
pass
for k in (
"KNOE_HOME",
"KNOE_CONF",
"PROLE_DATA",
"PROLE_LOGS",
"KNOE_SERVICE",
):
if os.environ.get(k):
env_vals[k] = os.environ.get(k)
if hasattr(self, "_env_entries"):
for k, ent in self._env_entries.items():
try:
v = ent.get().strip()
if v:
env_vals[k] = v
except Exception:
pass
# New policy: store filesystem paths as resolved literals and only when
# overridden (do not duplicate defaults, and never persist `$...`/`${...}`).
resolved_defaults: dict[str, str] = {}
resolved_now: dict[str, str] = {}
try:
if defaults_raw:
resolved_defaults = self._resolve_env_paths_for_fs(defaults_raw) or {}
resolved_now = self._resolve_env_paths_for_fs({**defaults_raw, **env_vals}) or {}
else:
# Best-effort fallback
resolved_defaults = self._resolve_env_paths_for_fs(self._env_defaults()) or {}
resolved_now = self._resolve_env_paths_for_fs(env_vals) or {}
except Exception:
resolved_defaults = {}
resolved_now = {}
for k in (
"KNOE_HOME",
"KNOE_CONF",
"PROLE_DATA",
"PROLE_LOGS",
"KNOE_SERVICE",
):
rv = (resolved_now.get(k) or "").strip()
dv = (resolved_defaults.get(k) or "").strip()
if not rv:
continue
if "$" in rv:
continue
if dv and rv == dv:
continue
_set(f"env_setup.{k}", rv)
ns_val = (
(str(_get_var(self.db_namespace, "")).strip())
if hasattr(self, "db_namespace")
else ""
)
if not ns_val:
ns_val = env_vals.get("DATABASE_NAMESPACE", "")
if not ns_val:
ns_val = env_vals.get("NAMESPACE", "")
_set("env_setup.DATABASE_NAMESPACE", ns_val)
cluster_name = (
(str(_get_var(self.cnpg_cluster_name, "")).strip())
if hasattr(self, "cnpg_cluster_name")
else ""
)
if not cluster_name:
cluster_name = env_vals.get("CLUSTER_NAME", "")
if not cluster_name:
cluster_name = "knoe-db"
_set("env_setup.CLUSTER_NAME", cluster_name)
# Database creation
_set("init_password.db_namespace", ns_val)
_set("init_password.cluster_name", cluster_name)
_set("init_password.db_username", _get_var(self.db_username, ""))
db_pw = _get_var(self.db_password, "")
db_pw_confirm = _get_var(self.db_password_confirm, "") or db_pw
db_pw_cfg = self._secret_cfg_value(
"Inputs", "init_password.db_password", db_pw, "db", "password"
)
_set("init_password.db_password", db_pw_cfg)
_set("init_password.db_password_confirm", db_pw_cfg or db_pw_confirm)
_set("init_password.db_host_port", _get_var(self.db_host_port, "5432"))
_set_bool(
"init_password.generate_ssh_key",
_action(
"init_password.generate_ssh_key",
DEFAULT_ACTION_FLAGS.get("init_password.generate_ssh_key", True),
),
)
# Database options (distribution, version, extensions)
_set("database_options.distribution", _get_var(self.db_distribution, "percona"))
_set("database_options.version_type", _get_var(self.db_version_type, "v18"))
for _ext in getattr(self, "extensions_list", []):
_set_bool(
f"database_options.ext.{_ext['id']}",
_get_var(self.db_extensions.get(_ext["id"]), False),
)
# Build DB image
_set_bool(
"init_db_build.run_build",
_action(
"init_db_build.run_build",
DEFAULT_ACTION_FLAGS.get("init_db_build.run_build", True),
),
)
# Cluster init + optional features
_set("init_cluster.cluster_env", _get_var(self.cluster_env, "dev"))
_set(
"init_cluster.mode",
_deployment_mode_from_env(_get_var(self.cluster_env, "dev")),
)
_set(
"init_cluster.deployment_target",
_deployment_target_label(_get_var(self.cluster_env, "dev")),
)
_set("init_cluster.k3s_server_url", _get_var(self.k3s_server_url, ""))
_set(
"init_cluster.k3s_token", _encrypt_cfg_secret(_get_var(self.k3s_token, ""))
)
_set_bool(
"init_cluster.supabase_enabled", _get_var(self.supabase_enabled, False)
)
_set_bool("init_cluster.gitops_enabled", _get_var(self.gitops_enabled, False))
_set_bool(
"init_cluster.argocd_enabled", _get_var(self.argocd_enabled, False)
)
_set_bool(
"init_cluster.kerberos_enabled", _get_var(self.kerberos_enabled, False)
)
_set_bool(
"init_cluster.at_rest_encryption_enabled",
_get_var(self.at_rest_encryption_enabled, False),
)
_set_bool(
"init_cluster.start_cluster",
_action(
"init_cluster.start_cluster",
DEFAULT_ACTION_FLAGS.get("init_cluster.start_cluster", True),
),
)
_set("supabase_config.pv_node", _get_var(self.supabase_pv_node, ""))
_set("supabase_config.pv_base_dir", _get_var(self.supabase_pv_base_dir, ""))
# Kerberos config
_set_bool("kerberos_config.enabled", _get_var(self.kerberos_enabled, False))
_set("kerberos_config.realm", _get_var(self.kerberos_realm, ""))
_set("kerberos_config.kdc", _get_var(self.kerberos_kdc, ""))
_set("kerberos_config.user", _get_var(self.kerberos_user, ""))
krb_pw = _get_var(self.kerberos_password, "")
_set(
"kerberos_config.password",
self._secret_cfg_value(
"Inputs", "kerberos_config.password", krb_pw, "kerberos", "password"
),
)
_set_bool(
"kerberos_config.test_connection",
_action(
"kerberos_config.test_connection",
DEFAULT_ACTION_FLAGS.get("kerberos_config.test_connection", False),
),
)
# Ollama config
_set("ollama_config.server_host", _get_var(self.ollama_server_host, ""))
_set(
"ollama_config.server_port",
_get_var(self.ollama_server_port, DEFAULT_OLLAMA_PORT),
)
_set("ollama_config.model", _get_var(self.ollama_model, ""))
# Init scripts + deploy
_set_bool(
"init_scripts.run_scripts",
_action(
"init_scripts.run_scripts",
DEFAULT_ACTION_FLAGS.get("init_scripts.run_scripts", True),
),
)
_set_bool(
"init_cnpg_deploy.run_deploy",
_action(
"init_cnpg_deploy.run_deploy",
DEFAULT_ACTION_FLAGS.get("init_cnpg_deploy.run_deploy", True),
),
)
_set_bool(
"init_cnpg_deploy.force_rollout",
_action(
"init_cnpg_deploy.force_rollout",
DEFAULT_ACTION_FLAGS.get("init_cnpg_deploy.force_rollout", False),
),
)
# GitOps
_set("gitops.namespace", _get_var(self.gitops_namespace, "gitea"))
_set("gitops.node_selector", _get_var(self.gitops_node_selector, ""))
_set("gitops.git_provider", _get_var(self.gitops_git_provider, "Gitea"))
# ArgoCD
_set("argocd.namespace", _get_var(self.argocd_namespace, "argocd"))
_set("argocd.node_selector", _get_var(self.argocd_node_selector, ""))
# Disk selection (installer packaging)
_set("disk_selection.disk_type", _get_var(self.selected_disk_type, "local"))
_set(
"disk_selection.removable_mount", _get_var(self.selected_removable_disk, "")
)
_set(
"disk_selection.local_path",
_get_var(self.selected_local_path, str(Path.home())),
)
# Build tools app
_set("build.deploy_env", getattr(self, "deploy_env_value", "Dev"))
_set_bool(
"build.run_build",
_action(
"build.run_build", DEFAULT_ACTION_FLAGS.get("build.run_build", False)
),
)
return inputs
def _sync_port_forward_mappings(self):
try:
mode = self._deployment_mode()
except Exception:
mode = ""
if not mode:
try:
mode = _deployment_mode_from_env(self.cluster_env.get())
except Exception:
mode = ""
if not mode:
mode = "k3d"
prefix = (
"PORT_FORWARD_K3S_MAPPING_"
if mode == "k3s"
else "PORT_FORWARD_K3D_MAPPING_"
)
pf_section = self.knoe_cfg_data.get("Port Forwards", {})
if pf_section is None:
pf_section = {}
try:
service_ns = (self._get_service_namespace() or "").strip()
except Exception:
service_ns = ""
if not service_ns:
service_ns = "default"
argocd_ns = (
(self.knoe_cfg_data.get("Global", {}) or {})
.get("ARGOCD_NAMESPACE", "")
.strip()
)
if not argocd_ns:
argocd_ns = (os.environ.get("ARGOCD_NAMESPACE") or "").strip()
if not argocd_ns:
argocd_ns = "argocd"
try:
db_ns = (self.db_namespace.get() or "").strip()
except Exception:
db_ns = ""
if not db_ns:
db_ns = (
(self.knoe_cfg_data.get("Global", {}) or {})
.get("DATABASE_NAMESPACE", "")
.strip()
)
if not db_ns:
db_ns = (
(self.knoe_cfg_data.get("Global", {}) or {})
.get("NAMESPACE", "")
.strip()
)
if not db_ns:
db_ns = (os.environ.get("DATABASE_NAMESPACE") or "").strip()
if not db_ns:
db_ns = (os.environ.get("NAMESPACE") or "").strip()
if not db_ns:
db_ns = "default"
try:
db_host_port = (self.db_host_port.get() or "").strip()
except Exception:
db_host_port = ""
if not db_host_port:
db_host_port = (
(self.knoe_cfg_data.get("Global", {}) or {})
.get("DB_HOST_PORT", "")
.strip()
)
if not db_host_port:
db_host_port = "5432"
supabase_enabled = False
try:
supabase_enabled = bool(self.supabase_enabled.get())
except Exception:
supabase_enabled = False
supabase_ns = (os.environ.get("SUPABASE_NAMESPACE") or "").strip()
if not supabase_ns:
supabase_ns = (
(self.knoe_cfg_data.get("Supabase", {}) or {})
.get("NAMESPACE", "")
.strip()
)
if not supabase_ns:
supabase_ns = "supabase"
gitops_enabled = False
try:
gitops_enabled = bool(self.gitops_enabled.get())
except Exception:
gitops_enabled = False
gitops_ns = (os.environ.get("GITEA_NAMESPACE") or "").strip()
if not gitops_ns:
gitops_ns = (
(self.knoe_cfg_data.get("GitOps", {}) or {})
.get("GITOPS_NAMESPACE", "")
.strip()
)
if not gitops_ns:
gitops_ns = (self.gitops_namespace.get() or "").strip() or "gitea"
mappings = _build_required_port_forwards(
mode=mode,
service_ns=service_ns,
argocd_ns=argocd_ns,
db_ns=db_ns,
db_host_port=db_host_port,
supabase_enabled=supabase_enabled,
supabase_namespace=supabase_ns,
gitops_enabled=gitops_enabled,
gitops_namespace=gitops_ns,
)
for mapping in mappings:
_pf_upsert_mapping(pf_section, prefix, mapping)
self.knoe_cfg_data["Port Forwards"] = pf_section
self._update_legacy_port_mapping()
def _update_legacy_port_mapping(self):
conf_dir = PROJECT_ROOT / "conf"
mapping_path = conf_dir / "port-mapping.cfg"
pf_section = self.knoe_cfg_data.get("Port Forwards", {})
if pf_section is None:
pf_section = {}
try:
mode = self._deployment_mode()
except Exception:
mode = "k3d"
prefix = (
"PORT_FORWARD_K3S_MAPPING_"
if mode == "k3s"
else "PORT_FORWARD_K3D_MAPPING_"
)
# Resolve ${DATABASE_NAMESPACE} for port-mapping entries
try:
db_ns = self.db_namespace.get().strip()
except Exception:
db_ns = ""
if not db_ns:
db_ns = (
(self.knoe_cfg_data.get("Global", {}) or {})
.get("DATABASE_NAMESPACE", "")
.strip()
)
if not db_ns:
db_ns = (
(self.knoe_cfg_data.get("Global", {}) or {})
.get("NAMESPACE", "")
.strip()
)
if not db_ns:
db_ns = "default"
lines = [
"# Port mappings for Knoe Tools (generated).",
"# Format: key: local=... remote=... ns=... svc=... address=...",
"",
]
# We want to maintain some order or just dump them
for k, v in sorted(pf_section.items()):
if k.startswith(prefix):
# Parse the mapping string: id=...;namespace=...;target=...;address=...;hostPort=...;servicePort=...;protocol=...;description=...
parts = {}
for p in str(v).split(";"):
if "=" in p:
key_val = p.split("=", 1)
if len(key_val) == 2:
parts[key_val[0].strip()] = key_val[1].strip()
if "id" in parts:
m_id = parts["id"]
local = parts.get("hostPort", "")
remote = parts.get("servicePort", "")
ns = parts.get("namespace", "")
target = parts.get("target", "")
addr = parts.get("address", "0.0.0.0")
# Resolve namespace references
if ns.startswith("${") and (
"DATABASE_NAMESPACE" in ns or "NAMESPACE" in ns
):
ns = db_ns
# Sanitize: skip entries with non-string or mock values
vals = [m_id, local, remote, ns, target, addr]
if any(
"MagicMock" in str(x) or not isinstance(x, str) for x in vals
):
continue
if local.startswith("${"):
continue
svc = target
if svc.startswith("svc/"):
svc = svc[4:]
lines.append(
f"{m_id}: local={local} remote={remote} ns={ns} svc={svc} address={addr}"
)
try:
mapping_path.write_text("\n".join(lines) + "\n")
print(f"[DEBUG] port-mapping.cfg updated at {mapping_path}")
except Exception as e:
print(f"[ERROR] Failed to update port-mapping.cfg: {e}")
def _add_port_mapping(self, mapping_str):
mode = self._deployment_mode()
prefix = (
"PORT_FORWARD_K3S_MAPPING_"
if mode == "k3s"
else "PORT_FORWARD_K3D_MAPPING_"
)
pf_section = self.knoe_cfg_data.get("Port Forwards", {})
if pf_section is None:
pf_section = {}
if not _pf_upsert_mapping(pf_section, prefix, mapping_str):
return
self.knoe_cfg_data["Port Forwards"] = pf_section
self._save_knoe_cfg()
def _process_script_output_line(self, line):
if "GRAFANA_ADMIN_PASSWORD=" in line:
pwd = line.split("GRAFANA_ADMIN_PASSWORD=")[1].strip()
if pwd:
mon = self.knoe_cfg_data.get("Monitoring", {})
if mon is None:
mon = {}
mon["GRAFANA_ADMIN_PASSWORD"] = pwd
self.knoe_cfg_data["Monitoring"] = mon
self._save_knoe_cfg()
if "PORT_FORWARD_MAPPING:" in line:
mapping = line.split("PORT_FORWARD_MAPPING:")[1].strip()
if mapping:
if _pf_extract_id(mapping) == "registry":
return
self._add_port_mapping(mapping)
def _is_port_available(self, port: int) -> bool:
"""Check if a port is available on localhost."""
import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
try:
s.bind(("127.0.0.1", port))
return True
except socket.error:
return False
def _pf_extract_field(self, mapping_str: str, field: str) -> str:
if not mapping_str or not field:
return ""
needle = f"{field}="
for part in str(mapping_str).split(";"):
part = part.strip()
if part.startswith(needle):
return part[len(needle) :].strip()
return ""
def _validate_port_forward_overlaps(self) -> bool:
try:
self._sync_port_forward_mappings()
except Exception:
pass
try:
mode = self._deployment_mode()
except Exception:
mode = ""
if not mode:
try:
mode = _deployment_mode_from_env(self.cluster_env.get())
except Exception:
mode = ""
if not mode:
mode = "k3d"
prefix = (
"PORT_FORWARD_K3S_MAPPING_"
if mode == "k3s"
else "PORT_FORWARD_K3D_MAPPING_"
)
pf_section = self.knoe_cfg_data.get("Port Forwards", {}) or {}
port_index = {}
for key, mapping in pf_section.items():
if not key.startswith(prefix):
continue
host_port = self._pf_extract_field(mapping, "hostPort")
if not host_port:
continue
mapping_id = _pf_extract_id(mapping)
target = self._pf_extract_field(mapping, "target")
namespace = self._pf_extract_field(mapping, "namespace")
label = mapping_id or target or key
if namespace and target:
label = f"{label} ({namespace}/{target})"
port_index.setdefault(host_port, []).append(label)
collisions = {
port: labels for port, labels in port_index.items() if len(labels) > 1
}
if not collisions:
return True
lines = []
for port, labels in sorted(
collisions.items(),
key=lambda item: int(item[0]) if str(item[0]).isdigit() else item[0],
):
lines.append(f"{port}: {', '.join(sorted(labels))}")
try:
messagebox.showerror(
"Port Collision",
"Port forward mappings overlap:\n\n"
+ "\n".join(lines)
+ "\n\nPlease choose unique host ports before continuing.",
)
except Exception:
pass
return False
def _show_port_error(self, port: int):
"""Show error message when port is in use, with option to see lsof output."""
msg = f"Port {port} is already in use on the local host.\n\nPlease choose a different port or stop the service using it."
dialog = tk.Toplevel(self.root)
dialog.title("Port In Use")
dialog.geometry("450x200")
dialog.configure(bg="white")
dialog.transient(self.root)
dialog.grab_set()
tk.Label(
dialog,
text="Port Conflict",
font=("SF Pro Text", 14, "bold"),
bg="white",
fg="#ff3b30",
).pack(pady=(20, 10))
tk.Label(
dialog,
text=msg,
font=("SF Pro Text", 11),
bg="white",
fg="black",
wraplength=400,
).pack(pady=10)
btn_frame = tk.Frame(dialog, bg="white")
btn_frame.pack(pady=20)
def show_lsof():
log_file = PROJECT_ROOT / "logs" / f"port_{port}_lsof.log"
log_file.parent.mkdir(parents=True, exist_ok=True)
try:
res = subprocess.run(
["lsof", "-i", f":{port}"], capture_output=True, text=True
)
content = (
res.stdout
if res.stdout
else f"No lsof output for port {port}. (Maybe permission denied?)"
)
if res.stderr:
content += "\n\nError:\n" + res.stderr
log_file.write_text(content)
# Open the log file
if platform.system() == "Darwin":
subprocess.run(["open", str(log_file)])
elif platform.system() == "Windows":
os.startfile(str(log_file))
else:
subprocess.run(["xdg-open", str(log_file)])
except Exception as e:
messagebox.showerror("Error", f"Failed to run lsof: {e}")
tk.Button(
btn_frame,
text="Show Logs (lsof)",
command=show_lsof,
bg="#F5F5DC",
relief="flat",
padx=10,
).pack(side="left", padx=10)
tk.Button(
btn_frame,
text="OK",
command=dialog.destroy,
bg="#007aff",
fg="white",
relief="flat",
padx=20,
).pack(side="left", padx=10)
def _apply_ansible_topology_defaults(self):
info = _detect_ansible_topology(PROJECT_ROOT)
if not info:
return
self.ansible_topology = info
self.ansible_topology_summary = _format_ansible_topology_summary(info)
try:
net = self.knoe_cfg_data.get("Network", {})
if info.get("topology_json"):
net["ANSIBLE_TOPOLOGY"] = info["topology_json"]
if info.get("inventory_path"):
net["ANSIBLE_INVENTORY"] = info["inventory_path"]
if info.get("infrastructure_path"):
net["ANSIBLE_INFRASTRUCTURE"] = info["infrastructure_path"]
if info.get("domain"):
net["ANSIBLE_DOMAIN"] = info["domain"]
if info.get("realm"):
net["ANSIBLE_REALM"] = info["realm"]
if info.get("ad_dc_host"):
net["AD_DC_HOST"] = info["ad_dc_host"]
if info.get("ad_dc_ip"):
net["AD_DC_IP"] = info["ad_dc_ip"]
if info.get("kdc_ip"):
net["KDC_ANSIBLE_DETECTED"] = info["kdc_ip"]
self.knoe_cfg_data["Network"] = net
except Exception:
pass
try:
if not self.kerberos_kdc.get().strip() and info.get("kdc_ip"):
self.kerberos_kdc.set(info["kdc_ip"])
self.kerberos_enabled.set(True)
if not self.kerberos_realm.get().strip() and info.get("realm"):
self.kerberos_realm.set(info["realm"])
except Exception:
pass
def _read_k3s_cfg_values(self) -> tuple[str, str, str]:
"""Return (cluster_env, server_url, token) from knoe.cfg if present.
Delegates to the shared ``_read_k3s_cfg`` so every presentation
layer (Tk, ncurses, silent) parses the same cfg sections.
"""
cfg_path = None
try:
if self._cfg_path_override is not None:
cfg_path = self._cfg_path_override
if cfg_path.is_dir():
cfg_path = cfg_path / "knoe.cfg"
else:
cfg_path = self._resolve_knoe_conf_dir() / "knoe.cfg"
except Exception:
cfg_path = None
return _read_k3s_cfg(cfg_path)
def _apply_cluster_env_default_from_cfg(self):
cfg_env, _cfg_server, _cfg_token = self._read_k3s_cfg_values()
if not cfg_env:
return
try:
current = (self.cluster_env.get() or "").strip()
except Exception:
current = ""
if not current or current == "dev":
try:
self.cluster_env.set(_cluster_env_radio_value(cfg_env))
except Exception:
pass
self._set_deploy_target_from_cluster_env()
def _apply_k3s_defaults(self):
# Environment overrides
env_server = (
os.environ.get("PROLE_K3S_SERVER") or os.environ.get("K3S_SERVER_URL") or ""
).strip()
env_token = (
os.environ.get("PROLE_K3S_TOKEN") or os.environ.get("K3S_TOKEN") or ""
).strip()
env_token = _normalize_k3s_token(env_token)
if env_server and not self.k3s_server_url.get().strip():
self.k3s_server_url.set(env_server)
if env_token and not self.k3s_token.get().strip():
self.k3s_token.set(env_token)
# Config defaults
_cfg_env, cfg_server, cfg_token = self._read_k3s_cfg_values()
if cfg_server and not self.k3s_server_url.get().strip():
self.k3s_server_url.set(cfg_server)
if cfg_token and not self.k3s_token.get().strip():
self.k3s_token.set(cfg_token)
# Ansible defaults
info = getattr(self, "ansible_topology", None) or _detect_ansible_topology(
PROJECT_ROOT
)
if info:
if info.get("k3s_server_url") and not self.k3s_server_url.get().strip():
self.k3s_server_url.set(info["k3s_server_url"])
token_val = _normalize_k3s_token(info.get("k3s_token"))
if token_val and not self.k3s_token.get().strip():
self.k3s_token.set(token_val)
def _propagate_namespace_change(self, *args):
"""Reacts to namespace change by updating dependent inputs and config files.
This is triggered by a `tk.StringVar` trace while the user types. We debounce
the propagation so we only apply a single stable transition from the last
synced namespace to the final value (instead of rewriting on each keystroke).
"""
try:
new_ns = (self.db_namespace.get() or "").strip()
except Exception:
return
if not new_ns:
return
# If there is no Tk root (e.g. some test harnesses), fall back to immediate.
root = getattr(self, "root", None)
if root is None:
self._apply_pending_namespace_change(new_ns)
return
# Debounce: cancel any in-flight scheduled propagation and reschedule.
after_id = getattr(self, "_namespace_propagate_after_id", None)
if after_id:
try:
root.after_cancel(after_id)
except Exception:
pass
self._pending_namespace_value = new_ns
delay_ms = getattr(self, "_namespace_debounce_ms", None)
if delay_ms is None:
delay_ms = getattr(self, "NAMESPACE_PROPAGATE_DEBOUNCE_MS", 250)
try:
self._namespace_propagate_after_id = root.after(
int(delay_ms), self._apply_pending_namespace_change
)
except Exception:
# If scheduling fails, apply immediately.
self._apply_pending_namespace_change(new_ns)
def _apply_pending_namespace_change(self, namespace: str | None = None):
"""Apply the pending namespace propagation using a stable old namespace."""
# Clear the timer id if we were invoked from `after`.
try:
self._namespace_propagate_after_id = None
except Exception:
pass
if namespace is None:
try:
namespace = (getattr(self, "_pending_namespace_value", "") or "").strip()
except Exception:
namespace = ""
new_ns = (namespace or "").strip()
if not new_ns:
return
old_ns = (getattr(self, "_last_synced_ns", "") or "").strip()
if not old_ns:
# Initialize sync baseline and persist.
try:
self._last_synced_ns = new_ns
except Exception:
pass
if new_ns == old_ns:
return
# Propagate to inputs and StringVars (e.g. OpenBao paths)
if old_ns:
# Do not rewrite independent namespaces.
exclude_var_names = {
"db_namespace",
"service_namespace",
"gitops_namespace",
"argocd_namespace",
}
# 1. Sweep StringVars
for attr_name in dir(self):
if attr_name in exclude_var_names:
continue
try:
attr = getattr(self, attr_name)
if isinstance(attr, tk.StringVar) and attr != self.db_namespace:
val = attr.get()
if val and old_ns in val:
attr.set(val.replace(old_ns, new_ns))
except Exception:
pass
# 2. Sweep inputs
if hasattr(self, "inputs"):
for k, v in list(self.inputs.items()):
if not isinstance(v, str) or old_ns not in v:
continue
lk = (k or "").lower()
if "service_namespace" in lk or lk.endswith("service_namespace"):
continue
self.inputs[k] = v.replace(old_ns, new_ns)
# Update environment and save knoe.cfg
try:
self._last_synced_ns = new_ns
except Exception:
pass
try:
self._update_env_namespace(new_ns)
# Re-save knoe.cfg so it's always in sync
self._save_knoe_cfg()
except Exception:
pass
Reference in New Issue View Git Blame Copy Permalink