chrisfu/prole
1
0
Fork 0
mirror of https://github.com/dredx/prole.git synced 2026-09-24 11:34:31 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Update Prole-DB and improve Supabase integration

Browse Source 
- Bumped Prole-DB image version to 17.7-053 in scripts, Dockerfile, and manifests.
- Replaced `prole-scan` with `prole-agent` throughout scripts and tests.
- Refined Kubernetes setup for Supabase to use namespace 'supabase'.
- Introduced conversion of Supabase Docker Compose to Kubernetes manifests with `kompose`.
- Added support for Kerberos toggle via environment variables in `init_kerberos.sh`.
- Improved error handling and logging in scripts for better maintainability.
This commit is contained in:
chrisfu 2026-02-01 23:50:09 -08:00
parent 9c1f58caf4
commit fff18fdbe4
30 changed files with 1027 additions and 188 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

34
docs/EMBEDDED-RESOURCES.md
View File

@ -14,7 +14,7 @@ The Prole Installer package includes several embedded resources that must be acc
- **proleIconblueprint.png** (1.6 MB) - Blueprint icon variant - **proleIconblueprint.png** (1.6 MB) - Blueprint icon variant
### 2. Binary Executables ### 2. Binary Executables
- **prole-net/prole-scan** (6.8 MB) - Network scanner - **prole-net/prole-agent** (6.8 MB) - Network scanner
- Universal binary (x86_64 + arm64) - Universal binary (x86_64 + arm64)
- Used by network scan screen - Used by network scan screen
- Detects Kerberos, Active Directory, etc. - Detects Kerberos, Active Directory, etc.
@ -54,7 +54,7 @@ if bg_path.exists():
**Binary Execution:** **Binary Execution:**
```python ```python
scan_binary = get_resource_path("prole-net/prole-scan") scan_binary = get_resource_path("prole-net/prole-agent")
if scan_binary.exists(): if scan_binary.exists():
process = subprocess.Popen([str(scan_binary)], ...) process = subprocess.Popen([str(scan_binary)], ...)
``` ```
@ -86,7 +86,7 @@ datas = [
**Binaries:** **Binaries:**
```python ```python
binaries = [ binaries = [
('prole-net/prole-scan', 'prole-net'), ('prole-net/prole-agent', 'prole-net'),
] ]
``` ```
@ -98,7 +98,7 @@ Total embedded resources: ~30-35 MB
Breakdown: Breakdown:
- Images: ~11 MB - Images: ~11 MB
- prole-scan: 6.8 MB - prole-agent: 6.8 MB
- Prole Tools.app: ~12 MB - Prole Tools.app: ~12 MB
- Other resources: ~5-10 MB - Other resources: ~5-10 MB
@ -108,10 +108,10 @@ Final installer bundle: ~50-100 MB (includes Python runtime)
### Network Scan Screen ### Network Scan Screen
The network scan screen uses `prole-scan` to detect services: The network scan screen uses `prole-agent` to detect services:
```python ```python
scan_binary = get_resource_path("prole-net/prole-scan") scan_binary = get_resource_path("prole-net/prole-agent")
process = subprocess.Popen([str(scan_binary)], process = subprocess.Popen([str(scan_binary)],
stdout=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, stderr=subprocess.STDOUT,
@ -140,7 +140,7 @@ python3 test_embedded_resources.sh
This tests: This tests:
- ✓ All image files exist - ✓ All image files exist
- ✓ prole-scan binary exists and is executable - ✓ prole-agent binary exists and is executable
- ✓ Prole Tools.app bundle exists - ✓ Prole Tools.app bundle exists
- ✓ Spec file includes all resources - ✓ Spec file includes all resources
- ✓ Resource sizes - ✓ Resource sizes
@ -163,18 +163,18 @@ ls -la "/tmp/_MEI*/prole-app/dist/Prole Tools.app"
### Binary Not Found Error ### Binary Not Found Error
**Error:** `Scan binary not found at /var/folders/.../prole-net/prole-scan` **Error:** `Scan binary not found at /var/folders/.../prole-net/prole-agent`
**Cause:** Binary not included in package or path not using `get_resource_path()` **Cause:** Binary not included in package or path not using `get_resource_path()`
**Solution:** **Solution:**
1. Verify spec includes binary: `grep prole-scan installer.spec` 1. Verify spec includes binary: `grep prole-agent installer.spec`
2. Check code uses `get_resource_path()`: `grep "get_resource_path.*prole-scan" install.py` 2. Check code uses `get_resource_path()`: `grep "get_resource_path.*prole-agent" install.py`
3. Rebuild: `make clean && make package` 3. Rebuild: `make clean && make package`
### Binary Not Executable ### Binary Not Executable
**Error:** `Permission denied` when running prole-scan **Error:** `Permission denied` when running prole-agent
**Cause:** Binary permissions not preserved in package **Cause:** Binary permissions not preserved in package
@ -182,10 +182,10 @@ ls -la "/tmp/_MEI*/prole-app/dist/Prole Tools.app"
Ensure binary is in `binaries` list (not `datas`) in spec file: Ensure binary is in `binaries` list (not `datas`) in spec file:
```python ```python
binaries = [ binaries = [
('prole-net/prole-scan', 'prole-net'), # Correct - preserves +x ('prole-net/prole-agent', 'prole-net'), # Correct - preserves +x
] ]
# NOT in datas: # NOT in datas:
# datas = [('prole-net/prole-scan', 'prole-net')] # Wrong - loses +x # datas = [('prole-net/prole-agent', 'prole-net')] # Wrong - loses +x
``` ```
### App Bundle Not Found ### App Bundle Not Found
@ -217,7 +217,7 @@ binaries = [
3. **Exclude debug symbols:** 3. **Exclude debug symbols:**
Strip binaries before packaging: Strip binaries before packaging:
```bash ```bash
strip prole-net/prole-scan strip prole-net/prole-agent
``` ```
## Build Process ## Build Process
@ -250,7 +250,7 @@ When the packaged installer runs:
Embedded binaries should be code signed: Embedded binaries should be code signed:
```bash ```bash
codesign --force --sign "Developer ID Application: Your Name" prole-net/prole-scan codesign --force --sign "Developer ID Application: Your Name" prole-net/prole-agent
``` ```
Then build the installer - the signed binary will be included. Then build the installer - the signed binary will be included.
@ -260,8 +260,8 @@ Then build the installer - the signed binary will be included.
Users can verify embedded binaries: Users can verify embedded binaries:
```bash ```bash
# Check signature of prole-scan after extraction # Check signature of prole-agent after extraction
codesign --verify --verbose /tmp/_MEI*/prole-net/prole-scan codesign --verify --verbose /tmp/_MEI*/prole-net/prole-agent
# Check installer bundle signature # Check installer bundle signature
codesign --verify --verbose "dist/Prole Installer.app" codesign --verify --verbose "dist/Prole Installer.app"

4
docs/PROLE-HOME-DIRECTORY.md
View File

@ -50,7 +50,7 @@ subprocess.Popen(['docker', 'build', '-t', 'prole-db:TAG', '.'], cwd=build_dir)
**Purpose:** Writable working directory for network scan operations **Purpose:** Writable working directory for network scan operations
**Why:** The `prole-scan` binary may need to write output files, cache data, or store temporary results. Running from a read-only directory causes failures. **Why:** The `prole-agent` binary may need to write output files, cache data, or store temporary results. Running from a read-only directory causes failures.
**Usage:** **Usage:**
```python ```python
@ -65,7 +65,7 @@ subprocess.Popen([str(scan_binary)], cwd=str(scan_dir))
**Contents:** **Contents:**
- Network scan results (temporary) - Network scan results (temporary)
- Ollama API interaction cache - Ollama API interaction cache
- Any intermediate files created by prole-scan - Any intermediate files created by prole-agent
**Size:** Varies, typically < 1 MB **Size:** Varies, typically < 1 MB

2
docs/build-system.md
View File

@ -97,7 +97,7 @@ A Python script (`scripts/generate_spec.py`) creates the PyInstaller specificati
- `prole-app/dist/Prole Tools.app` - Pre-built Prole Tools application bundle (entire .app) - `prole-app/dist/Prole Tools.app` - Pre-built Prole Tools application bundle (entire .app)
**Included Binaries:** **Included Binaries:**
- `prole-net/prole-scan` - Network scanner binary (universal: x86_64 + arm64, 6.8 MB) - `prole-net/prole-agent` - Network scanner binary (universal: x86_64 + arm64, 6.8 MB)
**Resource Path Resolution:** **Resource Path Resolution:**
The installer uses a `get_resource_path()` helper function that automatically resolves paths correctly whether running from source or from a PyInstaller bundle: The installer uses a `get_resource_path()` helper function that automatically resolves paths correctly whether running from source or from a PyInstaller bundle:

61
etc/init_cloudnative_pg.sh
View File

@ -103,7 +103,7 @@ ensure_prole_stack_resources() {
dir="$SCRIPT_DIR/../k8s/prole" dir="$SCRIPT_DIR/../k8s/prole"
for file in "$dir"/*.yaml; do for file in "$dir"/*.yaml; do
case "$(basename "$file")" in case "$(basename "$file")" in
prole-db.yaml|kustomization.yaml) prole-db.yaml|kustomization.yaml|supabase-*.yaml)
continue continue
;; ;;
esac esac
@ -115,7 +115,7 @@ ensure_prole_stack_resources() {
dir="$SCRIPT_DIR/../k8s/prole" dir="$SCRIPT_DIR/../k8s/prole"
for file in "$dir"/*.yaml; do for file in "$dir"/*.yaml; do
case "$(basename "$file")" in case "$(basename "$file")" in
kustomization.yaml) kustomization.yaml|supabase-*.yaml)
continue continue
;; ;;
esac esac
@ -147,21 +147,46 @@ wait_for_cnpg_pods() {
local start_time local start_time
start_time=$(date +%s) start_time=$(date +%s)
local target_pods="${CNPG_TARGET_PODS:-}"
if [[ -z "$target_pods" ]]; then
target_pods=$(kubectl -n "$NAMESPACE" get cluster "$CNPG_CLUSTER_NAME" -o jsonpath='{.spec.instances}' 2>/dev/null || true)
fi
if [[ -z "$target_pods" ]]; then
target_pods=3
fi
echo "Waiting for $target_pods CNPG pods for cluster '$CNPG_CLUSTER_NAME' in namespace '$NAMESPACE' to be Running..."
while true; do while true; do
if kubectl -n "$NAMESPACE" get cluster "$CNPG_CLUSTER_NAME" >/dev/null 2>&1; then local pods
if kubectl -n "$NAMESPACE" get pods -l "cnpg.io/cluster=$CNPG_CLUSTER_NAME" --no-headers 2>/dev/null | grep -q .; then pods=$(kubectl -n "$NAMESPACE" get pods -l "cnpg.io/cluster=$CNPG_CLUSTER_NAME" --no-headers 2>/dev/null || true)
if kubectl -n "$NAMESPACE" wait --for=condition=Ready pod -l "cnpg.io/cluster=$CNPG_CLUSTER_NAME" --timeout=60s >/dev/null 2>&1; then
return 0 if [[ -n "$pods" ]]; then
fi # Check for Error or CrashLoopBackOff
if echo "$pods" | grep -E "Error|CrashLoopBackOff" >/dev/null; then
echo "ERROR: Some CNPG pods are in Error or CrashLoopBackOff state:" >&2
echo "$pods" | grep -E "Error|CrashLoopBackOff" >&2
return 1
fi
# Count Running pods by name (exclude initdb)
local running_pods
running_pods=$(kubectl -n "$NAMESPACE" get pods --no-headers 2>/dev/null \
| grep "^${CNPG_CLUSTER_NAME}-" \
| grep -v "initdb" \
| grep "Running" \
| wc -l | xargs)
if [[ "$running_pods" -ge "$target_pods" ]]; then
echo "All $running_pods/$target_pods pods are Running."
return 0
fi fi
fi fi
if (( $(date +%s) - start_time > timeout )); then if (( $(date +%s) - start_time > timeout )); then
echo "ERROR: Timed out waiting for CNPG pods for cluster '$CNPG_CLUSTER_NAME' in namespace '$NAMESPACE'." >&2 echo "ERROR: Timed out waiting for $target_pods CNPG pods for cluster '$CNPG_CLUSTER_NAME' in namespace '$NAMESPACE'." >&2
return 1 return 1
fi fi
echo "Waiting for CNPG pods for cluster '$CNPG_CLUSTER_NAME'..." sleep 5
sleep 3
done done
} }
@ -299,13 +324,19 @@ initialize() {
return 1 return 1
fi fi
if [[ -x "$SCRIPT_DIR/init_kerberos.sh" ]]; then # Support both names for the toggle from prole.cfg
echo "Configuring Kerberos for CNPG pods ..." local kerberos_enabled="${KERBEROS_ENABLED:-${ENABLED:-false}}"
if ! "$SCRIPT_DIR/init_kerberos.sh" initialize; then if [[ "$kerberos_enabled" == "true" || "$kerberos_enabled" == "True" || "$kerberos_enabled" == "1" ]]; then
echo "WARN: Kerberos initialization did not complete successfully." if [[ -x "$SCRIPT_DIR/init_kerberos.sh" ]]; then
echo "Configuring Kerberos for CNPG pods ..."
if ! "$SCRIPT_DIR/init_kerberos.sh" initialize; then
echo "WARN: Kerberos initialization did not complete successfully."
fi
else
echo "WARN: init_kerberos.sh not found; skipping Kerberos configuration."
fi fi
else else
echo "WARN: init_kerberos.sh not found; skipping Kerberos configuration." echo "Kerberos is disabled; skipping Kerberos configuration."
fi fi
# Ensure port-forward is running for Postgres (local access) # Ensure port-forward is running for Postgres (local access)

29
etc/init_garage_store.sh
View File

@ -215,13 +215,28 @@ init_layout() {
assign_count=$((assign_count + 1)) assign_count=$((assign_count + 1))
done done
# Get current version for apply # Re-fetch layout and apply staged changes using the current layout version
local version local layout_after version apply_ok
version=$(echo "$layout" | grep "Version:" | awk '{print $2}' || echo "0") layout_after=$(kubectl exec -n "$NAMESPACE" "$pod" -- /garage layout show 2>/dev/null || true)
if [[ -z "$version" ]]; then version=0; fi version=$(echo "$layout_after" | awk -F: '/[Vv]ersion/ {gsub(/ /,"",$2); print $2; exit}' || true)
local next_version=$((version + 1)) if echo "$layout_after" | grep -qi "staged"; then
echo "Applying Garage layout (version $next_version) ..." if [[ -n "$version" ]]; then
kubectl exec -n "$NAMESPACE" "$pod" -- /garage layout apply --version "$next_version" || true echo "Applying Garage layout (version $version) ..."
if kubectl exec -n "$NAMESPACE" "$pod" -- /garage layout apply --version "$version"; then
apply_ok=1
else
apply_ok=0
fi
else
apply_ok=0
fi
if [[ "${apply_ok:-0}" -eq 0 ]]; then
echo "WARN: layout apply with explicit version failed; retrying without version..."
kubectl exec -n "$NAMESPACE" "$pod" -- /garage layout apply || true
fi
else
echo "No staged layout changes detected; skipping layout apply."
fi
else else
echo "Garage layout already assigned for node $short_id." echo "Garage layout already assigned for node $short_id."
fi fi

43
etc/init_kerberos.sh
View File

@ -57,6 +57,15 @@ ensure_namespace() {
} }
resolve_krb5_defaults() { resolve_krb5_defaults() {
# If 'dog' KDC is available in 'authority' namespace, use it as default
if kubectl get pod -n "authority" -l app=dog >/dev/null 2>&1; then
KRB5_KDC="dog.authority.svc.cluster.local"
KRB5_ADMIN="dog.authority.svc.cluster.local"
if [[ -z "${KRB5_REALM}" ]]; then
KRB5_REALM="PROLE.ORG"
fi
fi
if [[ -z "${KRB5_REALM}" && -n "${REALM:-}" ]]; then if [[ -z "${KRB5_REALM}" && -n "${REALM:-}" ]]; then
KRB5_REALM="$REALM" KRB5_REALM="$REALM"
fi fi
@ -184,16 +193,29 @@ patch_cnpg_cluster_for_auth() {
# Best-effort patch; schema may vary with CNPG version. # Best-effort patch; schema may vary with CNPG version.
# We remove krb_srvname as it is unrecognized in some PG 17 builds. # We remove krb_srvname as it is unrecognized in some PG 17 builds.
# We revert certificates to default to avoid operator TLS issues. # We revert certificates to default to avoid operator TLS issues.
if ! kubectl -n "$NAMESPACE" patch cluster "$CNPG_CLUSTER_NAME" --type merge -p "{
\"spec\": { local kerberos_enabled="${KERBEROS_ENABLED:-${ENABLED:-false}}"
\"postgresql\": { local pg_hba
\"parameters\": {\"krb_srvname\": null}, if [[ "$kerberos_enabled" == "true" || "$kerberos_enabled" == "True" || "$kerberos_enabled" == "1" ]]; then
\"pg_hba\": [ pg_hba="[
\"local all postgres trust\", \"local all postgres trust\",
\"host all postgres all scram-sha-256\", \"host all postgres all scram-sha-256\",
\"host all all all gss include_realm=1 krb_realm=$KRB5_REALM\", \"host all all all gss include_realm=1 krb_realm=$KRB5_REALM\",
\"host all all all scram-sha-256\" \"host all all all scram-sha-256\"
] ]"
else
pg_hba="[
\"local all postgres trust\",
\"host all postgres all scram-sha-256\",
\"host all all all scram-sha-256\"
]"
fi
if ! kubectl -n "$NAMESPACE" patch cluster "$CNPG_CLUSTER_NAME" --type merge -p "{
\"spec\": {
\"postgresql\": {
\"parameters\": {\"krb_srvname\": null},
\"pg_hba\": $pg_hba
}, },
\"certificates\": { \"certificates\": {
\"serverCASecret\": null, \"serverCASecret\": null,
@ -323,6 +345,15 @@ initialize() {
ensure_namespace ensure_namespace
resolve_krb5_defaults resolve_krb5_defaults
# Support both names for the toggle from prole.cfg
local enabled="${KERBEROS_ENABLED:-${ENABLED:-false}}"
if [[ "$enabled" == "false" || "$enabled" == "0" || "$enabled" == "False" ]]; then
log "Kerberos is disabled (KERBEROS_ENABLED=$enabled). Skipping initialization."
# Cleanup any existing Kerberos config if it was previously enabled
kubectl -n "$NAMESPACE" delete configmap prole-krb5-conf --ignore-not-found
return 0
fi
if [[ -z "${KRB5_REALM}" || -z "${KRB5_KDC}" ]]; then if [[ -z "${KRB5_REALM}" || -z "${KRB5_KDC}" ]]; then
err "ERROR: KRB5_REALM and KRB5_KDC must be set for Kerberos initialization." err "ERROR: KRB5_REALM and KRB5_KDC must be set for Kerberos initialization."
err " Set KRB5_REALM/KRB5_KDC or REALM/DOMAIN in env or config." err " Set KRB5_REALM/KRB5_KDC or REALM/DOMAIN in env or config."

9
etc/init_kerberos_test.sh
View File

@ -81,7 +81,7 @@ detect_image() {
if [[ -n "$release_file" && -f "$release_file" ]]; then if [[ -n "$release_file" && -f "$release_file" ]]; then
release=$(cat "$release_file" | tr -d '[:space:]') release=$(cat "$release_file" | tr -d '[:space:]')
else else
release="41" release="43"
fi fi
if [[ "$release" =~ ^[0-9]+$ ]]; then if [[ "$release" =~ ^[0-9]+$ ]]; then
@ -142,6 +142,13 @@ run_test() {
ensure_tools ensure_tools
ensure_namespace ensure_namespace
# Support both names for the toggle from prole.cfg
local enabled="${KERBEROS_ENABLED:-${ENABLED:-false}}"
if [[ "$enabled" == "false" || "$enabled" == "0" || "$enabled" == "False" ]]; then
echo "Kerberos is disabled (KERBEROS_ENABLED=$enabled). Skipping test."
return 0
fi
if [[ -z "$KRB5_REALM" || -z "$KRB5_USER" || -z "$KRB5_PASSWORD" || -z "$KRB5_KDC" ]]; then if [[ -z "$KRB5_REALM" || -z "$KRB5_USER" || -z "$KRB5_PASSWORD" || -z "$KRB5_KDC" ]]; then
echo "ERROR: Missing Kerberos configuration. Ensure KRB5_REALM, KRB5_KDC, KRB5_USER, KRB5_PASSWORD are set." >&2 echo "ERROR: Missing Kerberos configuration. Ensure KRB5_REALM, KRB5_KDC, KRB5_USER, KRB5_PASSWORD are set." >&2
exit 1 exit 1

22
etc/init_prole-db.sh
View File

@ -68,7 +68,7 @@ get_latest_image() {
if [[ -f "$release_file" ]]; then if [[ -f "$release_file" ]]; then
release=$(cat "$release_file" | tr -d '[:space:]') release=$(cat "$release_file" | tr -d '[:space:]')
else else
release="41" release="43"
fi fi
if [[ "$release" =~ ^[0-9]+$ ]]; then if [[ "$release" =~ ^[0-9]+$ ]]; then
@ -78,6 +78,24 @@ get_latest_image() {
echo "prole-db:${pg_version}-${release}" echo "prole-db:${pg_version}-${release}"
} }
sync_manifest_image() {
local image="$1"
local manifest_dir
manifest_dir=$(dirname "$CNPG_MANIFEST")
local files=("$CNPG_MANIFEST")
if [[ -f "$manifest_dir/prole-db-recovery.yaml.tpl" ]]; then
files+=("$manifest_dir/prole-db-recovery.yaml.tpl")
fi
local f tmp
for f in "${files[@]}"; do
if [[ -f "$f" ]] && grep -qE '^[[:space:]]*imageName:' "$f"; then
tmp=$(mktemp)
sed -E "s|^([[:space:]]*imageName:).*|\\1 ${image}|" "$f" > "$tmp"
mv "$tmp" "$f"
fi
done
}
start() { start() {
ensure_tools ensure_tools
ensure_namespace ensure_namespace
@ -116,6 +134,7 @@ start() {
# Compare latest image with deployed # Compare latest image with deployed
local latest_image local latest_image
latest_image=$(get_latest_image) latest_image=$(get_latest_image)
sync_manifest_image "$latest_image"
local current_image local current_image
current_image=$(kubectl get cluster "$CNPG_CLUSTER_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.imageName}' 2>/dev/null || echo "") current_image=$(kubectl get cluster "$CNPG_CLUSTER_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.imageName}' 2>/dev/null || echo "")
@ -190,6 +209,7 @@ deploy() {
else else
image="prole-db:$VERSION" image="prole-db:$VERSION"
fi fi
sync_manifest_image "$image"
echo "Deploying $image to cluster $CNPG_CLUSTER_NAME..." echo "Deploying $image to cluster $CNPG_CLUSTER_NAME..."

389
etc/init_supabase.sh
View File

@ -4,9 +4,10 @@ set -euo pipefail
# init_supabase.sh # init_supabase.sh
# Purpose: # Purpose:
# - Deploy Supabase in Kubernetes and connect it to the CNPG Postgres cluster. # - Deploy Supabase stack in Kubernetes using the official supabase/docker compose
# - Rewire Supabase to use the CNPG Postgres service
# - Disable image pulls (imagePullPolicy: Never) for now
# Initialize SCRIPT_DIR
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
# Load environment and config via prole_cfg.sh # Load environment and config via prole_cfg.sh
@ -16,103 +17,389 @@ source "$SCRIPT_DIR/prole_cfg.sh"
ACTION=${1:-} ACTION=${1:-}
CNPG_CLUSTER_NAME=${CNPG_CLUSTER_NAME:-prole-db} CNPG_CLUSTER_NAME=${CNPG_CLUSTER_NAME:-prole-db}
SUPABASE_HOME=${SUPABASE_HOME:-}
# Support both PROLE_HOME/k8s and sibling k8s directory SUPABASE_USE_DEV_COMPOSE=${SUPABASE_USE_DEV_COMPOSE:-0}
if [[ -d "$SCRIPT_DIR/../k8s/prole" ]]; then SUPABASE_IMAGE_PULL_POLICY=${SUPABASE_IMAGE_PULL_POLICY:-Never}
SUPABASE_MANIFEST_DIR="$SCRIPT_DIR/../k8s/prole" SUPABASE_K8S_DIR=${SUPABASE_K8S_DIR:-${PROLE_HOME:-$SCRIPT_DIR/..}/build/supabase-k8s}
elif [[ -n "${PROLE_HOME:-}" && -d "$PROLE_HOME/k8s/prole" ]]; then SUPABASE_POSTGRES_HOST=${SUPABASE_POSTGRES_HOST:-db}
SUPABASE_MANIFEST_DIR="$PROLE_HOME/k8s/prole" SUPABASE_POSTGRES_DB=${SUPABASE_POSTGRES_DB:-postgres}
else SUPABASE_POSTGRES_PORT=${SUPABASE_POSTGRES_PORT:-5432}
SUPABASE_MANIFEST_DIR="$SCRIPT_DIR/../k8s/prole" SUPABASE_APPLY_DB_MIGRATIONS=${SUPABASE_APPLY_DB_MIGRATIONS:-1}
fi
SUPABASE_FILES=(
"$SUPABASE_MANIFEST_DIR/supabase-configmap.yaml"
"$SUPABASE_MANIFEST_DIR/supabase-deployment.yaml"
"$SUPABASE_MANIFEST_DIR/supabase-service.yaml"
)
usage() { usage() {
cat <<EOF cat <<EOF
Usage: $0 [start|stop|restart|status] Usage: $0 [start|stop|restart|status]
Actions: Actions:
start Apply Supabase manifests start Convert Supabase docker-compose to K8s and apply in namespace
stop Delete Supabase manifests stop Delete Supabase K8s resources from namespace
restart Re-apply Supabase manifests restart Re-apply Supabase manifests
status Show Supabase resources status Show Supabase resources
Env overrides:
SUPABASE_HOME, SUPABASE_USE_DEV_COMPOSE, SUPABASE_K8S_DIR
SUPABASE_IMAGE_PULL_POLICY (default: Never)
SUPABASE_POSTGRES_HOST (default: db)
SUPABASE_POSTGRES_DB (default: postgres)
SUPABASE_POSTGRES_PORT (default: 5432)
SUPABASE_APPLY_DB_MIGRATIONS (default: 1)
EOF EOF
exit 1 exit 1
} }
log() { printf '%s\n' "$*"; }
err() { printf '%s\n' "$*" >&2; }
ensure_tools() { ensure_tools() {
for t in kubectl; do for t in kubectl kompose python3 sed awk base64; do
command -v "$t" >/dev/null || { echo "Missing required tool: $t" >&2; exit 1; } command -v "$t" >/dev/null || { err "Missing required tool: $t"; exit 1; }
done done
} }
ensure_namespace() { ensure_namespace() {
if ! kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then if ! kubectl get namespace "supabase" >/dev/null 2>&1; then
echo "Creating namespace '$NAMESPACE' ..." err "ERROR: namespace 'supabase' not found. Supabase must be deployed in its own namespace."
kubectl create namespace "$NAMESPACE" >/dev/null 2>&1 || true exit 1
fi fi
} }
ensure_prereqs() { ensure_prereqs() {
ensure_namespace ensure_namespace
if ! kubectl get cluster "$CNPG_CLUSTER_NAME" -n "$NAMESPACE" >/dev/null 2>&1; then # We still want to ensure prole-db-superuser secret is in the CURRENT namespace (where prole-db is)
echo "ERROR: CNPG cluster '$CNPG_CLUSTER_NAME' not found in namespace '$NAMESPACE'." >&2 # but supabase itself will be in 'supabase' namespace.
exit 1 # The issue description says: "connect supabase postgres network ports to our new namespace"
fi # This implies we might need to create services in the CURRENT namespace that point to supabase.
if ! kubectl get secret prole-db-user -n "$NAMESPACE" >/dev/null 2>&1; then if ! kubectl get secret prole-db-superuser -n "$NAMESPACE" >/dev/null 2>&1; then
echo "ERROR: Secret 'prole-db-user' not found in namespace '$NAMESPACE'." >&2 err "ERROR: Secret 'prole-db-superuser' not found in namespace '$NAMESPACE'."
exit 1 exit 1
fi fi
} }
apply_manifests() { detect_supabase_home() {
for f in "${SUPABASE_FILES[@]}"; do if [[ -n "$SUPABASE_HOME" && -d "$SUPABASE_HOME" ]]; then
echo "$SUPABASE_HOME"
return 0
fi
if [[ -d "$HOME/prole/supabase" ]]; then
echo "$HOME/prole/supabase"
return 0
fi
if [[ -d "$HOME/dev/supabase" ]]; then
echo "$HOME/dev/supabase"
return 0
fi
if [[ -n "${PROLE_HOME:-}" && -d "$PROLE_HOME/../supabase" ]]; then
echo "$PROLE_HOME/../supabase"
return 0
fi
return 1
}
get_db_password() {
kubectl -n "$NAMESPACE" get secret prole-db-superuser -o jsonpath='{.data.password}' 2>/dev/null | base64 -d
}
set_env_kv() {
local file="$1" key="$2" value="$3"
# Ensure value is quoted if it contains spaces and is not already quoted
if [[ "$value" == *" "* && ! "$value" =~ ^\".*\"$ && ! "$value" =~ ^\'.*\'$ ]]; then
value="\"$value\""
fi
if grep -q "^${key}=" "$file" 2>/dev/null; then
sed -i.bak "s|^${key}=.*|${key}=${value}|" "$file"
else
printf "%s=%s\n" "$key" "$value" >> "$file"
fi
}
build_env_file() {
local supa_home="$1"
local docker_dir="$supa_home/docker"
local env_base="$docker_dir/.env"
local env_example="$docker_dir/.env.example"
local env_out="$SUPABASE_K8S_DIR/.env"
local db_pass
db_pass=$(get_db_password || true)
mkdir -p "$SUPABASE_K8S_DIR"
if [[ -f "$env_base" ]]; then
cp "$env_base" "$env_out"
elif [[ -f "$env_example" ]]; then
cp "$env_example" "$env_out"
else
err "ERROR: Supabase .env or .env.example not found in $docker_dir"
exit 1
fi
# If we used .env.example, generate required secrets
if [[ ! -f "$env_base" ]]; then
if [[ -x "$docker_dir/utils/generate-keys.sh" ]]; then
log "Generating Supabase secrets from utils/generate-keys.sh ..."
local gen_out
gen_out=$(bash "$docker_dir/utils/generate-keys.sh" </dev/null || true)
while IFS= read -r line; do
[[ "$line" =~ ^[A-Z0-9_]+= ]] || continue
local k="${line%%=*}"
local v="${line#*=}"
set_env_kv "$env_out" "$k" "$v"
done <<< "$gen_out"
fi
fi
if [[ -n "$db_pass" ]]; then
set_env_kv "$env_out" "POSTGRES_PASSWORD" "$db_pass"
fi
set_env_kv "$env_out" "POSTGRES_HOST" "$SUPABASE_POSTGRES_HOST"
set_env_kv "$env_out" "POSTGRES_DB" "$SUPABASE_POSTGRES_DB"
set_env_kv "$env_out" "POSTGRES_PORT" "$SUPABASE_POSTGRES_PORT"
}
convert_compose_to_k8s() {
local supa_home="$1"
local docker_dir="$supa_home/docker"
local compose_file="$docker_dir/docker-compose.yml"
local dev_compose="$docker_dir/dev/docker-compose.dev.yml"
if [[ ! -f "$compose_file" ]]; then
err "ERROR: docker-compose.yml not found at $compose_file"
exit 1
fi
rm -rf "$SUPABASE_K8S_DIR"
mkdir -p "$SUPABASE_K8S_DIR"
build_env_file "$supa_home"
# Export .env for kompose interpolation
set -a
# Fix unquoted values with spaces in the .env file before sourcing
# Use python for robust .env parsing and quoting
python3 - <<PY
import pathlib
import re
env_file = pathlib.Path("$SUPABASE_K8S_DIR/.env")
if env_file.exists():
content = env_file.read_text()
new_lines = []
for line in content.splitlines():
# Match KEY=VALUE where VALUE has spaces and is not quoted
m = re.match(r'^([A-Z0-9_]+)=([^"\'].* .*)$', line)
if m:
key, val = m.groups()
new_lines.append(f'{key}="{val}"')
else:
new_lines.append(line)
env_file.write_text("\n".join(new_lines) + "\n")
PY
# shellcheck disable=SC1090
source "$SUPABASE_K8S_DIR/.env"
set +a
local files=("-f" "$compose_file")
if [[ "$SUPABASE_USE_DEV_COMPOSE" == "1" && -f "$dev_compose" ]]; then
files+=("-f" "$dev_compose")
fi
log "Converting Supabase docker-compose to Kubernetes manifests for namespace 'supabase' ..."
kompose "${files[@]}" convert -n "supabase" -o "$SUPABASE_K8S_DIR" --volumes=configMap --suppress-warnings
# Remove the built-in Supabase DB workloads (we use CNPG instead)
rm -f "$SUPABASE_K8S_DIR"/db-*.yaml "$SUPABASE_K8S_DIR"/db*.yaml 2>/dev/null || true
# Enforce imagePullPolicy
python3 - <<PY
import pathlib
import re
k8s_dir = pathlib.Path("$SUPABASE_K8S_DIR")
policy = "$SUPABASE_IMAGE_PULL_POLICY"
def patch_file(path: pathlib.Path):
lines = path.read_text().splitlines()
out = []
for line in lines:
out.append(line)
m = re.match(r'^(\s*)image:\s*.+', line)
if m:
indent = m.group(1)
out.append(f"{indent}imagePullPolicy: {policy}")
path.write_text("\n".join(out) + "\n")
# Sanitize container names (must not contain dots)
def sanitize_container_names(path: pathlib.Path):
content = path.read_text()
# Find container names that contain dots and replace dots with hyphens
# Look for "name: some.name" within container blocks
def repl(match):
indent = match.group(1)
name = match.group(2)
sanitized = name.replace('.', '-')
return f"{indent}name: {sanitized}"
# Match name: value where value has dots, but only if it looks like a container name
# We can be more specific by looking for lines that start with - name: or just name: inside containers
# Updated regex to be more comprehensive for container names in various K8s objects
# 1. Matches '- name: realtime-dev.supabase-realtime'
new_content = re.sub(r'(^\s+-\s+name:\s+)([a-zA-Z0-9.-]+)', repl, content, flags=re.MULTILINE)
# 2. Matches 'name: realtime-dev.supabase-realtime' (if not already matched by 1)
new_content = re.sub(r'(^\s+name:\s+)([a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+)', repl, new_content, flags=re.MULTILINE)
# 3. Final safety check: if we are in a containers: block, dots are NOT allowed in names.
# This is a bit more aggressive but safer.
lines = new_content.splitlines()
final_lines = []
in_containers = False
for line in lines:
if 'containers:' in line:
in_containers = True
# If we are in containers block, look for - name: and sanitize it
if in_containers:
# Use a specific match for - name: to avoid mangling other things
if re.match(r'^\s+-\s+name:\s+', line):
line = re.sub(r'(\s+-\s+name:\s+)([a-zA-Z0-9.-]+)', repl, line)
# If we hit another top level key (no indent), we might be out of spec/containers
elif line and not line.startswith(' '):
in_containers = False
final_lines.append(line)
new_content = "\n".join(final_lines) + "\n"
if content != new_content:
path.write_text(new_content)
for path in k8s_dir.glob("*.yaml"):
patch_file(path)
sanitize_container_names(path)
PY
}
apply_k8s_resources() {
log "Applying Supabase resources to namespace 'supabase' ..."
kubectl apply -n "supabase" -f "$SUPABASE_K8S_DIR"
# Create an alias service 'supabase-db' in the CURRENT namespace that points to Supabase Postgres in 'supabase' namespace
# This allows prole-db (in current namespace) to connect to Supabase
cat <<EOF | kubectl apply -n "$NAMESPACE" -f -
apiVersion: v1
kind: Service
metadata:
name: supabase-db
labels:
app: supabase-db-federated
spec:
type: ExternalName
externalName: db.supabase.svc.cluster.local
EOF
# Also create an alias 'db' in 'supabase' namespace pointing to prole-db in CURRENT namespace
# This allows Supabase components to use prole-db as their primary DB
cat <<EOF | kubectl apply -n "supabase" -f -
apiVersion: v1
kind: Service
metadata:
name: db
labels:
app: prole-db-alias
spec:
type: ExternalName
externalName: prole-db-rw.${NAMESPACE}.svc.cluster.local
EOF
}
apply_db_migrations() {
if [[ "$SUPABASE_APPLY_DB_MIGRATIONS" != "1" ]]; then
log "SUPABASE_APPLY_DB_MIGRATIONS=0; skipping Supabase DB migrations."
return
fi
local supa_home="$1"
local sql_dir="$supa_home/docker/volumes/db"
if [[ ! -d "$sql_dir" ]]; then
err "WARN: Supabase SQL directory not found: $sql_dir"
return
fi
local primary
primary=$(kubectl -n "$NAMESPACE" get cluster "$CNPG_CLUSTER_NAME" -o jsonpath='{.status.currentPrimary}' 2>/dev/null || true)
if [[ -z "$primary" ]]; then
primary=$(kubectl -n "$NAMESPACE" get pods -l "cnpg.io/cluster=$CNPG_CLUSTER_NAME" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)
fi
if [[ -z "$primary" ]]; then
err "WARN: Unable to locate primary CNPG pod; skipping migrations."
return
fi
local db_pass
db_pass=$(get_db_password || true)
if [[ -z "$db_pass" ]]; then
err "WARN: Unable to read prole-db-superuser password; skipping migrations."
return
fi
log "Applying Supabase SQL migrations to CNPG (${SUPABASE_POSTGRES_DB}) ..."
local files=(
"$sql_dir/_supabase.sql"
"$sql_dir/webhooks.sql"
"$sql_dir/realtime.sql"
"$sql_dir/logs.sql"
"$sql_dir/pooler.sql"
"$sql_dir/jwt.sql"
"$sql_dir/roles.sql"
)
for f in "${files[@]}"; do
if [[ -f "$f" ]]; then if [[ -f "$f" ]]; then
kubectl apply -n "$NAMESPACE" -f "$f" log "Applying $(basename "$f") ..."
else kubectl -n "$NAMESPACE" exec -i "$primary" -c postgres -- \
echo "ERROR: Missing manifest: $f" >&2 env PGPASSWORD="$db_pass" psql -U postgres -d "$SUPABASE_POSTGRES_DB" -v ON_ERROR_STOP=1 -f - < "$f" || true
exit 1
fi fi
done done
} }
delete_manifests() { delete_k8s_resources() {
for f in "${SUPABASE_FILES[@]}"; do if [[ -d "$SUPABASE_K8S_DIR" ]]; then
if [[ -f "$f" ]]; then kubectl delete -n "supabase" -f "$SUPABASE_K8S_DIR" --ignore-not-found || true
kubectl delete -n "$NAMESPACE" -f "$f" --ignore-not-found fi
fi kubectl delete -n "$NAMESPACE" svc/supabase-db --ignore-not-found || true
done kubectl delete -n "supabase" svc/db --ignore-not-found || true
} }
status() { status() {
ensure_tools ensure_tools
echo "Supabase status in namespace '$NAMESPACE':" log "Supabase status in namespace 'supabase':"
kubectl get deploy supabase -n "$NAMESPACE" || true kubectl get deploy,svc -n "supabase" | grep -E "supabase|kong|auth|rest|realtime|storage|meta|analytics|vector|imgproxy|functions|edge|pooler|db" || true
kubectl get svc supabase -n "$NAMESPACE" || true log "Federated services in namespace '$NAMESPACE':"
kubectl get svc -n "$NAMESPACE" | grep supabase-db || true
} }
case "$ACTION" in case "$ACTION" in
start) start)
ensure_tools ensure_tools
ensure_prereqs ensure_prereqs
echo "Applying Supabase manifests in namespace '$NAMESPACE'..." SUPABASE_HOME="$(detect_supabase_home)" || { err "ERROR: Supabase repo not found. Set SUPABASE_HOME or symlink ~/prole/supabase."; exit 1; }
apply_manifests log "Using Supabase repo: $SUPABASE_HOME"
log "Using namespace: $NAMESPACE"
convert_compose_to_k8s "$SUPABASE_HOME"
apply_k8s_resources
apply_db_migrations "$SUPABASE_HOME"
;; ;;
stop) stop)
ensure_tools ensure_tools
echo "Deleting Supabase manifests from namespace '$NAMESPACE'..." ensure_namespace
delete_manifests delete_k8s_resources
;; ;;
restart) restart)
ensure_tools ensure_tools
ensure_prereqs ensure_prereqs
echo "Re-applying Supabase manifests in namespace '$NAMESPACE'..." SUPABASE_HOME="$(detect_supabase_home)" || { err "ERROR: Supabase repo not found. Set SUPABASE_HOME or symlink ~/prole/supabase."; exit 1; }
apply_manifests convert_compose_to_k8s "$SUPABASE_HOME"
delete_k8s_resources
apply_k8s_resources
apply_db_migrations "$SUPABASE_HOME"
;; ;;
status) status)
status status

527
install.py
View File

@ -105,6 +105,330 @@ def _expand_path(val: str | None) -> str:
return os.path.expandvars(os.path.expanduser(str(val))) return os.path.expandvars(os.path.expanduser(str(val)))
def _resolve_supabase_home(project_root: Path) -> Path | None:
env_home = (os.environ.get('SUPABASE_HOME') or '').strip()
if env_home:
try:
candidate = Path(_expand_path(env_home))
if candidate.is_dir():
return candidate
except Exception:
pass
candidates = [
Path.home() / 'prole' / 'supabase',
Path.home() / 'dev' / 'supabase',
project_root / 'supabase',
]
prole_home = (os.environ.get('PROLE_HOME') or '').strip()
if prole_home:
try:
candidates.append(Path(_expand_path(prole_home)).parent / 'supabase')
except Exception:
pass
for candidate in candidates:
try:
if candidate.is_dir():
return candidate
except Exception:
continue
return None
def _collect_images_from_files(paths: list[Path]) -> set[str]:
images = set()
for path in paths:
if not path or not path.exists():
continue
try:
for line in path.read_text().splitlines():
m = re.match(r'^\s*image:\s*([^\s#]+)', line)
if not m:
continue
img = m.group(1).strip().strip('"').strip("'")
if not img:
continue
img = os.path.expandvars(img)
if '${' in img or '}' in img or '$' in img:
continue
images.add(img)
except Exception:
continue
return images
def _clean_yaml_value(raw: str) -> str:
if raw is None:
return ''
val = raw.strip()
if '#' in val:
val = val.split('#', 1)[0].strip()
if len(val) >= 2 and ((val[0] == val[-1]) and val.startswith(("'", '"'))):
val = val[1:-1]
return val.strip()
def _parse_yaml_scalar_values(path: Path, keys: set[str]) -> dict:
data = {}
if not path.exists():
return data
try:
for raw in path.read_text().splitlines():
line = raw.strip()
if not line or line.startswith('#') or line == '---':
continue
if line.startswith('- '):
continue
if ':' not in line:
continue
key, val = line.split(':', 1)
key = key.strip()
if key not in keys:
continue
clean_val = _clean_yaml_value(val)
if clean_val:
data[key] = clean_val
except Exception:
pass
return data
def _parse_internal_a_records(path: Path) -> tuple[str, dict, dict]:
domain = ''
ip_map: dict[str, str] = {}
fqdn_map: dict[str, str] = {}
if not path.exists():
return domain, ip_map, fqdn_map
try:
lines = path.read_text().splitlines()
except Exception:
return domain, ip_map, fqdn_map
in_block = False
current: dict[str, str] = {}
records = []
for raw in lines:
if not in_block:
s = raw.strip()
if s.startswith('prole_domain:'):
domain = _clean_yaml_value(s.split(':', 1)[1])
if s.startswith('prole_internal_a_records:'):
in_block = True
continue
if raw and not raw.startswith((' ', '\t')):
break
s = raw.strip()
if not s:
continue
if s.startswith('- '):
if current:
records.append(current)
current = {}
s = s[2:].strip()
if ':' in s:
key, val = s.split(':', 1)
key = key.strip()
if key in ('fqdn', 'ipv4'):
current[key] = _clean_yaml_value(val)
if current:
records.append(current)
for rec in records:
fqdn = rec.get('fqdn')
ip = rec.get('ipv4')
if not fqdn or not ip:
continue
fqdn_map[fqdn] = ip
ip_map[fqdn] = ip
short = fqdn.split('.')[0]
if short and short not in ip_map:
ip_map[short] = ip
return domain, ip_map, fqdn_map
def _parse_ansible_inventory_hosts(path: Path) -> dict:
groups: dict[str, list[str]] = {}
if not path.exists():
return groups
try:
current_group = None
for raw in path.read_text().splitlines():
line = raw.strip()
if not line or line.startswith(('#', ';')):
continue
if line.startswith('[') and line.endswith(']'):
current_group = line[1:-1].strip()
groups.setdefault(current_group, [])
continue
if current_group is None:
continue
host = line.split()[0]
if host and host not in groups[current_group]:
groups[current_group].append(host)
except Exception:
pass
return groups
def _resolve_host_ip(host: str, domain: str, ip_map: dict) -> str:
if not host:
return ''
if host in ip_map:
return ip_map[host]
if domain:
if '.' not in host:
fqdn = f"{host}.{domain}"
if fqdn in ip_map:
return ip_map[fqdn]
else:
short = host.split('.')[0]
if short in ip_map:
return ip_map[short]
return ''
def _detect_ansible_topology(project_root: Path) -> dict:
prole_home = (os.environ.get('PROLE_HOME') or '').strip()
base_candidates = []
if prole_home:
base_candidates.append(Path(_expand_path(prole_home)))
base_candidates.append(project_root)
infra_path = None
for base in base_candidates:
try:
candidate = base / 'infrastructure'
if candidate.is_dir():
infra_path = candidate
break
except Exception:
continue
if infra_path is None:
return {}
inventory_path = infra_path / 'inventory'
if not inventory_path.is_dir():
return {}
groups = _parse_ansible_inventory_hosts(inventory_path / 'hosts.ini')
vars_vals = _parse_yaml_scalar_values(
inventory_path / 'group_vars' / 'all' / 'vars.yml',
{'ad_dc_ip', 'prole_domain', 'kerberos_realm', 'krb5_realm', 'kerberos_kdc', 'krb5_kdc', 'kerberos_kdc_ip'}
)
domain = vars_vals.get('prole_domain', '')
ad_dc_ip = vars_vals.get('ad_dc_ip', '')
explicit_realm = vars_vals.get('kerberos_realm') or vars_vals.get('krb5_realm') or ''
explicit_kdc = vars_vals.get('kerberos_kdc') or vars_vals.get('krb5_kdc') or vars_vals.get('kerberos_kdc_ip') or ''
if explicit_kdc and not ad_dc_ip:
ad_dc_ip = explicit_kdc
dns_domain, ip_map, fqdn_records = _parse_internal_a_records(inventory_path / 'group_vars' / 'all' / 'dns.yml')
if not domain and dns_domain:
domain = dns_domain
ad_vars = _parse_yaml_scalar_values(
inventory_path / 'group_vars' / 'ad_dc' / 'vars.yml',
{'samba_dns_server'}
)
samba_dns_server = ad_vars.get('samba_dns_server', '')
ad_dc_host = ''
ad_dc_hosts = groups.get('ad_dc') or []
if ad_dc_hosts:
ad_dc_host = ad_dc_hosts[0]
if samba_dns_server:
ad_dc_host = ad_dc_host or samba_dns_server
if not ad_dc_ip:
ad_dc_ip = _resolve_host_ip(samba_dns_server, domain, ip_map)
if not ad_dc_ip and ad_dc_host:
ad_dc_ip = _resolve_host_ip(ad_dc_host, domain, ip_map)
kdc_ip = explicit_kdc or ad_dc_ip or ''
realm = explicit_realm or (domain.upper() if domain else '')
all_hosts = set()
for hosts in groups.values():
all_hosts.update(hosts)
if ad_dc_host:
all_hosts.add(ad_dc_host)
host_ip_map = {}
unmapped_hosts = []
for host in sorted(all_hosts):
ip = _resolve_host_ip(host, domain, ip_map)
if ip:
host_ip_map[host] = ip
else:
unmapped_hosts.append(host)
topology = {
'domain': domain,
'realm': realm,
'internal_records': fqdn_records,
'ad_dc': {
'host': ad_dc_host,
'ip': ad_dc_ip,
},
'groups': groups,
'hosts': host_ip_map,
'unmapped_hosts': unmapped_hosts,
}
try:
topology_json = json.dumps(topology, separators=(',', ':'))
except Exception:
topology_json = ''
return {
'infrastructure_path': str(infra_path),
'inventory_path': str(inventory_path),
'domain': domain,
'realm': realm,
'ad_dc_ip': ad_dc_ip,
'ad_dc_host': ad_dc_host,
'kdc_ip': kdc_ip,
'groups': groups,
'hosts': host_ip_map,
'unmapped_hosts': unmapped_hosts,
'topology': topology,
'topology_json': topology_json,
}
def _format_ansible_topology_summary(info: dict) -> str:
if not info:
return ''
lines = []
inv = info.get('inventory_path') or ''
if inv:
lines.append(f"Ansible inventory detected at {inv}.")
domain = info.get('domain') or ''
realm = info.get('realm') or ''
if domain or realm:
if domain and realm:
lines.append(f"Domain: {domain} (Realm: {realm})")
elif domain:
lines.append(f"Domain: {domain}")
else:
lines.append(f"Realm: {realm}")
ad_dc_host = info.get('ad_dc_host') or ''
ad_dc_ip = info.get('ad_dc_ip') or ''
if ad_dc_host or ad_dc_ip:
if ad_dc_host and ad_dc_ip:
lines.append(f"AD DC: {ad_dc_host} -> {ad_dc_ip}")
elif ad_dc_ip:
lines.append(f"AD DC IP: {ad_dc_ip}")
else:
lines.append(f"AD DC Host: {ad_dc_host}")
groups = info.get('groups') or {}
if groups:
group_bits = []
for name in sorted(groups.keys()):
group_bits.append(f"{name}({len(groups[name])})")
lines.append("Groups: " + ", ".join(group_bits))
hosts = info.get('hosts') or {}
if hosts:
host_items = sorted(hosts.items())
preview = host_items[:8]
host_str = ", ".join([f"{h}={ip}" for h, ip in preview])
if len(host_items) > len(preview):
host_str += f", +{len(host_items) - len(preview)} more"
lines.append("Hosts: " + host_str)
unmapped = info.get('unmapped_hosts') or []
if unmapped:
lines.append(f"Hosts without IPs: {len(unmapped)}")
return "\n".join(lines)
def _render_prole_cfg(inputs: dict, globals_to_save: dict, sections: dict, generated_at: str | None = None) -> str: def _render_prole_cfg(inputs: dict, globals_to_save: dict, sections: dict, generated_at: str | None = None) -> str:
content = [] content = []
content.append('; Prole Master Configuration File') content.append('; Prole Master Configuration File')
@ -164,7 +488,7 @@ class ProleController:
pg_version_file = self.project_root / "conf" / "postgresql" / ".version" pg_version_file = self.project_root / "conf" / "postgresql" / ".version"
release_file = self.project_root / "prole-db" / ".version" release_file = self.project_root / "prole-db" / ".version"
pg_version = pg_version_file.read_text().strip() if pg_version_file.exists() else "17.7" pg_version = pg_version_file.read_text().strip() if pg_version_file.exists() else "17.7"
release = release_file.read_text().strip() if release_file.exists() else "41" release = release_file.read_text().strip() if release_file.exists() else "43"
if release.isdigit(): if release.isdigit():
release = release.zfill(3) release = release.zfill(3)
return f"{pg_version}-{release}" return f"{pg_version}-{release}"
@ -499,6 +823,12 @@ class ProleInstaller:
self.kerberos_kdc = tk.StringVar() self.kerberos_kdc = tk.StringVar()
self.supabase_enabled = tk.BooleanVar(value=False) self.supabase_enabled = tk.BooleanVar(value=False)
self.at_rest_encryption_enabled = tk.BooleanVar(value=False) self.at_rest_encryption_enabled = tk.BooleanVar(value=False)
self.ansible_topology = {}
self.ansible_topology_summary = ''
try:
self._apply_ansible_topology_defaults()
except Exception:
pass
# Create pages # Create pages
self.page_frames = {} self.page_frames = {}
@ -684,6 +1014,43 @@ class ProleInstaller:
except Exception: except Exception:
pass pass
def _apply_ansible_topology_defaults(self):
info = _detect_ansible_topology(PROJECT_ROOT)
if not info:
return
self.ansible_topology = info
self.ansible_topology_summary = _format_ansible_topology_summary(info)
try:
net = self.prole_cfg_data.get('Network', {})
if info.get('topology_json'):
net['ANSIBLE_TOPOLOGY'] = info['topology_json']
if info.get('inventory_path'):
net['ANSIBLE_INVENTORY'] = info['inventory_path']
if info.get('infrastructure_path'):
net['ANSIBLE_INFRASTRUCTURE'] = info['infrastructure_path']
if info.get('domain'):
net['ANSIBLE_DOMAIN'] = info['domain']
if info.get('realm'):
net['ANSIBLE_REALM'] = info['realm']
if info.get('ad_dc_host'):
net['AD_DC_HOST'] = info['ad_dc_host']
if info.get('ad_dc_ip'):
net['AD_DC_IP'] = info['ad_dc_ip']
if info.get('kdc_ip'):
net['KDC_ANSIBLE_DETECTED'] = info['kdc_ip']
self.prole_cfg_data['Network'] = net
except Exception:
pass
try:
if not self.kerberos_kdc.get().strip() and info.get('kdc_ip'):
self.kerberos_kdc.set(info['kdc_ip'])
self.kerberos_enabled.set(True)
if not self.kerberos_realm.get().strip() and info.get('realm'):
self.kerberos_realm.set(info['realm'])
except Exception:
pass
def _create_sidebar_nav(self): def _create_sidebar_nav(self):
"""Create the left-hand navigation menu.""" """Create the left-hand navigation menu."""
tk.Label(self.sidebar, text="INSTALLER", bg='#F5F5DC', fg='#8B8B7A', tk.Label(self.sidebar, text="INSTALLER", bg='#F5F5DC', fg='#8B8B7A',
@ -1901,12 +2268,12 @@ class ProleInstaller:
font=('SF Pro Text', 18), anchor='ne') font=('SF Pro Text', 18), anchor='ne')
self._render_title('Network Configuration Scan', y=150) self._render_title('Network Configuration Scan', y=150)
self._render_paragraph("The network prole-scan will detect Kerberos services, Active Directory controllers, and other configuration details required for deployment.", y=210) self._render_paragraph("The network prole-agent will detect Kerberos services, Active Directory controllers, and other configuration details required for deployment.", y=210)
y = 280 y = 280
self.scan_status_var = tk.StringVar(value="Ready to prole-scan") self.scan_status_var = tk.StringVar(value="Ready to prole-agent")
# Ready to scan text (drawn on canvas) # Ready to scan text (drawn on canvas)
status_item = ui.canvas_text(self, 48, y, "Ready to prole-scan", fill='black', font=('SF Pro Text', 12)) status_item = ui.canvas_text(self, 48, y, "Ready to prole-agent", fill='black', font=('SF Pro Text', 12))
# Link variable to canvas text # Link variable to canvas text
def update_status_text(*args): def update_status_text(*args):
@ -1919,6 +2286,10 @@ class ProleInstaller:
# Standardized Console Output # Standardized Console Output
self.scan_results_console = self._create_console_output(y=320, title="Scan Output", width=880, height=380) self.scan_results_console = self._create_console_output(y=320, title="Scan Output", width=880, height=380)
self.scan_results_text = self.scan_results_console.text self.scan_results_text = self.scan_results_console.text
if getattr(self, 'ansible_topology_summary', ''):
self.scan_results_console.write(self.ansible_topology_summary + "\n")
self.scan_results_console.write("Ansible topology loaded; network scan can refine detection.\n")
y = 740 y = 740
@ -1973,7 +2344,7 @@ class ProleInstaller:
self._scan_running = True self._scan_running = True
self.scan_results_console.clear() self.scan_results_console.clear()
self.scan_results_console.write("Initializing network scan using prole-net/prole-scan ...\n") self.scan_results_console.write("Initializing network scan using prole-net/prole-agent ...\n")
self.scan_status_var.set("Scanning...") self.scan_status_var.set("Scanning...")
# Start animation # Start animation
@ -1983,7 +2354,7 @@ class ProleInstaller:
def worker(): def worker():
try: try:
# Use the new scan binary # Use the new scan binary
scan_binary = get_resource_path("prole-net/prole-scan") scan_binary = get_resource_path("prole-net/prole-agent")
if not scan_binary.exists(): if not scan_binary.exists():
self.safe_after(lambda: self.scan_results_console.write(f"Scan binary not found at {scan_binary}\n")) self.safe_after(lambda: self.scan_results_console.write(f"Scan binary not found at {scan_binary}\n"))
self.safe_after(lambda: self.scan_status_var.set("Scan failed")) self.safe_after(lambda: self.scan_status_var.set("Scan failed"))
@ -1998,7 +2369,7 @@ class ProleInstaller:
# Run scan from writable directory with 10s timeout and summary analysis # Run scan from writable directory with 10s timeout and summary analysis
# Set bufsize=0 for truly unbuffered binary stream reading # Set bufsize=0 for truly unbuffered binary stream reading
# Ensure PYTHONUNBUFFERED=1 is set in case prole-scan is/uses Python # Ensure PYTHONUNBUFFERED=1 is set in case prole-agent is/uses Python
env = os.environ.copy() env = os.environ.copy()
env['PYTHONUNBUFFERED'] = '1' env['PYTHONUNBUFFERED'] = '1'
process = subprocess.Popen([str(scan_binary), "-t", "10", "-s"], process = subprocess.Popen([str(scan_binary), "-t", "10", "-s"],
@ -3189,6 +3560,7 @@ class ProleInstaller:
('CloudNative-PG', 'init_cloudnative_pg.sh'), ('CloudNative-PG', 'init_cloudnative_pg.sh'),
] ]
if self.kerberos_enabled.get(): if self.kerberos_enabled.get():
scripts.append(('Authority', 'init_authority.sh'))
scripts.append(('Kerberos Realm', 'init_kerberos.sh')) scripts.append(('Kerberos Realm', 'init_kerberos.sh'))
scripts.extend([ scripts.extend([
('Prole DB', 'init_prole-db.sh'), ('Prole DB', 'init_prole-db.sh'),
@ -3548,7 +3920,49 @@ class ProleInstaller:
else: else:
self.script_consoles["init_cloudnative_pg.sh"].write("Skipping CloudNative-PG initialization because previous steps failed.\n") self.script_consoles["init_cloudnative_pg.sh"].write("Skipping CloudNative-PG initialization because previous steps failed.\n")
# 4. init_kerberos.sh initialize (optional) # 4. init_authority.sh start (optional)
if overall_success and self.kerberos_enabled.get():
script = "init_authority.sh"
if script in self.script_consoles:
_select_tab(script)
self.script_consoles[script].clear()
self.script_consoles[script].write(f"Running {script} start...\n")
log_path = _log_path_for(script)
self._record_install_log(log_path)
try:
log_fp = log_path.open('a', encoding='utf-8')
except Exception:
log_fp = None
def _auth_line(line):
self.script_consoles[script].write(line)
if log_fp:
try:
log_fp.write(line)
log_fp.flush()
except Exception:
pass
rc_auth = self.controller.run_script(
script,
args=['start'],
env=env,
on_line=_auth_line
)
if log_fp:
try:
log_fp.close()
except Exception:
pass
if rc_auth != 0:
self.script_consoles[script].write(f"\nERROR: {script} start failed with code {rc_auth}\n")
overall_success = False
else:
self.script_consoles[script].write(f"\n{script} completed successfully.\n")
# 5. init_kerberos.sh initialize (optional)
if overall_success and self.kerberos_enabled.get(): if overall_success and self.kerberos_enabled.get():
script = "init_kerberos.sh" script = "init_kerberos.sh"
if script in self.script_consoles: if script in self.script_consoles:
@ -3596,7 +4010,7 @@ class ProleInstaller:
elif not self.kerberos_enabled.get(): elif not self.kerberos_enabled.get():
if "init_kerberos.sh" in self.script_consoles: if "init_kerberos.sh" in self.script_consoles:
self.script_consoles["init_kerberos.sh"].write("Kerberos auth disabled; skipping init_kerberos.sh.\n") self.script_consoles["init_kerberos.sh"].write("Kerberos auth disabled; skipping init_kerberos.sh.\n")
# 5. init_prole-db.sh start # 6. init_prole-db.sh start
if overall_success: if overall_success:
script = "init_prole-db.sh" script = "init_prole-db.sh"
_select_tab(script) _select_tab(script)
@ -3639,7 +4053,7 @@ class ProleInstaller:
else: else:
self.script_consoles["init_prole-db.sh"].write("Skipping Prole DB initialization because previous steps failed.\n") self.script_consoles["init_prole-db.sh"].write("Skipping Prole DB initialization because previous steps failed.\n")
# 6. init_prole-db-backup.sh start # 7. init_prole-db-backup.sh start
if overall_success: if overall_success:
script = "init_prole-db-backup.sh" script = "init_prole-db-backup.sh"
_select_tab(script) _select_tab(script)
@ -3682,7 +4096,7 @@ class ProleInstaller:
else: else:
self.script_consoles["init_prole-db-backup.sh"].write("Skipping Prole DB backup because previous steps failed.\n") self.script_consoles["init_prole-db-backup.sh"].write("Skipping Prole DB backup because previous steps failed.\n")
# 7. init_supabase.sh start (optional) # 8. init_supabase.sh start
if overall_success: if overall_success:
script = "init_supabase.sh" script = "init_supabase.sh"
_select_tab(script) _select_tab(script)
@ -3718,7 +4132,7 @@ class ProleInstaller:
pass pass
if rc_sb != 0: if rc_sb != 0:
self.script_consoles[script].write(f"\nERROR: {script} start failed with code {rc_sb}\n") self.script_consoles[script].write(f"\nERROR: {script} start failed with code {rc_sb}\n")
overall_success = False # overall_success = False # Keep as False if you want to block port forwards on failure
else: else:
self.script_consoles[script].write(f"\n{script} completed successfully.\n") self.script_consoles[script].write(f"\n{script} completed successfully.\n")
else: else:
@ -3726,7 +4140,7 @@ class ProleInstaller:
else: else:
self.script_consoles["init_supabase.sh"].write("Skipping Supabase because previous steps failed.\n") self.script_consoles["init_supabase.sh"].write("Skipping Supabase because previous steps failed.\n")
# 8. init_port_forwards.sh start # 9. init_port_forwards.sh start
if overall_success: if overall_success:
script = "init_port_forwards.sh" script = "init_port_forwards.sh"
_select_tab(script) _select_tab(script)
@ -6585,16 +6999,16 @@ esac
base_dir = PROJECT_ROOT / rel_dir base_dir = PROJECT_ROOT / rel_dir
if not base_dir.exists(): if not base_dir.exists():
continue continue
for path in base_dir.glob('*.yaml'): images.update(_collect_images_from_files(list(base_dir.glob('*.yaml'))))
try:
for line in path.read_text().splitlines(): if include_supabase:
m = re.match(r'^\s*image:\s*([^\s#]+)', line) supa_home = _resolve_supabase_home(PROJECT_ROOT)
if m: if supa_home:
img = m.group(1).strip().strip('"').strip("'") docker_dir = supa_home / 'docker'
if img: compose_files = [docker_dir / 'docker-compose.yml']
images.add(img) if os.environ.get('SUPABASE_USE_DEV_COMPOSE') == '1':
except Exception: compose_files.append(docker_dir / 'dev' / 'docker-compose.dev.yml')
continue images.update(_collect_images_from_files(compose_files))
if not include_supabase: if not include_supabase:
images = {img for img in images if 'supabase' not in img} images = {img for img in images if 'supabase' not in img}
@ -6757,7 +7171,7 @@ esac
if not manifest_path.exists(): if not manifest_path.exists():
continue continue
content = manifest_path.read_text() content = manifest_path.read_text()
# Update imageName: prole-db:17.7-041 # Update imageName: prole-db:17.7-043
new_content = re.sub(r'imageName:\s*.*', f'imageName: {image_tag}', content) new_content = re.sub(r'imageName:\s*.*', f'imageName: {image_tag}', content)
# Update Kerberos realm in manifest if enabled # Update Kerberos realm in manifest if enabled
@ -7460,16 +7874,16 @@ class ProleSilentInstaller:
base_dir = self.project_root / rel_dir base_dir = self.project_root / rel_dir
if not base_dir.exists(): if not base_dir.exists():
continue continue
for path in base_dir.glob('*.yaml'): images.update(_collect_images_from_files(list(base_dir.glob('*.yaml'))))
try:
for line in path.read_text().splitlines(): if include_supabase:
m = re.match(r'^\s*image:\s*([^\s#]+)', line) supa_home = _resolve_supabase_home(self.project_root)
if m: if supa_home:
img = m.group(1).strip().strip('"').strip("'") docker_dir = supa_home / 'docker'
if img: compose_files = [docker_dir / 'docker-compose.yml']
images.add(img) if os.environ.get('SUPABASE_USE_DEV_COMPOSE') == '1':
except Exception: compose_files.append(docker_dir / 'dev' / 'docker-compose.dev.yml')
continue images.update(_collect_images_from_files(compose_files))
if not include_supabase: if not include_supabase:
images = {img for img in images if 'supabase' not in img} images = {img for img in images if 'supabase' not in img}
@ -7531,11 +7945,52 @@ class ProleSilentInstaller:
defaults = self._default_inputs() defaults = self._default_inputs()
loaded = self._load_inputs_from_cfg() loaded = self._load_inputs_from_cfg()
self.inputs = {**defaults, **loaded} self.inputs = {**defaults, **loaded}
try:
self._apply_ansible_defaults(set(loaded.keys()))
except Exception:
pass
for k in ('env_setup.PROLE_HOME', 'env_setup.PROLE_CONF', 'env_setup.PROLE_DATA', 'env_setup.PROLE_LOGS', 'env_setup.PROLE_SERVICE'): for k in ('env_setup.PROLE_HOME', 'env_setup.PROLE_CONF', 'env_setup.PROLE_DATA', 'env_setup.PROLE_LOGS', 'env_setup.PROLE_SERVICE'):
if k in self.inputs: if k in self.inputs:
self.inputs[k] = _expand_path(self.inputs[k]) self.inputs[k] = _expand_path(self.inputs[k])
def _apply_ansible_defaults(self, loaded_keys: set[str] | None = None):
info = _detect_ansible_topology(self.project_root)
if not info:
return
loaded_keys = loaded_keys or set()
try:
net = self.prole_cfg_data.get('Network', {})
if info.get('topology_json'):
net['ANSIBLE_TOPOLOGY'] = info['topology_json']
if info.get('inventory_path'):
net['ANSIBLE_INVENTORY'] = info['inventory_path']
if info.get('infrastructure_path'):
net['ANSIBLE_INFRASTRUCTURE'] = info['infrastructure_path']
if info.get('domain'):
net['ANSIBLE_DOMAIN'] = info['domain']
if info.get('realm'):
net['ANSIBLE_REALM'] = info['realm']
if info.get('ad_dc_host'):
net['AD_DC_HOST'] = info['ad_dc_host']
if info.get('ad_dc_ip'):
net['AD_DC_IP'] = info['ad_dc_ip']
if info.get('kdc_ip'):
net['KDC_ANSIBLE_DETECTED'] = info['kdc_ip']
self.prole_cfg_data['Network'] = net
except Exception:
pass
if 'kerberos_config.kdc' not in loaded_keys:
if not self._get_input('kerberos_config.kdc', '').strip() and info.get('kdc_ip'):
self.inputs['kerberos_config.kdc'] = info['kdc_ip']
if 'kerberos_config.realm' not in loaded_keys:
if not self._get_input('kerberos_config.realm', '').strip() and info.get('realm'):
self.inputs['kerberos_config.realm'] = info['realm']
if 'kerberos_config.enabled' not in loaded_keys:
if info.get('kdc_ip') and not self._get_input_bool('kerberos_config.enabled', False):
self.inputs['kerberos_config.enabled'] = _bool_str(True)
def _write_cfg(self): def _write_cfg(self):
globals_to_save = { globals_to_save = {
'PROLE_HOME': self._get_input('env_setup.PROLE_HOME', ''), 'PROLE_HOME': self._get_input('env_setup.PROLE_HOME', ''),
@ -7608,8 +8063,8 @@ class ProleSilentInstaller:
if not self._get_input_bool('network_scan.run', DEFAULT_ACTION_FLAGS.get('network_scan.run', True)): if not self._get_input_bool('network_scan.run', DEFAULT_ACTION_FLAGS.get('network_scan.run', True)):
self.log("[SKIP] Network scan disabled.") self.log("[SKIP] Network scan disabled.")
return return
self.log("==> Network scan (prole-scan)") self.log("==> Network scan (prole-agent)")
scan_binary = get_resource_path("prole-net/prole-scan") scan_binary = get_resource_path("prole-net/prole-agent")
if not scan_binary.exists(): if not scan_binary.exists():
self.err(f"[ERROR] Scan binary not found at {scan_binary}") self.err(f"[ERROR] Scan binary not found at {scan_binary}")
return return

2
installer.spec
View File

@ -22,7 +22,7 @@ datas = [
# Binaries to include (with execute permissions) # Binaries to include (with execute permissions)
binaries = [ binaries = [
('prole-net/prole-scan', 'prole-net'), ('prole-net/prole-agent', 'prole-net'),
] ]
# Hidden imports # Hidden imports

4
k8s/openbao/kerberos-configmap.yaml
View File

@ -12,8 +12,8 @@ data:
[realms] [realms]
PROLE.ORG = { PROLE.ORG = {
kdc = 10.0.0.3 kdc = kdc.prole.org
admin_server = 10.0.0.3 admin_server = kdc.prole.org
} }
[domain_realm] [domain_realm]

2
k8s/prole/prole-db-recovery.yaml.tpl
View File

@ -5,7 +5,7 @@ metadata:
name: prole-db name: prole-db
spec: spec:
instances: 3 instances: 3
imageName: prole-db:17.7-041 imageName: prole-db:17.7-053
postgresUID: 100 postgresUID: 100
postgresGID: 101 postgresGID: 101
maxSyncReplicas: 1 maxSyncReplicas: 1

2
k8s/prole/prole-db.yaml
View File

@ -4,7 +4,7 @@ metadata:
name: prole-db name: prole-db
spec: spec:
instances: 3 instances: 3
imageName: prole-db:17.7-041 imageName: prole-db:17.7-053
postgresUID: 100 postgresUID: 100
postgresGID: 101 postgresGID: 101
maxSyncReplicas: 1 maxSyncReplicas: 1

2
prole-db/.version
View File

@ -1 +1 @@
41 53

14
prole-db/Dockerfile
View File

@ -62,16 +62,6 @@ RUN set -eux; \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
cmake \ cmake \
libssl-dev \ libssl-dev \
libkrb5-dev \
krb5-user \
sssd \
sssd-krb5 \
sssd-krb5-common \
sssd-tools \
libpam-sss \
libnss-sss \
oddjob \
oddjob-mkhomedir \
libgdal-dev \ libgdal-dev \
libproj-dev \ libproj-dev \
libgeos-dev \ libgeos-dev \
@ -84,7 +74,7 @@ RUN set -eux; \
# Copy and install pg_prolelog from local .deb package # Copy and install pg_prolelog from local .deb package
ARG TARGETARCH ARG TARGETARCH
COPY pg_prolelog/percona-postgresql-17-prolelog_1.0-1_${TARGETARCH:-amd64}.deb /tmp/pg_prolelog.deb COPY pg_prolelog/percona-postgresql-17-pg-prolelog_1.0-1_${TARGETARCH:-amd64}.deb /tmp/pg_prolelog.deb
RUN set -eux; \ RUN set -eux; \
apt-get update; \ apt-get update; \
apt-get install -y --no-install-recommends /tmp/pg_prolelog.deb; \ apt-get install -y --no-install-recommends /tmp/pg_prolelog.deb; \
@ -99,6 +89,7 @@ RUN set -eux; \
percona-postgresql-17-pgvector \ percona-postgresql-17-pgvector \
percona-postgresql-17-postgis-3 \ percona-postgresql-17-postgis-3 \
percona-postgresql-contrib-17 \ percona-postgresql-contrib-17 \
percona-postgresql-contrib \
freetds-dev \ freetds-dev \
; \ ; \
rm -rf /var/lib/apt/lists/* rm -rf /var/lib/apt/lists/*
@ -123,7 +114,6 @@ RUN set -eux; \
echo " CREATE EXTENSION IF NOT EXISTS tds_fdw;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \ echo " CREATE EXTENSION IF NOT EXISTS tds_fdw;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \
echo " CREATE EXTENSION IF NOT EXISTS postgis;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \ echo " CREATE EXTENSION IF NOT EXISTS postgis;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \
echo " CREATE EXTENSION IF NOT EXISTS postgis_topology;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \ echo " CREATE EXTENSION IF NOT EXISTS postgis_topology;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \
echo " CREATE EXTENSION IF NOT EXISTS postgis_tiger_geocoder;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \
echo " CREATE EXTENSION IF NOT EXISTS pg_prolelog;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \ echo " CREATE EXTENSION IF NOT EXISTS pg_prolelog;" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \
echo "EOSQL" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \ echo "EOSQL" >> /docker-entrypoint-initdb.d/20_create_extensions.sh; \
chmod +x /docker-entrypoint-initdb.d/20_create_extensions.sh chmod +x /docker-entrypoint-initdb.d/20_create_extensions.sh

BIN
prole-db/pg_prolelog/percona-postgresql-17-pg-prolelog_1.0-1_arm64.deb Normal file
View File

Binary file not shown.

BIN
prole-db/pg_prolelog/percona-postgresql-17-prolelog_1.0-1_amd64.deb
View File

Binary file not shown.

BIN
prole-db/pg_prolelog/percona-postgresql-17-prolelog_1.0-1_arm64.deb
View File

Binary file not shown.

BIN
prole-net/prole-scan → prole-net/prole-agent
View File

Binary file not shown.

1
prole_requirements.txt
View File

@ -41,6 +41,7 @@ httpcore==1.0.9
httpx==0.28.1 httpx==0.28.1
idna==3.10 idna==3.10
importlib_metadata==8.7.0 importlib_metadata==8.7.0
langchain
isodate==0.6.1 isodate==0.6.1
Jinja2==3.1.6 Jinja2==3.1.6
jiter==0.10.0 jiter==0.10.0

1
requirements.txt
View File

@ -8,6 +8,7 @@ cryptography
ghp-import ghp-import
griffe griffe
idna idna
langchain
Jinja2 Jinja2
jsonpatch jsonpatch
jsonpointer jsonpointer

2
scripts/docker-root-prole-db.sh
View File

@ -7,7 +7,7 @@ export POSTGRES_PASSWORD=$(cat $PROLE_HOME/postgres-password.txt)
PG_VERSION=$(cat "$PROLE_HOME/conf/postgresql/.version" 2>/dev/null | tr -d '[:space:]') PG_VERSION=$(cat "$PROLE_HOME/conf/postgresql/.version" 2>/dev/null | tr -d '[:space:]')
RELEASE=$(cat "$PROLE_HOME/prole-db/.version" 2>/dev/null | tr -d '[:space:]') RELEASE=$(cat "$PROLE_HOME/prole-db/.version" 2>/dev/null | tr -d '[:space:]')
if [[ "$RELEASE" =~ ^[0-9]+$ ]]; then RELEASE=$(printf "%03d" "$RELEASE"); fi if [[ "$RELEASE" =~ ^[0-9]+$ ]]; then RELEASE=$(printf "%03d" "$RELEASE"); fi
IMAGE="prole-db:${PG_VERSION:-17.7}-${RELEASE:-041}" IMAGE="prole-db:${PG_VERSION:-17.7}-${RELEASE:-043}"
docker run --rm -it -u root \ docker run --rm -it -u root \
--hostname=prole-db-00n \ --hostname=prole-db-00n \

3
scripts/docker-run-prole-db.sh
View File

@ -1,3 +1,4 @@
set -eux
# export POSTGRES_PASSWORD=$(openssl rand -base64 32) # export POSTGRES_PASSWORD=$(openssl rand -base64 32)
# echo "$POSTGRES_PASSWORD" > postgres-password.txt # echo "$POSTGRES_PASSWORD" > postgres-password.txt
export POSTGRES_PASSWORD=$(cat postgres-password.txt) export POSTGRES_PASSWORD=$(cat postgres-password.txt)
@ -6,7 +7,7 @@ PROLE_HOME=~/dev/prole
PG_VERSION=$(cat "$PROLE_HOME/conf/postgresql/.version" 2>/dev/null | tr -d '[:space:]') PG_VERSION=$(cat "$PROLE_HOME/conf/postgresql/.version" 2>/dev/null | tr -d '[:space:]')
RELEASE=$(cat "$PROLE_HOME/prole-db/.version" 2>/dev/null | tr -d '[:space:]') RELEASE=$(cat "$PROLE_HOME/prole-db/.version" 2>/dev/null | tr -d '[:space:]')
if [[ "$RELEASE" =~ ^[0-9]+$ ]]; then RELEASE=$(printf "%03d" "$RELEASE"); fi if [[ "$RELEASE" =~ ^[0-9]+$ ]]; then RELEASE=$(printf "%03d" "$RELEASE"); fi
IMAGE="prole-db:${PG_VERSION:-17.7}-${RELEASE:-041}" IMAGE="prole-db:${PG_VERSION:-17.7}-${RELEASE:-043}"
docker run --rm -it --hostname=prole-db-00n \ docker run --rm -it --hostname=prole-db-00n \
-e POSTGRES_PASSWORD="$POSTGRES_PASSWORD" \ -e POSTGRES_PASSWORD="$POSTGRES_PASSWORD" \

2
scripts/generate_spec.py
View File

@ -27,7 +27,7 @@ datas = [
# Binaries to include (with execute permissions) # Binaries to include (with execute permissions)
binaries = [ binaries = [
('prole-net/prole-scan', 'prole-net'), ('prole-net/prole-agent', 'prole-net'),
] ]
# Hidden imports # Hidden imports

8
tests/final_comprehensive_test.sh
View File

@ -40,7 +40,7 @@ for img in ['img/proleIcon.png', 'img/proleLogo.png', 'img/proleLogoSepia.png']:
sys.exit(1) sys.exit(1)
" "
run_test "Binary executable" test -x prole-net/prole-scan run_test "Binary executable" test -x prole-net/prole-agent
run_test "App bundle" test -d "prole-app/dist/Prole Tools.app" run_test "App bundle" test -d "prole-app/dist/Prole Tools.app"
# Build system tests # Build system tests
@ -51,7 +51,7 @@ run_test "Icon conversion" make build/prole.icns
echo -n "[$((TESTS_TOTAL + 1))] Spec includes all resources... " echo -n "[$((TESTS_TOTAL + 1))] Spec includes all resources... "
TESTS_TOTAL=$((TESTS_TOTAL + 1)) TESTS_TOTAL=$((TESTS_TOTAL + 1))
if grep -q "prole-app/dist/Prole Tools.app" installer.spec && \ if grep -q "prole-app/dist/Prole Tools.app" installer.spec && \
grep -q "prole-net/prole-scan" installer.spec && \ grep -q "prole-net/prole-agent" installer.spec && \
grep -q "img.*img" installer.spec; then grep -q "img.*img" installer.spec; then
echo "✓" echo "✓"
TESTS_PASSED=$((TESTS_PASSED + 1)) TESTS_PASSED=$((TESTS_PASSED + 1))
@ -73,13 +73,13 @@ if [ $TESTS_PASSED -eq $TESTS_TOTAL ]; then
echo " ✓ Ncurses terminal interface" echo " ✓ Ncurses terminal interface"
echo " ✓ Automatic display detection" echo " ✓ Automatic display detection"
echo " ✓ Image resources embedded" echo " ✓ Image resources embedded"
echo " ✓ Binary executables embedded (prole-scan)" echo " ✓ Binary executables embedded (prole-agent)"
echo " ✓ App bundles embedded (Prole Tools.app)" echo " ✓ App bundles embedded (Prole Tools.app)"
echo " ✓ Build system configured" echo " ✓ Build system configured"
echo "" echo ""
echo "Embedded resources:" echo "Embedded resources:"
echo " • Images: ~11 MB" echo " • Images: ~11 MB"
echo " • prole-scan: 6.8 MB (universal)" echo " • prole-agent: 6.8 MB (universal)"
echo " • Prole Tools.app: ~12 MB" echo " • Prole Tools.app: ~12 MB"
echo " • Expected final size: ~50-100 MB" echo " • Expected final size: ~50-100 MB"
echo "" echo ""

12
tests/final_test.sh
View File

@ -47,8 +47,8 @@ test_item "proleLogoSepia.png" test -f img/proleLogoSepia.png
# Binary Resources # Binary Resources
echo "" echo ""
echo "BINARY RESOURCES" echo "BINARY RESOURCES"
test_item "prole-scan exists" test -f prole-net/prole-scan test_item "prole-agent exists" test -f prole-net/prole-agent
test_item "prole-scan executable" test -x prole-net/prole-scan test_item "prole-agent executable" test -x prole-net/prole-agent
# App Bundle # App Bundle
echo "" echo ""
@ -67,7 +67,7 @@ echo "SPEC FILE VERIFICATION"
python3 scripts/generate_spec.py > /dev/null 2>&1 python3 scripts/generate_spec.py > /dev/null 2>&1
test_item "Spec includes img" grep -q "img" installer.spec test_item "Spec includes img" grep -q "img" installer.spec
test_item "Spec includes prole-db" grep -q "prole-db" installer.spec test_item "Spec includes prole-db" grep -q "prole-db" installer.spec
test_item "Spec includes prole-scan" grep -q "prole-scan" installer.spec test_item "Spec includes prole-agent" grep -q "prole-agent" installer.spec
test_item "Spec includes Prole Tools" grep -q "Prole Tools" installer.spec test_item "Spec includes Prole Tools" grep -q "Prole Tools" installer.spec
# Resource Path Tests # Resource Path Tests
@ -85,7 +85,7 @@ from pathlib import Path
import sys import sys
sys.path.insert(0, str(Path.cwd())) sys.path.insert(0, str(Path.cwd()))
from install import get_resource_path from install import get_resource_path
assert get_resource_path('prole-net/prole-scan').exists() assert get_resource_path('prole-net/prole-agent').exists()
" "
test_item "Docker context works" python3 -c " test_item "Docker context works" python3 -c "
from pathlib import Path from pathlib import Path
@ -113,14 +113,14 @@ if [ $TESTS_PASSED -eq $TESTS_TOTAL ]; then
echo " ✓ Automatic display detection" echo " ✓ Automatic display detection"
echo " ✓ Static universal binary build system" echo " ✓ Static universal binary build system"
echo " ✓ Image resources embedded" echo " ✓ Image resources embedded"
echo " ✓ prole-scan binary embedded" echo " ✓ prole-agent binary embedded"
echo " ✓ Prole Tools.app embedded" echo " ✓ Prole Tools.app embedded"
echo " ✓ Docker build context embedded" echo " ✓ Docker build context embedded"
echo " ✓ Docker build uses ~/.prole/build (writable)" echo " ✓ Docker build uses ~/.prole/build (writable)"
echo "" echo ""
echo "PACKAGE WILL INCLUDE:" echo "PACKAGE WILL INCLUDE:"
echo " • Images (~11 MB)" echo " • Images (~11 MB)"
echo " • prole-scan (6.8 MB)" echo " • prole-agent (6.8 MB)"
echo " • Prole Tools.app (~12 MB)" echo " • Prole Tools.app (~12 MB)"
echo " • prole-db build context (~1-5 MB)" echo " • prole-db build context (~1-5 MB)"
echo " • Python runtime + deps (~30-50 MB)" echo " • Python runtime + deps (~30-50 MB)"

4
tests/test_all_prole_home_fixes.sh
View File

@ -36,7 +36,7 @@ test_item "Build directory writable" python3 -c "from pathlib import Path; f = P
echo "" echo ""
echo "NETWORK SCAN FIX" echo "NETWORK SCAN FIX"
test_item "prole-scan binary exists" test -x prole-net/prole-scan test_item "prole-agent binary exists" test -x prole-net/prole-agent
test_item "Scan directory creation" python3 -c "from pathlib import Path; d = Path.home() / '.prole' / 'scan'; d.mkdir(parents=True, exist_ok=True); assert d.exists()" test_item "Scan directory creation" python3 -c "from pathlib import Path; d = Path.home() / '.prole' / 'scan'; d.mkdir(parents=True, exist_ok=True); assert d.exists()"
test_item "Scan directory writable" python3 -c "from pathlib import Path; f = Path.home() / '.prole' / 'scan' / 'test.txt'; f.write_text('test'); f.unlink()" test_item "Scan directory writable" python3 -c "from pathlib import Path; f = Path.home() / '.prole' / 'scan' / 'test.txt'; f.write_text('test'); f.unlink()"
@ -51,7 +51,7 @@ echo ""
echo "SPEC FILE VERIFICATION" echo "SPEC FILE VERIFICATION"
python3 scripts/generate_spec.py > /dev/null 2>&1 python3 scripts/generate_spec.py > /dev/null 2>&1
test_item "Spec includes prole-db" grep -q "prole-db" installer.spec test_item "Spec includes prole-db" grep -q "prole-db" installer.spec
test_item "Spec includes prole-scan" grep -q "prole-scan" installer.spec test_item "Spec includes prole-agent" grep -q "prole-agent" installer.spec
# Cleanup # Cleanup
echo "" echo ""

14
tests/test_embedded_resources.sh
View File

@ -28,10 +28,10 @@ if [ $? -ne 0 ]; then exit 1; fi
# Test 2: Binaries # Test 2: Binaries
echo "2. Testing binary resources..." echo "2. Testing binary resources..."
if [ -f "prole-net/prole-scan" ] && [ -x "prole-net/prole-scan" ]; then if [ -f "prole-net/prole-agent" ] && [ -x "prole-net/prole-agent" ]; then
echo " ✓ prole-scan binary present and executable" echo " ✓ prole-agent binary present and executable"
else else
echo " ✗ prole-scan missing or not executable" echo " ✗ prole-agent missing or not executable"
exit 1 exit 1
fi fi
@ -57,7 +57,7 @@ fi
# Test 5: Verify spec includes everything # Test 5: Verify spec includes everything
echo "5. Verifying spec includes all resources..." echo "5. Verifying spec includes all resources..."
grep -q "prole-app/dist/Prole Tools.app" installer.spec && \ grep -q "prole-app/dist/Prole Tools.app" installer.spec && \
grep -q "prole-net/prole-scan" installer.spec && \ grep -q "prole-net/prole-agent" installer.spec && \
grep -q "img" installer.spec grep -q "img" installer.spec
if [ $? -eq 0 ]; then if [ $? -eq 0 ]; then
echo " ✓ All resources included in spec" echo " ✓ All resources included in spec"
@ -68,8 +68,8 @@ fi
# Test 6: Check binary size # Test 6: Check binary size
echo "6. Checking binary sizes..." echo "6. Checking binary sizes..."
SCAN_SIZE=$(ls -lh prole-net/prole-scan | awk '{print $5}') SCAN_SIZE=$(ls -lh prole-net/prole-agent | awk '{print $5}')
echo " prole-scan: $SCAN_SIZE (universal binary)" echo " prole-agent: $SCAN_SIZE (universal binary)"
APP_SIZE=$(du -sh "prole-app/dist/Prole Tools.app" | awk '{print $1}') APP_SIZE=$(du -sh "prole-app/dist/Prole Tools.app" | awk '{print $1}')
echo " Prole Tools.app: $APP_SIZE" echo " Prole Tools.app: $APP_SIZE"
@ -78,7 +78,7 @@ echo "=== All Embedded Resource Tests Passed ==="
echo "" echo ""
echo "Resources ready for packaging:" echo "Resources ready for packaging:"
echo " ✓ Images (proleIcon.png, proleLogo.png, proleLogoSepia.png)" echo " ✓ Images (proleIcon.png, proleLogo.png, proleLogoSepia.png)"
echo " ✓ Binary (prole-net/prole-scan)" echo " ✓ Binary (prole-net/prole-agent)"
echo " ✓ App bundle (prole-app/dist/Prole Tools.app)" echo " ✓ App bundle (prole-app/dist/Prole Tools.app)"
echo "" echo ""
echo "Build with: make package" echo "Build with: make package"

22
tests/test_network_scan_fix.sh
View File

@ -36,28 +36,28 @@ except Exception as e:
PYTEST PYTEST
if [ $? -ne 0 ]; then exit 1; fi if [ $? -ne 0 ]; then exit 1; fi
# Test 3: prole-scan binary exists # Test 3: prole-agent binary exists
echo "3. Testing prole-scan binary..." echo "3. Testing prole-agent binary..."
if [ -x "prole-net/prole-scan" ]; then if [ -x "prole-net/prole-agent" ]; then
echo " ✓ prole-scan binary exists and is executable" echo " ✓ prole-agent binary exists and is executable"
else else
echo " ✗ prole-scan binary missing or not executable" echo " ✗ prole-agent binary missing or not executable"
exit 1 exit 1
fi fi
# Test 4: prole-scan can run from scan directory # Test 4: prole-agent can run from scan directory
echo "4. Testing prole-scan runs from writable directory..." echo "4. Testing prole-agent runs from writable directory..."
mkdir -p /tmp/scan-test mkdir -p /tmp/scan-test
cd /tmp/scan-test cd /tmp/scan-test
/Users/chrisfu/dev/prole/prole-net/prole-scan 2>&1 & /Users/chrisfu/dev/prole/prole-net/prole-agent 2>&1 &
SCAN_PID=$! SCAN_PID=$!
sleep 2 sleep 2
if ps -p $SCAN_PID > /dev/null 2>&1; then if ps -p $SCAN_PID > /dev/null 2>&1; then
echo " ✓ prole-scan runs successfully" echo " ✓ prole-agent runs successfully"
kill $SCAN_PID 2>/dev/null kill $SCAN_PID 2>/dev/null
wait 2>/dev/null wait 2>/dev/null
else else
echo " ✗ prole-scan failed to start" echo " ✗ prole-agent failed to start"
exit 1 exit 1
fi fi
cd /Users/chrisfu/dev/prole cd /Users/chrisfu/dev/prole
@ -85,5 +85,5 @@ echo " • From source: Uses writable cwd"
echo " • From package: Uses ~/.prole/scan" echo " • From package: Uses ~/.prole/scan"
echo "" echo ""
echo "Scan directory: ~/.prole/scan" echo "Scan directory: ~/.prole/scan"
echo "Purpose: Writable working directory for prole-scan" echo "Purpose: Writable working directory for prole-agent"
echo "" echo ""