prole/docs/DOCKER-BUILD-FIX.md

Docker Build Fix for PyInstaller Bundle

Problem

When the Knoe Installer is packaged with PyInstaller and run as a standalone binary, Docker builds fail with the error:

Building knoe-db:17.7-037 in /var/folders/rt/pywlnmxj3dn7t5552vwdcpp80000gn/T/_MEIGozuj4/knoe-db...

The build hangs or fails because:

1. PyInstaller extracts resources to a temporary read-only directory (/tmp/_MEIxxxxxx/)
2. Docker build requires a writable context directory
3. The temporary directory has restrictive permissions
4. Docker cannot create the build context properly

Solution

Copy the Docker build context to a writable user directory before building.

Implementation

Location: install.py, run_db_build() method (line ~1791)

Before:

def worker():
    tag = self.get_knoe_db_version()
    cwd = PROJECT_ROOT / "knoe-db"  # ✗ Points to read-only _MEIPASS

    cmd = ['docker', 'build', '-t', f"knoe-db:{tag}", '.']
    proc = subprocess.Popen(cmd, cwd=cwd, ...)

After:

def worker():
    tag = self.get_knoe_db_version()

    # Use $KNOE_HOME/build for Docker build context
    knoe_home = resolve_knoe_home()
    # Use a mode-scoped build dir to avoid cross-mode interference (k3d vs k3s)
    mode_key = _deployment_mode_from_env(env_key) or "default"
    build_dir = knoe_home / "build" / mode_key / "knoe-db"
    build_dir.mkdir(parents=True, exist_ok=True)

    # Copy DB build context to writable location
    source_dir = get_resource_path("knoe-db")
    if not source_dir.exists():
        source_dir = get_resource_path("knoe-db")
    if source_dir.exists():
        import shutil
        if build_dir.exists():
            shutil.rmtree(build_dir)
        shutil.copytree(source_dir, build_dir)

    cwd = build_dir  # ✓ Writable user directory

    cmd = ['docker', 'build', '-t', f"knoe-db:{tag}", '.']
    proc = subprocess.Popen(cmd, cwd=cwd, ...)

Directory Structure (Mode-scoped)

The installer now creates and uses:

$HOME/.knoe/
└── build/
    └── k3d/               # or k3s, k8s
        └── knoe-db/       # Docker build context
            ├── Dockerfile
            ├── conf/
            ├── scripts/
            ├── ...
            └── .knoe_build_context_ready  # marker to preserve generated Dockerfile

Version metadata is also mode-scoped:

$HOME/.knoe/ └── modes/ └── k3d/ # or k3s, k8s ├── conf/postgresql/.version └── knoe-db/.version

Why This Works

1. Writable Location: $HOME/.knoe/ is user-writable
2. Persistent: Files remain between runs (can be cached)
3. Clean State: Each build starts fresh (old dir removed)
4. Docker Compatible: Standard directory Docker can access

Resource Path Resolution

The knoe-db directory is:

• Included in spec: ('knoe-db', 'knoe-db')
• Extracted by PyInstaller: To _MEIPASS/knoe-db/
• Copied to writable location: $HOME/.knoe/build/<mode>/knoe-db/
• Used for build: Docker builds from writable copy

Performance

First Build:

• Copy knoe-db (~1-5 MB): < 1 second
• Docker build: 30-60 seconds (varies)

Subsequent Builds:

• Directory recreation: < 1 second
• Docker build: May use layer cache

Testing

Test from Source

python3 install.py --gui
# Navigate to "Build Container" screen
# Click "Build Database Image"
# Should succeed without hanging

Test from Package

make package
./dist/Knoe\ Installer.app/Contents/MacOS/knoe-installer --gui
# Navigate to "Build Container" screen
# Click "Build Database Image"
# Should create ~/.knoe/build/<mode>/knoe-db and succeed

Verify Directory Creation

# After build starts
ls -la ~/.knoe/build/<mode>/knoe-db/
# Should show Dockerfile and other files

Cleanup

The build directory persists after the installer exits. To clean up:

rm -rf ~/.knoe/build

Or include in the installer:

def cleanup_build_dirs():
    """Clean up temporary build directories."""
    build_dir = Path.home() / ".knoe" / "build"
    if build_dir.exists():
        shutil.rmtree(build_dir)

Other Potential Issues

If Docker build still hangs, check:

1. Docker daemon running:

   docker info
   

2. Docker context accessible:

   ls -la ~/.knoe/build/<mode>/knoe-db/Dockerfile
   

3. Disk space:

   df -h ~
   

4. Docker permissions:

   docker ps
   # Should not require sudo
   

Alternative Solutions Considered

1. Use /tmp with Unique Names

import tempfile
build_dir = Path(tempfile.mkdtemp(prefix="knoe-build-"))

Pros: Automatic cleanup Cons: Lost between runs, no caching

2. Build in Place (_MEIPASS)

cwd = get_resource_path("knoe-db")

Pros: No copying needed Cons: ✗ Fails due to read-only permissions

3. Use PROJECT_ROOT for Both

cwd = PROJECT_ROOT / "knoe-db"

Pros: Works from source Cons: ✗ Fails from package (_MEIPASS is read-only)

4. Chosen: Copy to ~/.knoe/build ✓

Pros: Works from both source and package, writable, persistent Cons: Requires disk space, manual cleanup

Related Changes

• Spec file updated: Added ('knoe-db', 'knoe-db') to data files
• Resource path: Uses get_resource_path("knoe-db") for source
• Build output: Console shows build directory location

Future Enhancements

1. Add cleanup option:

   # In exit handler or menu
   if messagebox.askyesno("Cleanup", "Remove build directories?"):
       cleanup_build_dirs()
   

2. Progress indication:

   self._db_build_console.write("Copying build context...\n")
   shutil.copytree(source_dir, build_dir)
   self._db_build_console.write("Build context ready.\n")
   

3. Cache detection:

   if build_dir.exists() and not force_clean:
       self._db_build_console.write("Using cached build context...\n")
   else:
       shutil.copytree(source_dir, build_dir)
   

Multi-version deploy safety (CNPG)

etc/init_cloudnative_pg.sh now enforces:

• Multiple DB image versions must be deployed into different namespaces (for parallel installs)
• Reusing the same namespace with a different image/version requires an explicit upgrade flag: --upgrade

Summary

The Docker build now works correctly from both source and packaged binary by:

1. Creating ~/.knoe/build/<mode>/knoe-db/ directory
2. Copying build context from embedded resources
3. Running docker build in the writable directory
4. Avoiding PyInstaller's read-only temporary extraction directory