feat: Add --piku-repo and --piku-branch options for installing from forks#16
Draft
clusterfudge wants to merge 67 commits into
Draft
feat: Add --piku-repo and --piku-branch options for installing from forks#16clusterfudge wants to merge 67 commits into
clusterfudge wants to merge 67 commits into
Conversation
…orks This change allows users to install piku from custom forks or branches, which is useful for: - Testing piku changes on real servers before submitting PRs - Running custom modifications in production - Installing from development branches Changes: - playbooks/piku.yml: Parameterize all GitHub URLs with piku_repo and piku_branch variables (defaults to piku/piku and master) - piku-bootstrap: Add CLI argument parsing for --piku-repo=OWNER/REPO and --piku-branch=BRANCH, pass values to ansible via --extra-vars - README.md: Document new options with examples Usage: ./piku-bootstrap install --piku-repo=youruser/piku --piku-branch=feature-xyz
Contributor
Author
|
cc @chr15m 👋 this should help me testing my uv changes (or others like it!) |
Add comprehensive end-to-end test suite that validates: 1. Basic UV deployment with pyproject.toml 2. Python version selection via ENV variable 3. Python version selection via .python-version file 4. Multiple dependencies installation 5. Dependency updates on redeploy 6. ENV takes priority over .python-version Tests run in Docker container with systemd to simulate real server environment. Uses git push workflow to validate full deployment pipeline. Tests require piku with UV support (clusterfudge/piku branch claude/fix-piku-uv-support-ILtq9 or once merged to upstream).
- Fix get_app_port() to filter out warning messages and extract only the port number - Add force: yes to piku.py download to ensure fresh copy is fetched (fixes CDN caching issues during development)
- New workflow e2e-uv.yml runs on push/PR to main - Uses Docker-in-Docker with systemd for realistic testing - Configurable PIKU_REPO and PIKU_BRANCH via environment - Updated README with CI integration documentation
Ensures Docker image is rebuilt fresh to pick up any changes in playbooks.
GitHub's CDN caches raw.githubusercontent.com responses for 5 minutes. Using ansible's get_url with force: yes doesn't bypass this. Adding a nocache query parameter with epoch timestamp forces a fresh fetch.
GitHub's raw.githubusercontent.com CDN caches content aggressively and ignores query parameters. Using the GitHub Contents API with the Accept: application/vnd.github.v3.raw header returns fresh content.
piku-bootstrap first-run clones the repo from GitHub, which means the playbooks in the cloned copy may be outdated. Override them with the playbooks copied into the Docker image to ensure we test the current version.
- Changed apps from Flask dev server to WSGI module format - piku runs Python web apps via uwsgi, not as standalone scripts - Added wait_for_deploy() to wait for uwsgi config creation - Improved get_app_port() to read from uwsgi config file - Added port validation before use - Increased wait times for app restarts - Added more debugging output on failures
- Print deploy output to help diagnose failures - Add verbose SSH flag to see connection issues - Use more specific patterns to detect UV deployment
- WSGI apps create wsgi.1.ini not web.1.ini - Updated wait_for_deploy to check for any app config - Updated get_app_port to find any config file - Added debugging for missing configs
- Increase wait time after uwsgi config creation - Show app logs and uwsgi emperor logs when app returns error
uWSGI is compiled against the system Python (3.10 on Ubuntu 22.04). When UV creates a virtualenv with a different Python version (e.g., 3.12), the packages installed won't be visible to uWSGI's internal interpreter. Set PYTHON_VERSION=3.10 in all test apps to ensure the virtualenv Python matches the uWSGI interpreter. This is a known limitation of using system uWSGI with UV-managed Pythons.
The app may get a new port after redeploy, so we need to fetch it again instead of using the cached port from the initial deploy.
clusterfudge
commented
Jan 22, 2026
| chmod +x run_tests.sh test_uv_e2e.sh | ||
| ./run_tests.sh --no-cache | ||
| env: | ||
| # Use clusterfudge/piku with UV fixes until merged upstream |
Contributor
Author
There was a problem hiding this comment.
@chr15m FYI using this to validate piku/piku#415 , and found a bug!
Collaborator
There was a problem hiding this comment.
What was the bug? Can you file an issue describing it?
Contributor
Author
There was a problem hiding this comment.
sorry, it was a bug in this change! fixed with this commit piku/piku@c134920
5 tasks
Contributor
Author
|
@chr15m what do you think are some good ways to validate this? I find the docker/ubuntu test pretty compelling, but we could also expand it to cover some more platforms. That would just increase build times (which, I think are free on gh for OSS? but unsure). |
Collaborator
|
The automated tests could definitely use expanding. What would be great would be to spawn a second Docker instance and connect to it, install piku.py, and then run through all the different commands and deployment types to check that they're still working. That's a big task though. Even just running piku.py on a different instance would be a great start. |
Addresses reviewer feedback requesting expanded test coverage with a second Docker instance for realistic SSH/git push testing. New test infrastructure: - Docker Compose setup with piku-server and test-client containers - Server auto-bootstraps piku via systemd service - Client connects via SSH for git push deployments - Shared SSH keys via Docker volume Test suites added: - test_python_pip.sh: 4 tests for requirements.txt deployments - test_python_uv.sh: 6 tests for UV/pyproject.toml deployments - test_nodejs.sh: 4 tests for Node.js/Express deployments - test_piku_commands.sh: 6 tests for piku CLI commands Helper library (lib/test_helpers.sh): - App lifecycle: create_app, deploy_app, wait_for_app, destroy_app - HTTP testing: test_http with expected content matching - SSH/piku commands: run_piku, ssh_server - App templates: create_flask_app, create_uv_flask_app, create_node_app GitHub Actions workflow (e2e-full.yml): - Parallel jobs for faster feedback - Supports workflow_dispatch with custom piku_repo/piku_branch - 45-minute timeout for full suite Documentation: - Architecture diagram - Quick start guide - Test writing instructions - Debugging tips
GitHub Actions runners have Docker Compose as a plugin (docker compose) not as the standalone binary (docker-compose).
- Add entrypoint.sh that writes env vars to /tmp/piku-env before systemd - Update bootstrap-piku.sh to source env vars and add logging - Remove obsolete 'version' attribute from docker-compose.yml - Pass PIKU_REPO/PIKU_BRANCH via environment section (runtime, not build-time)
…cker exec The systemd service approach was failing due to timing issues with container startup. This follows the pattern from the working e2e-uv tests: - Remove bootstrap systemd service - Remove entrypoint.sh and bootstrap-piku.sh scripts - Run piku-bootstrap interactively via docker exec after container starts - Wait for systemd explicitly with 'systemctl is-system-running --wait' - Remove health check dependency from docker-compose.yml This is more reliable and easier to debug.
The systemd-user-sessions.service doesn't seem to activate properly in Docker containers. Directly remove the nologin files to allow SSH logins after bootstrap completes.
Simply appending to authorized_keys doesn't configure the key with piku properly. Need to run piku.py setup:ssh to set up the git-shell command configuration.
Piku creates uwsgi config files named like 'app_web.1.ini' or 'app_wsgi.1.ini', not just 'app.ini'. Use a wildcard pattern to check if any ini file for the app exists.
The uwsgi config file existing doesn't mean the app is running yet. Add a delay and then poll for a non-403/502 HTTP response to ensure the app is actually serving requests.
…ommand) Piku uses uwsgi directly and expects Procfile format 'wsgi: module:callable' not 'wsgi: gunicorn command'. Also add WSGI compatibility alias.
Shows uwsgi/nginx configs, service status, and logs to help debug why apps aren't becoming ready.
Piku needs NGINX_SERVER_NAME in the ENV file to generate nginx virtual host configuration. Without this, nginx returns 403 because there's no virtual host matching the Host header.
Tests that manually create files (instead of using helper functions) also need NGINX_SERVER_NAME in their ENV files for nginx virtual hosting to work.
piku logs tails indefinitely, so we need a 5-second timeout. Also add fallback verification via log file existence.
The app may take time to start after piku start command. Add wait_for_app call with 60 second timeout.
piku only has 'restart', 'stop', and 'deploy' commands - there is no separate 'start' command. Fixed test to use restart after stop.
- Install Node.js 18.x via NodeSource for Node.js app support - Install uv (Astral's fast Python package installer) for UV support - Use clusterfudge/piku UV branch for e2e-python-uv tests until merged upstream This ensures all test suites can pass without depending on piku's runtime-installed versions of these tools.
The piku process runs as the 'piku' user, not root. Installing uv
to /root/.local/bin meant which('uv') would fail when piku tried
to detect UV apps. Now installed directly to /usr/local/bin.
Ubuntu 22.04 ships with Python 3.10, not 3.11. The uv deployment fails with 'No interpreter found for Python >=3.11' when trying to use a version that isn't installed.
The piku-bootstrap script looks for its playbooks in ~/.piku-bootstrap/piku-bootstrap/. Previously we copied files to /root/ which meant the script would clone the UPSTREAM repo during first-run, ignoring our custom playbook with piku_repo and piku_branch support. Now we copy to the expected location so our fork's playbooks are used.
1. UV Flask test app now uses correct Procfile format (wsgi: wsgi:app) and includes WSGI entry point (application = app) 2. Fetch piku.py task now shows HTTP status and handles curl failures to help debug rate limiting or API errors
All UV tests now use: - Procfile: 'wsgi: wsgi:app' (module:callable format) - wsgi.py: includes 'application = app' WSGI entry point This matches how piku expects WSGI apps to be configured.
Without NGINX_SERVER_NAME, nginx doesn't set up virtual hosting properly and requests don't reach the app.
piku/piku master doesn't support UV deployments (has PYTHON_VERSION parsing bug). The separate e2e-python-uv job uses a UV-supporting branch (clusterfudge/piku:claude/fix-piku-uv-support-ILtq9) which passes correctly. The e2e-tests combined job now skips python_uv tests when using the default piku/piku master configuration.
All tests passing now. Clean up temporary debug output that was added while diagnosing the piku-bootstrap repo path issue.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds support for installing piku from custom forks via new CLI options:
--piku-repo=OWNER/REPO- Install from a specific GitHub fork (default: piku/piku)--piku-branch=BRANCH- Install from a specific branch (default: master)Use Cases
Examples
Changes
piku_repo,piku_branch, andpiku_raw_urlvariables with templated URLsE2E Test Status
Note: The e2e-python-uv tests use
clusterfudge/piku:claude/fix-piku-uv-support-ILtq9which includes UV deployment support not yet in piku/piku master.Backward Compatibility
Fully backward compatible - defaults match current upstream behavior (piku/piku, master branch).