feat(plugins): add per-plugin Python virtual environments using UV#3429
Open
mcosgriff wants to merge 48 commits into
Open
feat(plugins): add per-plugin Python virtual environments using UV#3429mcosgriff wants to merge 48 commits into
mcosgriff wants to merge 48 commits into
Conversation
Isolate Python dependencies per plugin to prevent conflicts between plugins sharing a single venv. Each plugin gets its own venv at /gems/plugin_venvs//.venv, with automatic cleanup on uninstall. - Add uvinstall script for per-plugin venv creation (uv.lock or pip fallback) - Route microservice and script processes to their plugin's venv - Return per-plugin package listing in admin UI grouped by plugin - Migrate demo plugin from requirements.txt to pyproject.toml + uv.lock - Pin UV version in Dockerfiles, replace cd with WORKDIR in Dockerfile-ubi - Sort apk packages alphabetically in Dockerfile BREAKING CHANGE: PythonPackageModel.names now returns a hash keyed by plugin name instead of a flat array. The /openc3-api/packages endpoint response shape for python packages changes accordingly. Co-Authored-By: Claude noreply@anthropic.com
Collapsible per-plugin groups in PackagesTab with formatted names and package count badges. Exclude dev deps from plugin venvs. Add --no-build to uv sync in Dockerfile, remove stale gem files before rake, and drop unnecessary smoke test. Co-Authored-By: Claude noreply@anthropic.com
Show all plugin venv packages expanded with subheaders instead of collapsible groups. Revert --no-build from uv sync since some deps like watchdog require source builds on Alpine.
Add a /packages/trees endpoint that runs uv tree per plugin venv to show the dependency hierarchy. Plugin sections are collapsible (default collapsed) and fall back to the flat package list for non-UV venvs.
When uploading a .whl file via Admin > Packages, users are now prompted to select which plugin virtual environment to install into. The backend threads the plugin name through to pipinstall via a PIPINSTALL_VENV environment variable, targeting the correct per-plugin venv.
`uv tree` only shows manifest dependencies, hiding pip-installed packages. `uv pip list` shows all installed packages in the venv, giving users accurate visibility into what is available in each plugin environment.
Replace raw uv pip list table output and dist-info names with uniform name==version format (e.g. numpy==2.4.4) for all sections.
Add CLI command, API endpoint, and Admin UI button to migrate existing plugins from the shared Python venv to per-plugin UV virtual environments. Also show venv path in packages list.
…eature Add tests covering UV-related code paths across plugin_model, microservice_operator, python_package_model, packages_controller, and running_script. Fix Logger.debug call in running_script.rb that used bare Logger instead of OpenC3::Logger.
…on packages view Show system venv packages (/openc3/python/.venv), UV download cache contents (/gems/uv), and legacy shared venv in the admin Packages tab. Sections are ordered: System > Cached > Plugins > Shared. Helps users in airgapped environments see which package versions are available.
…tion Bake system Python wheels into /openc3/uv_cache during Docker build and seed them into the /gems/uv volume at init. This ensures plugins reuse cached wheels without re-downloading, critical for airgapped environments. Replace separate System/Cached sections with a single Cached view showing all available packages. Normalize package names to PEP 503 canonical form.
…acing docs Add documentation for the UV per-plugin virtual environment feature to plugins, admin, generators, microservices, and installation pages. Covers dependency isolation, pyproject.toml vs requirements.txt, runtime environment variables, and offline/airgapped cache seeding.
Add explanatory comments to the new UV per-plugin virtual environment code across controllers, models, and operators so the intent is clear without tracing through the full call chain.
…lugin uninstall Copy uploaded .whl files to UV uploads directory so they appear in the cached packages listing. Add delete button for packages in plugin venvs with controller, model, and pipuninstall script support for targeted uninstall. Keep empty plugin venvs visible as install targets.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3429 +/- ##
==========================================
+ Coverage 79.28% 79.41% +0.13%
==========================================
Files 670 671 +1
Lines 56846 57119 +273
Branches 728 728
==========================================
+ Hits 45072 45363 +291
+ Misses 11696 11678 -18
Partials 78 78
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Remove overlapping quantifiers that caused super-linear backtracking: \s* to literal space in wheel filename parser, and \d+ to \d in plugin name formatter where [\d_a-z]* already covers subsequent digits.
Upgrade urllib3 2.6.3 to 2.7.0 to fix two high-severity advisories (GHSA-qccp-gfcp-xxvc, GHSA-mf9v-mfxr-j63j). Add uv lock commands to the plugin generators doc so developers know how to manage lockfiles.
Upgrade idna 3.13 to 3.17 to resolve GHSA-65pc-fj4g-8rjx, a moderate severity bypass of the CVE-2024-3651 fix in idna.encode().
Eliminate polynomial regex backtracking in wheel filename parser by splitting into two non-backtracking steps. Use ENV.fetch with defaults for PATH, PYTHONPATH, and PYPI_URL. Use Set for reserved name lookup and prefix unused block parameter with underscore.
Use ENV.fetch for PYTHONUSERBASE access, prefix remaining unused block parameters with underscore, and split long Dockerfile-ubi lines with backslash continuations.
Fix high-severity rustls-webpki vulnerability (GHSA-82j2-j2ch-gfr8) flagged by Trivy in the uv binary's bundled Rust dependencies.
Cover PythonPackageModel.put whl upload copying, PluginsController migrate_to_uv action (success, error, auth), and PluginModel migrate_to_uv! exception rescue path to address Codecov gaps.
Add images showing the cached packages list, plugin venv contents, package installation, and multiple cached versions to illustrate the per-plugin virtual environment feature.
Temp scripts run under TEMP/ and cannot resolve a plugin virtual environment from their path. Add a "Plugin Venv" dropdown to the Script Runner toolbar that lists installed plugin venvs by scanning /gems/plugin_venvs/ for .uv_managed markers. The selected venv name is passed directly to RunningScript.spawn, bypassing the target lookup. - Add GET /script-api/scripts/plugin_python_venvs endpoint - Add pluginVenv parameter through controller → Script.run → spawn - Show dropdown with venv path subtitles, persist selection across runs - Document Python virtual environment behavior in Script Runner docs
Contributor
Author
|
Contributor
Author
|
Contributor
Author
|
Contributor
Author
|
Contributor
Author
|
Contributor
Author
|
Rename plugin_venv to python_venv across frontend, API, and backend for clarity. Add system venv as default option in the dropdown and set VIRTUAL_ENV consistently for all script environments. Fix API Dockerfiles to use the locally-built openc3 gem from the base image via OPENC3_PATH build arg instead of re-fetching from rubygems.org, which caused source changes to be silently dropped.
Contributor
Author
|
No /gems/plugin_venvs/openc3-cosmos-tsdb-migration-1_1_0_gem__0 $ ls -alt
total 12
drwxr-xr-x 3 501 dialout 4096 Jun 2 22:17 .
-rw-r--r-- 1 501 dialout 0 Jun 2 22:17 .uv_managed
drwxrwxrwx 4 openc3 openc3 4096 Jun 2 22:17 ..
drwxr-xr-x 4 501 dialout 4096 Jun 2 22:17 .venvWas converted to a uv project so /gems/plugin_venvs/openc3-cosmos-demo-7_2_1_pre_beta0_20260602215918_gem__0 $ ls -alt
total 232
drwxrwxrwx 4 openc3 openc3 4096 Jun 2 22:17 ..
drwxr-xr-x 3 501 dialout 4096 Jun 2 22:01 .
-rw-r--r-- 1 501 dialout 0 Jun 2 22:01 .uv_managed
drwxr-xr-x 4 501 dialout 4096 Jun 2 22:01 .venv
-rw-r--r-- 1 501 dialout 2116 Jun 2 22:01 pyproject.toml
-rw-r--r-- 1 501 dialout 219425 Jun 2 22:01 uv.lock |
The mkdir in the Dockerfile conflicts with the /gems volume on subsequent container starts, preventing the operator from starting. The directory is created on demand by uvinstall via mkdir -p.
The previous order copied only the Gemfile, ran bundle install with OPENC3_PATH, then copied the full source — which overwrote the generated Gemfile.lock with the committed one, causing a mismatch that crashed both API containers on startup.
Add a destroy command to openc3.sh that removes only the Docker images belonging to the current installation (Core or Enterprise), allowing users to switch between editions without rebuilding. Includes confirmation prompt, force option, and automatic dangling image prune. Update zsh completion to include the new command.
The ARG OPENC3_PATH line caused bundle install to resolve the openc3 gem as a PATH source in the lockfile, but the ARG was not available at runtime, so the Gemfile fell back to the rubygems version. This mismatch caused Bundler::ProductionError on read-only containers. Removing the ARG lets bundle install use the gem already installed in the openc3-base image, keeping the lockfile consistent.
Add list and status commands to openc3.sh, openc3.bat, and zsh completions. The list command shows Docker images scoped to the current installation; status shows container state via compose ps. Fix per-plugin venv package discovery by using PYTHONPATH instead of VIRTUAL_ENV/PYTHONUSERBASE (which CPython ignores inside venvs). Expose resolved Python env vars in microservice API detail view. Also add default case to select prompts and strip docker.io/ prefix from compose image names for compatibility with docker CLI filters.
…aversal The python_venv parameter from the frontend was interpolated directly into a file path without sanitization. Apply the same tr() sanitization used elsewhere to strip path traversal characters.
Replace String#tr with File.basename to sanitize the python_venv parameter. File.basename is recognized by CodeQL as a path traversal sanitizer, resolving the rb/path-injection alert (CWE-22).
Member
|
closes #1335 |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.venvcreated and managed viauv sync, ensuring dependency isolation between plugins.uvinstallscript and CLI tooling (openc3cli migratetouv) to create per-plugin venvs and migrate legacy plugins fromrequirements.txttopyproject.toml/uv.lock.PluginModelandPythonPackageModelto manage per-plugin venvs during plugin install/uninstall lifecycle, including venv creation, dependency resolution, cleanup, and migration of legacy plugins via migrate_to_uv.MicroserviceOperatorandRunningScriptto activate the correct plugin-specific venv (viaVIRTUAL_ENV,PATH, andPYTHONPATH) when launching microservices and running scripts.PluginModel,PythonPackageModel,MicroserviceOperator, andRunningScriptcovering the new venv behavior.Motivation
Previously, all plugins shared a single global Python environment. This caused dependency conflicts when plugins required different versions of the same package and made it difficult to cleanly uninstall a plugin's Python dependencies. Per-plugin virtual environments provide full isolation, reproducible builds via lockfiles, and clean teardown on plugin removal.
Key files
openc3/bin/uvinstall— new script to create and manage plugin venvsopenc3/lib/openc3/models/plugin_model.rb— venv lifecycle during plugin install/uninstallopenc3/lib/openc3/models/python_package_model.rb— UV-based package managementopenc3/lib/openc3/operators/microservice_operator.rb— venv activation for microservicesopenc3/lib/openc3/utilities/running_script.rb— venv activation for script executionopenc3-cosmos-init/plugins/packages/openc3-vue-common/src/tools/admin/tabs/PackagesTab.vue— redesigned admin UIopenc3-cosmos-init/plugins/packages/openc3-cosmos-demo/pyproject.toml— demo plugin migrated to UVopenc3/lib/openc3/utilities/script.rb— passes python_venv through Script.run and adds the script-existence checkopenc3-cosmos-script-runner-api/app/controllers/scripts_controller.rb— new plugin_python_venvs endpointopenc3-cosmos-init/plugins/packages/openc3-cosmos-tool-scriptrunner/src/tools/scriptrunner/ScriptRunner.vue— venv selector dropdown in Script Runner UIopenc3/bin/openc3cli— new uvmigrate CLI commandopenc3/lib/openc3/models/microservice_model.rb— exposes venv info for microservicesDocumentation
Plugin configuration
configuration/_plugins.md— Phase 2: Deploy section rewritten to describe UV venv-per-plugin install flow, two supported dependency formats (pyproject.toml recommended, requirements.txt), cache pre-seeding, and pip fallbackconfiguration/plugins.md— Same changes (generated output of _plugins.md)getting-started/cli.md— documents the new migratetouv commandTools
tools/admin.md— Plugins tab: "Migrate to UV" button. Packages tab: new Cached, Plugin venvs, and Shared (legacy) subsections with 6 screenshots (packages.png updated, packages_cache.png, packages_cache_versions.png, packages_venv.png, packages_venv_added.png, packages_venv_full.png added)tools/script-runner.md— New "Python Virtual Environments" section covering automatic venv resolution for saved scripts and the Target selector dropdown for temp scriptsDevelopment
development/microservices.md— New "Python Runtime Environment" section documenting env vars set by the operator (VIRTUAL_ENV, PATH, PYTHONUSERBASE, PYTHONPATH)Getting started
getting-started/generators.md— New "Language Flag Handling" table, updated plugin generator docs for # LANGUAGE comment in plugin.txt, requirements.txt description updated to recommend pyproject.toml + uv.lock, new "Python Dependency Management" note with uv lock commands, new "Command Validator Generator" section, updated language resolution docs for target/microservice/conversion/processor/limits_response generatorsgetting-started/installation.md— New "Python Dependencies in Offline Environments" section explaining UV cache pre-seeding and .whl upload via Admin UI