From e850dd1ee6e69d69a33b110ffe08072c1b3c4369 Mon Sep 17 00:00:00 2001 From: Johnny Greco Date: Thu, 23 Jul 2026 19:01:57 +0000 Subject: [PATCH 1/6] Add middleware project update workflow --- projects/README.md | 4 +- .../AGENTS.md | 9 +- .../README.md | 66 +++- .../pyproject.toml | 12 +- .../src/middleware_kit/__init__.py | 3 + .../src/middleware_kit}/cli.py | 50 ++- .../src/middleware_kit}/generator.py | 302 +++++++++++++++- .../templates/python/.gitignore | 0 .../templates/python/README.md | 10 +- .../templates/python/pyproject.toml | 0 .../templates/python/src/package/__init__.py | 0 .../python/src/package/bindings/__init__.py | 0 .../templates/python/src/package/server.py | 0 .../templates/python/tests/test_server.py | 0 .../middleware_kit}/templates/rust/.gitignore | 0 .../middleware_kit}/templates/rust/Cargo.toml | 0 .../middleware_kit}/templates/rust/README.md | 9 +- .../middleware_kit}/templates/rust/build.rs | 0 .../middleware_kit}/templates/rust/src/lib.rs | 0 .../templates/rust/src/main.rs | 0 .../tests/test_cli.py | 61 +++- .../tests/test_generator.py | 328 +++++++++++++++++- .../uv.lock | 2 +- .../src/openshell_middleware_init/__init__.py | 3 - 24 files changed, 782 insertions(+), 77 deletions(-) rename projects/{openshell-middleware-init => middleware-kit}/AGENTS.md (85%) rename projects/{openshell-middleware-init => middleware-kit}/README.md (54%) rename projects/{openshell-middleware-init => middleware-kit}/pyproject.toml (72%) create mode 100644 projects/middleware-kit/src/middleware_kit/__init__.py rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/cli.py (59%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/generator.py (65%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/templates/python/.gitignore (100%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/templates/python/README.md (92%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/templates/python/pyproject.toml (100%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/templates/python/src/package/__init__.py (100%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/templates/python/src/package/bindings/__init__.py (100%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/templates/python/src/package/server.py (100%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/templates/python/tests/test_server.py (100%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/templates/rust/.gitignore (100%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/templates/rust/Cargo.toml (100%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/templates/rust/README.md (92%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/templates/rust/build.rs (100%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/templates/rust/src/lib.rs (100%) rename projects/{openshell-middleware-init/src/openshell_middleware_init => middleware-kit/src/middleware_kit}/templates/rust/src/main.rs (100%) rename projects/{openshell-middleware-init => middleware-kit}/tests/test_cli.py (51%) rename projects/{openshell-middleware-init => middleware-kit}/tests/test_generator.py (70%) rename projects/{openshell-middleware-init => middleware-kit}/uv.lock (99%) delete mode 100644 projects/openshell-middleware-init/src/openshell_middleware_init/__init__.py diff --git a/projects/README.md b/projects/README.md index 674946c2..84c451ff 100644 --- a/projects/README.md +++ b/projects/README.md @@ -6,8 +6,8 @@ layout. Current projects: -- `openshell-middleware-init`: Typer CLI that generates version-matched Python - and Rust OpenShell supervisor middleware projects. +- `middleware-kit`: `mkit` CLI that creates and updates version-matched + Python and Rust OpenShell supervisor middleware projects. - `python-project-template`: Minimal, production-ready Python project scaffold managed with uv. - `reachy-mini-openshell`: Reachy Mini conversation demo for OpenShell. diff --git a/projects/openshell-middleware-init/AGENTS.md b/projects/middleware-kit/AGENTS.md similarity index 85% rename from projects/openshell-middleware-init/AGENTS.md rename to projects/middleware-kit/AGENTS.md index 864d071c..d14f987a 100644 --- a/projects/openshell-middleware-init/AGENTS.md +++ b/projects/middleware-kit/AGENTS.md @@ -4,11 +4,12 @@ Read `README.md` and `pyproject.toml` before changing this project. ## Preserve these invariants -- Keep initialization non-destructive. Never merge into, follow, or replace an +- Keep creation non-destructive. Never merge into, follow, or replace an existing output path, including a symlink. - Build and validate in a hidden sibling staging directory. Publish only after all generation and validation steps succeed. -- Preserve reservation ownership checks and atomic no-replace publication. +- Preserve reservation ownership checks, atomic no-replace creation, and atomic + exchange publication for updates. - Support Linux and macOS explicitly. Do not weaken publication guarantees to add another platform implicitly. - Keep every generated project version-matched: the OpenShell tag, downloaded @@ -25,7 +26,7 @@ Read `README.md` and `pyproject.toml` before changing this project. ## Change templates carefully -- Keep templates under `src/openshell_middleware_init/templates/` runnable as +- Keep templates under `src/middleware_kit/templates/` runnable as standalone projects. - Use `__UPPER_SNAKE_CASE__` for template markers. Add every marker to `TemplateContext.replacements` and cover it with a rendering test. @@ -36,7 +37,7 @@ Read `README.md` and `pyproject.toml` before changing this project. ## Test behavior, not implementation details -- Keep initializer unit tests hermetic. Inject protocol downloads and project +- Keep project-tool unit tests hermetic. Inject protocol downloads and project preparation instead of contacting GitHub or invoking uv or Cargo. - Add regression tests for changes to output safety, failure cleanup, naming, manifests, network behavior, or rendered files. diff --git a/projects/openshell-middleware-init/README.md b/projects/middleware-kit/README.md similarity index 54% rename from projects/openshell-middleware-init/README.md rename to projects/middleware-kit/README.md index bbad8952..e54620ca 100644 --- a/projects/openshell-middleware-init/README.md +++ b/projects/middleware-kit/README.md @@ -1,11 +1,11 @@ -# OpenShell Middleware Init +# OpenShell Middleware Kit -`openshell-middleware-init` creates a runnable Python or Rust starter for an -OpenShell supervisor middleware service. The starter implements the complete -gRPC service as a pass-through, pins its protocol contract to one OpenShell -release, and includes tests, dependency locks, and registration guidance. +`middleware-kit` creates and updates runnable Python or Rust OpenShell +supervisor middleware services. A new project implements the complete gRPC +service as a pass-through, pins its protocol contract to one OpenShell release, +and includes tests, dependency locks, and registration guidance. -The initializer does not install or replace OpenShell. +The project tool does not install or replace OpenShell. ## Requirements @@ -20,24 +20,24 @@ Install the command in an isolated tool environment from GitHub: ```sh uv tool install \ - "openshell-middleware-init @ git+https://github.com/NVIDIA/OpenShell-Research.git#subdirectory=projects/openshell-middleware-init" + "middleware-kit @ git+https://github.com/NVIDIA/OpenShell-Research.git#subdirectory=projects/middleware-kit" ``` If you already have this repository checked out, install from its local path instead: ```sh -uv tool install /path/to/OpenShell-Research/projects/openshell-middleware-init +uv tool install /path/to/OpenShell-Research/projects/middleware-kit ``` -Both forms make `openshell-middleware-init` available outside the source tree +Both forms make `mkit` available outside the source tree without running `uv sync` in this project. Contributors working on the CLI should use the locked project environment: ```sh uv sync --locked -uv run openshell-middleware-init --help +uv run mkit --help ``` ## Quick start @@ -45,7 +45,7 @@ uv run openshell-middleware-init --help Generate and run a Python starter with the installed command: ```sh -openshell-middleware-init audit-headers \ +mkit create audit-headers \ --language python \ --openshell-version v0.0.86 \ --output /tmp/audit-headers @@ -58,7 +58,7 @@ uv run audit-headers Or generate and run a Rust starter: ```sh -openshell-middleware-init audit-headers \ +mkit create audit-headers \ --language rust \ --openshell-version v0.0.86 \ --output /tmp/audit-headers-rust @@ -72,9 +72,31 @@ The output path must not already exist. Use a pinned OpenShell tag for reproducible projects; `--openshell-version latest` is available for experimentation. -Run `openshell-middleware-init --help` for all options. Python package names +Run `mkit --help` for all options. Python package names default to a normalized project name and can be changed with `--package-name`. +## Update a project + +From a generated project, refresh to the latest OpenShell release: + +```sh +mkit update +``` + +To select a release or update a project from another directory: + +```sh +mkit update /path/to/audit-headers \ + --openshell-version v0.0.90 +``` + +The update command reads `middleware-dev-manifest.json` to discover the +project language and Python package. It downloads the selected +`supervisor_middleware.proto`, regenerates Python protobuf and gRPC bindings +when applicable, refreshes `uv.lock` or `Cargo.lock`, and records the new +version and protocol checksum in the manifest. Projects created under the +former `middleware-project` and `openshell-middleware-init` names are supported. + ## What you get Each generated project contains: @@ -94,19 +116,25 @@ service and register it with OpenShell. ## Safety and failure behavior -Generation is non-destructive. The initializer validates a hidden sibling +Creation is non-destructive. The project tool validates a hidden sibling staging directory, then publishes it atomically. It refuses an existing output, -including a symlink, and uses a per-output reservation to prevent concurrent -writers. A normal failure removes the initializer's own staging and reservation -without publishing a partial project. +including a symlink. Updates copy the complete existing project into a hidden +sibling staging directory, change only generator-owned protocol artifacts +there, validate the staged project, then atomically exchange those artifacts +in place. User implementation files and the project directory itself are +preserved. If publication fails, completed exchanges are rolled back in reverse +order. Both operations use a per-project reservation to prevent concurrent +writers; a normal failure removes the tool's own staging and reservation +without leaving partial changes. If the process is killed, it may leave -`..openshell-middleware-init.lock` and a hidden staging directory. Before +`..middleware-kit.lock` and a hidden staging directory. Before removing either one: 1. Read `metadata.json` in the reservation. 2. On the recorded host, confirm that the recorded PID is no longer the same - initializer process and that the final output does not exist. + project tool process. For a create operation, confirm that the final output + does not exist. For an update, do not remove the final project. 3. Inspect and remove only the recorded staging directory. 4. Remove `owner` and `metadata.json`, then remove the empty reservation with `rmdir`. Stop if it contains anything unexpected. diff --git a/projects/openshell-middleware-init/pyproject.toml b/projects/middleware-kit/pyproject.toml similarity index 72% rename from projects/openshell-middleware-init/pyproject.toml rename to projects/middleware-kit/pyproject.toml index 6c47e66c..7b0c24f9 100644 --- a/projects/openshell-middleware-init/pyproject.toml +++ b/projects/middleware-kit/pyproject.toml @@ -1,7 +1,7 @@ [project] -name = "openshell-middleware-init" +name = "middleware-kit" version = "0.1.0" -description = "Generate version-matched OpenShell supervisor middleware projects." +description = "Create and update version-matched OpenShell middleware projects." readme = "README.md" requires-python = ">=3.10" license = "Apache-2.0" @@ -13,7 +13,7 @@ dependencies = [ ] [project.scripts] -openshell-middleware-init = "openshell_middleware_init.cli:main" +mkit = "middleware_kit.cli:main" [project.urls] Repository = "https://github.com/NVIDIA/OpenShell-Research" @@ -34,7 +34,7 @@ build-backend = "uv_build" addopts = [ "--strict-config", "--strict-markers", - "--cov=openshell_middleware_init", + "--cov=middleware_kit", "--cov-report=term-missing", "--cov-fail-under=95", ] @@ -43,14 +43,14 @@ testpaths = ["tests"] [tool.ruff] line-length = 100 target-version = "py310" -extend-exclude = ["src/openshell_middleware_init/templates"] +extend-exclude = ["src/middleware_kit/templates"] [tool.ruff.lint] select = ["B", "E", "F", "I", "RUF", "SIM", "UP"] [tool.ty.src] include = ["src", "tests"] -exclude = ["src/openshell_middleware_init/templates"] +exclude = ["src/middleware_kit/templates"] [tool.uv] required-version = ">=0.11.0" diff --git a/projects/middleware-kit/src/middleware_kit/__init__.py b/projects/middleware-kit/src/middleware_kit/__init__.py new file mode 100644 index 00000000..0020d1e3 --- /dev/null +++ b/projects/middleware-kit/src/middleware_kit/__init__.py @@ -0,0 +1,3 @@ +"""Create and update version-matched OpenShell middleware projects.""" + +__version__ = "0.1.0" diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/cli.py b/projects/middleware-kit/src/middleware_kit/cli.py similarity index 59% rename from projects/openshell-middleware-init/src/openshell_middleware_init/cli.py rename to projects/middleware-kit/src/middleware_kit/cli.py index a6e3de5a..173b243c 100644 --- a/projects/openshell-middleware-init/src/openshell_middleware_init/cli.py +++ b/projects/middleware-kit/src/middleware_kit/cli.py @@ -1,4 +1,4 @@ -"""Typer command-line interface for the middleware project generator.""" +"""Typer command-line interface for OpenShell middleware projects.""" from __future__ import annotations @@ -8,7 +8,11 @@ import typer -from openshell_middleware_init.generator import InitializationError, initialize_project +from middleware_kit.generator import ( + InitializationError, + initialize_project, + update_project, +) class Language(str, Enum): @@ -22,12 +26,12 @@ class Language(str, Enum): add_completion=False, no_args_is_help=True, pretty_exceptions_enable=False, - help="Generate a runnable, version-matched OpenShell middleware project.", + help="Create or update a version-matched OpenShell middleware project.", ) @app.command() -def init( +def create( name: Annotated[ str, typer.Argument(help="Project name, such as audit-headers."), @@ -71,14 +75,48 @@ def init( package_name=package_name, ) except InitializationError as error: - typer.echo(f"openshell-middleware-init: error: {error}", err=True) - raise typer.Exit(code=1) from error + _report_error(error) typer.echo(f"Created {result.language} middleware project at {result.destination}") typer.echo(f"OpenShell contract: {result.openshell_version}") typer.echo(f"Next: cd {result.destination} && {result.run_command}") +@app.command() +def update( + project: Annotated[ + Path, + typer.Argument( + help="Existing generated middleware project. Defaults to the current directory." + ), + ] = Path("."), + openshell_version: Annotated[ + str, + typer.Option( + "--openshell-version", + "--version", + help="OpenShell release tag (for example v0.0.86), or latest.", + ), + ] = "latest", +) -> None: + """Update an existing middleware project's OpenShell contract and generated files.""" + try: + result = update_project( + project_dir=project, + requested_version=openshell_version, + ) + except InitializationError as error: + _report_error(error) + + typer.echo(f"Updated {result.language} middleware project at {result.destination}") + typer.echo(f"OpenShell contract: {result.openshell_version}") + + +def _report_error(error: InitializationError) -> None: + typer.echo(f"mkit: error: {error}", err=True) + raise typer.Exit(code=1) from error + + def main() -> None: """Run the command-line application.""" app() diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/generator.py b/projects/middleware-kit/src/middleware_kit/generator.py similarity index 65% rename from projects/openshell-middleware-init/src/openshell_middleware_init/generator.py rename to projects/middleware-kit/src/middleware_kit/generator.py index 60267c37..2f2d4400 100644 --- a/projects/openshell-middleware-init/src/openshell_middleware_init/generator.py +++ b/projects/middleware-kit/src/middleware_kit/generator.py @@ -1,4 +1,4 @@ -"""Safe, version-matched project generation.""" +"""Safe, version-matched project creation and updates.""" from __future__ import annotations @@ -26,7 +26,7 @@ from importlib.resources import files from pathlib import Path -from openshell_middleware_init import __version__ +from middleware_kit import __version__ _REPOSITORY_URL = "https://github.com/NVIDIA/OpenShell" _RAW_URL = "https://raw.githubusercontent.com/NVIDIA/OpenShell" @@ -36,6 +36,8 @@ _PYTHON_PACKAGE_PATTERN = re.compile(r"^[a-z][a-z0-9_]*$") _PROJECT_NAME_PATTERN = re.compile(r"^[a-z0-9](?:[a-z0-9._-]*[a-z0-9])?$") _NETWORK_ATTEMPTS = 4 +_TOOL_NAME = "middleware-kit" +_LEGACY_TOOL_NAMES = {"middleware-project", "openshell-middleware-init"} _RUST_KEYWORDS = { "abstract", "as", @@ -105,12 +107,16 @@ class InitializationError(RuntimeError): - """A user-actionable project initialization failure.""" + """A user-actionable middleware project operation failure.""" + + +class PublicationRollbackError(InitializationError): + """An update failure whose recovery artifacts must be preserved.""" @dataclass(frozen=True) class InitializationResult: - """Details about a successfully generated project.""" + """Details about a successful middleware project operation.""" destination: Path language: str @@ -118,6 +124,14 @@ class InitializationResult: run_command: str +@dataclass(frozen=True) +class ProjectMetadata: + """Metadata needed to refresh a generated middleware project.""" + + language: str + python_package: str | None + + @dataclass(frozen=True) class OutputReservation: """Identity and recovery data for an output-path reservation.""" @@ -184,7 +198,7 @@ def initialize_project( runner = command_runner if command_runner is not None else _prepare_project destination.parent.mkdir(parents=True, exist_ok=True) - lock_path = destination.parent / f".{destination.name}.openshell-middleware-init.lock" + lock_path = destination.parent / f".{destination.name}.{_TOOL_NAME}.lock" lock_token = secrets.token_hex(16) reservation = _acquire_lock(lock_path, lock_token, destination, version) staging_path: Path | None = None @@ -232,11 +246,80 @@ def initialize_project( ) +def update_project( + *, + project_dir: Path, + requested_version: str = "latest", + download_proto: DownloadProto | None = None, + command_runner: CommandRunner | None = None, +) -> InitializationResult: + """Refresh generator-owned artifacts and atomically publish a validated update.""" + _validate_platform() + project_dir = project_dir.expanduser() + _validate_existing_project(project_dir) + project_dir = project_dir.resolve() + project_stat = project_dir.stat(follow_symlinks=False) + metadata = _read_project_metadata(project_dir) + if command_runner is None: + _preflight_language(metadata.language) + version = _normalize_version(requested_version) + downloader = download_proto if download_proto is not None else _download_proto + runner = command_runner if command_runner is not None else _prepare_project + + lock_path = project_dir.parent / f".{project_dir.name}.{_TOOL_NAME}.lock" + lock_token = secrets.token_hex(16) + reservation = _acquire_lock(lock_path, lock_token, project_dir, version) + staging_path: Path | None = None + published = False + preserve_recovery = False + try: + _validate_existing_project(project_dir) + shutil.copytree(project_dir, reservation.staging_path, symlinks=True) + staging_path = reservation.staging_path + proto, proto_url = downloader(version) + _validate_proto(proto, version) + _refresh_generated_artifacts( + staging_path, + metadata=metadata, + version=version, + proto_url=proto_url, + proto=proto, + ) + runner(metadata.language, staging_path, metadata.python_package or "unused") + _verify_lock(reservation) + _verify_project_identity(project_dir, project_stat.st_dev, project_stat.st_ino) + _publish_generated_artifacts(staging_path, project_dir, metadata) + published = True + except PublicationRollbackError: + preserve_recovery = True + raise + except InitializationError: + raise + except (OSError, subprocess.SubprocessError) as error: + raise InitializationError(str(error)) from error + finally: + if preserve_recovery: + with suppress(OSError): + os.close(reservation.directory_fd) + else: + if staging_path is not None: + shutil.rmtree(staging_path, ignore_errors=True) + _release_lock(reservation) + + if not published: # pragma: no cover - defensive; failures raise above + raise AssertionError("updated project was not published") + return InitializationResult( + destination=project_dir, + language=metadata.language, + openshell_version=version, + run_command="", + ) + + def _validate_platform() -> None: if sys.platform != "darwin" and not sys.platform.startswith("linux"): raise InitializationError( - "openshell-middleware-init supports Linux and macOS; " - f"unsupported platform: {sys.platform}" + f"{_TOOL_NAME} supports Linux and macOS; unsupported platform: {sys.platform}" ) @@ -294,7 +377,7 @@ def _normalize_version(requested: str) -> str: def _resolve_latest_version() -> str: request = urllib.request.Request( f"{_REPOSITORY_URL}/releases/latest", - headers={"User-Agent": f"openshell-middleware-init/{__version__}"}, + headers={"User-Agent": f"{_TOOL_NAME}/{__version__}"}, ) try: _, resolved_url = _fetch_url(request) @@ -313,7 +396,7 @@ def _download_proto(version: str) -> tuple[bytes, str]: url = f"{_RAW_URL}/{version}/{_PROTO_PATH}" request = urllib.request.Request( url, - headers={"User-Agent": f"openshell-middleware-init/{__version__}"}, + headers={"User-Agent": f"{_TOOL_NAME}/{__version__}"}, ) try: body, _ = _fetch_url(request) @@ -376,6 +459,112 @@ def _validate_destination(destination: Path) -> None: raise InitializationError(f"invalid output path: {destination}") +def _validate_existing_project(project_dir: Path) -> None: + if project_dir.is_symlink(): + raise InitializationError(f"project path must not be a symlink: {project_dir}") + if not project_dir.is_dir(): + raise InitializationError(f"project path must be an existing directory: {project_dir}") + + +def _verify_project_identity(project_dir: Path, device: int, inode: int) -> None: + try: + current = project_dir.stat(follow_symlinks=False) + except OSError as error: + raise InitializationError( + "project path changed during update; refusing to publish" + ) from error + if not stat.S_ISDIR(current.st_mode) or current.st_dev != device or current.st_ino != inode: + raise InitializationError("project path changed during update; refusing to publish") + + +def _read_project_metadata(project_dir: Path) -> ProjectMetadata: + manifest_path = project_dir / "middleware-dev-manifest.json" + if manifest_path.is_symlink() or not manifest_path.is_file(): + raise InitializationError( + f"not a generated middleware project; missing regular manifest: {manifest_path}" + ) + try: + manifest = json.loads(manifest_path.read_text()) + except (OSError, json.JSONDecodeError) as error: + raise InitializationError(f"could not read project manifest: {error}") from error + if not isinstance(manifest, dict): + raise InitializationError("project manifest must contain a JSON object") + + generator = manifest.get("generator") + generator_name = generator.get("name") if isinstance(generator, dict) else None + if generator_name != _TOOL_NAME and generator_name not in _LEGACY_TOOL_NAMES: + raise InitializationError( + "project manifest was not created by middleware-kit, middleware-project, " + "or openshell-middleware-init" + ) + languages = manifest.get("languages") + if languages not in (["python"], ["rust"]): + raise InitializationError("project manifest must identify exactly one supported language") + language = languages[0] + python_package = manifest.get("python_package") + if language == "python": + if not isinstance(python_package, str) or not _PYTHON_PACKAGE_PATTERN.fullmatch( + python_package + ): + raise InitializationError( + "Python project manifest has an invalid or missing python_package" + ) + elif python_package is not None: + raise InitializationError("Rust project manifest must set python_package to null") + + _validate_refresh_targets(project_dir, language, python_package) + return ProjectMetadata(language=language, python_package=python_package) + + +def _validate_refresh_targets(project_dir: Path, language: str, python_package: str | None) -> None: + regular_files = [ + project_dir / "middleware-dev-manifest.json", + project_dir / "proto" / "supervisor_middleware.proto", + ] + regular_files.append(project_dir / ("uv.lock" if language == "python" else "Cargo.lock")) + for path in regular_files: + if path.is_symlink() or not path.is_file(): + raise InitializationError(f"generated artifact must be a regular file: {path}") + + if language == "python": + if python_package is None: # pragma: no cover - checked by caller + raise AssertionError("Python package is required") + bindings_dir = project_dir / "src" / python_package / "bindings" + if bindings_dir.is_symlink() or not bindings_dir.is_dir(): + raise InitializationError( + f"generated bindings must be an existing directory: {bindings_dir}" + ) + + +def _refresh_generated_artifacts( + project_dir: Path, + *, + metadata: ProjectMetadata, + version: str, + proto_url: str, + proto: bytes, +) -> None: + proto_path = project_dir / "proto" / "supervisor_middleware.proto" + proto_path.write_bytes(proto) + if metadata.language == "python": + if metadata.python_package is None: # pragma: no cover - checked during discovery + raise AssertionError("Python package is required") + bindings_dir = project_dir / "src" / metadata.python_package / "bindings" + shutil.rmtree(bindings_dir) + bindings_dir.mkdir() + bindings_dir.joinpath("__init__.py").write_text( + '"""Generated OpenShell supervisor middleware bindings. Do not edit."""\n' + ) + _write_manifest( + project_dir, + version=version, + proto_url=proto_url, + proto=proto, + language=metadata.language, + python_package=metadata.python_package, + ) + + def _acquire_lock( lock_path: Path, token: str, destination: Path, version: str ) -> OutputReservation: @@ -383,9 +572,9 @@ def _acquire_lock( lock_path.mkdir(mode=0o700) except FileExistsError as error: raise InitializationError( - f"output path is reserved by another initializer: {destination}; " + f"project path is reserved by another {_TOOL_NAME} process: {destination}; " f"inspect {lock_path / 'metadata.json'} and follow the stale-reservation " - "recovery steps in the openshell-middleware-init README" + f"recovery steps in the {_TOOL_NAME} README" ) from error directory_fd = -1 try: @@ -405,9 +594,7 @@ def _acquire_lock( device=lock_stat.st_dev, inode=lock_stat.st_ino, destination=destination, - staging_path=( - destination.parent / f".{destination.name}.openshell-middleware-init.{token}" - ), + staging_path=(destination.parent / f".{destination.name}.{_TOOL_NAME}.{token}"), version=version, started_at=datetime.now(timezone.utc).isoformat(), ) @@ -567,8 +754,85 @@ def _publish_no_replace(source: Path, destination: Path) -> None: raise OSError(error_number, os.strerror(error_number), destination) +def _publish_exchange(source: Path, destination: Path) -> None: + """Atomically exchange a validated staged project with its existing project.""" + source_bytes = os.fsencode(source) + destination_bytes = os.fsencode(destination) + if sys.platform.startswith("linux"): + library = ctypes.CDLL(None, use_errno=True) + try: + rename = library.renameat2 + except AttributeError as error: # pragma: no cover - old Linux libc + raise InitializationError( + "this Linux runtime cannot atomically publish a project update" + ) from error + rename.argtypes = ( + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_uint, + ) + rename.restype = ctypes.c_int + result = rename(-100, source_bytes, -100, destination_bytes, 2) + elif sys.platform == "darwin": # pragma: no cover - platform-specific + library = ctypes.CDLL(None, use_errno=True) + rename = library.renamex_np + rename.argtypes = (ctypes.c_char_p, ctypes.c_char_p, ctypes.c_uint) + rename.restype = ctypes.c_int + result = rename(source_bytes, destination_bytes, 0x00000002) + else: # pragma: no cover - unsupported platform + raise InitializationError("this platform cannot atomically publish a project update") + + if result == 0: + return + error_number = ctypes.get_errno() + if error_number in {errno.EINVAL, errno.ENOSYS, errno.EOPNOTSUPP}: + raise InitializationError( + "the project filesystem does not support atomic update publication" + ) + raise OSError(error_number, os.strerror(error_number), destination) + + +def _publish_generated_artifacts( + staged_project: Path, project_dir: Path, metadata: ProjectMetadata +) -> None: + """Exchange refreshed artifacts in place, rolling back a normal publication failure.""" + relative_paths = [ + Path("proto/supervisor_middleware.proto"), + Path("uv.lock" if metadata.language == "python" else "Cargo.lock"), + ] + if metadata.language == "python": + if metadata.python_package is None: # pragma: no cover - checked during discovery + raise AssertionError("Python package is required") + relative_paths.append(Path("src") / metadata.python_package / "bindings") + relative_paths.append(Path("middleware-dev-manifest.json")) + + exchanged: list[Path] = [] + try: + for relative_path in relative_paths: + _publish_exchange( + staged_project / relative_path, + project_dir / relative_path, + ) + exchanged.append(relative_path) + except (InitializationError, OSError) as publish_error: + try: + for relative_path in reversed(exchanged): + _publish_exchange( + staged_project / relative_path, + project_dir / relative_path, + ) + except (InitializationError, OSError) as rollback_error: + raise PublicationRollbackError( + "artifact publication and rollback both failed; inspect the project and " + f"{staged_project} before removing the reservation" + ) from rollback_error + raise publish_error + + def _render_project(destination: Path, language: str, context: TemplateContext) -> None: - template_root = files("openshell_middleware_init").joinpath("templates").joinpath(language) + template_root = files("middleware_kit").joinpath("templates").joinpath(language) template_paths = { "python": ( ".gitignore", @@ -614,7 +878,7 @@ def _write_manifest( "languages": [language], "python_package": python_package, "generator": { - "name": "openshell-middleware-init", + "name": _TOOL_NAME, "version": __version__, }, } @@ -631,7 +895,7 @@ def _prepare_project(language: str, project_dir: Path, package_name: str) -> Non def _require_command(command: str) -> str: resolved = shutil.which(command) if resolved is None: - raise InitializationError(f"'{command}' is required to initialize this project") + raise InitializationError(f"'{command}' is required to prepare this project") return resolved @@ -691,7 +955,7 @@ def _prepare_python_project(project_dir: Path, package_name: str) -> None: ) grpc_module.write_text(generated.replace(absolute_import, relative_import, 1)) - with tempfile.TemporaryDirectory(prefix="openshell-middleware-init-python-") as environment: + with tempfile.TemporaryDirectory(prefix=f"{_TOOL_NAME}-python-") as environment: process_environment = os.environ.copy() process_environment.pop("VIRTUAL_ENV", None) process_environment["UV_PROJECT_ENVIRONMENT"] = environment @@ -717,7 +981,7 @@ def _prepare_python_project(project_dir: Path, package_name: str) -> None: def _prepare_rust_project(project_dir: Path) -> None: cargo = _require_command("cargo") - with tempfile.TemporaryDirectory(prefix="openshell-middleware-init-rust-") as target_dir: + with tempfile.TemporaryDirectory(prefix=f"{_TOOL_NAME}-rust-") as target_dir: process_environment = os.environ.copy() process_environment["CARGO_TARGET_DIR"] = target_dir _run( diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/.gitignore b/projects/middleware-kit/src/middleware_kit/templates/python/.gitignore similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/.gitignore rename to projects/middleware-kit/src/middleware_kit/templates/python/.gitignore diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/README.md b/projects/middleware-kit/src/middleware_kit/templates/python/README.md similarity index 92% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/README.md rename to projects/middleware-kit/src/middleware_kit/templates/python/README.md index 86ee41ba..3ebc6976 100644 --- a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/README.md +++ b/projects/middleware-kit/src/middleware_kit/templates/python/README.md @@ -62,9 +62,13 @@ for the policy syntax supported by your pinned OpenShell release. - `middleware-dev-manifest.json` records the release, source URL, and SHA-256. - `uv.lock` records the Python dependency solution. -Commit these files. When changing the OpenShell version, regenerate the project -or deliberately regenerate all four artifacts together; do not mix bindings and -contracts from different releases. +Commit these files. Refresh all version-matched artifacts together with: + +```sh +mkit update --openshell-version latest +``` + +Use a release tag instead of `latest` for a reproducible update. The starter is deliberately permissive. Before deployment, validate untrusted configuration, bound request and response work, avoid logging request content, diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/pyproject.toml b/projects/middleware-kit/src/middleware_kit/templates/python/pyproject.toml similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/pyproject.toml rename to projects/middleware-kit/src/middleware_kit/templates/python/pyproject.toml diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/src/package/__init__.py b/projects/middleware-kit/src/middleware_kit/templates/python/src/package/__init__.py similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/src/package/__init__.py rename to projects/middleware-kit/src/middleware_kit/templates/python/src/package/__init__.py diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/src/package/bindings/__init__.py b/projects/middleware-kit/src/middleware_kit/templates/python/src/package/bindings/__init__.py similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/src/package/bindings/__init__.py rename to projects/middleware-kit/src/middleware_kit/templates/python/src/package/bindings/__init__.py diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/src/package/server.py b/projects/middleware-kit/src/middleware_kit/templates/python/src/package/server.py similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/src/package/server.py rename to projects/middleware-kit/src/middleware_kit/templates/python/src/package/server.py diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/tests/test_server.py b/projects/middleware-kit/src/middleware_kit/templates/python/tests/test_server.py similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/tests/test_server.py rename to projects/middleware-kit/src/middleware_kit/templates/python/tests/test_server.py diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/.gitignore b/projects/middleware-kit/src/middleware_kit/templates/rust/.gitignore similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/.gitignore rename to projects/middleware-kit/src/middleware_kit/templates/rust/.gitignore diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/Cargo.toml b/projects/middleware-kit/src/middleware_kit/templates/rust/Cargo.toml similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/Cargo.toml rename to projects/middleware-kit/src/middleware_kit/templates/rust/Cargo.toml diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/README.md b/projects/middleware-kit/src/middleware_kit/templates/rust/README.md similarity index 92% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/README.md rename to projects/middleware-kit/src/middleware_kit/templates/rust/README.md index 8400f056..d8e9f83c 100644 --- a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/README.md +++ b/projects/middleware-kit/src/middleware_kit/templates/rust/README.md @@ -59,8 +59,13 @@ for the policy syntax supported by your pinned OpenShell release. - `middleware-dev-manifest.json` records the release, source URL, and SHA-256. - `Cargo.lock` records the Rust dependency solution. -Commit these files. When changing the OpenShell version, regenerate the project -or deliberately update the contract and manifest together. +Commit these files. Refresh all version-matched artifacts together with: + +```sh +mkit update --openshell-version latest +``` + +Use a release tag instead of `latest` for a reproducible update. The starter is deliberately permissive. Before deployment, validate untrusted configuration, bound request and response work, avoid logging request content, diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/build.rs b/projects/middleware-kit/src/middleware_kit/templates/rust/build.rs similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/build.rs rename to projects/middleware-kit/src/middleware_kit/templates/rust/build.rs diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/src/lib.rs b/projects/middleware-kit/src/middleware_kit/templates/rust/src/lib.rs similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/src/lib.rs rename to projects/middleware-kit/src/middleware_kit/templates/rust/src/lib.rs diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/src/main.rs b/projects/middleware-kit/src/middleware_kit/templates/rust/src/main.rs similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/src/main.rs rename to projects/middleware-kit/src/middleware_kit/templates/rust/src/main.rs diff --git a/projects/openshell-middleware-init/tests/test_cli.py b/projects/middleware-kit/tests/test_cli.py similarity index 51% rename from projects/openshell-middleware-init/tests/test_cli.py rename to projects/middleware-kit/tests/test_cli.py index 40d876a9..d578d212 100644 --- a/projects/openshell-middleware-init/tests/test_cli.py +++ b/projects/middleware-kit/tests/test_cli.py @@ -2,8 +2,8 @@ from typer.testing import CliRunner -from openshell_middleware_init import cli -from openshell_middleware_init.generator import InitializationError, InitializationResult +from middleware_kit import cli +from middleware_kit.generator import InitializationError, InitializationResult runner = CliRunner() @@ -12,8 +12,14 @@ def test_help_describes_required_choices() -> None: result = runner.invoke(cli.app, ["--help"]) assert result.exit_code == 0 - assert "--language" in result.stdout - assert "--openshell-version" in result.stdout + assert "create" in result.stdout + assert "update" in result.stdout + + create_help = runner.invoke(cli.app, ["create", "--help"]) + + assert create_help.exit_code == 0 + assert "--language" in create_help.stdout + assert "--openshell-version" in create_help.stdout def test_cli_reports_success(monkeypatch, tmp_path: Path) -> None: @@ -35,6 +41,7 @@ def fake_initialize_project(**options): result = runner.invoke( cli.app, [ + "create", "audit", "--language", "python", @@ -60,6 +67,7 @@ def fake_initialize_project(**options): result = runner.invoke( cli.app, [ + "create", "audit", "--language", "rust", @@ -72,3 +80,48 @@ def fake_initialize_project(**options): assert result.exit_code == 1 assert "error: output exists" in result.stderr + + +def test_cli_reports_update_success(monkeypatch, tmp_path: Path) -> None: + destination = tmp_path / "audit" + + def fake_update_project(**options): + assert options == { + "project_dir": destination, + "requested_version": "v1.2.3", + } + return InitializationResult( + destination=destination, + language="rust", + openshell_version="v1.2.3", + run_command="", + ) + + monkeypatch.setattr(cli, "update_project", fake_update_project) + + result = runner.invoke( + cli.app, + [ + "update", + str(destination), + "--openshell-version", + "v1.2.3", + ], + ) + + assert result.exit_code == 0 + assert "Updated rust middleware project" in result.stdout + assert "OpenShell contract: v1.2.3" in result.stdout + + +def test_cli_reports_update_error(monkeypatch, tmp_path: Path) -> None: + def fake_update_project(**options): + del options + raise InitializationError("not generated") + + monkeypatch.setattr(cli, "update_project", fake_update_project) + + result = runner.invoke(cli.app, ["update", str(tmp_path)]) + + assert result.exit_code == 1 + assert "error: not generated" in result.stderr diff --git a/projects/openshell-middleware-init/tests/test_generator.py b/projects/middleware-kit/tests/test_generator.py similarity index 70% rename from projects/openshell-middleware-init/tests/test_generator.py rename to projects/middleware-kit/tests/test_generator.py index d337561e..9fe4b795 100644 --- a/projects/openshell-middleware-init/tests/test_generator.py +++ b/projects/middleware-kit/tests/test_generator.py @@ -14,8 +14,8 @@ import pytest -from openshell_middleware_init import generator -from openshell_middleware_init.generator import InitializationError, initialize_project +from middleware_kit import generator +from middleware_kit.generator import InitializationError, initialize_project, update_project PROTO = b"""syntax = "proto3"; package openshell.middleware.v1; @@ -25,6 +25,7 @@ message HttpRequestEvaluation {} message HttpRequestResult {} """ +UPDATED_PROTO = PROTO + b"// refreshed contract\n" def local_proto(version: str) -> tuple[bytes, str]: @@ -63,7 +64,7 @@ def test_generates_python_project_with_provenance(tmp_path: Path) -> None: assert manifest["languages"] == ["python"] assert manifest["python_package"] == "audit_headers" assert len(manifest["proto_sha256"]) == 64 - assert not (tmp_path / ".audit-headers.openshell-middleware-init.lock").exists() + assert not (tmp_path / ".audit-headers.middleware-kit.lock").exists() def test_generates_rust_project_with_normalized_crate_name(tmp_path: Path) -> None: @@ -89,6 +90,313 @@ def test_generates_rust_project_with_normalized_crate_name(tmp_path: Path) -> No assert manifest["python_package"] is None +def test_updates_python_generated_artifacts_and_preserves_user_code(tmp_path: Path) -> None: + destination = tmp_path / "audit-headers" + initialize_project( + name="audit-headers", + language="python", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + server_path = destination / "src/audit_headers/server.py" + server_path.write_text(server_path.read_text() + "\n# user policy\n") + old_binding = destination / "src/audit_headers/bindings/old_generated.py" + old_binding.write_text("old generated code\n") + project_inode = destination.stat().st_ino + + def updated_proto(version: str) -> tuple[bytes, str]: + return UPDATED_PROTO, f"https://example.test/OpenShell/{version}/proto" + + result = update_project( + project_dir=destination, + requested_version="1.2.3", + download_proto=updated_proto, + command_runner=no_op_runner, + ) + + assert result.destination == destination + assert result.language == "python" + assert result.openshell_version == "v1.2.3" + assert destination.stat().st_ino == project_inode + assert server_path.read_text().endswith("# user policy\n") + assert (destination / "proto/supervisor_middleware.proto").read_bytes() == UPDATED_PROTO + assert not old_binding.exists() + assert (destination / "src/audit_headers/bindings/__init__.py").is_file() + manifest = json.loads((destination / "middleware-dev-manifest.json").read_text()) + assert manifest["openshell_version"] == "v1.2.3" + assert manifest["generator"]["name"] == "middleware-kit" + assert not (tmp_path / ".audit-headers.middleware-kit.lock").exists() + assert not list(tmp_path.glob(".audit-headers.middleware-kit.*")) + + +@pytest.mark.parametrize( + "legacy_name", + ["middleware-project", "openshell-middleware-init"], +) +def test_updates_rust_project_created_by_legacy_tool(tmp_path: Path, legacy_name: str) -> None: + destination = tmp_path / "audit" + initialize_project( + name="audit", + language="rust", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + manifest_path = destination / "middleware-dev-manifest.json" + manifest = json.loads(manifest_path.read_text()) + manifest["generator"]["name"] = legacy_name + manifest_path.write_text(json.dumps(manifest)) + + result = update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=no_op_runner, + ) + + assert result.language == "rust" + assert (destination / "proto/supervisor_middleware.proto").read_bytes() == UPDATED_PROTO + updated_manifest = json.loads(manifest_path.read_text()) + assert updated_manifest["generator"]["name"] == "middleware-kit" + + +def test_failed_update_keeps_original_project_unchanged(tmp_path: Path) -> None: + destination = tmp_path / "audit" + initialize_project( + name="audit", + language="rust", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + original_manifest = (destination / "middleware-dev-manifest.json").read_bytes() + original_proto = (destination / "proto/supervisor_middleware.proto").read_bytes() + + def fail_runner(language: str, project: Path, package: str) -> None: + del language, project, package + raise InitializationError("validation failed") + + with pytest.raises(InitializationError, match="validation failed"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=fail_runner, + ) + + assert (destination / "middleware-dev-manifest.json").read_bytes() == original_manifest + assert (destination / "proto/supervisor_middleware.proto").read_bytes() == original_proto + assert not (tmp_path / ".audit.middleware-kit.lock").exists() + assert not list(tmp_path.glob(".audit.middleware-kit.*")) + + +def test_publication_failure_rolls_back_exchanged_artifacts( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + destination = tmp_path / "audit" + initialize_project( + name="audit", + language="rust", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + original_manifest = (destination / "middleware-dev-manifest.json").read_bytes() + original_proto = (destination / "proto/supervisor_middleware.proto").read_bytes() + original_exchange = generator._publish_exchange + calls = 0 + + def fail_second_exchange(source: Path, target: Path) -> None: + nonlocal calls + calls += 1 + if calls == 2: + raise InitializationError("publication failed") + original_exchange(source, target) + + monkeypatch.setattr(generator, "_publish_exchange", fail_second_exchange) + + with pytest.raises(InitializationError, match="publication failed"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=no_op_runner, + ) + + assert calls == 3 + assert (destination / "middleware-dev-manifest.json").read_bytes() == original_manifest + assert (destination / "proto/supervisor_middleware.proto").read_bytes() == original_proto + + +def test_failed_publication_rollback_preserves_recovery_artifacts( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + destination = tmp_path / "audit" + initialize_project( + name="audit", + language="rust", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + original_exchange = generator._publish_exchange + calls = 0 + + def fail_publication_and_rollback(source: Path, target: Path) -> None: + nonlocal calls + calls += 1 + if calls >= 2: + raise InitializationError("exchange failed") + original_exchange(source, target) + + monkeypatch.setattr(generator, "_publish_exchange", fail_publication_and_rollback) + + with pytest.raises(generator.PublicationRollbackError, match="rollback both failed"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=no_op_runner, + ) + + assert calls == 3 + assert (tmp_path / ".audit.middleware-kit.lock").is_dir() + assert len(list(tmp_path.glob(".audit.middleware-kit.*"))) == 2 + + +def test_update_rejects_non_generated_project(tmp_path: Path) -> None: + destination = tmp_path / "not-generated" + destination.mkdir() + + with pytest.raises(InitializationError, match="missing regular manifest"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=local_proto, + command_runner=no_op_runner, + ) + + +def test_update_rejects_symlink_and_missing_project_paths(tmp_path: Path) -> None: + missing = tmp_path / "missing" + symlink = tmp_path / "symlink" + symlink.symlink_to(missing, target_is_directory=True) + + with pytest.raises(InitializationError, match="must not be a symlink"): + update_project( + project_dir=symlink, + download_proto=local_proto, + command_runner=no_op_runner, + ) + with pytest.raises(InitializationError, match="existing directory"): + update_project( + project_dir=missing, + download_proto=local_proto, + command_runner=no_op_runner, + ) + + +@pytest.mark.parametrize( + ("manifest", "message"), + [ + ("not json", "could not read"), + ("[]", "JSON object"), + ('{"generator": {"name": "other"}}', "was not created"), + ( + '{"generator": {"name": "middleware-kit"}, "languages": ["python", "rust"]}', + "exactly one", + ), + ( + '{"generator": {"name": "middleware-kit"}, ' + '"languages": ["python"], "python_package": "Bad-Package"}', + "invalid or missing", + ), + ( + '{"generator": {"name": "middleware-kit"}, ' + '"languages": ["rust"], "python_package": "unexpected"}', + "must set python_package", + ), + ], +) +def test_update_rejects_invalid_manifest(tmp_path: Path, manifest: str, message: str) -> None: + destination = tmp_path / "project" + destination.mkdir() + (destination / "middleware-dev-manifest.json").write_text(manifest) + + with pytest.raises(InitializationError, match=message): + update_project( + project_dir=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + + +def test_update_requires_regular_generated_artifacts(tmp_path: Path) -> None: + destination = tmp_path / "project" + destination.mkdir() + (destination / "middleware-dev-manifest.json").write_text( + json.dumps( + { + "generator": {"name": "middleware-kit"}, + "languages": ["rust"], + "python_package": None, + } + ) + ) + + with pytest.raises(InitializationError, match="regular file"): + update_project( + project_dir=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + + +def test_update_requires_python_bindings_directory(tmp_path: Path) -> None: + destination = tmp_path / "project" + (destination / "proto").mkdir(parents=True) + (destination / "proto/supervisor_middleware.proto").write_bytes(PROTO) + (destination / "uv.lock").write_text("test lock\n") + (destination / "middleware-dev-manifest.json").write_text( + json.dumps( + { + "generator": {"name": "middleware-kit"}, + "languages": ["python"], + "python_package": "audit", + } + ) + ) + + with pytest.raises(InitializationError, match="bindings must be"): + update_project( + project_dir=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + + +def test_update_detects_replaced_project_before_publication(tmp_path: Path) -> None: + project = tmp_path / "project" + replacement = tmp_path / "replacement" + project.mkdir() + original = project.stat() + project.rename(replacement) + project.mkdir() + + with pytest.raises(InitializationError, match="changed during update"): + generator._verify_project_identity(project, original.st_dev, original.st_ino) + + project.rmdir() + with pytest.raises(InitializationError, match="changed during update"): + generator._verify_project_identity(project, original.st_dev, original.st_ino) + + def test_python_package_name_can_be_overridden(tmp_path: Path) -> None: destination = tmp_path / "project" @@ -246,9 +554,9 @@ def test_refuses_a_dangling_destination_symlink(tmp_path: Path) -> None: def test_refuses_a_reserved_destination(tmp_path: Path) -> None: destination = tmp_path / "reserved" - (tmp_path / ".reserved.openshell-middleware-init.lock").mkdir() + (tmp_path / ".reserved.middleware-kit.lock").mkdir() - with pytest.raises(InitializationError, match="reserved by another initializer"): + with pytest.raises(InitializationError, match="reserved by another middleware-kit"): initialize_project( name="reserved", language="rust", @@ -283,7 +591,7 @@ def collide_before_publish(source: Path, final_output: Path) -> None: ) assert (destination / "owned-by-other-process").read_text() == "keep me\n" - assert not (tmp_path / ".contended.openshell-middleware-init.lock").exists() + assert not (tmp_path / ".contended.middleware-kit.lock").exists() def test_failure_cleans_staging_and_owned_reservation(tmp_path: Path) -> None: @@ -304,8 +612,8 @@ def fail_runner(language: str, project: Path, package: str) -> None: ) assert not destination.exists() - assert not (tmp_path / ".failing.openshell-middleware-init.lock").exists() - assert not list(tmp_path.glob(".failing.openshell-middleware-init.*")) + assert not (tmp_path / ".failing.middleware-kit.lock").exists() + assert not list(tmp_path.glob(".failing.middleware-kit.*")) def test_rejects_an_unexpected_proto(tmp_path: Path) -> None: @@ -435,6 +743,10 @@ def fail(*args: object, **kwargs: object) -> None: generator._download_proto("v1.2.3") +def test_network_error_reason_handles_plain_os_error() -> None: + assert generator._network_error_reason(OSError("offline")) == "offline" + + def test_download_retries_transient_failures(monkeypatch: pytest.MonkeyPatch) -> None: attempts = 0 response = FakeResponse(body=PROTO) diff --git a/projects/openshell-middleware-init/uv.lock b/projects/middleware-kit/uv.lock similarity index 99% rename from projects/openshell-middleware-init/uv.lock rename to projects/middleware-kit/uv.lock index bfd92049..1a28bb8b 100644 --- a/projects/openshell-middleware-init/uv.lock +++ b/projects/middleware-kit/uv.lock @@ -166,7 +166,7 @@ wheels = [ ] [[package]] -name = "openshell-middleware-init" +name = "middleware-kit" version = "0.1.0" source = { editable = "." } dependencies = [ diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/__init__.py b/projects/openshell-middleware-init/src/openshell_middleware_init/__init__.py deleted file mode 100644 index 03f94f25..00000000 --- a/projects/openshell-middleware-init/src/openshell_middleware_init/__init__.py +++ /dev/null @@ -1,3 +0,0 @@ -"""Generate version-matched OpenShell middleware projects.""" - -__version__ = "0.1.0" From dbff748923d16cd150118cdca2c05d74d2fa611d Mon Sep 17 00:00:00 2001 From: Johnny Greco Date: Thu, 23 Jul 2026 19:16:00 +0000 Subject: [PATCH 2/6] Harden middleware project updates --- projects/middleware-kit/README.md | 23 +++ .../src/middleware_kit/generator.py | 63 +++++++- .../middleware_kit/templates/rust/src/main.rs | 2 +- .../middleware-kit/tests/test_generator.py | 135 ++++++++++++++++-- 4 files changed, 205 insertions(+), 18 deletions(-) diff --git a/projects/middleware-kit/README.md b/projects/middleware-kit/README.md index e54620ca..015505cf 100644 --- a/projects/middleware-kit/README.md +++ b/projects/middleware-kit/README.md @@ -40,6 +40,29 @@ uv sync --locked uv run mkit --help ``` +## Migrate from the initializer + +The distribution, executable, and creation syntax have changed: + +| Aspect | Before | Now | +| --- | --- | --- | +| Distribution | `openshell-middleware-init` | `middleware-kit` | +| Executable | `openshell-middleware-init` | `mkit` | +| Repository path | `projects/openshell-middleware-init` | `projects/middleware-kit` | +| Create syntax | `openshell-middleware-init ...` | `mkit create ...` | + +Replace an existing tool installation with: + +```sh +uv tool uninstall openshell-middleware-init +uv tool install \ + "middleware-kit @ git+https://github.com/NVIDIA/OpenShell-Research.git#subdirectory=projects/middleware-kit" +``` + +Generated projects do not need to be recreated. `mkit update` recognizes +manifests written by both `openshell-middleware-init` and the interim +`middleware-project` name. + ## Quick start Generate and run a Python starter with the installed command: diff --git a/projects/middleware-kit/src/middleware_kit/generator.py b/projects/middleware-kit/src/middleware_kit/generator.py index 2f2d4400..2e0f69b8 100644 --- a/projects/middleware-kit/src/middleware_kit/generator.py +++ b/projects/middleware-kit/src/middleware_kit/generator.py @@ -38,6 +38,18 @@ _NETWORK_ATTEMPTS = 4 _TOOL_NAME = "middleware-kit" _LEGACY_TOOL_NAMES = {"middleware-project", "openshell-middleware-init"} +_STAGING_IGNORED_ROOT_ENTRIES = { + ".coverage", + ".git", + ".mypy_cache", + ".pytest_cache", + ".ruff_cache", + ".ty_cache", + ".venv", + "dist", + "htmlcov", + "target", +} _RUST_KEYWORDS = { "abstract", "as", @@ -274,7 +286,7 @@ def update_project( preserve_recovery = False try: _validate_existing_project(project_dir) - shutil.copytree(project_dir, reservation.staging_path, symlinks=True) + _copy_project_to_staging(project_dir, reservation.staging_path) staging_path = reservation.staging_path proto, proto_url = downloader(version) _validate_proto(proto, version) @@ -523,6 +535,7 @@ def _validate_refresh_targets(project_dir: Path, language: str, python_package: ] regular_files.append(project_dir / ("uv.lock" if language == "python" else "Cargo.lock")) for path in regular_files: + _reject_symlink_components(project_dir, path) if path.is_symlink() or not path.is_file(): raise InitializationError(f"generated artifact must be a regular file: {path}") @@ -530,12 +543,44 @@ def _validate_refresh_targets(project_dir: Path, language: str, python_package: if python_package is None: # pragma: no cover - checked by caller raise AssertionError("Python package is required") bindings_dir = project_dir / "src" / python_package / "bindings" + _reject_symlink_components(project_dir, bindings_dir) if bindings_dir.is_symlink() or not bindings_dir.is_dir(): raise InitializationError( f"generated bindings must be an existing directory: {bindings_dir}" ) +def _reject_symlink_components(project_dir: Path, target: Path) -> None: + relative_target = target.relative_to(project_dir) + current = project_dir + for component in relative_target.parts: + current /= component + try: + current_stat = current.lstat() + except FileNotFoundError: + return + if stat.S_ISLNK(current_stat.st_mode): + raise InitializationError( + f"generated artifact path must not contain symlinks: {current}" + ) + + +def _copy_project_to_staging(project_dir: Path, staging_path: Path) -> None: + def ignore_disposable_entries(directory: str, names: list[str]) -> set[str]: + ignored = {"__pycache__"} & set(names) + if Path(directory) == project_dir: + ignored.update(_STAGING_IGNORED_ROOT_ENTRIES & set(names)) + return ignored + + shutil.copytree( + project_dir, + staging_path, + symlinks=True, + ignore=ignore_disposable_entries, + ) + staging_path.chmod(0o700) + + def _refresh_generated_artifacts( project_dir: Path, *, @@ -544,6 +589,11 @@ def _refresh_generated_artifacts( proto_url: str, proto: bytes, ) -> None: + _validate_refresh_targets( + project_dir, + metadata.language, + metadata.python_package, + ) proto_path = project_dir / "proto" / "supervisor_middleware.proto" proto_path.write_bytes(proto) if metadata.language == "python": @@ -970,9 +1020,7 @@ def _prepare_python_project(project_dir: Path, package_name: str) -> None: "run", "--project", str(project_dir), - "python", - "-c", - f"from {package_name}.server import Middleware", + "pytest", ), cwd=project_dir, environment=process_environment, @@ -985,7 +1033,12 @@ def _prepare_rust_project(project_dir: Path) -> None: process_environment = os.environ.copy() process_environment["CARGO_TARGET_DIR"] = target_dir _run( - (cargo, "check", "--manifest-path", str(project_dir / "Cargo.toml")), + (cargo, "fmt", "--manifest-path", str(project_dir / "Cargo.toml"), "--check"), + cwd=project_dir, + environment=process_environment, + ) + _run( + (cargo, "test", "--manifest-path", str(project_dir / "Cargo.toml")), cwd=project_dir, environment=process_environment, ) diff --git a/projects/middleware-kit/src/middleware_kit/templates/rust/src/main.rs b/projects/middleware-kit/src/middleware_kit/templates/rust/src/main.rs index d0ea5378..911ddb51 100644 --- a/projects/middleware-kit/src/middleware_kit/templates/rust/src/main.rs +++ b/projects/middleware-kit/src/middleware_kit/templates/rust/src/main.rs @@ -1,7 +1,7 @@ use std::{env, error::Error, net::SocketAddr}; -use __RUST_LIB_NAME__::middleware_service; use tonic::transport::Server; +use __RUST_LIB_NAME__::middleware_service; #[tokio::main] async fn main() -> Result<(), Box> { diff --git a/projects/middleware-kit/tests/test_generator.py b/projects/middleware-kit/tests/test_generator.py index 9fe4b795..966fe58d 100644 --- a/projects/middleware-kit/tests/test_generator.py +++ b/projects/middleware-kit/tests/test_generator.py @@ -381,6 +381,108 @@ def test_update_requires_python_bindings_directory(tmp_path: Path) -> None: ) +def test_update_rejects_symlinked_proto_directory_without_touching_target( + tmp_path: Path, +) -> None: + destination = tmp_path / "project" + initialize_project( + name="project", + language="rust", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + (destination / "proto").rename(destination / "original-proto") + external_proto = tmp_path / "external-proto" + external_proto.mkdir() + sentinel = external_proto / "supervisor_middleware.proto" + sentinel.write_text("external contract\n") + (destination / "proto").symlink_to(external_proto, target_is_directory=True) + + with pytest.raises(InitializationError, match="must not contain symlinks"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=no_op_runner, + ) + + assert sentinel.read_text() == "external contract\n" + + +def test_update_rejects_symlinked_python_package_without_touching_target( + tmp_path: Path, +) -> None: + destination = tmp_path / "project" + initialize_project( + name="project", + language="python", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + package_dir = destination / "src/project" + package_dir.rename(destination / "original-package") + external_package = tmp_path / "external-package" + bindings = external_package / "bindings" + bindings.mkdir(parents=True) + sentinel = bindings / "keep.txt" + sentinel.write_text("external binding\n") + package_dir.symlink_to(external_package, target_is_directory=True) + + with pytest.raises(InitializationError, match="must not contain symlinks"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=no_op_runner, + ) + + assert sentinel.read_text() == "external binding\n" + + +def test_update_omits_disposable_directories_from_staging(tmp_path: Path) -> None: + destination = tmp_path / "project" + initialize_project( + name="project", + language="python", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + for directory_name in (".git", ".venv", ".pytest_cache", "dist", "target"): + disposable = destination / directory_name + disposable.mkdir() + (disposable / "large-cache").write_text("not needed\n") + nested_cache = destination / "src/project/__pycache__" + nested_cache.mkdir() + (nested_cache / "server.pyc").write_bytes(b"cache") + user_file = destination / "policy-notes.txt" + user_file.write_text("preserve me\n") + + def inspect_staging(language: str, project: Path, package: str) -> None: + assert language == "python" + assert package == "project" + assert user_file.name in {path.name for path in project.iterdir()} + for directory_name in (".git", ".venv", ".pytest_cache", "dist", "target"): + assert not (project / directory_name).exists() + assert not (project / "src/project/__pycache__").exists() + no_op_runner(language, project, package) + + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=inspect_staging, + ) + + assert user_file.read_text() == "preserve me\n" + assert (destination / ".venv/large-cache").read_text() == "not needed\n" + + def test_update_detects_replaced_project_before_publication(tmp_path: Path) -> None: project = tmp_path / "project" replacement = tmp_path / "replacement" @@ -1089,6 +1191,7 @@ def fake_run(command, *, cwd, environment=None) -> None: assert generated.startswith("from . import supervisor_middleware_pb2") assert len(calls) == 3 assert calls[1][1] == "sync" + assert calls[2][-1] == "pytest" def test_prepare_python_rejects_unexpected_generated_import( @@ -1113,23 +1216,31 @@ def fake_run(command, *, cwd, environment=None) -> None: def test_prepare_rust_runs_cargo_with_temporary_target( monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: - observed: dict[str, object] = {} + observed: list[tuple[tuple[str, ...], Path, dict[str, str]]] = [] monkeypatch.setattr(generator, "_require_command", lambda command: f"/tools/{command}") def fake_run(command, *, cwd, environment=None) -> None: - observed.update(command=command, cwd=cwd, environment=environment) + assert environment is not None + observed.append((tuple(command), cwd, environment)) monkeypatch.setattr(generator, "_run", fake_run) generator._prepare_rust_project(tmp_path) - assert observed["command"] == ( - "/tools/cargo", - "check", - "--manifest-path", - str(tmp_path / "Cargo.toml"), - ) - assert observed["cwd"] == tmp_path - environment = observed["environment"] - assert isinstance(environment, dict) - assert "CARGO_TARGET_DIR" in environment + assert [command for command, _, _ in observed] == [ + ( + "/tools/cargo", + "fmt", + "--manifest-path", + str(tmp_path / "Cargo.toml"), + "--check", + ), + ( + "/tools/cargo", + "test", + "--manifest-path", + str(tmp_path / "Cargo.toml"), + ), + ] + assert all(cwd == tmp_path for _, cwd, _ in observed) + assert all("CARGO_TARGET_DIR" in environment for _, _, environment in observed) From 46bf4eac02207f3479b3679790bae690d13379cd Mon Sep 17 00:00:00 2001 From: Johnny Greco Date: Thu, 23 Jul 2026 19:20:26 +0000 Subject: [PATCH 3/6] Secure staged artifact publication --- .../src/middleware_kit/generator.py | 150 +++++++++++++----- .../middleware_kit/templates/rust/src/main.rs | 3 +- .../middleware-kit/tests/test_generator.py | 94 +++++++++-- 3 files changed, 199 insertions(+), 48 deletions(-) diff --git a/projects/middleware-kit/src/middleware_kit/generator.py b/projects/middleware-kit/src/middleware_kit/generator.py index 2e0f69b8..67a1fdf3 100644 --- a/projects/middleware-kit/src/middleware_kit/generator.py +++ b/projects/middleware-kit/src/middleware_kit/generator.py @@ -300,6 +300,16 @@ def update_project( runner(metadata.language, staging_path, metadata.python_package or "unused") _verify_lock(reservation) _verify_project_identity(project_dir, project_stat.st_dev, project_stat.st_ino) + _validate_refresh_targets( + staging_path, + metadata.language, + metadata.python_package, + ) + _validate_refresh_targets( + project_dir, + metadata.language, + metadata.python_package, + ) _publish_generated_artifacts(staging_path, project_dir, metadata) published = True except PublicationRollbackError: @@ -806,42 +816,113 @@ def _publish_no_replace(source: Path, destination: Path) -> None: def _publish_exchange(source: Path, destination: Path) -> None: """Atomically exchange a validated staged project with its existing project.""" - source_bytes = os.fsencode(source) - destination_bytes = os.fsencode(destination) - if sys.platform.startswith("linux"): - library = ctypes.CDLL(None, use_errno=True) - try: - rename = library.renameat2 - except AttributeError as error: # pragma: no cover - old Linux libc - raise InitializationError( - "this Linux runtime cannot atomically publish a project update" - ) from error - rename.argtypes = ( - ctypes.c_int, - ctypes.c_char_p, - ctypes.c_int, - ctypes.c_char_p, - ctypes.c_uint, + source_parent_fd = _open_parent_directory_no_follow(source) + destination_parent_fd = _open_parent_directory_no_follow(destination) + try: + _validate_exchange_entries( + source_parent_fd, + source.name, + destination_parent_fd, + destination.name, ) - rename.restype = ctypes.c_int - result = rename(-100, source_bytes, -100, destination_bytes, 2) - elif sys.platform == "darwin": # pragma: no cover - platform-specific - library = ctypes.CDLL(None, use_errno=True) - rename = library.renamex_np - rename.argtypes = (ctypes.c_char_p, ctypes.c_char_p, ctypes.c_uint) - rename.restype = ctypes.c_int - result = rename(source_bytes, destination_bytes, 0x00000002) - else: # pragma: no cover - unsupported platform - raise InitializationError("this platform cannot atomically publish a project update") + source_name = os.fsencode(source.name) + destination_name = os.fsencode(destination.name) + if sys.platform.startswith("linux"): + library = ctypes.CDLL(None, use_errno=True) + try: + rename = library.renameat2 + except AttributeError as error: # pragma: no cover - old Linux libc + raise InitializationError( + "this Linux runtime cannot atomically publish a project update" + ) from error + rename.argtypes = ( + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_uint, + ) + rename.restype = ctypes.c_int + result = rename( + source_parent_fd, + source_name, + destination_parent_fd, + destination_name, + 2, + ) + elif sys.platform == "darwin": # pragma: no cover - platform-specific + library = ctypes.CDLL(None, use_errno=True) + rename = library.renameatx_np + rename.argtypes = ( + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_uint, + ) + rename.restype = ctypes.c_int + result = rename( + source_parent_fd, + source_name, + destination_parent_fd, + destination_name, + 0x00000002, + ) + else: # pragma: no cover - unsupported platform + raise InitializationError("this platform cannot atomically publish a project update") - if result == 0: - return - error_number = ctypes.get_errno() - if error_number in {errno.EINVAL, errno.ENOSYS, errno.EOPNOTSUPP}: + if result == 0: + return + error_number = ctypes.get_errno() + if error_number in {errno.EINVAL, errno.ENOSYS, errno.EOPNOTSUPP}: + raise InitializationError( + "the project filesystem does not support atomic update publication" + ) + raise OSError(error_number, os.strerror(error_number), destination) + finally: + os.close(source_parent_fd) + os.close(destination_parent_fd) + + +def _open_parent_directory_no_follow(path: Path) -> int: + if not path.is_absolute(): + raise InitializationError(f"artifact path must be absolute: {path}") + flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0) + descriptor = os.open(path.anchor, flags) + try: + for component in path.parent.parts[1:]: + next_descriptor = os.open(component, flags, dir_fd=descriptor) + os.close(descriptor) + descriptor = next_descriptor + except OSError: + os.close(descriptor) + raise + return descriptor + + +def _validate_exchange_entries( + source_parent_fd: int, + source_name: str, + destination_parent_fd: int, + destination_name: str, +) -> None: + source_stat = os.stat(source_name, dir_fd=source_parent_fd, follow_symlinks=False) + destination_stat = os.stat( + destination_name, + dir_fd=destination_parent_fd, + follow_symlinks=False, + ) + source_is_directory = stat.S_ISDIR(source_stat.st_mode) + destination_is_directory = stat.S_ISDIR(destination_stat.st_mode) + source_is_regular = stat.S_ISREG(source_stat.st_mode) + destination_is_regular = stat.S_ISREG(destination_stat.st_mode) + if not ( + (source_is_directory and destination_is_directory) + or (source_is_regular and destination_is_regular) + ): raise InitializationError( - "the project filesystem does not support atomic update publication" + "generated artifacts changed type during update; refusing to publish" ) - raise OSError(error_number, os.strerror(error_number), destination) def _publish_generated_artifacts( @@ -1032,11 +1113,6 @@ def _prepare_rust_project(project_dir: Path) -> None: with tempfile.TemporaryDirectory(prefix=f"{_TOOL_NAME}-rust-") as target_dir: process_environment = os.environ.copy() process_environment["CARGO_TARGET_DIR"] = target_dir - _run( - (cargo, "fmt", "--manifest-path", str(project_dir / "Cargo.toml"), "--check"), - cwd=project_dir, - environment=process_environment, - ) _run( (cargo, "test", "--manifest-path", str(project_dir / "Cargo.toml")), cwd=project_dir, diff --git a/projects/middleware-kit/src/middleware_kit/templates/rust/src/main.rs b/projects/middleware-kit/src/middleware_kit/templates/rust/src/main.rs index 911ddb51..7af6266b 100644 --- a/projects/middleware-kit/src/middleware_kit/templates/rust/src/main.rs +++ b/projects/middleware-kit/src/middleware_kit/templates/rust/src/main.rs @@ -1,7 +1,6 @@ use std::{env, error::Error, net::SocketAddr}; use tonic::transport::Server; -use __RUST_LIB_NAME__::middleware_service; #[tokio::main] async fn main() -> Result<(), Box> { @@ -12,7 +11,7 @@ async fn main() -> Result<(), Box> { println!("serving __SERVICE_NAME__ on {address}"); Server::builder() - .add_service(middleware_service()) + .add_service(__RUST_LIB_NAME__::middleware_service()) .serve(address) .await?; Ok(()) diff --git a/projects/middleware-kit/tests/test_generator.py b/projects/middleware-kit/tests/test_generator.py index 966fe58d..02d4d091 100644 --- a/projects/middleware-kit/tests/test_generator.py +++ b/projects/middleware-kit/tests/test_generator.py @@ -83,7 +83,10 @@ def test_generates_rust_project_with_normalized_crate_name(tmp_path: Path) -> No cargo = (destination / "Cargo.toml").read_text() assert 'name = "request-audit"' in cargo assert '[lib]\nname = "request_audit"' in cargo - assert "use request_audit::" in (destination / "src/main.rs").read_text() + assert ( + ".add_service(request_audit::middleware_service())" + in (destination / "src/main.rs").read_text() + ) assert stat.S_IMODE(destination.stat().st_mode) == 0o755 manifest = json.loads((destination / "middleware-dev-manifest.json").read_text()) assert manifest["languages"] == ["rust"] @@ -483,6 +486,83 @@ def inspect_staging(language: str, project: Path, package: str) -> None: assert (destination / ".venv/large-cache").read_text() == "not needed\n" +@pytest.mark.parametrize("replace_live_path", [False, True]) +def test_update_revalidates_symlink_ancestors_after_project_validation( + tmp_path: Path, replace_live_path: bool +) -> None: + destination = tmp_path / "project" + initialize_project( + name="project", + language="rust", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + original_manifest = (destination / "middleware-dev-manifest.json").read_bytes() + external_proto = tmp_path / "external-proto" + external_proto.mkdir() + sentinel = external_proto / "supervisor_middleware.proto" + sentinel.write_text("external contract\n") + + def replace_proto_ancestor(language: str, staged_project: Path, package: str) -> None: + no_op_runner(language, staged_project, package) + project_to_change = destination if replace_live_path else staged_project + (project_to_change / "proto").rename(project_to_change / "original-proto") + (project_to_change / "proto").symlink_to(external_proto, target_is_directory=True) + + with pytest.raises(InitializationError, match="must not contain symlinks"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=replace_proto_ancestor, + ) + + assert sentinel.read_text() == "external contract\n" + assert (destination / "middleware-dev-manifest.json").read_bytes() == original_manifest + + +def test_exchange_refuses_symlinked_parent_created_immediately_before_publish( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + destination = tmp_path / "project" + initialize_project( + name="project", + language="rust", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + external_proto = tmp_path / "external-proto" + external_proto.mkdir() + sentinel = external_proto / "supervisor_middleware.proto" + sentinel.write_text("external contract\n") + original_exchange = generator._publish_exchange + first_exchange = True + + def replace_parent_then_exchange(source: Path, target: Path) -> None: + nonlocal first_exchange + if first_exchange: + first_exchange = False + target.parent.rename(destination / "original-proto") + target.parent.symlink_to(external_proto, target_is_directory=True) + original_exchange(source, target) + + monkeypatch.setattr(generator, "_publish_exchange", replace_parent_then_exchange) + + with pytest.raises(InitializationError): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=no_op_runner, + ) + + assert sentinel.read_text() == "external contract\n" + + def test_update_detects_replaced_project_before_publication(tmp_path: Path) -> None: project = tmp_path / "project" replacement = tmp_path / "replacement" @@ -556,7 +636,10 @@ def test_rust_project_names_get_valid_explicit_library_names( cargo = (destination / "Cargo.toml").read_text() assert f'name = "{crate}"' in cargo assert f'[lib]\nname = "{library}"' in cargo - assert f"use {library}::middleware_service;" in (destination / "src/main.rs").read_text() + assert ( + f".add_service({library}::middleware_service())" + in (destination / "src/main.rs").read_text() + ) def test_unsupported_platform_fails_before_filesystem_changes( @@ -1228,13 +1311,6 @@ def fake_run(command, *, cwd, environment=None) -> None: generator._prepare_rust_project(tmp_path) assert [command for command, _, _ in observed] == [ - ( - "/tools/cargo", - "fmt", - "--manifest-path", - str(tmp_path / "Cargo.toml"), - "--check", - ), ( "/tools/cargo", "test", From 5f8e4ed08643d785b16de959be1943ac4bb91352 Mon Sep 17 00:00:00 2001 From: Johnny Greco Date: Thu, 23 Jul 2026 19:28:50 +0000 Subject: [PATCH 4/6] Verify anchored artifact exchanges --- .../src/middleware_kit/generator.py | 148 ++++++++++++------ .../middleware-kit/tests/test_generator.py | 30 ++++ 2 files changed, 127 insertions(+), 51 deletions(-) diff --git a/projects/middleware-kit/src/middleware_kit/generator.py b/projects/middleware-kit/src/middleware_kit/generator.py index 67a1fdf3..1816c2d9 100644 --- a/projects/middleware-kit/src/middleware_kit/generator.py +++ b/projects/middleware-kit/src/middleware_kit/generator.py @@ -827,70 +827,100 @@ def _publish_exchange(source: Path, destination: Path) -> None: ) source_name = os.fsencode(source.name) destination_name = os.fsencode(destination.name) - if sys.platform.startswith("linux"): - library = ctypes.CDLL(None, use_errno=True) - try: - rename = library.renameat2 - except AttributeError as error: # pragma: no cover - old Linux libc - raise InitializationError( - "this Linux runtime cannot atomically publish a project update" - ) from error - rename.argtypes = ( - ctypes.c_int, - ctypes.c_char_p, - ctypes.c_int, - ctypes.c_char_p, - ctypes.c_uint, - ) - rename.restype = ctypes.c_int - result = rename( - source_parent_fd, - source_name, - destination_parent_fd, - destination_name, - 2, - ) - elif sys.platform == "darwin": # pragma: no cover - platform-specific - library = ctypes.CDLL(None, use_errno=True) - rename = library.renameatx_np - rename.argtypes = ( - ctypes.c_int, - ctypes.c_char_p, - ctypes.c_int, - ctypes.c_char_p, - ctypes.c_uint, - ) - rename.restype = ctypes.c_int - result = rename( - source_parent_fd, - source_name, - destination_parent_fd, - destination_name, - 0x00000002, - ) - else: # pragma: no cover - unsupported platform - raise InitializationError("this platform cannot atomically publish a project update") - - if result == 0: + result = _exchange_at( + source_parent_fd, + source_name, + destination_parent_fd, + destination_name, + ) + if result != 0: + _raise_exchange_error(destination) + source_attached = _directory_fd_matches_path(source.parent, source_parent_fd) + destination_attached = _directory_fd_matches_path( + destination.parent, + destination_parent_fd, + ) + if source_attached and destination_attached: return - error_number = ctypes.get_errno() - if error_number in {errno.EINVAL, errno.ENOSYS, errno.EOPNOTSUPP}: - raise InitializationError( - "the project filesystem does not support atomic update publication" + + reverse_result = _exchange_at( + source_parent_fd, + source_name, + destination_parent_fd, + destination_name, + ) + if reverse_result != 0: + error_number = ctypes.get_errno() + raise PublicationRollbackError( + "an artifact parent changed during publication and the anchored exchange " + f"could not be reversed: {os.strerror(error_number)}" ) - raise OSError(error_number, os.strerror(error_number), destination) + raise InitializationError( + "an artifact parent changed during publication; the exchange was reversed" + ) finally: os.close(source_parent_fd) os.close(destination_parent_fd) +def _exchange_at( + source_parent_fd: int, + source_name: bytes, + destination_parent_fd: int, + destination_name: bytes, +) -> int: + if sys.platform.startswith("linux"): + library = ctypes.CDLL(None, use_errno=True) + try: + rename = library.renameat2 + except AttributeError as error: # pragma: no cover - old Linux libc + raise InitializationError( + "this Linux runtime cannot atomically publish a project update" + ) from error + exchange_flag = 2 + elif sys.platform == "darwin": # pragma: no cover - platform-specific + library = ctypes.CDLL(None, use_errno=True) + rename = library.renameatx_np + exchange_flag = 0x00000002 + else: # pragma: no cover - unsupported platform + raise InitializationError("this platform cannot atomically publish a project update") + rename.argtypes = ( + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_uint, + ) + rename.restype = ctypes.c_int + return rename( + source_parent_fd, + source_name, + destination_parent_fd, + destination_name, + exchange_flag, + ) + + +def _raise_exchange_error(destination: Path) -> None: + error_number = ctypes.get_errno() + if error_number in {errno.EINVAL, errno.ENOSYS, errno.EOPNOTSUPP}: + raise InitializationError( + "the project filesystem does not support atomic update publication" + ) + raise OSError(error_number, os.strerror(error_number), destination) + + def _open_parent_directory_no_follow(path: Path) -> int: + return _open_directory_no_follow(path.parent) + + +def _open_directory_no_follow(path: Path) -> int: if not path.is_absolute(): raise InitializationError(f"artifact path must be absolute: {path}") flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0) descriptor = os.open(path.anchor, flags) try: - for component in path.parent.parts[1:]: + for component in path.parts[1:]: next_descriptor = os.open(component, flags, dir_fd=descriptor) os.close(descriptor) descriptor = next_descriptor @@ -900,6 +930,22 @@ def _open_parent_directory_no_follow(path: Path) -> int: return descriptor +def _directory_fd_matches_path(path: Path, expected_descriptor: int) -> bool: + try: + current_descriptor = _open_directory_no_follow(path) + except (InitializationError, OSError): + return False + try: + expected_stat = os.fstat(expected_descriptor) + current_stat = os.fstat(current_descriptor) + return ( + expected_stat.st_dev == current_stat.st_dev + and expected_stat.st_ino == current_stat.st_ino + ) + finally: + os.close(current_descriptor) + + def _validate_exchange_entries( source_parent_fd: int, source_name: str, diff --git a/projects/middleware-kit/tests/test_generator.py b/projects/middleware-kit/tests/test_generator.py index 02d4d091..98d8c698 100644 --- a/projects/middleware-kit/tests/test_generator.py +++ b/projects/middleware-kit/tests/test_generator.py @@ -1162,6 +1162,36 @@ def test_publish_no_replace_moves_into_absent_destination(tmp_path: Path) -> Non assert (destination / "generated").read_text() == "ready\n" +def test_exchange_reverses_when_open_parent_is_detached( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + source_parent = tmp_path / "staged/proto" + destination_parent = tmp_path / "project/proto" + source_parent.mkdir(parents=True) + destination_parent.mkdir(parents=True) + source = source_parent / "supervisor_middleware.proto" + destination = destination_parent / "supervisor_middleware.proto" + source.write_text("new contract\n") + destination.write_text("old contract\n") + moved_destination_parent = tmp_path / "project/moved-proto" + validate_exchange_entries = generator._validate_exchange_entries + + def validate_then_detach(*arguments) -> None: + validate_exchange_entries(*arguments) + destination_parent.rename(moved_destination_parent) + destination_parent.mkdir() + (destination_parent / destination.name).write_text("concurrent replacement\n") + + monkeypatch.setattr(generator, "_validate_exchange_entries", validate_then_detach) + + with pytest.raises(InitializationError, match=r"parent changed.*exchange was reversed"): + generator._publish_exchange(source, destination) + + assert source.read_text() == "new contract\n" + assert (moved_destination_parent / destination.name).read_text() == "old contract\n" + assert destination.read_text() == "concurrent replacement\n" + + class FakeRename: def __init__(self, error_number: int) -> None: self.error_number = error_number From f5a69c535beb30974bb205fa95534ba04670b3f5 Mon Sep 17 00:00:00 2001 From: Johnny Greco Date: Thu, 23 Jul 2026 19:38:14 +0000 Subject: [PATCH 5/6] Remove legacy middleware compatibility --- projects/middleware-kit/README.md | 27 +-- .../middleware-kit/src/middleware_kit/cli.py | 12 +- .../src/middleware_kit/generator.py | 144 +++++++-------- projects/middleware-kit/tests/test_cli.py | 20 +- .../middleware-kit/tests/test_generator.py | 174 +++++++----------- 5 files changed, 151 insertions(+), 226 deletions(-) diff --git a/projects/middleware-kit/README.md b/projects/middleware-kit/README.md index 015505cf..b018bb4d 100644 --- a/projects/middleware-kit/README.md +++ b/projects/middleware-kit/README.md @@ -40,29 +40,6 @@ uv sync --locked uv run mkit --help ``` -## Migrate from the initializer - -The distribution, executable, and creation syntax have changed: - -| Aspect | Before | Now | -| --- | --- | --- | -| Distribution | `openshell-middleware-init` | `middleware-kit` | -| Executable | `openshell-middleware-init` | `mkit` | -| Repository path | `projects/openshell-middleware-init` | `projects/middleware-kit` | -| Create syntax | `openshell-middleware-init ...` | `mkit create ...` | - -Replace an existing tool installation with: - -```sh -uv tool uninstall openshell-middleware-init -uv tool install \ - "middleware-kit @ git+https://github.com/NVIDIA/OpenShell-Research.git#subdirectory=projects/middleware-kit" -``` - -Generated projects do not need to be recreated. `mkit update` recognizes -manifests written by both `openshell-middleware-init` and the interim -`middleware-project` name. - ## Quick start Generate and run a Python starter with the installed command: @@ -117,8 +94,8 @@ The update command reads `middleware-dev-manifest.json` to discover the project language and Python package. It downloads the selected `supervisor_middleware.proto`, regenerates Python protobuf and gRPC bindings when applicable, refreshes `uv.lock` or `Cargo.lock`, and records the new -version and protocol checksum in the manifest. Projects created under the -former `middleware-project` and `openshell-middleware-init` names are supported. +version and protocol checksum in the manifest. The update command accepts only +manifests that identify `middleware-kit` as their generator. ## What you get diff --git a/projects/middleware-kit/src/middleware_kit/cli.py b/projects/middleware-kit/src/middleware_kit/cli.py index 173b243c..2fdf215d 100644 --- a/projects/middleware-kit/src/middleware_kit/cli.py +++ b/projects/middleware-kit/src/middleware_kit/cli.py @@ -9,8 +9,8 @@ import typer from middleware_kit.generator import ( - InitializationError, - initialize_project, + ProjectError, + create_project, update_project, ) @@ -67,14 +67,14 @@ def create( """Create a new OpenShell supervisor middleware project.""" destination = output if output is not None else Path.cwd() / name try: - result = initialize_project( + result = create_project( name=name, language=language.value, requested_version=openshell_version, destination=destination, package_name=package_name, ) - except InitializationError as error: + except ProjectError as error: _report_error(error) typer.echo(f"Created {result.language} middleware project at {result.destination}") @@ -105,14 +105,14 @@ def update( project_dir=project, requested_version=openshell_version, ) - except InitializationError as error: + except ProjectError as error: _report_error(error) typer.echo(f"Updated {result.language} middleware project at {result.destination}") typer.echo(f"OpenShell contract: {result.openshell_version}") -def _report_error(error: InitializationError) -> None: +def _report_error(error: ProjectError) -> None: typer.echo(f"mkit: error: {error}", err=True) raise typer.Exit(code=1) from error diff --git a/projects/middleware-kit/src/middleware_kit/generator.py b/projects/middleware-kit/src/middleware_kit/generator.py index 1816c2d9..72a2256b 100644 --- a/projects/middleware-kit/src/middleware_kit/generator.py +++ b/projects/middleware-kit/src/middleware_kit/generator.py @@ -37,7 +37,6 @@ _PROJECT_NAME_PATTERN = re.compile(r"^[a-z0-9](?:[a-z0-9._-]*[a-z0-9])?$") _NETWORK_ATTEMPTS = 4 _TOOL_NAME = "middleware-kit" -_LEGACY_TOOL_NAMES = {"middleware-project", "openshell-middleware-init"} _STAGING_IGNORED_ROOT_ENTRIES = { ".coverage", ".git", @@ -118,16 +117,16 @@ } -class InitializationError(RuntimeError): +class ProjectError(RuntimeError): """A user-actionable middleware project operation failure.""" -class PublicationRollbackError(InitializationError): +class PublicationRollbackError(ProjectError): """An update failure whose recovery artifacts must be preserved.""" @dataclass(frozen=True) -class InitializationResult: +class ProjectResult: """Details about a successful middleware project operation.""" destination: Path @@ -186,7 +185,7 @@ def replacements(self) -> Mapping[str, str]: CommandRunner = Callable[[str, Path, str], None] -def initialize_project( +def create_project( *, name: str, language: str, @@ -195,7 +194,7 @@ def initialize_project( package_name: str | None = None, download_proto: DownloadProto | None = None, command_runner: CommandRunner | None = None, -) -> InitializationResult: +) -> ProjectResult: """Generate and validate a project, then publish it atomically.""" _validate_platform() context = _template_context(name, language, package_name) @@ -237,16 +236,16 @@ def initialize_project( _verify_lock(reservation) _publish_no_replace(staging_path, destination) staging_path = None - except InitializationError: + except ProjectError: raise except (OSError, subprocess.SubprocessError) as error: - raise InitializationError(str(error)) from error + raise ProjectError(str(error)) from error finally: if staging_path is not None: shutil.rmtree(staging_path, ignore_errors=True) _release_lock(reservation) - return InitializationResult( + return ProjectResult( destination=destination, language=language, openshell_version=version, @@ -264,7 +263,7 @@ def update_project( requested_version: str = "latest", download_proto: DownloadProto | None = None, command_runner: CommandRunner | None = None, -) -> InitializationResult: +) -> ProjectResult: """Refresh generator-owned artifacts and atomically publish a validated update.""" _validate_platform() project_dir = project_dir.expanduser() @@ -315,10 +314,10 @@ def update_project( except PublicationRollbackError: preserve_recovery = True raise - except InitializationError: + except ProjectError: raise except (OSError, subprocess.SubprocessError) as error: - raise InitializationError(str(error)) from error + raise ProjectError(str(error)) from error finally: if preserve_recovery: with suppress(OSError): @@ -330,7 +329,7 @@ def update_project( if not published: # pragma: no cover - defensive; failures raise above raise AssertionError("updated project was not published") - return InitializationResult( + return ProjectResult( destination=project_dir, language=metadata.language, openshell_version=version, @@ -340,7 +339,7 @@ def update_project( def _validate_platform() -> None: if sys.platform != "darwin" and not sys.platform.startswith("linux"): - raise InitializationError( + raise ProjectError( f"{_TOOL_NAME} supports Linux and macOS; unsupported platform: {sys.platform}" ) @@ -348,13 +347,13 @@ def _validate_platform() -> None: def _template_context(name: str, language: str, package_name: str | None) -> TemplateContext: normalized_name = name.strip().lower() if not _PROJECT_NAME_PATTERN.fullmatch(normalized_name): - raise InitializationError( + raise ProjectError( "project name must use lowercase letters, digits, dots, hyphens, or underscores" ) if language not in {"python", "rust"}: - raise InitializationError("language must be 'python' or 'rust'") + raise ProjectError("language must be 'python' or 'rust'") if language == "rust" and package_name is not None: - raise InitializationError("--package-name is only valid with --language python") + raise ProjectError("--package-name is only valid with --language python") distribution_name = re.sub(r"[._]+", "-", normalized_name) identifier = re.sub(r"[^a-z0-9]+", "_", normalized_name).strip("_") @@ -363,7 +362,7 @@ def _template_context(name: str, language: str, package_name: str | None) -> Tem derived_package = f"middleware_{derived_package}".rstrip("_") effective_package = package_name if package_name is not None else derived_package if not _PYTHON_PACKAGE_PATTERN.fullmatch(effective_package): - raise InitializationError( + raise ProjectError( "Python package name must start with a lowercase letter and contain only " "lowercase letters, digits, and underscores" ) @@ -390,7 +389,7 @@ def _normalize_version(requested: str) -> str: if not value.startswith("v"): value = f"v{value}" if not _VERSION_PATTERN.fullmatch(value): - raise InitializationError( + raise ProjectError( f"invalid OpenShell version '{requested}'; expected a tag such as v0.0.86" ) return value @@ -404,13 +403,13 @@ def _resolve_latest_version() -> str: try: _, resolved_url = _fetch_url(request) except (OSError, urllib.error.URLError, http.client.IncompleteRead) as error: - raise InitializationError("could not resolve OpenShell's latest release") from error + raise ProjectError("could not resolve OpenShell's latest release") from error prefix = f"{_REPOSITORY_URL}/releases/tag/" if not resolved_url.startswith(prefix): - raise InitializationError(f"unexpected latest-release redirect: {resolved_url}") + raise ProjectError(f"unexpected latest-release redirect: {resolved_url}") version = resolved_url.removeprefix(prefix) if not _VERSION_PATTERN.fullmatch(version): - raise InitializationError(f"latest release has an unexpected tag: {version}") + raise ProjectError(f"latest release has an unexpected tag: {version}") return version @@ -425,14 +424,14 @@ def _download_proto(version: str) -> tuple[bytes, str]: return body, url except urllib.error.HTTPError as error: if error.code == 404: - raise InitializationError( + raise ProjectError( f"{version} does not expose {_PROTO_PATH}; choose a middleware-capable release" ) from error - raise InitializationError( + raise ProjectError( f"could not download {_PROTO_PATH} for {version}: HTTP {error.code}" ) from error except (OSError, urllib.error.URLError, http.client.IncompleteRead) as error: - raise InitializationError( + raise ProjectError( f"could not download {_PROTO_PATH} for {version}: {_network_error_reason(error)}" ) from error @@ -469,70 +468,63 @@ def _validate_proto(proto: bytes, version: str) -> None: b"rpc EvaluateHttpRequest", ) if not proto or any(fragment not in proto for fragment in required_fragments): - raise InitializationError( + raise ProjectError( f"downloaded contract for {version} is not a supported supervisor middleware proto" ) def _validate_destination(destination: Path) -> None: if os.path.lexists(destination): - raise InitializationError(f"output path must not already exist: {destination}") + raise ProjectError(f"output path must not already exist: {destination}") if destination.name in {"", ".", ".."}: - raise InitializationError(f"invalid output path: {destination}") + raise ProjectError(f"invalid output path: {destination}") def _validate_existing_project(project_dir: Path) -> None: if project_dir.is_symlink(): - raise InitializationError(f"project path must not be a symlink: {project_dir}") + raise ProjectError(f"project path must not be a symlink: {project_dir}") if not project_dir.is_dir(): - raise InitializationError(f"project path must be an existing directory: {project_dir}") + raise ProjectError(f"project path must be an existing directory: {project_dir}") def _verify_project_identity(project_dir: Path, device: int, inode: int) -> None: try: current = project_dir.stat(follow_symlinks=False) except OSError as error: - raise InitializationError( - "project path changed during update; refusing to publish" - ) from error + raise ProjectError("project path changed during update; refusing to publish") from error if not stat.S_ISDIR(current.st_mode) or current.st_dev != device or current.st_ino != inode: - raise InitializationError("project path changed during update; refusing to publish") + raise ProjectError("project path changed during update; refusing to publish") def _read_project_metadata(project_dir: Path) -> ProjectMetadata: manifest_path = project_dir / "middleware-dev-manifest.json" if manifest_path.is_symlink() or not manifest_path.is_file(): - raise InitializationError( + raise ProjectError( f"not a generated middleware project; missing regular manifest: {manifest_path}" ) try: manifest = json.loads(manifest_path.read_text()) except (OSError, json.JSONDecodeError) as error: - raise InitializationError(f"could not read project manifest: {error}") from error + raise ProjectError(f"could not read project manifest: {error}") from error if not isinstance(manifest, dict): - raise InitializationError("project manifest must contain a JSON object") + raise ProjectError("project manifest must contain a JSON object") generator = manifest.get("generator") generator_name = generator.get("name") if isinstance(generator, dict) else None - if generator_name != _TOOL_NAME and generator_name not in _LEGACY_TOOL_NAMES: - raise InitializationError( - "project manifest was not created by middleware-kit, middleware-project, " - "or openshell-middleware-init" - ) + if generator_name != _TOOL_NAME: + raise ProjectError("project manifest generator must be middleware-kit") languages = manifest.get("languages") if languages not in (["python"], ["rust"]): - raise InitializationError("project manifest must identify exactly one supported language") + raise ProjectError("project manifest must identify exactly one supported language") language = languages[0] python_package = manifest.get("python_package") if language == "python": if not isinstance(python_package, str) or not _PYTHON_PACKAGE_PATTERN.fullmatch( python_package ): - raise InitializationError( - "Python project manifest has an invalid or missing python_package" - ) + raise ProjectError("Python project manifest has an invalid or missing python_package") elif python_package is not None: - raise InitializationError("Rust project manifest must set python_package to null") + raise ProjectError("Rust project manifest must set python_package to null") _validate_refresh_targets(project_dir, language, python_package) return ProjectMetadata(language=language, python_package=python_package) @@ -547,7 +539,7 @@ def _validate_refresh_targets(project_dir: Path, language: str, python_package: for path in regular_files: _reject_symlink_components(project_dir, path) if path.is_symlink() or not path.is_file(): - raise InitializationError(f"generated artifact must be a regular file: {path}") + raise ProjectError(f"generated artifact must be a regular file: {path}") if language == "python": if python_package is None: # pragma: no cover - checked by caller @@ -555,9 +547,7 @@ def _validate_refresh_targets(project_dir: Path, language: str, python_package: bindings_dir = project_dir / "src" / python_package / "bindings" _reject_symlink_components(project_dir, bindings_dir) if bindings_dir.is_symlink() or not bindings_dir.is_dir(): - raise InitializationError( - f"generated bindings must be an existing directory: {bindings_dir}" - ) + raise ProjectError(f"generated bindings must be an existing directory: {bindings_dir}") def _reject_symlink_components(project_dir: Path, target: Path) -> None: @@ -570,9 +560,7 @@ def _reject_symlink_components(project_dir: Path, target: Path) -> None: except FileNotFoundError: return if stat.S_ISLNK(current_stat.st_mode): - raise InitializationError( - f"generated artifact path must not contain symlinks: {current}" - ) + raise ProjectError(f"generated artifact path must not contain symlinks: {current}") def _copy_project_to_staging(project_dir: Path, staging_path: Path) -> None: @@ -631,7 +619,7 @@ def _acquire_lock( try: lock_path.mkdir(mode=0o700) except FileExistsError as error: - raise InitializationError( + raise ProjectError( f"project path is reserved by another {_TOOL_NAME} process: {destination}; " f"inspect {lock_path / 'metadata.json'} and follow the stale-reservation " f"recovery steps in the {_TOOL_NAME} README" @@ -718,9 +706,9 @@ def _verify_lock(reservation: OutputReservation) -> None: raise OSError("reservation owner is not a regular file") recorded = owner.read() except OSError as error: - raise InitializationError("output reservation was lost; refusing to publish") from error + raise ProjectError("output reservation was lost; refusing to publish") from error if recorded != reservation.token: - raise InitializationError("output reservation ownership changed; refusing to publish") + raise ProjectError("output reservation ownership changed; refusing to publish") def _remove_reservation_files(reservation: OutputReservation) -> bool: @@ -761,7 +749,7 @@ def _cleanup_reservation(reservation: OutputReservation) -> None: def _release_lock(reservation: OutputReservation) -> None: try: _verify_lock(reservation) - except InitializationError: + except ProjectError: with suppress(OSError): os.close(reservation.directory_fd) return @@ -777,7 +765,7 @@ def _publish_no_replace(source: Path, destination: Path) -> None: try: rename = library.renameat2 except AttributeError as error: # pragma: no cover - old Linux libc - raise InitializationError( + raise ProjectError( "this Linux runtime cannot publish atomically without replacing an output" ) from error rename.argtypes = ( @@ -796,21 +784,17 @@ def _publish_no_replace(source: Path, destination: Path) -> None: rename.restype = ctypes.c_int result = rename(source_bytes, destination_bytes, 0x00000004) else: # pragma: no cover - unsupported platform - raise InitializationError( - "this platform cannot publish atomically without replacing an output" - ) + raise ProjectError("this platform cannot publish atomically without replacing an output") if result == 0: return error_number = ctypes.get_errno() if error_number in {errno.EEXIST, errno.ENOTEMPTY}: - raise InitializationError( + raise ProjectError( f"output path appeared during setup; refusing to overwrite it: {destination}" ) if error_number in {errno.EINVAL, errno.ENOSYS, errno.EOPNOTSUPP}: - raise InitializationError( - "the output filesystem does not support atomic no-replace publication" - ) + raise ProjectError("the output filesystem does not support atomic no-replace publication") raise OSError(error_number, os.strerror(error_number), destination) @@ -855,7 +839,7 @@ def _publish_exchange(source: Path, destination: Path) -> None: "an artifact parent changed during publication and the anchored exchange " f"could not be reversed: {os.strerror(error_number)}" ) - raise InitializationError( + raise ProjectError( "an artifact parent changed during publication; the exchange was reversed" ) finally: @@ -874,7 +858,7 @@ def _exchange_at( try: rename = library.renameat2 except AttributeError as error: # pragma: no cover - old Linux libc - raise InitializationError( + raise ProjectError( "this Linux runtime cannot atomically publish a project update" ) from error exchange_flag = 2 @@ -883,7 +867,7 @@ def _exchange_at( rename = library.renameatx_np exchange_flag = 0x00000002 else: # pragma: no cover - unsupported platform - raise InitializationError("this platform cannot atomically publish a project update") + raise ProjectError("this platform cannot atomically publish a project update") rename.argtypes = ( ctypes.c_int, ctypes.c_char_p, @@ -904,9 +888,7 @@ def _exchange_at( def _raise_exchange_error(destination: Path) -> None: error_number = ctypes.get_errno() if error_number in {errno.EINVAL, errno.ENOSYS, errno.EOPNOTSUPP}: - raise InitializationError( - "the project filesystem does not support atomic update publication" - ) + raise ProjectError("the project filesystem does not support atomic update publication") raise OSError(error_number, os.strerror(error_number), destination) @@ -916,7 +898,7 @@ def _open_parent_directory_no_follow(path: Path) -> int: def _open_directory_no_follow(path: Path) -> int: if not path.is_absolute(): - raise InitializationError(f"artifact path must be absolute: {path}") + raise ProjectError(f"artifact path must be absolute: {path}") flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0) descriptor = os.open(path.anchor, flags) try: @@ -933,7 +915,7 @@ def _open_directory_no_follow(path: Path) -> int: def _directory_fd_matches_path(path: Path, expected_descriptor: int) -> bool: try: current_descriptor = _open_directory_no_follow(path) - except (InitializationError, OSError): + except (ProjectError, OSError): return False try: expected_stat = os.fstat(expected_descriptor) @@ -966,9 +948,7 @@ def _validate_exchange_entries( (source_is_directory and destination_is_directory) or (source_is_regular and destination_is_regular) ): - raise InitializationError( - "generated artifacts changed type during update; refusing to publish" - ) + raise ProjectError("generated artifacts changed type during update; refusing to publish") def _publish_generated_artifacts( @@ -993,14 +973,14 @@ def _publish_generated_artifacts( project_dir / relative_path, ) exchanged.append(relative_path) - except (InitializationError, OSError) as publish_error: + except (ProjectError, OSError) as publish_error: try: for relative_path in reversed(exchanged): _publish_exchange( staged_project / relative_path, project_dir / relative_path, ) - except (InitializationError, OSError) as rollback_error: + except (ProjectError, OSError) as rollback_error: raise PublicationRollbackError( "artifact publication and rollback both failed; inspect the project and " f"{staged_project} before removing the reservation" @@ -1072,7 +1052,7 @@ def _prepare_project(language: str, project_dir: Path, package_name: str) -> Non def _require_command(command: str) -> str: resolved = shutil.which(command) if resolved is None: - raise InitializationError(f"'{command}' is required to prepare this project") + raise ProjectError(f"'{command}' is required to prepare this project") return resolved @@ -1094,7 +1074,7 @@ def _run( check=True, ) except subprocess.CalledProcessError as error: - raise InitializationError( + raise ProjectError( f"validation command failed with exit code {error.returncode}: {' '.join(command)}" ) from error @@ -1127,7 +1107,7 @@ def _prepare_python_project(project_dir: Path, package_name: str) -> None: absolute_import = "import supervisor_middleware_pb2 as supervisor__middleware__pb2" relative_import = "from . import supervisor_middleware_pb2 as supervisor__middleware__pb2" if absolute_import not in generated: - raise InitializationError( + raise ProjectError( "generated gRPC module has an unexpected import layout; no project was published" ) grpc_module.write_text(generated.replace(absolute_import, relative_import, 1)) diff --git a/projects/middleware-kit/tests/test_cli.py b/projects/middleware-kit/tests/test_cli.py index d578d212..f6ec6b3f 100644 --- a/projects/middleware-kit/tests/test_cli.py +++ b/projects/middleware-kit/tests/test_cli.py @@ -3,7 +3,7 @@ from typer.testing import CliRunner from middleware_kit import cli -from middleware_kit.generator import InitializationError, InitializationResult +from middleware_kit.generator import ProjectError, ProjectResult runner = CliRunner() @@ -25,18 +25,18 @@ def test_help_describes_required_choices() -> None: def test_cli_reports_success(monkeypatch, tmp_path: Path) -> None: destination = tmp_path / "audit" - def fake_initialize_project(**options): + def fake_create_project(**options): assert options["name"] == "audit" assert options["language"] == "python" assert options["destination"] == destination - return InitializationResult( + return ProjectResult( destination=destination, language="python", openshell_version="v0.0.86", run_command="uv run audit", ) - monkeypatch.setattr(cli, "initialize_project", fake_initialize_project) + monkeypatch.setattr(cli, "create_project", fake_create_project) result = runner.invoke( cli.app, @@ -57,12 +57,12 @@ def fake_initialize_project(**options): assert "OpenShell contract: v0.0.86" in result.stdout -def test_cli_reports_initialization_error(monkeypatch, tmp_path: Path) -> None: - def fake_initialize_project(**options): +def test_cli_reports_project_error(monkeypatch, tmp_path: Path) -> None: + def fake_create_project(**options): del options - raise InitializationError("output exists") + raise ProjectError("output exists") - monkeypatch.setattr(cli, "initialize_project", fake_initialize_project) + monkeypatch.setattr(cli, "create_project", fake_create_project) result = runner.invoke( cli.app, @@ -90,7 +90,7 @@ def fake_update_project(**options): "project_dir": destination, "requested_version": "v1.2.3", } - return InitializationResult( + return ProjectResult( destination=destination, language="rust", openshell_version="v1.2.3", @@ -117,7 +117,7 @@ def fake_update_project(**options): def test_cli_reports_update_error(monkeypatch, tmp_path: Path) -> None: def fake_update_project(**options): del options - raise InitializationError("not generated") + raise ProjectError("not generated") monkeypatch.setattr(cli, "update_project", fake_update_project) diff --git a/projects/middleware-kit/tests/test_generator.py b/projects/middleware-kit/tests/test_generator.py index 98d8c698..ed86ada0 100644 --- a/projects/middleware-kit/tests/test_generator.py +++ b/projects/middleware-kit/tests/test_generator.py @@ -15,7 +15,7 @@ import pytest from middleware_kit import generator -from middleware_kit.generator import InitializationError, initialize_project, update_project +from middleware_kit.generator import ProjectError, create_project, update_project PROTO = b"""syntax = "proto3"; package openshell.middleware.v1; @@ -43,7 +43,7 @@ def no_op_runner(language: str, project: Path, package: str) -> None: def test_generates_python_project_with_provenance(tmp_path: Path) -> None: destination = tmp_path / "audit-headers" - result = initialize_project( + result = create_project( name="audit-headers", language="python", requested_version="0.0.86", @@ -70,7 +70,7 @@ def test_generates_python_project_with_provenance(tmp_path: Path) -> None: def test_generates_rust_project_with_normalized_crate_name(tmp_path: Path) -> None: destination = tmp_path / "request.audit" - result = initialize_project( + result = create_project( name="request.audit", language="rust", requested_version="v0.0.86", @@ -95,7 +95,7 @@ def test_generates_rust_project_with_normalized_crate_name(tmp_path: Path) -> No def test_updates_python_generated_artifacts_and_preserves_user_code(tmp_path: Path) -> None: destination = tmp_path / "audit-headers" - initialize_project( + create_project( name="audit-headers", language="python", requested_version="v0.0.86", @@ -134,41 +134,9 @@ def updated_proto(version: str) -> tuple[bytes, str]: assert not list(tmp_path.glob(".audit-headers.middleware-kit.*")) -@pytest.mark.parametrize( - "legacy_name", - ["middleware-project", "openshell-middleware-init"], -) -def test_updates_rust_project_created_by_legacy_tool(tmp_path: Path, legacy_name: str) -> None: - destination = tmp_path / "audit" - initialize_project( - name="audit", - language="rust", - requested_version="v0.0.86", - destination=destination, - download_proto=local_proto, - command_runner=no_op_runner, - ) - manifest_path = destination / "middleware-dev-manifest.json" - manifest = json.loads(manifest_path.read_text()) - manifest["generator"]["name"] = legacy_name - manifest_path.write_text(json.dumps(manifest)) - - result = update_project( - project_dir=destination, - requested_version="v1.2.3", - download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), - command_runner=no_op_runner, - ) - - assert result.language == "rust" - assert (destination / "proto/supervisor_middleware.proto").read_bytes() == UPDATED_PROTO - updated_manifest = json.loads(manifest_path.read_text()) - assert updated_manifest["generator"]["name"] == "middleware-kit" - - def test_failed_update_keeps_original_project_unchanged(tmp_path: Path) -> None: destination = tmp_path / "audit" - initialize_project( + create_project( name="audit", language="rust", requested_version="v0.0.86", @@ -181,9 +149,9 @@ def test_failed_update_keeps_original_project_unchanged(tmp_path: Path) -> None: def fail_runner(language: str, project: Path, package: str) -> None: del language, project, package - raise InitializationError("validation failed") + raise ProjectError("validation failed") - with pytest.raises(InitializationError, match="validation failed"): + with pytest.raises(ProjectError, match="validation failed"): update_project( project_dir=destination, requested_version="v1.2.3", @@ -201,7 +169,7 @@ def test_publication_failure_rolls_back_exchanged_artifacts( monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: destination = tmp_path / "audit" - initialize_project( + create_project( name="audit", language="rust", requested_version="v0.0.86", @@ -218,12 +186,12 @@ def fail_second_exchange(source: Path, target: Path) -> None: nonlocal calls calls += 1 if calls == 2: - raise InitializationError("publication failed") + raise ProjectError("publication failed") original_exchange(source, target) monkeypatch.setattr(generator, "_publish_exchange", fail_second_exchange) - with pytest.raises(InitializationError, match="publication failed"): + with pytest.raises(ProjectError, match="publication failed"): update_project( project_dir=destination, requested_version="v1.2.3", @@ -240,7 +208,7 @@ def test_failed_publication_rollback_preserves_recovery_artifacts( monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: destination = tmp_path / "audit" - initialize_project( + create_project( name="audit", language="rust", requested_version="v0.0.86", @@ -255,7 +223,7 @@ def fail_publication_and_rollback(source: Path, target: Path) -> None: nonlocal calls calls += 1 if calls >= 2: - raise InitializationError("exchange failed") + raise ProjectError("exchange failed") original_exchange(source, target) monkeypatch.setattr(generator, "_publish_exchange", fail_publication_and_rollback) @@ -277,7 +245,7 @@ def test_update_rejects_non_generated_project(tmp_path: Path) -> None: destination = tmp_path / "not-generated" destination.mkdir() - with pytest.raises(InitializationError, match="missing regular manifest"): + with pytest.raises(ProjectError, match="missing regular manifest"): update_project( project_dir=destination, requested_version="v1.2.3", @@ -291,13 +259,13 @@ def test_update_rejects_symlink_and_missing_project_paths(tmp_path: Path) -> Non symlink = tmp_path / "symlink" symlink.symlink_to(missing, target_is_directory=True) - with pytest.raises(InitializationError, match="must not be a symlink"): + with pytest.raises(ProjectError, match="must not be a symlink"): update_project( project_dir=symlink, download_proto=local_proto, command_runner=no_op_runner, ) - with pytest.raises(InitializationError, match="existing directory"): + with pytest.raises(ProjectError, match="existing directory"): update_project( project_dir=missing, download_proto=local_proto, @@ -310,7 +278,7 @@ def test_update_rejects_symlink_and_missing_project_paths(tmp_path: Path) -> Non [ ("not json", "could not read"), ("[]", "JSON object"), - ('{"generator": {"name": "other"}}', "was not created"), + ('{"generator": {"name": "other"}}', "generator must be middleware-kit"), ( '{"generator": {"name": "middleware-kit"}, "languages": ["python", "rust"]}', "exactly one", @@ -332,7 +300,7 @@ def test_update_rejects_invalid_manifest(tmp_path: Path, manifest: str, message: destination.mkdir() (destination / "middleware-dev-manifest.json").write_text(manifest) - with pytest.raises(InitializationError, match=message): + with pytest.raises(ProjectError, match=message): update_project( project_dir=destination, download_proto=local_proto, @@ -353,7 +321,7 @@ def test_update_requires_regular_generated_artifacts(tmp_path: Path) -> None: ) ) - with pytest.raises(InitializationError, match="regular file"): + with pytest.raises(ProjectError, match="regular file"): update_project( project_dir=destination, download_proto=local_proto, @@ -376,7 +344,7 @@ def test_update_requires_python_bindings_directory(tmp_path: Path) -> None: ) ) - with pytest.raises(InitializationError, match="bindings must be"): + with pytest.raises(ProjectError, match="bindings must be"): update_project( project_dir=destination, download_proto=local_proto, @@ -388,7 +356,7 @@ def test_update_rejects_symlinked_proto_directory_without_touching_target( tmp_path: Path, ) -> None: destination = tmp_path / "project" - initialize_project( + create_project( name="project", language="rust", requested_version="v0.0.86", @@ -403,7 +371,7 @@ def test_update_rejects_symlinked_proto_directory_without_touching_target( sentinel.write_text("external contract\n") (destination / "proto").symlink_to(external_proto, target_is_directory=True) - with pytest.raises(InitializationError, match="must not contain symlinks"): + with pytest.raises(ProjectError, match="must not contain symlinks"): update_project( project_dir=destination, requested_version="v1.2.3", @@ -418,7 +386,7 @@ def test_update_rejects_symlinked_python_package_without_touching_target( tmp_path: Path, ) -> None: destination = tmp_path / "project" - initialize_project( + create_project( name="project", language="python", requested_version="v0.0.86", @@ -435,7 +403,7 @@ def test_update_rejects_symlinked_python_package_without_touching_target( sentinel.write_text("external binding\n") package_dir.symlink_to(external_package, target_is_directory=True) - with pytest.raises(InitializationError, match="must not contain symlinks"): + with pytest.raises(ProjectError, match="must not contain symlinks"): update_project( project_dir=destination, requested_version="v1.2.3", @@ -448,7 +416,7 @@ def test_update_rejects_symlinked_python_package_without_touching_target( def test_update_omits_disposable_directories_from_staging(tmp_path: Path) -> None: destination = tmp_path / "project" - initialize_project( + create_project( name="project", language="python", requested_version="v0.0.86", @@ -491,7 +459,7 @@ def test_update_revalidates_symlink_ancestors_after_project_validation( tmp_path: Path, replace_live_path: bool ) -> None: destination = tmp_path / "project" - initialize_project( + create_project( name="project", language="rust", requested_version="v0.0.86", @@ -511,7 +479,7 @@ def replace_proto_ancestor(language: str, staged_project: Path, package: str) -> (project_to_change / "proto").rename(project_to_change / "original-proto") (project_to_change / "proto").symlink_to(external_proto, target_is_directory=True) - with pytest.raises(InitializationError, match="must not contain symlinks"): + with pytest.raises(ProjectError, match="must not contain symlinks"): update_project( project_dir=destination, requested_version="v1.2.3", @@ -527,7 +495,7 @@ def test_exchange_refuses_symlinked_parent_created_immediately_before_publish( monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: destination = tmp_path / "project" - initialize_project( + create_project( name="project", language="rust", requested_version="v0.0.86", @@ -552,7 +520,7 @@ def replace_parent_then_exchange(source: Path, target: Path) -> None: monkeypatch.setattr(generator, "_publish_exchange", replace_parent_then_exchange) - with pytest.raises(InitializationError): + with pytest.raises(ProjectError): update_project( project_dir=destination, requested_version="v1.2.3", @@ -571,18 +539,18 @@ def test_update_detects_replaced_project_before_publication(tmp_path: Path) -> N project.rename(replacement) project.mkdir() - with pytest.raises(InitializationError, match="changed during update"): + with pytest.raises(ProjectError, match="changed during update"): generator._verify_project_identity(project, original.st_dev, original.st_ino) project.rmdir() - with pytest.raises(InitializationError, match="changed during update"): + with pytest.raises(ProjectError, match="changed during update"): generator._verify_project_identity(project, original.st_dev, original.st_ino) def test_python_package_name_can_be_overridden(tmp_path: Path) -> None: destination = tmp_path / "project" - initialize_project( + create_project( name="project", language="python", requested_version="v0.0.86", @@ -598,7 +566,7 @@ def test_python_package_name_can_be_overridden(tmp_path: Path) -> None: def test_numeric_project_name_gets_importable_python_package(tmp_path: Path) -> None: destination = tmp_path / "123" - initialize_project( + create_project( name="123", language="python", requested_version="v0.0.86", @@ -624,7 +592,7 @@ def test_rust_project_names_get_valid_explicit_library_names( ) -> None: destination = tmp_path / name - initialize_project( + create_project( name=name, language="rust", requested_version="v0.0.86", @@ -648,8 +616,8 @@ def test_unsupported_platform_fails_before_filesystem_changes( destination = tmp_path / "output" monkeypatch.setattr(generator.sys, "platform", "win32") - with pytest.raises(InitializationError, match="supports Linux and macOS"): - initialize_project( + with pytest.raises(ProjectError, match="supports Linux and macOS"): + create_project( name="project", language="python", requested_version="v0.0.86", @@ -678,8 +646,8 @@ def test_rejects_invalid_project_choices( package_name: str | None, message: str, ) -> None: - with pytest.raises(InitializationError, match=message): - initialize_project( + with pytest.raises(ProjectError, match=message): + create_project( name=name, language=language, requested_version="v0.0.86", @@ -692,8 +660,8 @@ def test_rejects_invalid_project_choices( @pytest.mark.parametrize("version", ["", "main", "v1", "v1.2", "v1.2.x"]) def test_rejects_invalid_versions(tmp_path: Path, version: str) -> None: - with pytest.raises(InitializationError, match="invalid OpenShell version"): - initialize_project( + with pytest.raises(ProjectError, match="invalid OpenShell version"): + create_project( name="project", language="python", requested_version=version, @@ -707,8 +675,8 @@ def test_refuses_an_existing_destination(tmp_path: Path) -> None: destination = tmp_path / "existing" destination.mkdir() - with pytest.raises(InitializationError, match="must not already exist"): - initialize_project( + with pytest.raises(ProjectError, match="must not already exist"): + create_project( name="existing", language="python", requested_version="v0.0.86", @@ -723,8 +691,8 @@ def test_refuses_a_dangling_destination_symlink(tmp_path: Path) -> None: target = tmp_path / "symlink-target" destination.symlink_to(target, target_is_directory=True) - with pytest.raises(InitializationError, match="must not already exist"): - initialize_project( + with pytest.raises(ProjectError, match="must not already exist"): + create_project( name="project", language="python", requested_version="v0.0.86", @@ -741,8 +709,8 @@ def test_refuses_a_reserved_destination(tmp_path: Path) -> None: destination = tmp_path / "reserved" (tmp_path / ".reserved.middleware-kit.lock").mkdir() - with pytest.raises(InitializationError, match="reserved by another middleware-kit"): - initialize_project( + with pytest.raises(ProjectError, match="reserved by another middleware-kit"): + create_project( name="reserved", language="rust", requested_version="v0.0.86", @@ -765,8 +733,8 @@ def collide_before_publish(source: Path, final_output: Path) -> None: monkeypatch.setattr(generator, "_publish_no_replace", collide_before_publish) - with pytest.raises(InitializationError, match="appeared during setup"): - initialize_project( + with pytest.raises(ProjectError, match="appeared during setup"): + create_project( name="contended", language="python", requested_version="v0.0.86", @@ -784,10 +752,10 @@ def test_failure_cleans_staging_and_owned_reservation(tmp_path: Path) -> None: def fail_runner(language: str, project: Path, package: str) -> None: del language, project, package - raise InitializationError("validation failed") + raise ProjectError("validation failed") - with pytest.raises(InitializationError, match="validation failed"): - initialize_project( + with pytest.raises(ProjectError, match="validation failed"): + create_project( name="failing", language="python", requested_version="v0.0.86", @@ -805,8 +773,8 @@ def test_rejects_an_unexpected_proto(tmp_path: Path) -> None: def invalid_proto(version: str) -> tuple[bytes, str]: return b"not a proto", f"https://example.test/{version}" - with pytest.raises(InitializationError, match="not a supported"): - initialize_project( + with pytest.raises(ProjectError, match="not a supported"): + create_project( name="invalid-proto", language="rust", requested_version="v0.0.86", @@ -819,12 +787,12 @@ def invalid_proto(version: str) -> tuple[bytes, str]: def test_missing_required_command_has_actionable_error(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(generator.shutil, "which", lambda command: None) - with pytest.raises(InitializationError, match="'uv' is required"): + with pytest.raises(ProjectError, match="'uv' is required"): generator._require_command("uv") def test_failed_subprocess_is_translated(tmp_path: Path) -> None: - with pytest.raises(InitializationError, match="validation command failed"): + with pytest.raises(ProjectError, match="validation command failed"): generator._run( (sys.executable, "-c", "raise SystemExit(7)"), cwd=tmp_path, @@ -836,8 +804,8 @@ def fail_runner(language: str, project: Path, package: str) -> None: del language, project, package raise subprocess.SubprocessError("tool failed") - with pytest.raises(InitializationError, match="tool failed"): - initialize_project( + with pytest.raises(ProjectError, match="tool failed"): + create_project( name="failure", language="rust", requested_version="v0.0.86", @@ -885,7 +853,7 @@ def test_latest_version_rejects_unexpected_redirects( response = FakeResponse(url=resolved_url) monkeypatch.setattr(generator.urllib.request, "urlopen", lambda *args, **kwargs: response) - with pytest.raises(InitializationError, match=message): + with pytest.raises(ProjectError, match=message): generator._resolve_latest_version() @@ -901,7 +869,7 @@ def fail(*args: object, **kwargs: object) -> None: monkeypatch.setattr(generator.urllib.request, "urlopen", fail) monkeypatch.setattr(generator.time, "sleep", lambda _: None) - with pytest.raises(InitializationError, match="could not resolve"): + with pytest.raises(ProjectError, match="could not resolve"): generator._resolve_latest_version() assert attempts == 4 @@ -924,7 +892,7 @@ def fail(*args: object, **kwargs: object) -> None: monkeypatch.setattr(generator.urllib.request, "urlopen", fail) monkeypatch.setattr(generator.time, "sleep", lambda _: None) - with pytest.raises(InitializationError, match=r"could not download.*missing"): + with pytest.raises(ProjectError, match=r"could not download.*missing"): generator._download_proto("v1.2.3") @@ -993,7 +961,7 @@ def fail(*args: object, **kwargs: object) -> None: monkeypatch.setattr(generator.time, "sleep", lambda _: None) expected_message = "middleware-capable release" if status == 404 else "HTTP 503" - with pytest.raises(InitializationError, match=expected_message): + with pytest.raises(ProjectError, match=expected_message): generator._download_proto("v1.2.3") assert attempts == expected_attempts @@ -1011,8 +979,8 @@ def test_toolchain_preflight_precedes_latest_resolution_and_output_changes( lambda: pytest.fail("latest must not be resolved before toolchain preflight"), ) - with pytest.raises(InitializationError, match=rf"'{command}' is required"): - generator.initialize_project( + with pytest.raises(ProjectError, match=rf"'{command}' is required"): + generator.create_project( name="audit-headers", language=language, requested_version="latest", @@ -1038,12 +1006,12 @@ def test_lock_verification_detects_loss_and_changed_owner(tmp_path: Path) -> Non started_at="2026-07-22T00:00:00+00:00", ) - with pytest.raises(InitializationError, match="reservation was lost"): + with pytest.raises(ProjectError, match="reservation was lost"): generator._verify_lock(missing) reservation = generator._acquire_lock(lock, "mine", tmp_path / "output", "v0.0.86") (lock / "owner").write_text("theirs") - with pytest.raises(InitializationError, match="ownership changed"): + with pytest.raises(ProjectError, match="ownership changed"): generator._verify_lock(reservation) generator._release_lock(reservation) @@ -1093,7 +1061,7 @@ def test_reservation_verification_rejects_changed_directory_identity(tmp_path: P started_at=reservation.started_at, ) - with pytest.raises(InitializationError, match="reservation was lost"): + with pytest.raises(ProjectError, match="reservation was lost"): generator._verify_lock(changed_identity) generator._cleanup_reservation(reservation) @@ -1103,7 +1071,7 @@ def test_reservation_verification_rejects_non_file_owner(tmp_path: Path) -> None (reservation.path / "owner").unlink() (reservation.path / "owner").mkdir() - with pytest.raises(InitializationError, match="reservation was lost"): + with pytest.raises(ProjectError, match="reservation was lost"): generator._verify_lock(reservation) generator._release_lock(reservation) @@ -1184,7 +1152,7 @@ def validate_then_detach(*arguments) -> None: monkeypatch.setattr(generator, "_validate_exchange_entries", validate_then_detach) - with pytest.raises(InitializationError, match=r"parent changed.*exchange was reversed"): + with pytest.raises(ProjectError, match=r"parent changed.*exchange was reversed"): generator._publish_exchange(source, destination) assert source.read_text() == "new contract\n" @@ -1217,7 +1185,7 @@ def test_publish_reports_filesystem_without_no_replace_support( ) -> None: monkeypatch.setattr(generator.ctypes, "CDLL", lambda *args, **kwargs: FakeLibrary(error_number)) - with pytest.raises(InitializationError, match="does not support"): + with pytest.raises(ProjectError, match="does not support"): generator._publish_no_replace(tmp_path / "source", tmp_path / "destination") @@ -1264,13 +1232,13 @@ def test_require_command_returns_resolved_path(monkeypatch: pytest.MonkeyPatch) def test_run_passes_environment_to_subprocess(tmp_path: Path) -> None: environment = os.environ.copy() - environment["MIDDLEWARE_INIT_TEST_VALUE"] = "present" + environment["MIDDLEWARE_KIT_TEST_VALUE"] = "present" generator._run( ( sys.executable, "-c", - "import os; assert os.environ['MIDDLEWARE_INIT_TEST_VALUE'] == 'present'", + "import os; assert os.environ['MIDDLEWARE_KIT_TEST_VALUE'] == 'present'", ), cwd=tmp_path, environment=environment, @@ -1322,7 +1290,7 @@ def fake_run(command, *, cwd, environment=None) -> None: monkeypatch.setattr(generator, "_run", fake_run) - with pytest.raises(InitializationError, match="unexpected import layout"): + with pytest.raises(ProjectError, match="unexpected import layout"): generator._prepare_python_project(tmp_path, "audit") From d2311c884de3830314bb0d182659e640be428b46 Mon Sep 17 00:00:00 2001 From: Johnny Greco Date: Thu, 23 Jul 2026 19:42:35 +0000 Subject: [PATCH 6/6] Clarify middleware kit documentation --- projects/middleware-kit/AGENTS.md | 84 ++++++++++++------------ projects/middleware-kit/README.md | 104 +++++++++++++++--------------- 2 files changed, 95 insertions(+), 93 deletions(-) diff --git a/projects/middleware-kit/AGENTS.md b/projects/middleware-kit/AGENTS.md index d14f987a..e3abdbe2 100644 --- a/projects/middleware-kit/AGENTS.md +++ b/projects/middleware-kit/AGENTS.md @@ -2,49 +2,49 @@ Read `README.md` and `pyproject.toml` before changing this project. -## Preserve these invariants - -- Keep creation non-destructive. Never merge into, follow, or replace an - existing output path, including a symlink. -- Build and validate in a hidden sibling staging directory. Publish only after - all generation and validation steps succeed. -- Preserve reservation ownership checks, atomic no-replace creation, and atomic - exchange publication for updates. -- Support Linux and macOS explicitly. Do not weaken publication guarantees to - add another platform implicitly. -- Keep every generated project version-matched: the OpenShell tag, downloaded - proto, bindings or build configuration, lockfile, and manifest must agree. -- Do not install, replace, or configure the user's OpenShell installation. - -## Use the project toolchain - -- Use `uv` for this Python project. Treat `pyproject.toml` and `uv.lock` as the - dependency sources of truth. +## Safety rules + +- `mkit create` must never write into, follow, or replace an existing output + path, including a symlink. +- Build and check the project in a temporary directory next to its destination. + Move it into place only after every check passes. +- Check lock ownership before writing. Creation must not overwrite an existing + path. Updates must swap each generated file atomically and undo earlier swaps + if one fails. +- Support only Linux and macOS. Do not add a platform unless it can provide the + same file-safety guarantees. +- Keep the OpenShell tag, downloaded proto, bindings or Rust build files, + lockfile, and manifest on the same version. +- Never install, replace, or configure OpenShell. + +## Dependencies + +- Use `uv`. `pyproject.toml` and `uv.lock` define the dependencies. - Do not add `requirements.txt` or another dependency export unless a documented - non-uv consumer requires it. -- Use `uv add` or `uv remove` for dependency changes; do not hand-edit the lock. - -## Change templates carefully - -- Keep templates under `src/middleware_kit/templates/` runnable as - standalone projects. -- Use `__UPPER_SNAKE_CASE__` for template markers. Add every marker to - `TemplateContext.replacements` and cover it with a rendering test. -- Treat generated Python protobuf and gRPC modules as generator-owned. Do not - format, type-check, or hand-edit them. -- When changing a template, generate the affected language project in isolated - scratch storage and run its documented checks when practical. - -## Test behavior, not implementation details - -- Keep project-tool unit tests hermetic. Inject protocol downloads and project - preparation instead of contacting GitHub or invoking uv or Cargo. -- Add regression tests for changes to output safety, failure cleanup, naming, - manifests, network behavior, or rendered files. -- Use isolated temporary paths for end-to-end generation. Never generate over an - existing directory. - -## Validate every change + tool needs one. +- Use `uv add` or `uv remove` to change dependencies. Do not edit `uv.lock` by + hand. + +## Templates + +- Templates in `src/middleware_kit/templates/` must produce working standalone + projects. +- Write template markers as `__UPPER_SNAKE_CASE__`. Add each marker to + `TemplateContext.replacements` and test its rendered value. +- Do not format, type-check, or edit generated Python protobuf and gRPC files. +- After changing a template, generate a project in a temporary directory and + run its documented checks when practical. + +## Tests + +- Unit tests must not contact GitHub or run `uv` or Cargo in generated projects. + Pass test doubles for downloads and command execution. +- Add regression tests when changing file safety, failure cleanup, names, + manifests, network handling, or generated files. +- Run end-to-end generation in a new temporary directory. Never generate over + an existing directory. + +## Checks Run these commands from this directory: diff --git a/projects/middleware-kit/README.md b/projects/middleware-kit/README.md index b018bb4d..5f96e2b3 100644 --- a/projects/middleware-kit/README.md +++ b/projects/middleware-kit/README.md @@ -1,11 +1,11 @@ # OpenShell Middleware Kit -`middleware-kit` creates and updates runnable Python or Rust OpenShell -supervisor middleware services. A new project implements the complete gRPC -service as a pass-through, pins its protocol contract to one OpenShell release, -and includes tests, dependency locks, and registration guidance. +`middleware-kit` creates and updates Python or Rust services for OpenShell +supervisor middleware. Each new project starts as a working pass-through gRPC +service. It includes the protocol file for one OpenShell release, tests, +dependency locks, and instructions for registering the service. -The project tool does not install or replace OpenShell. +The CLI does not install or change OpenShell. ## Requirements @@ -16,7 +16,7 @@ The project tool does not install or replace OpenShell. ## Install the CLI -Install the command in an isolated tool environment from GitHub: +Install `mkit` from GitHub with `uv`: ```sh uv tool install \ @@ -30,10 +30,9 @@ instead: uv tool install /path/to/OpenShell-Research/projects/middleware-kit ``` -Both forms make `mkit` available outside the source tree -without running `uv sync` in this project. +Both commands install `mkit` for use outside this repository. -Contributors working on the CLI should use the locked project environment: +To work on the CLI itself, use the locked project environment: ```sh uv sync --locked @@ -68,38 +67,36 @@ cargo test --locked cargo run --locked -- 127.0.0.1:50051 ``` -The output path must not already exist. Use a pinned OpenShell tag for -reproducible projects; `--openshell-version latest` is available for -experimentation. +The output path must not exist. Pin an OpenShell tag when you need repeatable +builds. Use `--openshell-version latest` when you want the newest release. -Run `mkit --help` for all options. Python package names -default to a normalized project name and can be changed with `--package-name`. +Run `mkit --help` for all options. By default, `mkit` derives the Python package +name from the project name. Use `--package-name` to set it yourself. ## Update a project -From a generated project, refresh to the latest OpenShell release: +Run this inside a generated project to use the latest OpenShell release: ```sh mkit update ``` -To select a release or update a project from another directory: +To choose a release or update a project in another directory: ```sh mkit update /path/to/audit-headers \ --openshell-version v0.0.90 ``` -The update command reads `middleware-dev-manifest.json` to discover the -project language and Python package. It downloads the selected -`supervisor_middleware.proto`, regenerates Python protobuf and gRPC bindings -when applicable, refreshes `uv.lock` or `Cargo.lock`, and records the new -version and protocol checksum in the manifest. The update command accepts only -manifests that identify `middleware-kit` as their generator. +`mkit update` reads `middleware-dev-manifest.json` to find the project language +and Python package. It downloads the selected `supervisor_middleware.proto`, +regenerates Python protobuf and gRPC bindings when needed, updates `uv.lock` or +`Cargo.lock`, and writes the version and protocol checksum to the manifest. +The manifest must name `middleware-kit` as its generator. ## What you get -Each generated project contains: +Each project contains: - a pass-through implementation of `Describe`, `ValidateConfig`, and `EvaluateHttpRequest`; @@ -108,40 +105,45 @@ Each generated project contains: - tests and lint/type-check configuration; - `uv.lock` or `Cargo.lock`; and - `middleware-dev-manifest.json` with the release, source URL, and protocol - SHA-256. + checksum. Start by implementing policy behavior in the generated `validate_config` and `evaluate_http_request` functions. The generated README explains how to run the service and register it with OpenShell. -## Safety and failure behavior - -Creation is non-destructive. The project tool validates a hidden sibling -staging directory, then publishes it atomically. It refuses an existing output, -including a symlink. Updates copy the complete existing project into a hidden -sibling staging directory, change only generator-owned protocol artifacts -there, validate the staged project, then atomically exchange those artifacts -in place. User implementation files and the project directory itself are -preserved. If publication fails, completed exchanges are rolled back in reverse -order. Both operations use a per-project reservation to prevent concurrent -writers; a normal failure removes the tool's own staging and reservation -without leaving partial changes. - -If the process is killed, it may leave -`..middleware-kit.lock` and a hidden staging directory. Before -removing either one: - -1. Read `metadata.json` in the reservation. -2. On the recorded host, confirm that the recorded PID is no longer the same - project tool process. For a create operation, confirm that the final output - does not exist. For an update, do not remove the final project. -3. Inspect and remove only the recorded staging directory. -4. Remove `owner` and `metadata.json`, then remove the empty reservation with - `rmdir`. Stop if it contains anything unexpected. +## How `mkit` protects your files + +`mkit create` builds and checks the project in a temporary directory next to +the output path. It moves the finished project into place only after every +check passes. If the output path already exists, including as a symlink, the +command stops without changing it. + +`mkit update` works on a temporary copy of the project. It changes only the +protocol, generated bindings or Rust build files, lockfile, and manifest. It +runs the project checks before replacing those files. Your implementation files +stay unchanged. If a file replacement fails, `mkit` restores the files it +already replaced. + +A lock prevents two `mkit` processes from changing the same path at once. +Normal failures remove the lock and temporary files. If an update and its +rollback both fail, `mkit` keeps the recovery files and prints their locations. + +If the process is killed, it may leave a `..middleware-kit.lock` +directory and a temporary project directory. Clean them up as follows: + +1. Open `metadata.json` in the lock directory. +2. On the host listed in that file, check that the listed PID is no longer an + `mkit` process. +3. For `create`, also check that the requested output path does not exist. + Never remove the project directory after an interrupted `update`. +4. Inspect the temporary directory listed in `metadata.json`, then remove only + that directory. +5. Remove `owner` and `metadata.json`. Use `rmdir` to remove the empty lock + directory. Stop if the lock directory contains any other files. ## Develop the CLI -Run the complete local gate from this directory: +Run these checks from this directory: ```sh uv run ruff format --check . @@ -151,5 +153,5 @@ uv run pytest uv build ``` -Unit tests are hermetic: they use local protocol fixtures and do not contact -GitHub or invoke uv or Cargo for generated projects. +Unit tests use local protocol fixtures. They do not contact GitHub or run `uv` +or Cargo inside generated projects.