diff --git a/projects/README.md b/projects/README.md index 674946c2..84c451ff 100644 --- a/projects/README.md +++ b/projects/README.md @@ -6,8 +6,8 @@ layout. Current projects: -- `openshell-middleware-init`: Typer CLI that generates version-matched Python - and Rust OpenShell supervisor middleware projects. +- `middleware-kit`: `mkit` CLI that creates and updates version-matched + Python and Rust OpenShell supervisor middleware projects. - `python-project-template`: Minimal, production-ready Python project scaffold managed with uv. - `reachy-mini-openshell`: Reachy Mini conversation demo for OpenShell. diff --git a/projects/middleware-kit/AGENTS.md b/projects/middleware-kit/AGENTS.md new file mode 100644 index 00000000..e3abdbe2 --- /dev/null +++ b/projects/middleware-kit/AGENTS.md @@ -0,0 +1,59 @@ +# Agent instructions + +Read `README.md` and `pyproject.toml` before changing this project. + +## Safety rules + +- `mkit create` must never write into, follow, or replace an existing output + path, including a symlink. +- Build and check the project in a temporary directory next to its destination. + Move it into place only after every check passes. +- Check lock ownership before writing. Creation must not overwrite an existing + path. Updates must swap each generated file atomically and undo earlier swaps + if one fails. +- Support only Linux and macOS. Do not add a platform unless it can provide the + same file-safety guarantees. +- Keep the OpenShell tag, downloaded proto, bindings or Rust build files, + lockfile, and manifest on the same version. +- Never install, replace, or configure OpenShell. + +## Dependencies + +- Use `uv`. `pyproject.toml` and `uv.lock` define the dependencies. +- Do not add `requirements.txt` or another dependency export unless a documented + tool needs one. +- Use `uv add` or `uv remove` to change dependencies. Do not edit `uv.lock` by + hand. + +## Templates + +- Templates in `src/middleware_kit/templates/` must produce working standalone + projects. +- Write template markers as `__UPPER_SNAKE_CASE__`. Add each marker to + `TemplateContext.replacements` and test its rendered value. +- Do not format, type-check, or edit generated Python protobuf and gRPC files. +- After changing a template, generate a project in a temporary directory and + run its documented checks when practical. + +## Tests + +- Unit tests must not contact GitHub or run `uv` or Cargo in generated projects. + Pass test doubles for downloads and command execution. +- Add regression tests when changing file safety, failure cleanup, names, + manifests, network handling, or generated files. +- Run end-to-end generation in a new temporary directory. Never generate over + an existing directory. + +## Checks + +Run these commands from this directory: + +```sh +uv run ruff format --check . +uv run ruff check . +uv run ty check +uv run pytest +uv build +``` + +Report any command that could not run and why. diff --git a/projects/middleware-kit/README.md b/projects/middleware-kit/README.md new file mode 100644 index 00000000..5f96e2b3 --- /dev/null +++ b/projects/middleware-kit/README.md @@ -0,0 +1,157 @@ +# OpenShell Middleware Kit + +`middleware-kit` creates and updates Python or Rust services for OpenShell +supervisor middleware. Each new project starts as a working pass-through gRPC +service. It includes the protocol file for one OpenShell release, tests, +dependency locks, and instructions for registering the service. + +The CLI does not install or change OpenShell. + +## Requirements + +- Linux or macOS +- [uv](https://docs.astral.sh/uv/) +- Network access to GitHub and the selected OpenShell release +- For Rust projects: Cargo and a Rust 1.90-compatible toolchain + +## Install the CLI + +Install `mkit` from GitHub with `uv`: + +```sh +uv tool install \ + "middleware-kit @ git+https://github.com/NVIDIA/OpenShell-Research.git#subdirectory=projects/middleware-kit" +``` + +If you already have this repository checked out, install from its local path +instead: + +```sh +uv tool install /path/to/OpenShell-Research/projects/middleware-kit +``` + +Both commands install `mkit` for use outside this repository. + +To work on the CLI itself, use the locked project environment: + +```sh +uv sync --locked +uv run mkit --help +``` + +## Quick start + +Generate and run a Python starter with the installed command: + +```sh +mkit create audit-headers \ + --language python \ + --openshell-version v0.0.86 \ + --output /tmp/audit-headers + +cd /tmp/audit-headers +uv run pytest +uv run audit-headers +``` + +Or generate and run a Rust starter: + +```sh +mkit create audit-headers \ + --language rust \ + --openshell-version v0.0.86 \ + --output /tmp/audit-headers-rust + +cd /tmp/audit-headers-rust +cargo test --locked +cargo run --locked -- 127.0.0.1:50051 +``` + +The output path must not exist. Pin an OpenShell tag when you need repeatable +builds. Use `--openshell-version latest` when you want the newest release. + +Run `mkit --help` for all options. By default, `mkit` derives the Python package +name from the project name. Use `--package-name` to set it yourself. + +## Update a project + +Run this inside a generated project to use the latest OpenShell release: + +```sh +mkit update +``` + +To choose a release or update a project in another directory: + +```sh +mkit update /path/to/audit-headers \ + --openshell-version v0.0.90 +``` + +`mkit update` reads `middleware-dev-manifest.json` to find the project language +and Python package. It downloads the selected `supervisor_middleware.proto`, +regenerates Python protobuf and gRPC bindings when needed, updates `uv.lock` or +`Cargo.lock`, and writes the version and protocol checksum to the manifest. +The manifest must name `middleware-kit` as its generator. + +## What you get + +Each project contains: + +- a pass-through implementation of `Describe`, `ValidateConfig`, and + `EvaluateHttpRequest`; +- the exact `supervisor_middleware.proto` from the selected OpenShell release; +- generated Python gRPC bindings or Rust Tonic build configuration; +- tests and lint/type-check configuration; +- `uv.lock` or `Cargo.lock`; and +- `middleware-dev-manifest.json` with the release, source URL, and protocol + checksum. + +Start by implementing policy behavior in the generated `validate_config` and +`evaluate_http_request` functions. The generated README explains how to run the +service and register it with OpenShell. + +## How `mkit` protects your files + +`mkit create` builds and checks the project in a temporary directory next to +the output path. It moves the finished project into place only after every +check passes. If the output path already exists, including as a symlink, the +command stops without changing it. + +`mkit update` works on a temporary copy of the project. It changes only the +protocol, generated bindings or Rust build files, lockfile, and manifest. It +runs the project checks before replacing those files. Your implementation files +stay unchanged. If a file replacement fails, `mkit` restores the files it +already replaced. + +A lock prevents two `mkit` processes from changing the same path at once. +Normal failures remove the lock and temporary files. If an update and its +rollback both fail, `mkit` keeps the recovery files and prints their locations. + +If the process is killed, it may leave a `..middleware-kit.lock` +directory and a temporary project directory. Clean them up as follows: + +1. Open `metadata.json` in the lock directory. +2. On the host listed in that file, check that the listed PID is no longer an + `mkit` process. +3. For `create`, also check that the requested output path does not exist. + Never remove the project directory after an interrupted `update`. +4. Inspect the temporary directory listed in `metadata.json`, then remove only + that directory. +5. Remove `owner` and `metadata.json`. Use `rmdir` to remove the empty lock + directory. Stop if the lock directory contains any other files. + +## Develop the CLI + +Run these checks from this directory: + +```sh +uv run ruff format --check . +uv run ruff check . +uv run ty check +uv run pytest +uv build +``` + +Unit tests use local protocol fixtures. They do not contact GitHub or run `uv` +or Cargo inside generated projects. diff --git a/projects/openshell-middleware-init/pyproject.toml b/projects/middleware-kit/pyproject.toml similarity index 72% rename from projects/openshell-middleware-init/pyproject.toml rename to projects/middleware-kit/pyproject.toml index 6c47e66c..7b0c24f9 100644 --- a/projects/openshell-middleware-init/pyproject.toml +++ b/projects/middleware-kit/pyproject.toml @@ -1,7 +1,7 @@ [project] -name = "openshell-middleware-init" +name = "middleware-kit" version = "0.1.0" -description = "Generate version-matched OpenShell supervisor middleware projects." +description = "Create and update version-matched OpenShell middleware projects." readme = "README.md" requires-python = ">=3.10" license = "Apache-2.0" @@ -13,7 +13,7 @@ dependencies = [ ] [project.scripts] -openshell-middleware-init = "openshell_middleware_init.cli:main" +mkit = "middleware_kit.cli:main" [project.urls] Repository = "https://github.com/NVIDIA/OpenShell-Research" @@ -34,7 +34,7 @@ build-backend = "uv_build" addopts = [ "--strict-config", "--strict-markers", - "--cov=openshell_middleware_init", + "--cov=middleware_kit", "--cov-report=term-missing", "--cov-fail-under=95", ] @@ -43,14 +43,14 @@ testpaths = ["tests"] [tool.ruff] line-length = 100 target-version = "py310" -extend-exclude = ["src/openshell_middleware_init/templates"] +extend-exclude = ["src/middleware_kit/templates"] [tool.ruff.lint] select = ["B", "E", "F", "I", "RUF", "SIM", "UP"] [tool.ty.src] include = ["src", "tests"] -exclude = ["src/openshell_middleware_init/templates"] +exclude = ["src/middleware_kit/templates"] [tool.uv] required-version = ">=0.11.0" diff --git a/projects/middleware-kit/src/middleware_kit/__init__.py b/projects/middleware-kit/src/middleware_kit/__init__.py new file mode 100644 index 00000000..0020d1e3 --- /dev/null +++ b/projects/middleware-kit/src/middleware_kit/__init__.py @@ -0,0 +1,3 @@ +"""Create and update version-matched OpenShell middleware projects.""" + +__version__ = "0.1.0" diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/cli.py b/projects/middleware-kit/src/middleware_kit/cli.py similarity index 58% rename from projects/openshell-middleware-init/src/openshell_middleware_init/cli.py rename to projects/middleware-kit/src/middleware_kit/cli.py index a6e3de5a..2fdf215d 100644 --- a/projects/openshell-middleware-init/src/openshell_middleware_init/cli.py +++ b/projects/middleware-kit/src/middleware_kit/cli.py @@ -1,4 +1,4 @@ -"""Typer command-line interface for the middleware project generator.""" +"""Typer command-line interface for OpenShell middleware projects.""" from __future__ import annotations @@ -8,7 +8,11 @@ import typer -from openshell_middleware_init.generator import InitializationError, initialize_project +from middleware_kit.generator import ( + ProjectError, + create_project, + update_project, +) class Language(str, Enum): @@ -22,12 +26,12 @@ class Language(str, Enum): add_completion=False, no_args_is_help=True, pretty_exceptions_enable=False, - help="Generate a runnable, version-matched OpenShell middleware project.", + help="Create or update a version-matched OpenShell middleware project.", ) @app.command() -def init( +def create( name: Annotated[ str, typer.Argument(help="Project name, such as audit-headers."), @@ -63,22 +67,56 @@ def init( """Create a new OpenShell supervisor middleware project.""" destination = output if output is not None else Path.cwd() / name try: - result = initialize_project( + result = create_project( name=name, language=language.value, requested_version=openshell_version, destination=destination, package_name=package_name, ) - except InitializationError as error: - typer.echo(f"openshell-middleware-init: error: {error}", err=True) - raise typer.Exit(code=1) from error + except ProjectError as error: + _report_error(error) typer.echo(f"Created {result.language} middleware project at {result.destination}") typer.echo(f"OpenShell contract: {result.openshell_version}") typer.echo(f"Next: cd {result.destination} && {result.run_command}") +@app.command() +def update( + project: Annotated[ + Path, + typer.Argument( + help="Existing generated middleware project. Defaults to the current directory." + ), + ] = Path("."), + openshell_version: Annotated[ + str, + typer.Option( + "--openshell-version", + "--version", + help="OpenShell release tag (for example v0.0.86), or latest.", + ), + ] = "latest", +) -> None: + """Update an existing middleware project's OpenShell contract and generated files.""" + try: + result = update_project( + project_dir=project, + requested_version=openshell_version, + ) + except ProjectError as error: + _report_error(error) + + typer.echo(f"Updated {result.language} middleware project at {result.destination}") + typer.echo(f"OpenShell contract: {result.openshell_version}") + + +def _report_error(error: ProjectError) -> None: + typer.echo(f"mkit: error: {error}", err=True) + raise typer.Exit(code=1) from error + + def main() -> None: """Run the command-line application.""" app() diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/generator.py b/projects/middleware-kit/src/middleware_kit/generator.py similarity index 53% rename from projects/openshell-middleware-init/src/openshell_middleware_init/generator.py rename to projects/middleware-kit/src/middleware_kit/generator.py index 60267c37..72a2256b 100644 --- a/projects/openshell-middleware-init/src/openshell_middleware_init/generator.py +++ b/projects/middleware-kit/src/middleware_kit/generator.py @@ -1,4 +1,4 @@ -"""Safe, version-matched project generation.""" +"""Safe, version-matched project creation and updates.""" from __future__ import annotations @@ -26,7 +26,7 @@ from importlib.resources import files from pathlib import Path -from openshell_middleware_init import __version__ +from middleware_kit import __version__ _REPOSITORY_URL = "https://github.com/NVIDIA/OpenShell" _RAW_URL = "https://raw.githubusercontent.com/NVIDIA/OpenShell" @@ -36,6 +36,19 @@ _PYTHON_PACKAGE_PATTERN = re.compile(r"^[a-z][a-z0-9_]*$") _PROJECT_NAME_PATTERN = re.compile(r"^[a-z0-9](?:[a-z0-9._-]*[a-z0-9])?$") _NETWORK_ATTEMPTS = 4 +_TOOL_NAME = "middleware-kit" +_STAGING_IGNORED_ROOT_ENTRIES = { + ".coverage", + ".git", + ".mypy_cache", + ".pytest_cache", + ".ruff_cache", + ".ty_cache", + ".venv", + "dist", + "htmlcov", + "target", +} _RUST_KEYWORDS = { "abstract", "as", @@ -104,13 +117,17 @@ } -class InitializationError(RuntimeError): - """A user-actionable project initialization failure.""" +class ProjectError(RuntimeError): + """A user-actionable middleware project operation failure.""" + + +class PublicationRollbackError(ProjectError): + """An update failure whose recovery artifacts must be preserved.""" @dataclass(frozen=True) -class InitializationResult: - """Details about a successfully generated project.""" +class ProjectResult: + """Details about a successful middleware project operation.""" destination: Path language: str @@ -118,6 +135,14 @@ class InitializationResult: run_command: str +@dataclass(frozen=True) +class ProjectMetadata: + """Metadata needed to refresh a generated middleware project.""" + + language: str + python_package: str | None + + @dataclass(frozen=True) class OutputReservation: """Identity and recovery data for an output-path reservation.""" @@ -160,7 +185,7 @@ def replacements(self) -> Mapping[str, str]: CommandRunner = Callable[[str, Path, str], None] -def initialize_project( +def create_project( *, name: str, language: str, @@ -169,7 +194,7 @@ def initialize_project( package_name: str | None = None, download_proto: DownloadProto | None = None, command_runner: CommandRunner | None = None, -) -> InitializationResult: +) -> ProjectResult: """Generate and validate a project, then publish it atomically.""" _validate_platform() context = _template_context(name, language, package_name) @@ -184,7 +209,7 @@ def initialize_project( runner = command_runner if command_runner is not None else _prepare_project destination.parent.mkdir(parents=True, exist_ok=True) - lock_path = destination.parent / f".{destination.name}.openshell-middleware-init.lock" + lock_path = destination.parent / f".{destination.name}.{_TOOL_NAME}.lock" lock_token = secrets.token_hex(16) reservation = _acquire_lock(lock_path, lock_token, destination, version) staging_path: Path | None = None @@ -211,16 +236,16 @@ def initialize_project( _verify_lock(reservation) _publish_no_replace(staging_path, destination) staging_path = None - except InitializationError: + except ProjectError: raise except (OSError, subprocess.SubprocessError) as error: - raise InitializationError(str(error)) from error + raise ProjectError(str(error)) from error finally: if staging_path is not None: shutil.rmtree(staging_path, ignore_errors=True) _release_lock(reservation) - return InitializationResult( + return ProjectResult( destination=destination, language=language, openshell_version=version, @@ -232,24 +257,103 @@ def initialize_project( ) +def update_project( + *, + project_dir: Path, + requested_version: str = "latest", + download_proto: DownloadProto | None = None, + command_runner: CommandRunner | None = None, +) -> ProjectResult: + """Refresh generator-owned artifacts and atomically publish a validated update.""" + _validate_platform() + project_dir = project_dir.expanduser() + _validate_existing_project(project_dir) + project_dir = project_dir.resolve() + project_stat = project_dir.stat(follow_symlinks=False) + metadata = _read_project_metadata(project_dir) + if command_runner is None: + _preflight_language(metadata.language) + version = _normalize_version(requested_version) + downloader = download_proto if download_proto is not None else _download_proto + runner = command_runner if command_runner is not None else _prepare_project + + lock_path = project_dir.parent / f".{project_dir.name}.{_TOOL_NAME}.lock" + lock_token = secrets.token_hex(16) + reservation = _acquire_lock(lock_path, lock_token, project_dir, version) + staging_path: Path | None = None + published = False + preserve_recovery = False + try: + _validate_existing_project(project_dir) + _copy_project_to_staging(project_dir, reservation.staging_path) + staging_path = reservation.staging_path + proto, proto_url = downloader(version) + _validate_proto(proto, version) + _refresh_generated_artifacts( + staging_path, + metadata=metadata, + version=version, + proto_url=proto_url, + proto=proto, + ) + runner(metadata.language, staging_path, metadata.python_package or "unused") + _verify_lock(reservation) + _verify_project_identity(project_dir, project_stat.st_dev, project_stat.st_ino) + _validate_refresh_targets( + staging_path, + metadata.language, + metadata.python_package, + ) + _validate_refresh_targets( + project_dir, + metadata.language, + metadata.python_package, + ) + _publish_generated_artifacts(staging_path, project_dir, metadata) + published = True + except PublicationRollbackError: + preserve_recovery = True + raise + except ProjectError: + raise + except (OSError, subprocess.SubprocessError) as error: + raise ProjectError(str(error)) from error + finally: + if preserve_recovery: + with suppress(OSError): + os.close(reservation.directory_fd) + else: + if staging_path is not None: + shutil.rmtree(staging_path, ignore_errors=True) + _release_lock(reservation) + + if not published: # pragma: no cover - defensive; failures raise above + raise AssertionError("updated project was not published") + return ProjectResult( + destination=project_dir, + language=metadata.language, + openshell_version=version, + run_command="", + ) + + def _validate_platform() -> None: if sys.platform != "darwin" and not sys.platform.startswith("linux"): - raise InitializationError( - "openshell-middleware-init supports Linux and macOS; " - f"unsupported platform: {sys.platform}" + raise ProjectError( + f"{_TOOL_NAME} supports Linux and macOS; unsupported platform: {sys.platform}" ) def _template_context(name: str, language: str, package_name: str | None) -> TemplateContext: normalized_name = name.strip().lower() if not _PROJECT_NAME_PATTERN.fullmatch(normalized_name): - raise InitializationError( + raise ProjectError( "project name must use lowercase letters, digits, dots, hyphens, or underscores" ) if language not in {"python", "rust"}: - raise InitializationError("language must be 'python' or 'rust'") + raise ProjectError("language must be 'python' or 'rust'") if language == "rust" and package_name is not None: - raise InitializationError("--package-name is only valid with --language python") + raise ProjectError("--package-name is only valid with --language python") distribution_name = re.sub(r"[._]+", "-", normalized_name) identifier = re.sub(r"[^a-z0-9]+", "_", normalized_name).strip("_") @@ -258,7 +362,7 @@ def _template_context(name: str, language: str, package_name: str | None) -> Tem derived_package = f"middleware_{derived_package}".rstrip("_") effective_package = package_name if package_name is not None else derived_package if not _PYTHON_PACKAGE_PATTERN.fullmatch(effective_package): - raise InitializationError( + raise ProjectError( "Python package name must start with a lowercase letter and contain only " "lowercase letters, digits, and underscores" ) @@ -285,7 +389,7 @@ def _normalize_version(requested: str) -> str: if not value.startswith("v"): value = f"v{value}" if not _VERSION_PATTERN.fullmatch(value): - raise InitializationError( + raise ProjectError( f"invalid OpenShell version '{requested}'; expected a tag such as v0.0.86" ) return value @@ -294,18 +398,18 @@ def _normalize_version(requested: str) -> str: def _resolve_latest_version() -> str: request = urllib.request.Request( f"{_REPOSITORY_URL}/releases/latest", - headers={"User-Agent": f"openshell-middleware-init/{__version__}"}, + headers={"User-Agent": f"{_TOOL_NAME}/{__version__}"}, ) try: _, resolved_url = _fetch_url(request) except (OSError, urllib.error.URLError, http.client.IncompleteRead) as error: - raise InitializationError("could not resolve OpenShell's latest release") from error + raise ProjectError("could not resolve OpenShell's latest release") from error prefix = f"{_REPOSITORY_URL}/releases/tag/" if not resolved_url.startswith(prefix): - raise InitializationError(f"unexpected latest-release redirect: {resolved_url}") + raise ProjectError(f"unexpected latest-release redirect: {resolved_url}") version = resolved_url.removeprefix(prefix) if not _VERSION_PATTERN.fullmatch(version): - raise InitializationError(f"latest release has an unexpected tag: {version}") + raise ProjectError(f"latest release has an unexpected tag: {version}") return version @@ -313,21 +417,21 @@ def _download_proto(version: str) -> tuple[bytes, str]: url = f"{_RAW_URL}/{version}/{_PROTO_PATH}" request = urllib.request.Request( url, - headers={"User-Agent": f"openshell-middleware-init/{__version__}"}, + headers={"User-Agent": f"{_TOOL_NAME}/{__version__}"}, ) try: body, _ = _fetch_url(request) return body, url except urllib.error.HTTPError as error: if error.code == 404: - raise InitializationError( + raise ProjectError( f"{version} does not expose {_PROTO_PATH}; choose a middleware-capable release" ) from error - raise InitializationError( + raise ProjectError( f"could not download {_PROTO_PATH} for {version}: HTTP {error.code}" ) from error except (OSError, urllib.error.URLError, http.client.IncompleteRead) as error: - raise InitializationError( + raise ProjectError( f"could not download {_PROTO_PATH} for {version}: {_network_error_reason(error)}" ) from error @@ -364,16 +468,149 @@ def _validate_proto(proto: bytes, version: str) -> None: b"rpc EvaluateHttpRequest", ) if not proto or any(fragment not in proto for fragment in required_fragments): - raise InitializationError( + raise ProjectError( f"downloaded contract for {version} is not a supported supervisor middleware proto" ) def _validate_destination(destination: Path) -> None: if os.path.lexists(destination): - raise InitializationError(f"output path must not already exist: {destination}") + raise ProjectError(f"output path must not already exist: {destination}") if destination.name in {"", ".", ".."}: - raise InitializationError(f"invalid output path: {destination}") + raise ProjectError(f"invalid output path: {destination}") + + +def _validate_existing_project(project_dir: Path) -> None: + if project_dir.is_symlink(): + raise ProjectError(f"project path must not be a symlink: {project_dir}") + if not project_dir.is_dir(): + raise ProjectError(f"project path must be an existing directory: {project_dir}") + + +def _verify_project_identity(project_dir: Path, device: int, inode: int) -> None: + try: + current = project_dir.stat(follow_symlinks=False) + except OSError as error: + raise ProjectError("project path changed during update; refusing to publish") from error + if not stat.S_ISDIR(current.st_mode) or current.st_dev != device or current.st_ino != inode: + raise ProjectError("project path changed during update; refusing to publish") + + +def _read_project_metadata(project_dir: Path) -> ProjectMetadata: + manifest_path = project_dir / "middleware-dev-manifest.json" + if manifest_path.is_symlink() or not manifest_path.is_file(): + raise ProjectError( + f"not a generated middleware project; missing regular manifest: {manifest_path}" + ) + try: + manifest = json.loads(manifest_path.read_text()) + except (OSError, json.JSONDecodeError) as error: + raise ProjectError(f"could not read project manifest: {error}") from error + if not isinstance(manifest, dict): + raise ProjectError("project manifest must contain a JSON object") + + generator = manifest.get("generator") + generator_name = generator.get("name") if isinstance(generator, dict) else None + if generator_name != _TOOL_NAME: + raise ProjectError("project manifest generator must be middleware-kit") + languages = manifest.get("languages") + if languages not in (["python"], ["rust"]): + raise ProjectError("project manifest must identify exactly one supported language") + language = languages[0] + python_package = manifest.get("python_package") + if language == "python": + if not isinstance(python_package, str) or not _PYTHON_PACKAGE_PATTERN.fullmatch( + python_package + ): + raise ProjectError("Python project manifest has an invalid or missing python_package") + elif python_package is not None: + raise ProjectError("Rust project manifest must set python_package to null") + + _validate_refresh_targets(project_dir, language, python_package) + return ProjectMetadata(language=language, python_package=python_package) + + +def _validate_refresh_targets(project_dir: Path, language: str, python_package: str | None) -> None: + regular_files = [ + project_dir / "middleware-dev-manifest.json", + project_dir / "proto" / "supervisor_middleware.proto", + ] + regular_files.append(project_dir / ("uv.lock" if language == "python" else "Cargo.lock")) + for path in regular_files: + _reject_symlink_components(project_dir, path) + if path.is_symlink() or not path.is_file(): + raise ProjectError(f"generated artifact must be a regular file: {path}") + + if language == "python": + if python_package is None: # pragma: no cover - checked by caller + raise AssertionError("Python package is required") + bindings_dir = project_dir / "src" / python_package / "bindings" + _reject_symlink_components(project_dir, bindings_dir) + if bindings_dir.is_symlink() or not bindings_dir.is_dir(): + raise ProjectError(f"generated bindings must be an existing directory: {bindings_dir}") + + +def _reject_symlink_components(project_dir: Path, target: Path) -> None: + relative_target = target.relative_to(project_dir) + current = project_dir + for component in relative_target.parts: + current /= component + try: + current_stat = current.lstat() + except FileNotFoundError: + return + if stat.S_ISLNK(current_stat.st_mode): + raise ProjectError(f"generated artifact path must not contain symlinks: {current}") + + +def _copy_project_to_staging(project_dir: Path, staging_path: Path) -> None: + def ignore_disposable_entries(directory: str, names: list[str]) -> set[str]: + ignored = {"__pycache__"} & set(names) + if Path(directory) == project_dir: + ignored.update(_STAGING_IGNORED_ROOT_ENTRIES & set(names)) + return ignored + + shutil.copytree( + project_dir, + staging_path, + symlinks=True, + ignore=ignore_disposable_entries, + ) + staging_path.chmod(0o700) + + +def _refresh_generated_artifacts( + project_dir: Path, + *, + metadata: ProjectMetadata, + version: str, + proto_url: str, + proto: bytes, +) -> None: + _validate_refresh_targets( + project_dir, + metadata.language, + metadata.python_package, + ) + proto_path = project_dir / "proto" / "supervisor_middleware.proto" + proto_path.write_bytes(proto) + if metadata.language == "python": + if metadata.python_package is None: # pragma: no cover - checked during discovery + raise AssertionError("Python package is required") + bindings_dir = project_dir / "src" / metadata.python_package / "bindings" + shutil.rmtree(bindings_dir) + bindings_dir.mkdir() + bindings_dir.joinpath("__init__.py").write_text( + '"""Generated OpenShell supervisor middleware bindings. Do not edit."""\n' + ) + _write_manifest( + project_dir, + version=version, + proto_url=proto_url, + proto=proto, + language=metadata.language, + python_package=metadata.python_package, + ) def _acquire_lock( @@ -382,10 +619,10 @@ def _acquire_lock( try: lock_path.mkdir(mode=0o700) except FileExistsError as error: - raise InitializationError( - f"output path is reserved by another initializer: {destination}; " + raise ProjectError( + f"project path is reserved by another {_TOOL_NAME} process: {destination}; " f"inspect {lock_path / 'metadata.json'} and follow the stale-reservation " - "recovery steps in the openshell-middleware-init README" + f"recovery steps in the {_TOOL_NAME} README" ) from error directory_fd = -1 try: @@ -405,9 +642,7 @@ def _acquire_lock( device=lock_stat.st_dev, inode=lock_stat.st_ino, destination=destination, - staging_path=( - destination.parent / f".{destination.name}.openshell-middleware-init.{token}" - ), + staging_path=(destination.parent / f".{destination.name}.{_TOOL_NAME}.{token}"), version=version, started_at=datetime.now(timezone.utc).isoformat(), ) @@ -471,9 +706,9 @@ def _verify_lock(reservation: OutputReservation) -> None: raise OSError("reservation owner is not a regular file") recorded = owner.read() except OSError as error: - raise InitializationError("output reservation was lost; refusing to publish") from error + raise ProjectError("output reservation was lost; refusing to publish") from error if recorded != reservation.token: - raise InitializationError("output reservation ownership changed; refusing to publish") + raise ProjectError("output reservation ownership changed; refusing to publish") def _remove_reservation_files(reservation: OutputReservation) -> bool: @@ -514,7 +749,7 @@ def _cleanup_reservation(reservation: OutputReservation) -> None: def _release_lock(reservation: OutputReservation) -> None: try: _verify_lock(reservation) - except InitializationError: + except ProjectError: with suppress(OSError): os.close(reservation.directory_fd) return @@ -530,7 +765,7 @@ def _publish_no_replace(source: Path, destination: Path) -> None: try: rename = library.renameat2 except AttributeError as error: # pragma: no cover - old Linux libc - raise InitializationError( + raise ProjectError( "this Linux runtime cannot publish atomically without replacing an output" ) from error rename.argtypes = ( @@ -549,26 +784,212 @@ def _publish_no_replace(source: Path, destination: Path) -> None: rename.restype = ctypes.c_int result = rename(source_bytes, destination_bytes, 0x00000004) else: # pragma: no cover - unsupported platform - raise InitializationError( - "this platform cannot publish atomically without replacing an output" - ) + raise ProjectError("this platform cannot publish atomically without replacing an output") if result == 0: return error_number = ctypes.get_errno() if error_number in {errno.EEXIST, errno.ENOTEMPTY}: - raise InitializationError( + raise ProjectError( f"output path appeared during setup; refusing to overwrite it: {destination}" ) if error_number in {errno.EINVAL, errno.ENOSYS, errno.EOPNOTSUPP}: - raise InitializationError( - "the output filesystem does not support atomic no-replace publication" + raise ProjectError("the output filesystem does not support atomic no-replace publication") + raise OSError(error_number, os.strerror(error_number), destination) + + +def _publish_exchange(source: Path, destination: Path) -> None: + """Atomically exchange a validated staged project with its existing project.""" + source_parent_fd = _open_parent_directory_no_follow(source) + destination_parent_fd = _open_parent_directory_no_follow(destination) + try: + _validate_exchange_entries( + source_parent_fd, + source.name, + destination_parent_fd, + destination.name, + ) + source_name = os.fsencode(source.name) + destination_name = os.fsencode(destination.name) + result = _exchange_at( + source_parent_fd, + source_name, + destination_parent_fd, + destination_name, + ) + if result != 0: + _raise_exchange_error(destination) + source_attached = _directory_fd_matches_path(source.parent, source_parent_fd) + destination_attached = _directory_fd_matches_path( + destination.parent, + destination_parent_fd, ) + if source_attached and destination_attached: + return + + reverse_result = _exchange_at( + source_parent_fd, + source_name, + destination_parent_fd, + destination_name, + ) + if reverse_result != 0: + error_number = ctypes.get_errno() + raise PublicationRollbackError( + "an artifact parent changed during publication and the anchored exchange " + f"could not be reversed: {os.strerror(error_number)}" + ) + raise ProjectError( + "an artifact parent changed during publication; the exchange was reversed" + ) + finally: + os.close(source_parent_fd) + os.close(destination_parent_fd) + + +def _exchange_at( + source_parent_fd: int, + source_name: bytes, + destination_parent_fd: int, + destination_name: bytes, +) -> int: + if sys.platform.startswith("linux"): + library = ctypes.CDLL(None, use_errno=True) + try: + rename = library.renameat2 + except AttributeError as error: # pragma: no cover - old Linux libc + raise ProjectError( + "this Linux runtime cannot atomically publish a project update" + ) from error + exchange_flag = 2 + elif sys.platform == "darwin": # pragma: no cover - platform-specific + library = ctypes.CDLL(None, use_errno=True) + rename = library.renameatx_np + exchange_flag = 0x00000002 + else: # pragma: no cover - unsupported platform + raise ProjectError("this platform cannot atomically publish a project update") + rename.argtypes = ( + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_uint, + ) + rename.restype = ctypes.c_int + return rename( + source_parent_fd, + source_name, + destination_parent_fd, + destination_name, + exchange_flag, + ) + + +def _raise_exchange_error(destination: Path) -> None: + error_number = ctypes.get_errno() + if error_number in {errno.EINVAL, errno.ENOSYS, errno.EOPNOTSUPP}: + raise ProjectError("the project filesystem does not support atomic update publication") raise OSError(error_number, os.strerror(error_number), destination) +def _open_parent_directory_no_follow(path: Path) -> int: + return _open_directory_no_follow(path.parent) + + +def _open_directory_no_follow(path: Path) -> int: + if not path.is_absolute(): + raise ProjectError(f"artifact path must be absolute: {path}") + flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0) + descriptor = os.open(path.anchor, flags) + try: + for component in path.parts[1:]: + next_descriptor = os.open(component, flags, dir_fd=descriptor) + os.close(descriptor) + descriptor = next_descriptor + except OSError: + os.close(descriptor) + raise + return descriptor + + +def _directory_fd_matches_path(path: Path, expected_descriptor: int) -> bool: + try: + current_descriptor = _open_directory_no_follow(path) + except (ProjectError, OSError): + return False + try: + expected_stat = os.fstat(expected_descriptor) + current_stat = os.fstat(current_descriptor) + return ( + expected_stat.st_dev == current_stat.st_dev + and expected_stat.st_ino == current_stat.st_ino + ) + finally: + os.close(current_descriptor) + + +def _validate_exchange_entries( + source_parent_fd: int, + source_name: str, + destination_parent_fd: int, + destination_name: str, +) -> None: + source_stat = os.stat(source_name, dir_fd=source_parent_fd, follow_symlinks=False) + destination_stat = os.stat( + destination_name, + dir_fd=destination_parent_fd, + follow_symlinks=False, + ) + source_is_directory = stat.S_ISDIR(source_stat.st_mode) + destination_is_directory = stat.S_ISDIR(destination_stat.st_mode) + source_is_regular = stat.S_ISREG(source_stat.st_mode) + destination_is_regular = stat.S_ISREG(destination_stat.st_mode) + if not ( + (source_is_directory and destination_is_directory) + or (source_is_regular and destination_is_regular) + ): + raise ProjectError("generated artifacts changed type during update; refusing to publish") + + +def _publish_generated_artifacts( + staged_project: Path, project_dir: Path, metadata: ProjectMetadata +) -> None: + """Exchange refreshed artifacts in place, rolling back a normal publication failure.""" + relative_paths = [ + Path("proto/supervisor_middleware.proto"), + Path("uv.lock" if metadata.language == "python" else "Cargo.lock"), + ] + if metadata.language == "python": + if metadata.python_package is None: # pragma: no cover - checked during discovery + raise AssertionError("Python package is required") + relative_paths.append(Path("src") / metadata.python_package / "bindings") + relative_paths.append(Path("middleware-dev-manifest.json")) + + exchanged: list[Path] = [] + try: + for relative_path in relative_paths: + _publish_exchange( + staged_project / relative_path, + project_dir / relative_path, + ) + exchanged.append(relative_path) + except (ProjectError, OSError) as publish_error: + try: + for relative_path in reversed(exchanged): + _publish_exchange( + staged_project / relative_path, + project_dir / relative_path, + ) + except (ProjectError, OSError) as rollback_error: + raise PublicationRollbackError( + "artifact publication and rollback both failed; inspect the project and " + f"{staged_project} before removing the reservation" + ) from rollback_error + raise publish_error + + def _render_project(destination: Path, language: str, context: TemplateContext) -> None: - template_root = files("openshell_middleware_init").joinpath("templates").joinpath(language) + template_root = files("middleware_kit").joinpath("templates").joinpath(language) template_paths = { "python": ( ".gitignore", @@ -614,7 +1035,7 @@ def _write_manifest( "languages": [language], "python_package": python_package, "generator": { - "name": "openshell-middleware-init", + "name": _TOOL_NAME, "version": __version__, }, } @@ -631,7 +1052,7 @@ def _prepare_project(language: str, project_dir: Path, package_name: str) -> Non def _require_command(command: str) -> str: resolved = shutil.which(command) if resolved is None: - raise InitializationError(f"'{command}' is required to initialize this project") + raise ProjectError(f"'{command}' is required to prepare this project") return resolved @@ -653,7 +1074,7 @@ def _run( check=True, ) except subprocess.CalledProcessError as error: - raise InitializationError( + raise ProjectError( f"validation command failed with exit code {error.returncode}: {' '.join(command)}" ) from error @@ -686,12 +1107,12 @@ def _prepare_python_project(project_dir: Path, package_name: str) -> None: absolute_import = "import supervisor_middleware_pb2 as supervisor__middleware__pb2" relative_import = "from . import supervisor_middleware_pb2 as supervisor__middleware__pb2" if absolute_import not in generated: - raise InitializationError( + raise ProjectError( "generated gRPC module has an unexpected import layout; no project was published" ) grpc_module.write_text(generated.replace(absolute_import, relative_import, 1)) - with tempfile.TemporaryDirectory(prefix="openshell-middleware-init-python-") as environment: + with tempfile.TemporaryDirectory(prefix=f"{_TOOL_NAME}-python-") as environment: process_environment = os.environ.copy() process_environment.pop("VIRTUAL_ENV", None) process_environment["UV_PROJECT_ENVIRONMENT"] = environment @@ -706,9 +1127,7 @@ def _prepare_python_project(project_dir: Path, package_name: str) -> None: "run", "--project", str(project_dir), - "python", - "-c", - f"from {package_name}.server import Middleware", + "pytest", ), cwd=project_dir, environment=process_environment, @@ -717,11 +1136,11 @@ def _prepare_python_project(project_dir: Path, package_name: str) -> None: def _prepare_rust_project(project_dir: Path) -> None: cargo = _require_command("cargo") - with tempfile.TemporaryDirectory(prefix="openshell-middleware-init-rust-") as target_dir: + with tempfile.TemporaryDirectory(prefix=f"{_TOOL_NAME}-rust-") as target_dir: process_environment = os.environ.copy() process_environment["CARGO_TARGET_DIR"] = target_dir _run( - (cargo, "check", "--manifest-path", str(project_dir / "Cargo.toml")), + (cargo, "test", "--manifest-path", str(project_dir / "Cargo.toml")), cwd=project_dir, environment=process_environment, ) diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/.gitignore b/projects/middleware-kit/src/middleware_kit/templates/python/.gitignore similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/.gitignore rename to projects/middleware-kit/src/middleware_kit/templates/python/.gitignore diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/README.md b/projects/middleware-kit/src/middleware_kit/templates/python/README.md similarity index 92% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/README.md rename to projects/middleware-kit/src/middleware_kit/templates/python/README.md index 86ee41ba..3ebc6976 100644 --- a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/README.md +++ b/projects/middleware-kit/src/middleware_kit/templates/python/README.md @@ -62,9 +62,13 @@ for the policy syntax supported by your pinned OpenShell release. - `middleware-dev-manifest.json` records the release, source URL, and SHA-256. - `uv.lock` records the Python dependency solution. -Commit these files. When changing the OpenShell version, regenerate the project -or deliberately regenerate all four artifacts together; do not mix bindings and -contracts from different releases. +Commit these files. Refresh all version-matched artifacts together with: + +```sh +mkit update --openshell-version latest +``` + +Use a release tag instead of `latest` for a reproducible update. The starter is deliberately permissive. Before deployment, validate untrusted configuration, bound request and response work, avoid logging request content, diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/pyproject.toml b/projects/middleware-kit/src/middleware_kit/templates/python/pyproject.toml similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/pyproject.toml rename to projects/middleware-kit/src/middleware_kit/templates/python/pyproject.toml diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/src/package/__init__.py b/projects/middleware-kit/src/middleware_kit/templates/python/src/package/__init__.py similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/src/package/__init__.py rename to projects/middleware-kit/src/middleware_kit/templates/python/src/package/__init__.py diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/src/package/bindings/__init__.py b/projects/middleware-kit/src/middleware_kit/templates/python/src/package/bindings/__init__.py similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/src/package/bindings/__init__.py rename to projects/middleware-kit/src/middleware_kit/templates/python/src/package/bindings/__init__.py diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/src/package/server.py b/projects/middleware-kit/src/middleware_kit/templates/python/src/package/server.py similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/src/package/server.py rename to projects/middleware-kit/src/middleware_kit/templates/python/src/package/server.py diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/tests/test_server.py b/projects/middleware-kit/src/middleware_kit/templates/python/tests/test_server.py similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/python/tests/test_server.py rename to projects/middleware-kit/src/middleware_kit/templates/python/tests/test_server.py diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/.gitignore b/projects/middleware-kit/src/middleware_kit/templates/rust/.gitignore similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/.gitignore rename to projects/middleware-kit/src/middleware_kit/templates/rust/.gitignore diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/Cargo.toml b/projects/middleware-kit/src/middleware_kit/templates/rust/Cargo.toml similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/Cargo.toml rename to projects/middleware-kit/src/middleware_kit/templates/rust/Cargo.toml diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/README.md b/projects/middleware-kit/src/middleware_kit/templates/rust/README.md similarity index 92% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/README.md rename to projects/middleware-kit/src/middleware_kit/templates/rust/README.md index 8400f056..d8e9f83c 100644 --- a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/README.md +++ b/projects/middleware-kit/src/middleware_kit/templates/rust/README.md @@ -59,8 +59,13 @@ for the policy syntax supported by your pinned OpenShell release. - `middleware-dev-manifest.json` records the release, source URL, and SHA-256. - `Cargo.lock` records the Rust dependency solution. -Commit these files. When changing the OpenShell version, regenerate the project -or deliberately update the contract and manifest together. +Commit these files. Refresh all version-matched artifacts together with: + +```sh +mkit update --openshell-version latest +``` + +Use a release tag instead of `latest` for a reproducible update. The starter is deliberately permissive. Before deployment, validate untrusted configuration, bound request and response work, avoid logging request content, diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/build.rs b/projects/middleware-kit/src/middleware_kit/templates/rust/build.rs similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/build.rs rename to projects/middleware-kit/src/middleware_kit/templates/rust/build.rs diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/src/lib.rs b/projects/middleware-kit/src/middleware_kit/templates/rust/src/lib.rs similarity index 100% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/src/lib.rs rename to projects/middleware-kit/src/middleware_kit/templates/rust/src/lib.rs diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/src/main.rs b/projects/middleware-kit/src/middleware_kit/templates/rust/src/main.rs similarity index 83% rename from projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/src/main.rs rename to projects/middleware-kit/src/middleware_kit/templates/rust/src/main.rs index d0ea5378..7af6266b 100644 --- a/projects/openshell-middleware-init/src/openshell_middleware_init/templates/rust/src/main.rs +++ b/projects/middleware-kit/src/middleware_kit/templates/rust/src/main.rs @@ -1,6 +1,5 @@ use std::{env, error::Error, net::SocketAddr}; -use __RUST_LIB_NAME__::middleware_service; use tonic::transport::Server; #[tokio::main] @@ -12,7 +11,7 @@ async fn main() -> Result<(), Box> { println!("serving __SERVICE_NAME__ on {address}"); Server::builder() - .add_service(middleware_service()) + .add_service(__RUST_LIB_NAME__::middleware_service()) .serve(address) .await?; Ok(()) diff --git a/projects/middleware-kit/tests/test_cli.py b/projects/middleware-kit/tests/test_cli.py new file mode 100644 index 00000000..f6ec6b3f --- /dev/null +++ b/projects/middleware-kit/tests/test_cli.py @@ -0,0 +1,127 @@ +from pathlib import Path + +from typer.testing import CliRunner + +from middleware_kit import cli +from middleware_kit.generator import ProjectError, ProjectResult + +runner = CliRunner() + + +def test_help_describes_required_choices() -> None: + result = runner.invoke(cli.app, ["--help"]) + + assert result.exit_code == 0 + assert "create" in result.stdout + assert "update" in result.stdout + + create_help = runner.invoke(cli.app, ["create", "--help"]) + + assert create_help.exit_code == 0 + assert "--language" in create_help.stdout + assert "--openshell-version" in create_help.stdout + + +def test_cli_reports_success(monkeypatch, tmp_path: Path) -> None: + destination = tmp_path / "audit" + + def fake_create_project(**options): + assert options["name"] == "audit" + assert options["language"] == "python" + assert options["destination"] == destination + return ProjectResult( + destination=destination, + language="python", + openshell_version="v0.0.86", + run_command="uv run audit", + ) + + monkeypatch.setattr(cli, "create_project", fake_create_project) + + result = runner.invoke( + cli.app, + [ + "create", + "audit", + "--language", + "python", + "--openshell-version", + "v0.0.86", + "--output", + str(destination), + ], + ) + + assert result.exit_code == 0 + assert "Created python middleware project" in result.stdout + assert "OpenShell contract: v0.0.86" in result.stdout + + +def test_cli_reports_project_error(monkeypatch, tmp_path: Path) -> None: + def fake_create_project(**options): + del options + raise ProjectError("output exists") + + monkeypatch.setattr(cli, "create_project", fake_create_project) + + result = runner.invoke( + cli.app, + [ + "create", + "audit", + "--language", + "rust", + "--openshell-version", + "v0.0.86", + "--output", + str(tmp_path / "audit"), + ], + ) + + assert result.exit_code == 1 + assert "error: output exists" in result.stderr + + +def test_cli_reports_update_success(monkeypatch, tmp_path: Path) -> None: + destination = tmp_path / "audit" + + def fake_update_project(**options): + assert options == { + "project_dir": destination, + "requested_version": "v1.2.3", + } + return ProjectResult( + destination=destination, + language="rust", + openshell_version="v1.2.3", + run_command="", + ) + + monkeypatch.setattr(cli, "update_project", fake_update_project) + + result = runner.invoke( + cli.app, + [ + "update", + str(destination), + "--openshell-version", + "v1.2.3", + ], + ) + + assert result.exit_code == 0 + assert "Updated rust middleware project" in result.stdout + assert "OpenShell contract: v1.2.3" in result.stdout + + +def test_cli_reports_update_error(monkeypatch, tmp_path: Path) -> None: + def fake_update_project(**options): + del options + raise ProjectError("not generated") + + monkeypatch.setattr(cli, "update_project", fake_update_project) + + result = runner.invoke(cli.app, ["update", str(tmp_path)]) + + assert result.exit_code == 1 + assert "error: not generated" in result.stderr diff --git a/projects/openshell-middleware-init/tests/test_generator.py b/projects/middleware-kit/tests/test_generator.py similarity index 53% rename from projects/openshell-middleware-init/tests/test_generator.py rename to projects/middleware-kit/tests/test_generator.py index d337561e..ed86ada0 100644 --- a/projects/openshell-middleware-init/tests/test_generator.py +++ b/projects/middleware-kit/tests/test_generator.py @@ -14,8 +14,8 @@ import pytest -from openshell_middleware_init import generator -from openshell_middleware_init.generator import InitializationError, initialize_project +from middleware_kit import generator +from middleware_kit.generator import ProjectError, create_project, update_project PROTO = b"""syntax = "proto3"; package openshell.middleware.v1; @@ -25,6 +25,7 @@ message HttpRequestEvaluation {} message HttpRequestResult {} """ +UPDATED_PROTO = PROTO + b"// refreshed contract\n" def local_proto(version: str) -> tuple[bytes, str]: @@ -42,7 +43,7 @@ def no_op_runner(language: str, project: Path, package: str) -> None: def test_generates_python_project_with_provenance(tmp_path: Path) -> None: destination = tmp_path / "audit-headers" - result = initialize_project( + result = create_project( name="audit-headers", language="python", requested_version="0.0.86", @@ -63,13 +64,13 @@ def test_generates_python_project_with_provenance(tmp_path: Path) -> None: assert manifest["languages"] == ["python"] assert manifest["python_package"] == "audit_headers" assert len(manifest["proto_sha256"]) == 64 - assert not (tmp_path / ".audit-headers.openshell-middleware-init.lock").exists() + assert not (tmp_path / ".audit-headers.middleware-kit.lock").exists() def test_generates_rust_project_with_normalized_crate_name(tmp_path: Path) -> None: destination = tmp_path / "request.audit" - result = initialize_project( + result = create_project( name="request.audit", language="rust", requested_version="v0.0.86", @@ -82,17 +83,474 @@ def test_generates_rust_project_with_normalized_crate_name(tmp_path: Path) -> No cargo = (destination / "Cargo.toml").read_text() assert 'name = "request-audit"' in cargo assert '[lib]\nname = "request_audit"' in cargo - assert "use request_audit::" in (destination / "src/main.rs").read_text() + assert ( + ".add_service(request_audit::middleware_service())" + in (destination / "src/main.rs").read_text() + ) assert stat.S_IMODE(destination.stat().st_mode) == 0o755 manifest = json.loads((destination / "middleware-dev-manifest.json").read_text()) assert manifest["languages"] == ["rust"] assert manifest["python_package"] is None +def test_updates_python_generated_artifacts_and_preserves_user_code(tmp_path: Path) -> None: + destination = tmp_path / "audit-headers" + create_project( + name="audit-headers", + language="python", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + server_path = destination / "src/audit_headers/server.py" + server_path.write_text(server_path.read_text() + "\n# user policy\n") + old_binding = destination / "src/audit_headers/bindings/old_generated.py" + old_binding.write_text("old generated code\n") + project_inode = destination.stat().st_ino + + def updated_proto(version: str) -> tuple[bytes, str]: + return UPDATED_PROTO, f"https://example.test/OpenShell/{version}/proto" + + result = update_project( + project_dir=destination, + requested_version="1.2.3", + download_proto=updated_proto, + command_runner=no_op_runner, + ) + + assert result.destination == destination + assert result.language == "python" + assert result.openshell_version == "v1.2.3" + assert destination.stat().st_ino == project_inode + assert server_path.read_text().endswith("# user policy\n") + assert (destination / "proto/supervisor_middleware.proto").read_bytes() == UPDATED_PROTO + assert not old_binding.exists() + assert (destination / "src/audit_headers/bindings/__init__.py").is_file() + manifest = json.loads((destination / "middleware-dev-manifest.json").read_text()) + assert manifest["openshell_version"] == "v1.2.3" + assert manifest["generator"]["name"] == "middleware-kit" + assert not (tmp_path / ".audit-headers.middleware-kit.lock").exists() + assert not list(tmp_path.glob(".audit-headers.middleware-kit.*")) + + +def test_failed_update_keeps_original_project_unchanged(tmp_path: Path) -> None: + destination = tmp_path / "audit" + create_project( + name="audit", + language="rust", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + original_manifest = (destination / "middleware-dev-manifest.json").read_bytes() + original_proto = (destination / "proto/supervisor_middleware.proto").read_bytes() + + def fail_runner(language: str, project: Path, package: str) -> None: + del language, project, package + raise ProjectError("validation failed") + + with pytest.raises(ProjectError, match="validation failed"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=fail_runner, + ) + + assert (destination / "middleware-dev-manifest.json").read_bytes() == original_manifest + assert (destination / "proto/supervisor_middleware.proto").read_bytes() == original_proto + assert not (tmp_path / ".audit.middleware-kit.lock").exists() + assert not list(tmp_path.glob(".audit.middleware-kit.*")) + + +def test_publication_failure_rolls_back_exchanged_artifacts( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + destination = tmp_path / "audit" + create_project( + name="audit", + language="rust", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + original_manifest = (destination / "middleware-dev-manifest.json").read_bytes() + original_proto = (destination / "proto/supervisor_middleware.proto").read_bytes() + original_exchange = generator._publish_exchange + calls = 0 + + def fail_second_exchange(source: Path, target: Path) -> None: + nonlocal calls + calls += 1 + if calls == 2: + raise ProjectError("publication failed") + original_exchange(source, target) + + monkeypatch.setattr(generator, "_publish_exchange", fail_second_exchange) + + with pytest.raises(ProjectError, match="publication failed"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=no_op_runner, + ) + + assert calls == 3 + assert (destination / "middleware-dev-manifest.json").read_bytes() == original_manifest + assert (destination / "proto/supervisor_middleware.proto").read_bytes() == original_proto + + +def test_failed_publication_rollback_preserves_recovery_artifacts( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + destination = tmp_path / "audit" + create_project( + name="audit", + language="rust", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + original_exchange = generator._publish_exchange + calls = 0 + + def fail_publication_and_rollback(source: Path, target: Path) -> None: + nonlocal calls + calls += 1 + if calls >= 2: + raise ProjectError("exchange failed") + original_exchange(source, target) + + monkeypatch.setattr(generator, "_publish_exchange", fail_publication_and_rollback) + + with pytest.raises(generator.PublicationRollbackError, match="rollback both failed"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=no_op_runner, + ) + + assert calls == 3 + assert (tmp_path / ".audit.middleware-kit.lock").is_dir() + assert len(list(tmp_path.glob(".audit.middleware-kit.*"))) == 2 + + +def test_update_rejects_non_generated_project(tmp_path: Path) -> None: + destination = tmp_path / "not-generated" + destination.mkdir() + + with pytest.raises(ProjectError, match="missing regular manifest"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=local_proto, + command_runner=no_op_runner, + ) + + +def test_update_rejects_symlink_and_missing_project_paths(tmp_path: Path) -> None: + missing = tmp_path / "missing" + symlink = tmp_path / "symlink" + symlink.symlink_to(missing, target_is_directory=True) + + with pytest.raises(ProjectError, match="must not be a symlink"): + update_project( + project_dir=symlink, + download_proto=local_proto, + command_runner=no_op_runner, + ) + with pytest.raises(ProjectError, match="existing directory"): + update_project( + project_dir=missing, + download_proto=local_proto, + command_runner=no_op_runner, + ) + + +@pytest.mark.parametrize( + ("manifest", "message"), + [ + ("not json", "could not read"), + ("[]", "JSON object"), + ('{"generator": {"name": "other"}}', "generator must be middleware-kit"), + ( + '{"generator": {"name": "middleware-kit"}, "languages": ["python", "rust"]}', + "exactly one", + ), + ( + '{"generator": {"name": "middleware-kit"}, ' + '"languages": ["python"], "python_package": "Bad-Package"}', + "invalid or missing", + ), + ( + '{"generator": {"name": "middleware-kit"}, ' + '"languages": ["rust"], "python_package": "unexpected"}', + "must set python_package", + ), + ], +) +def test_update_rejects_invalid_manifest(tmp_path: Path, manifest: str, message: str) -> None: + destination = tmp_path / "project" + destination.mkdir() + (destination / "middleware-dev-manifest.json").write_text(manifest) + + with pytest.raises(ProjectError, match=message): + update_project( + project_dir=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + + +def test_update_requires_regular_generated_artifacts(tmp_path: Path) -> None: + destination = tmp_path / "project" + destination.mkdir() + (destination / "middleware-dev-manifest.json").write_text( + json.dumps( + { + "generator": {"name": "middleware-kit"}, + "languages": ["rust"], + "python_package": None, + } + ) + ) + + with pytest.raises(ProjectError, match="regular file"): + update_project( + project_dir=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + + +def test_update_requires_python_bindings_directory(tmp_path: Path) -> None: + destination = tmp_path / "project" + (destination / "proto").mkdir(parents=True) + (destination / "proto/supervisor_middleware.proto").write_bytes(PROTO) + (destination / "uv.lock").write_text("test lock\n") + (destination / "middleware-dev-manifest.json").write_text( + json.dumps( + { + "generator": {"name": "middleware-kit"}, + "languages": ["python"], + "python_package": "audit", + } + ) + ) + + with pytest.raises(ProjectError, match="bindings must be"): + update_project( + project_dir=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + + +def test_update_rejects_symlinked_proto_directory_without_touching_target( + tmp_path: Path, +) -> None: + destination = tmp_path / "project" + create_project( + name="project", + language="rust", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + (destination / "proto").rename(destination / "original-proto") + external_proto = tmp_path / "external-proto" + external_proto.mkdir() + sentinel = external_proto / "supervisor_middleware.proto" + sentinel.write_text("external contract\n") + (destination / "proto").symlink_to(external_proto, target_is_directory=True) + + with pytest.raises(ProjectError, match="must not contain symlinks"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=no_op_runner, + ) + + assert sentinel.read_text() == "external contract\n" + + +def test_update_rejects_symlinked_python_package_without_touching_target( + tmp_path: Path, +) -> None: + destination = tmp_path / "project" + create_project( + name="project", + language="python", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + package_dir = destination / "src/project" + package_dir.rename(destination / "original-package") + external_package = tmp_path / "external-package" + bindings = external_package / "bindings" + bindings.mkdir(parents=True) + sentinel = bindings / "keep.txt" + sentinel.write_text("external binding\n") + package_dir.symlink_to(external_package, target_is_directory=True) + + with pytest.raises(ProjectError, match="must not contain symlinks"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=no_op_runner, + ) + + assert sentinel.read_text() == "external binding\n" + + +def test_update_omits_disposable_directories_from_staging(tmp_path: Path) -> None: + destination = tmp_path / "project" + create_project( + name="project", + language="python", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + for directory_name in (".git", ".venv", ".pytest_cache", "dist", "target"): + disposable = destination / directory_name + disposable.mkdir() + (disposable / "large-cache").write_text("not needed\n") + nested_cache = destination / "src/project/__pycache__" + nested_cache.mkdir() + (nested_cache / "server.pyc").write_bytes(b"cache") + user_file = destination / "policy-notes.txt" + user_file.write_text("preserve me\n") + + def inspect_staging(language: str, project: Path, package: str) -> None: + assert language == "python" + assert package == "project" + assert user_file.name in {path.name for path in project.iterdir()} + for directory_name in (".git", ".venv", ".pytest_cache", "dist", "target"): + assert not (project / directory_name).exists() + assert not (project / "src/project/__pycache__").exists() + no_op_runner(language, project, package) + + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=inspect_staging, + ) + + assert user_file.read_text() == "preserve me\n" + assert (destination / ".venv/large-cache").read_text() == "not needed\n" + + +@pytest.mark.parametrize("replace_live_path", [False, True]) +def test_update_revalidates_symlink_ancestors_after_project_validation( + tmp_path: Path, replace_live_path: bool +) -> None: + destination = tmp_path / "project" + create_project( + name="project", + language="rust", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + original_manifest = (destination / "middleware-dev-manifest.json").read_bytes() + external_proto = tmp_path / "external-proto" + external_proto.mkdir() + sentinel = external_proto / "supervisor_middleware.proto" + sentinel.write_text("external contract\n") + + def replace_proto_ancestor(language: str, staged_project: Path, package: str) -> None: + no_op_runner(language, staged_project, package) + project_to_change = destination if replace_live_path else staged_project + (project_to_change / "proto").rename(project_to_change / "original-proto") + (project_to_change / "proto").symlink_to(external_proto, target_is_directory=True) + + with pytest.raises(ProjectError, match="must not contain symlinks"): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=replace_proto_ancestor, + ) + + assert sentinel.read_text() == "external contract\n" + assert (destination / "middleware-dev-manifest.json").read_bytes() == original_manifest + + +def test_exchange_refuses_symlinked_parent_created_immediately_before_publish( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + destination = tmp_path / "project" + create_project( + name="project", + language="rust", + requested_version="v0.0.86", + destination=destination, + download_proto=local_proto, + command_runner=no_op_runner, + ) + external_proto = tmp_path / "external-proto" + external_proto.mkdir() + sentinel = external_proto / "supervisor_middleware.proto" + sentinel.write_text("external contract\n") + original_exchange = generator._publish_exchange + first_exchange = True + + def replace_parent_then_exchange(source: Path, target: Path) -> None: + nonlocal first_exchange + if first_exchange: + first_exchange = False + target.parent.rename(destination / "original-proto") + target.parent.symlink_to(external_proto, target_is_directory=True) + original_exchange(source, target) + + monkeypatch.setattr(generator, "_publish_exchange", replace_parent_then_exchange) + + with pytest.raises(ProjectError): + update_project( + project_dir=destination, + requested_version="v1.2.3", + download_proto=lambda version: (UPDATED_PROTO, f"https://example.test/{version}"), + command_runner=no_op_runner, + ) + + assert sentinel.read_text() == "external contract\n" + + +def test_update_detects_replaced_project_before_publication(tmp_path: Path) -> None: + project = tmp_path / "project" + replacement = tmp_path / "replacement" + project.mkdir() + original = project.stat() + project.rename(replacement) + project.mkdir() + + with pytest.raises(ProjectError, match="changed during update"): + generator._verify_project_identity(project, original.st_dev, original.st_ino) + + project.rmdir() + with pytest.raises(ProjectError, match="changed during update"): + generator._verify_project_identity(project, original.st_dev, original.st_ino) + + def test_python_package_name_can_be_overridden(tmp_path: Path) -> None: destination = tmp_path / "project" - initialize_project( + create_project( name="project", language="python", requested_version="v0.0.86", @@ -108,7 +566,7 @@ def test_python_package_name_can_be_overridden(tmp_path: Path) -> None: def test_numeric_project_name_gets_importable_python_package(tmp_path: Path) -> None: destination = tmp_path / "123" - initialize_project( + create_project( name="123", language="python", requested_version="v0.0.86", @@ -134,7 +592,7 @@ def test_rust_project_names_get_valid_explicit_library_names( ) -> None: destination = tmp_path / name - initialize_project( + create_project( name=name, language="rust", requested_version="v0.0.86", @@ -146,7 +604,10 @@ def test_rust_project_names_get_valid_explicit_library_names( cargo = (destination / "Cargo.toml").read_text() assert f'name = "{crate}"' in cargo assert f'[lib]\nname = "{library}"' in cargo - assert f"use {library}::middleware_service;" in (destination / "src/main.rs").read_text() + assert ( + f".add_service({library}::middleware_service())" + in (destination / "src/main.rs").read_text() + ) def test_unsupported_platform_fails_before_filesystem_changes( @@ -155,8 +616,8 @@ def test_unsupported_platform_fails_before_filesystem_changes( destination = tmp_path / "output" monkeypatch.setattr(generator.sys, "platform", "win32") - with pytest.raises(InitializationError, match="supports Linux and macOS"): - initialize_project( + with pytest.raises(ProjectError, match="supports Linux and macOS"): + create_project( name="project", language="python", requested_version="v0.0.86", @@ -185,8 +646,8 @@ def test_rejects_invalid_project_choices( package_name: str | None, message: str, ) -> None: - with pytest.raises(InitializationError, match=message): - initialize_project( + with pytest.raises(ProjectError, match=message): + create_project( name=name, language=language, requested_version="v0.0.86", @@ -199,8 +660,8 @@ def test_rejects_invalid_project_choices( @pytest.mark.parametrize("version", ["", "main", "v1", "v1.2", "v1.2.x"]) def test_rejects_invalid_versions(tmp_path: Path, version: str) -> None: - with pytest.raises(InitializationError, match="invalid OpenShell version"): - initialize_project( + with pytest.raises(ProjectError, match="invalid OpenShell version"): + create_project( name="project", language="python", requested_version=version, @@ -214,8 +675,8 @@ def test_refuses_an_existing_destination(tmp_path: Path) -> None: destination = tmp_path / "existing" destination.mkdir() - with pytest.raises(InitializationError, match="must not already exist"): - initialize_project( + with pytest.raises(ProjectError, match="must not already exist"): + create_project( name="existing", language="python", requested_version="v0.0.86", @@ -230,8 +691,8 @@ def test_refuses_a_dangling_destination_symlink(tmp_path: Path) -> None: target = tmp_path / "symlink-target" destination.symlink_to(target, target_is_directory=True) - with pytest.raises(InitializationError, match="must not already exist"): - initialize_project( + with pytest.raises(ProjectError, match="must not already exist"): + create_project( name="project", language="python", requested_version="v0.0.86", @@ -246,10 +707,10 @@ def test_refuses_a_dangling_destination_symlink(tmp_path: Path) -> None: def test_refuses_a_reserved_destination(tmp_path: Path) -> None: destination = tmp_path / "reserved" - (tmp_path / ".reserved.openshell-middleware-init.lock").mkdir() + (tmp_path / ".reserved.middleware-kit.lock").mkdir() - with pytest.raises(InitializationError, match="reserved by another initializer"): - initialize_project( + with pytest.raises(ProjectError, match="reserved by another middleware-kit"): + create_project( name="reserved", language="rust", requested_version="v0.0.86", @@ -272,8 +733,8 @@ def collide_before_publish(source: Path, final_output: Path) -> None: monkeypatch.setattr(generator, "_publish_no_replace", collide_before_publish) - with pytest.raises(InitializationError, match="appeared during setup"): - initialize_project( + with pytest.raises(ProjectError, match="appeared during setup"): + create_project( name="contended", language="python", requested_version="v0.0.86", @@ -283,7 +744,7 @@ def collide_before_publish(source: Path, final_output: Path) -> None: ) assert (destination / "owned-by-other-process").read_text() == "keep me\n" - assert not (tmp_path / ".contended.openshell-middleware-init.lock").exists() + assert not (tmp_path / ".contended.middleware-kit.lock").exists() def test_failure_cleans_staging_and_owned_reservation(tmp_path: Path) -> None: @@ -291,10 +752,10 @@ def test_failure_cleans_staging_and_owned_reservation(tmp_path: Path) -> None: def fail_runner(language: str, project: Path, package: str) -> None: del language, project, package - raise InitializationError("validation failed") + raise ProjectError("validation failed") - with pytest.raises(InitializationError, match="validation failed"): - initialize_project( + with pytest.raises(ProjectError, match="validation failed"): + create_project( name="failing", language="python", requested_version="v0.0.86", @@ -304,16 +765,16 @@ def fail_runner(language: str, project: Path, package: str) -> None: ) assert not destination.exists() - assert not (tmp_path / ".failing.openshell-middleware-init.lock").exists() - assert not list(tmp_path.glob(".failing.openshell-middleware-init.*")) + assert not (tmp_path / ".failing.middleware-kit.lock").exists() + assert not list(tmp_path.glob(".failing.middleware-kit.*")) def test_rejects_an_unexpected_proto(tmp_path: Path) -> None: def invalid_proto(version: str) -> tuple[bytes, str]: return b"not a proto", f"https://example.test/{version}" - with pytest.raises(InitializationError, match="not a supported"): - initialize_project( + with pytest.raises(ProjectError, match="not a supported"): + create_project( name="invalid-proto", language="rust", requested_version="v0.0.86", @@ -326,12 +787,12 @@ def invalid_proto(version: str) -> tuple[bytes, str]: def test_missing_required_command_has_actionable_error(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(generator.shutil, "which", lambda command: None) - with pytest.raises(InitializationError, match="'uv' is required"): + with pytest.raises(ProjectError, match="'uv' is required"): generator._require_command("uv") def test_failed_subprocess_is_translated(tmp_path: Path) -> None: - with pytest.raises(InitializationError, match="validation command failed"): + with pytest.raises(ProjectError, match="validation command failed"): generator._run( (sys.executable, "-c", "raise SystemExit(7)"), cwd=tmp_path, @@ -343,8 +804,8 @@ def fail_runner(language: str, project: Path, package: str) -> None: del language, project, package raise subprocess.SubprocessError("tool failed") - with pytest.raises(InitializationError, match="tool failed"): - initialize_project( + with pytest.raises(ProjectError, match="tool failed"): + create_project( name="failure", language="rust", requested_version="v0.0.86", @@ -392,7 +853,7 @@ def test_latest_version_rejects_unexpected_redirects( response = FakeResponse(url=resolved_url) monkeypatch.setattr(generator.urllib.request, "urlopen", lambda *args, **kwargs: response) - with pytest.raises(InitializationError, match=message): + with pytest.raises(ProjectError, match=message): generator._resolve_latest_version() @@ -408,7 +869,7 @@ def fail(*args: object, **kwargs: object) -> None: monkeypatch.setattr(generator.urllib.request, "urlopen", fail) monkeypatch.setattr(generator.time, "sleep", lambda _: None) - with pytest.raises(InitializationError, match="could not resolve"): + with pytest.raises(ProjectError, match="could not resolve"): generator._resolve_latest_version() assert attempts == 4 @@ -431,10 +892,14 @@ def fail(*args: object, **kwargs: object) -> None: monkeypatch.setattr(generator.urllib.request, "urlopen", fail) monkeypatch.setattr(generator.time, "sleep", lambda _: None) - with pytest.raises(InitializationError, match=r"could not download.*missing"): + with pytest.raises(ProjectError, match=r"could not download.*missing"): generator._download_proto("v1.2.3") +def test_network_error_reason_handles_plain_os_error() -> None: + assert generator._network_error_reason(OSError("offline")) == "offline" + + def test_download_retries_transient_failures(monkeypatch: pytest.MonkeyPatch) -> None: attempts = 0 response = FakeResponse(body=PROTO) @@ -496,7 +961,7 @@ def fail(*args: object, **kwargs: object) -> None: monkeypatch.setattr(generator.time, "sleep", lambda _: None) expected_message = "middleware-capable release" if status == 404 else "HTTP 503" - with pytest.raises(InitializationError, match=expected_message): + with pytest.raises(ProjectError, match=expected_message): generator._download_proto("v1.2.3") assert attempts == expected_attempts @@ -514,8 +979,8 @@ def test_toolchain_preflight_precedes_latest_resolution_and_output_changes( lambda: pytest.fail("latest must not be resolved before toolchain preflight"), ) - with pytest.raises(InitializationError, match=rf"'{command}' is required"): - generator.initialize_project( + with pytest.raises(ProjectError, match=rf"'{command}' is required"): + generator.create_project( name="audit-headers", language=language, requested_version="latest", @@ -541,12 +1006,12 @@ def test_lock_verification_detects_loss_and_changed_owner(tmp_path: Path) -> Non started_at="2026-07-22T00:00:00+00:00", ) - with pytest.raises(InitializationError, match="reservation was lost"): + with pytest.raises(ProjectError, match="reservation was lost"): generator._verify_lock(missing) reservation = generator._acquire_lock(lock, "mine", tmp_path / "output", "v0.0.86") (lock / "owner").write_text("theirs") - with pytest.raises(InitializationError, match="ownership changed"): + with pytest.raises(ProjectError, match="ownership changed"): generator._verify_lock(reservation) generator._release_lock(reservation) @@ -596,7 +1061,7 @@ def test_reservation_verification_rejects_changed_directory_identity(tmp_path: P started_at=reservation.started_at, ) - with pytest.raises(InitializationError, match="reservation was lost"): + with pytest.raises(ProjectError, match="reservation was lost"): generator._verify_lock(changed_identity) generator._cleanup_reservation(reservation) @@ -606,7 +1071,7 @@ def test_reservation_verification_rejects_non_file_owner(tmp_path: Path) -> None (reservation.path / "owner").unlink() (reservation.path / "owner").mkdir() - with pytest.raises(InitializationError, match="reservation was lost"): + with pytest.raises(ProjectError, match="reservation was lost"): generator._verify_lock(reservation) generator._release_lock(reservation) @@ -665,6 +1130,36 @@ def test_publish_no_replace_moves_into_absent_destination(tmp_path: Path) -> Non assert (destination / "generated").read_text() == "ready\n" +def test_exchange_reverses_when_open_parent_is_detached( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + source_parent = tmp_path / "staged/proto" + destination_parent = tmp_path / "project/proto" + source_parent.mkdir(parents=True) + destination_parent.mkdir(parents=True) + source = source_parent / "supervisor_middleware.proto" + destination = destination_parent / "supervisor_middleware.proto" + source.write_text("new contract\n") + destination.write_text("old contract\n") + moved_destination_parent = tmp_path / "project/moved-proto" + validate_exchange_entries = generator._validate_exchange_entries + + def validate_then_detach(*arguments) -> None: + validate_exchange_entries(*arguments) + destination_parent.rename(moved_destination_parent) + destination_parent.mkdir() + (destination_parent / destination.name).write_text("concurrent replacement\n") + + monkeypatch.setattr(generator, "_validate_exchange_entries", validate_then_detach) + + with pytest.raises(ProjectError, match=r"parent changed.*exchange was reversed"): + generator._publish_exchange(source, destination) + + assert source.read_text() == "new contract\n" + assert (moved_destination_parent / destination.name).read_text() == "old contract\n" + assert destination.read_text() == "concurrent replacement\n" + + class FakeRename: def __init__(self, error_number: int) -> None: self.error_number = error_number @@ -690,7 +1185,7 @@ def test_publish_reports_filesystem_without_no_replace_support( ) -> None: monkeypatch.setattr(generator.ctypes, "CDLL", lambda *args, **kwargs: FakeLibrary(error_number)) - with pytest.raises(InitializationError, match="does not support"): + with pytest.raises(ProjectError, match="does not support"): generator._publish_no_replace(tmp_path / "source", tmp_path / "destination") @@ -737,13 +1232,13 @@ def test_require_command_returns_resolved_path(monkeypatch: pytest.MonkeyPatch) def test_run_passes_environment_to_subprocess(tmp_path: Path) -> None: environment = os.environ.copy() - environment["MIDDLEWARE_INIT_TEST_VALUE"] = "present" + environment["MIDDLEWARE_KIT_TEST_VALUE"] = "present" generator._run( ( sys.executable, "-c", - "import os; assert os.environ['MIDDLEWARE_INIT_TEST_VALUE'] == 'present'", + "import os; assert os.environ['MIDDLEWARE_KIT_TEST_VALUE'] == 'present'", ), cwd=tmp_path, environment=environment, @@ -777,6 +1272,7 @@ def fake_run(command, *, cwd, environment=None) -> None: assert generated.startswith("from . import supervisor_middleware_pb2") assert len(calls) == 3 assert calls[1][1] == "sync" + assert calls[2][-1] == "pytest" def test_prepare_python_rejects_unexpected_generated_import( @@ -794,30 +1290,31 @@ def fake_run(command, *, cwd, environment=None) -> None: monkeypatch.setattr(generator, "_run", fake_run) - with pytest.raises(InitializationError, match="unexpected import layout"): + with pytest.raises(ProjectError, match="unexpected import layout"): generator._prepare_python_project(tmp_path, "audit") def test_prepare_rust_runs_cargo_with_temporary_target( monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: - observed: dict[str, object] = {} + observed: list[tuple[tuple[str, ...], Path, dict[str, str]]] = [] monkeypatch.setattr(generator, "_require_command", lambda command: f"/tools/{command}") def fake_run(command, *, cwd, environment=None) -> None: - observed.update(command=command, cwd=cwd, environment=environment) + assert environment is not None + observed.append((tuple(command), cwd, environment)) monkeypatch.setattr(generator, "_run", fake_run) generator._prepare_rust_project(tmp_path) - assert observed["command"] == ( - "/tools/cargo", - "check", - "--manifest-path", - str(tmp_path / "Cargo.toml"), - ) - assert observed["cwd"] == tmp_path - environment = observed["environment"] - assert isinstance(environment, dict) - assert "CARGO_TARGET_DIR" in environment + assert [command for command, _, _ in observed] == [ + ( + "/tools/cargo", + "test", + "--manifest-path", + str(tmp_path / "Cargo.toml"), + ), + ] + assert all(cwd == tmp_path for _, cwd, _ in observed) + assert all("CARGO_TARGET_DIR" in environment for _, _, environment in observed) diff --git a/projects/openshell-middleware-init/uv.lock b/projects/middleware-kit/uv.lock similarity index 99% rename from projects/openshell-middleware-init/uv.lock rename to projects/middleware-kit/uv.lock index bfd92049..1a28bb8b 100644 --- a/projects/openshell-middleware-init/uv.lock +++ b/projects/middleware-kit/uv.lock @@ -166,7 +166,7 @@ wheels = [ ] [[package]] -name = "openshell-middleware-init" +name = "middleware-kit" version = "0.1.0" source = { editable = "." } dependencies = [ diff --git a/projects/openshell-middleware-init/AGENTS.md b/projects/openshell-middleware-init/AGENTS.md deleted file mode 100644 index 864d071c..00000000 --- a/projects/openshell-middleware-init/AGENTS.md +++ /dev/null @@ -1,58 +0,0 @@ -# Agent instructions - -Read `README.md` and `pyproject.toml` before changing this project. - -## Preserve these invariants - -- Keep initialization non-destructive. Never merge into, follow, or replace an - existing output path, including a symlink. -- Build and validate in a hidden sibling staging directory. Publish only after - all generation and validation steps succeed. -- Preserve reservation ownership checks and atomic no-replace publication. -- Support Linux and macOS explicitly. Do not weaken publication guarantees to - add another platform implicitly. -- Keep every generated project version-matched: the OpenShell tag, downloaded - proto, bindings or build configuration, lockfile, and manifest must agree. -- Do not install, replace, or configure the user's OpenShell installation. - -## Use the project toolchain - -- Use `uv` for this Python project. Treat `pyproject.toml` and `uv.lock` as the - dependency sources of truth. -- Do not add `requirements.txt` or another dependency export unless a documented - non-uv consumer requires it. -- Use `uv add` or `uv remove` for dependency changes; do not hand-edit the lock. - -## Change templates carefully - -- Keep templates under `src/openshell_middleware_init/templates/` runnable as - standalone projects. -- Use `__UPPER_SNAKE_CASE__` for template markers. Add every marker to - `TemplateContext.replacements` and cover it with a rendering test. -- Treat generated Python protobuf and gRPC modules as generator-owned. Do not - format, type-check, or hand-edit them. -- When changing a template, generate the affected language project in isolated - scratch storage and run its documented checks when practical. - -## Test behavior, not implementation details - -- Keep initializer unit tests hermetic. Inject protocol downloads and project - preparation instead of contacting GitHub or invoking uv or Cargo. -- Add regression tests for changes to output safety, failure cleanup, naming, - manifests, network behavior, or rendered files. -- Use isolated temporary paths for end-to-end generation. Never generate over an - existing directory. - -## Validate every change - -Run these commands from this directory: - -```sh -uv run ruff format --check . -uv run ruff check . -uv run ty check -uv run pytest -uv build -``` - -Report any command that could not run and why. diff --git a/projects/openshell-middleware-init/README.md b/projects/openshell-middleware-init/README.md deleted file mode 100644 index bbad8952..00000000 --- a/projects/openshell-middleware-init/README.md +++ /dev/null @@ -1,127 +0,0 @@ -# OpenShell Middleware Init - -`openshell-middleware-init` creates a runnable Python or Rust starter for an -OpenShell supervisor middleware service. The starter implements the complete -gRPC service as a pass-through, pins its protocol contract to one OpenShell -release, and includes tests, dependency locks, and registration guidance. - -The initializer does not install or replace OpenShell. - -## Requirements - -- Linux or macOS -- [uv](https://docs.astral.sh/uv/) -- Network access to GitHub and the selected OpenShell release -- For Rust projects: Cargo and a Rust 1.90-compatible toolchain - -## Install the CLI - -Install the command in an isolated tool environment from GitHub: - -```sh -uv tool install \ - "openshell-middleware-init @ git+https://github.com/NVIDIA/OpenShell-Research.git#subdirectory=projects/openshell-middleware-init" -``` - -If you already have this repository checked out, install from its local path -instead: - -```sh -uv tool install /path/to/OpenShell-Research/projects/openshell-middleware-init -``` - -Both forms make `openshell-middleware-init` available outside the source tree -without running `uv sync` in this project. - -Contributors working on the CLI should use the locked project environment: - -```sh -uv sync --locked -uv run openshell-middleware-init --help -``` - -## Quick start - -Generate and run a Python starter with the installed command: - -```sh -openshell-middleware-init audit-headers \ - --language python \ - --openshell-version v0.0.86 \ - --output /tmp/audit-headers - -cd /tmp/audit-headers -uv run pytest -uv run audit-headers -``` - -Or generate and run a Rust starter: - -```sh -openshell-middleware-init audit-headers \ - --language rust \ - --openshell-version v0.0.86 \ - --output /tmp/audit-headers-rust - -cd /tmp/audit-headers-rust -cargo test --locked -cargo run --locked -- 127.0.0.1:50051 -``` - -The output path must not already exist. Use a pinned OpenShell tag for -reproducible projects; `--openshell-version latest` is available for -experimentation. - -Run `openshell-middleware-init --help` for all options. Python package names -default to a normalized project name and can be changed with `--package-name`. - -## What you get - -Each generated project contains: - -- a pass-through implementation of `Describe`, `ValidateConfig`, and - `EvaluateHttpRequest`; -- the exact `supervisor_middleware.proto` from the selected OpenShell release; -- generated Python gRPC bindings or Rust Tonic build configuration; -- tests and lint/type-check configuration; -- `uv.lock` or `Cargo.lock`; and -- `middleware-dev-manifest.json` with the release, source URL, and protocol - SHA-256. - -Start by implementing policy behavior in the generated `validate_config` and -`evaluate_http_request` functions. The generated README explains how to run the -service and register it with OpenShell. - -## Safety and failure behavior - -Generation is non-destructive. The initializer validates a hidden sibling -staging directory, then publishes it atomically. It refuses an existing output, -including a symlink, and uses a per-output reservation to prevent concurrent -writers. A normal failure removes the initializer's own staging and reservation -without publishing a partial project. - -If the process is killed, it may leave -`..openshell-middleware-init.lock` and a hidden staging directory. Before -removing either one: - -1. Read `metadata.json` in the reservation. -2. On the recorded host, confirm that the recorded PID is no longer the same - initializer process and that the final output does not exist. -3. Inspect and remove only the recorded staging directory. -4. Remove `owner` and `metadata.json`, then remove the empty reservation with - `rmdir`. Stop if it contains anything unexpected. - -## Develop the CLI - -Run the complete local gate from this directory: - -```sh -uv run ruff format --check . -uv run ruff check . -uv run ty check -uv run pytest -uv build -``` - -Unit tests are hermetic: they use local protocol fixtures and do not contact -GitHub or invoke uv or Cargo for generated projects. diff --git a/projects/openshell-middleware-init/src/openshell_middleware_init/__init__.py b/projects/openshell-middleware-init/src/openshell_middleware_init/__init__.py deleted file mode 100644 index 03f94f25..00000000 --- a/projects/openshell-middleware-init/src/openshell_middleware_init/__init__.py +++ /dev/null @@ -1,3 +0,0 @@ -"""Generate version-matched OpenShell middleware projects.""" - -__version__ = "0.1.0" diff --git a/projects/openshell-middleware-init/tests/test_cli.py b/projects/openshell-middleware-init/tests/test_cli.py deleted file mode 100644 index 40d876a9..00000000 --- a/projects/openshell-middleware-init/tests/test_cli.py +++ /dev/null @@ -1,74 +0,0 @@ -from pathlib import Path - -from typer.testing import CliRunner - -from openshell_middleware_init import cli -from openshell_middleware_init.generator import InitializationError, InitializationResult - -runner = CliRunner() - - -def test_help_describes_required_choices() -> None: - result = runner.invoke(cli.app, ["--help"]) - - assert result.exit_code == 0 - assert "--language" in result.stdout - assert "--openshell-version" in result.stdout - - -def test_cli_reports_success(monkeypatch, tmp_path: Path) -> None: - destination = tmp_path / "audit" - - def fake_initialize_project(**options): - assert options["name"] == "audit" - assert options["language"] == "python" - assert options["destination"] == destination - return InitializationResult( - destination=destination, - language="python", - openshell_version="v0.0.86", - run_command="uv run audit", - ) - - monkeypatch.setattr(cli, "initialize_project", fake_initialize_project) - - result = runner.invoke( - cli.app, - [ - "audit", - "--language", - "python", - "--openshell-version", - "v0.0.86", - "--output", - str(destination), - ], - ) - - assert result.exit_code == 0 - assert "Created python middleware project" in result.stdout - assert "OpenShell contract: v0.0.86" in result.stdout - - -def test_cli_reports_initialization_error(monkeypatch, tmp_path: Path) -> None: - def fake_initialize_project(**options): - del options - raise InitializationError("output exists") - - monkeypatch.setattr(cli, "initialize_project", fake_initialize_project) - - result = runner.invoke( - cli.app, - [ - "audit", - "--language", - "rust", - "--openshell-version", - "v0.0.86", - "--output", - str(tmp_path / "audit"), - ], - ) - - assert result.exit_code == 1 - assert "error: output exists" in result.stderr