diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f2ad36..1d103dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,8 @@ project intends to use Semantic Versioning once its public API reaches stability ## [Unreleased] +## [0.3.0] - 2026-07-21 + ### Added - `testenix tune` and its `testenix benchmark` alias for fresh-process, counterbalanced, diff --git a/README.md b/README.md index 33e4786..5e04134 100644 --- a/README.md +++ b/README.md @@ -119,10 +119,11 @@ rename, Testenix warns without pretending the already published output was rolle paths must be new, inside the project, and disjoint from both source and generated suites. There is no `--force` option, and old tests are never deleted or renamed. -The converter stops on semantics it cannot preserve. The v0.2 pytest subset covers module -functions, one static parametrization, simple local/adjacent-conftest fixtures, statically declared -autouse fixtures, bare `@pytest.mark.asyncio` coroutine tests through fresh function-scoped loop -wrappers, and simple pytest classes. Native `tmp_path` and a dependency-free `monkeypatch` +The converter stops on semantics it cannot preserve. The pytest subset introduced in v0.2 covers +module functions, one static parametrization, simple local/adjacent-conftest fixtures, statically +declared autouse fixtures, bare `@pytest.mark.asyncio` coroutine tests through fresh +function-scoped loop wrappers, and simple pytest classes. Native `tmp_path` and a dependency-free +`monkeypatch` implementation cover the common `setattr` and `setenv` forms with automatic per-test rollback, including calls through statically provable module-local helpers. Complex class lifecycle, async fixtures, unmarked async tests, configured async loop scopes or debug mode, custom @@ -337,7 +338,7 @@ test. ## Where Testenix is deliberately different -The `testenix run` engine is not a native drop-in reimplementation of pytest. Its v0.2 value is a +The `testenix run` engine is not a native drop-in reimplementation of pytest. Its v0.3 value is a smaller, coherent native stack: async tests and async fixtures need no plugin, parallel execution and duration-aware scheduling need no xdist, every retry remains visible, and a worker crash cannot silently erase tests that completed before it. The native runtime has no third-party dependencies. @@ -368,19 +369,20 @@ can copy its own text or the complete project reference for an LLM. ## Benchmarks The checked-in `3.15x` result is a **historical Testenix 0.1.0 synthetic baseline**, not a Testenix -0.2.1 measurement. On one M4 Pro/CPython 3.11 machine, native `testenix run` completed 100,000 +0.3.0 measurement. On one M4 Pro/CPython 3.11 machine, native `testenix run` completed 100,000 generated no-op tests across 16 modules in a median 8.04 seconds, compared with 25.33 seconds for pytest and 21.30 seconds for pytest-xdist. The run used four workers, `--no-history`, pytest-xdist 3.8's default `load` scheduler, one warm-up, and five counterbalanced measured rounds. It does not describe `testenix pytest`, which executes through pytest, or promise the same ratio for a real project. -No Testenix 0.2.1 scaling result is checked in yet. The provenance-gated matrix harness covers +No Testenix 0.3.0 scaling result is checked in yet. The provenance-gated matrix harness covers 100/500/1,000/3,000 tests, balanced/dominant/single-module layouts, 1/2/4/auto workers, and both default history and `--no-history`, plus explicit safe-module sharding. Until that clean five-round -matrix is published, the project does not claim a 0.2.1 speedup. A separate manifest harness can measure a private real project -without copying its code, stdout, environment values, or absolute paths into the result. It marks a -result publishable only when a current successful migration report proves exact per-test +matrix is published, the project does not claim a 0.3.0 speedup. A separate manifest harness can +measure a private real project without copying its code, stdout, environment values, or absolute +paths into the result. It marks a result publishable only when a current successful migration +report proves exact per-test inventory/outcome parity, complete source/generated Python-file inventories and hashes, and binds canonical pytest and Testenix commands to the migrated source/output roots. Without that report, the result is explicitly diagnostic-only. Publishable source roots must be directories so support @@ -404,7 +406,7 @@ pytest-xdist, unittest, or real project suites. ```bash # current-version synthetic matrix; refuses dirty/version-mismatched publication input uv run --no-editable python benchmarks/run_scaling_matrix.py \ - --output benchmarks/scaling_matrix_0_2_1.json + --output benchmarks/scaling_matrix_0_3_0.json # redaction-safe real-project comparison driven by an argument-array manifest cp benchmarks/real_project_manifest.example.json /tmp/testenix-project-benchmark.json @@ -442,7 +444,7 @@ See the [generated results and chart](https://polishdataengineer.github.io/teste JSON-safe representation when a value itself is not serializable. - Synchronous test and fixture bodies run outside Testenix's internal asyncio loop. APIs restricted to Python's main thread, such as installing signal handlers, are not supported inside those - bodies in v0.2. Migrated pytest-asyncio wrappers are synchronous from Testenix's perspective and + bodies in v0.3. Migrated pytest-asyncio wrappers are synchronous from Testenix's perspective and therefore share this restriction while creating a fresh event loop for each test or case. - On every supported platform, an executable script that calls the programmatic `run()`/`run_async()` API must use the standard `if __name__ == "__main__":` multiprocessing @@ -458,11 +460,11 @@ See the [generated results and chart](https://polishdataengineer.github.io/teste checks pass. Spread independent tests across modules or validate opt-in sharding, then run `testenix tune` before making a project-specific speed claim. - Test impact analysis, result caching, remote workers, and deep pytest-result aggregation are not - part of version 0.2. + part of version 0.3. ## Project status -Testenix 0.2.1 is pre-1.0 software. The distribution, import package, CLI, configuration namespace, +Testenix 0.3.0 is pre-1.0 software. The distribution, import package, CLI, configuration namespace, and state directory consistently use `testenix`. The project is licensed under MIT and releases are published to PyPI through Trusted Publishing. diff --git a/benchmarks/real_project_manifest.example.json b/benchmarks/real_project_manifest.example.json index f540c8d..aeedf97 100644 --- a/benchmarks/real_project_manifest.example.json +++ b/benchmarks/real_project_manifest.example.json @@ -1,7 +1,7 @@ { "schema_version": 1, "name": "private-project", - "testenix_version": "0.2.1", + "testenix_version": "0.3.0", "expected_tests": 118, "expected_passed": 118, "warmups": 1, diff --git a/docs/architecture.md b/docs/architecture.md index f9d811d..41361ee 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -18,7 +18,7 @@ Windows: testenix pytest =========> pytest.console_main -> collector/plugins/exe ``` The bridge is a CLI infrastructure adapter, not a native collection adapter. It does not emit -Testenix events or construct a `RunResult` in version 0.2. +Testenix events or construct a `RunResult` in version 0.3. The migration adapter is separate from that handoff. It statically converts a deliberately small pytest subset or generates SHA-pinned wrappers around the standard unittest protocol. Its @@ -76,7 +76,7 @@ trusted manifest +-- roots/inventory/SHA-256 verify - migration analyzers depend on serializable migration contracts, while shadow execution and atomic publication remain application/infrastructure concerns. -## Version 0.2 scope +## Version 0.3 scope - explicit `@test` and `@fixture` authoring API, plus conventional `test_*` discovery; - sync functions, coroutines, generators, and async-generator fixture teardown; @@ -99,7 +99,7 @@ trusted manifest +-- roots/inventory/SHA-256 verify wrappers, plus simple pytest classes. Remote workers, distributed storage, result caching, automatic quarantine, and a stable third-party -plugin SDK are deliberately outside version 0.2. +plugin SDK are deliberately outside version 0.3. ## Fixture scopes and process isolation @@ -114,7 +114,7 @@ globals, and import-time lifecycle behavior. Function-scoped fixtures can be rec Because arbitrary dynamic calls and external effects cannot be proven safe, passing this policy is a caller trust decision; ineligible modules keep normal affinity. -Scope therefore has the following concrete meaning in version 0.2: +Scope therefore has the following concrete meaning in version 0.3: | Scope | Lifetime | | --- | --- | diff --git a/docs/benchmarking.md b/docs/benchmarking.md index 434be3e..4734880 100644 --- a/docs/benchmarking.md +++ b/docs/benchmarking.md @@ -24,7 +24,7 @@ record throughput, mean, standard deviation, provenance, and raw samples. Pytest and its cache provider are disabled, pytest-xdist 3.8 is loaded explicitly with its default `load` distribution, and every tool runs from the generated suite directory so repository-level pytest configuration does not affect the comparison. Those historical records do not measure Testenix -0.2.1. +0.3.0. Schema-version 2 harness output additionally records the requested and resolved worker counts, balanced/dominant/single-module test distributions, default-history versus `--no-history`, the @@ -120,7 +120,7 @@ Generate the current-version dimension sweeps from a clean checkout with: ```bash uv run --no-editable python benchmarks/run_scaling_matrix.py \ - --output benchmarks/scaling_matrix_0_2_1.json + --output benchmarks/scaling_matrix_0_3_0.json ``` For an unpublished smoke test only, add `--quick --allow-dirty`. A publishable matrix must retain @@ -158,7 +158,7 @@ replacing the approved marketing baseline: their timing variance is outside this The checked-in baseline files are historical development evidence, not universal or current-version performance claims. See `docs/performance-analysis.md` for the large-suite results, optimization -profile, memory notes, and native-code decision. A clean Testenix 0.2.1 scaling matrix, publishable +profile, memory notes, and native-code decision. A clean Testenix 0.3.0 scaling matrix, publishable real-project suites, alternative pytest-xdist strategies, and cross-platform repetitions remain required before publishing broad comparative claims. diff --git a/docs/benchmarks/results.md b/docs/benchmarks/results.md index bb0eeed..76e9057 100644 --- a/docs/benchmarks/results.md +++ b/docs/benchmarks/results.md @@ -5,10 +5,10 @@ evidence for specific synthetic workloads, not a universal claim that Testenix i than pytest. `Testenix` in these results means the native `testenix run` engine. The `testenix pytest` compatibility bridge delegates to pytest and is not represented here. -## Testenix 0.2.1 scaling matrix +## Testenix 0.3.0 scaling matrix No current-version matrix is checked in yet. The historical results below must therefore not be -described as Testenix 0.2.1 performance. The new provenance-gated harness covers +described as Testenix 0.3.0 performance. The new provenance-gated harness covers 100/500/1,000/3,000 tests, balanced/dominant/single-module layouts, 1/2/4/auto workers, and both default history and `--no-history`, plus explicit safe-module sharding. Its default design uses dimension sweeps; use @@ -20,7 +20,7 @@ logical CPU count. ```console $ uv run --no-editable python benchmarks/run_scaling_matrix.py \ - --output benchmarks/scaling_matrix_0_2_1.json + --output benchmarks/scaling_matrix_0_3_0.json ``` The command refuses a dirty worktree or an installed Testenix version that differs from @@ -62,7 +62,7 @@ sensitive argument indexes in `redact_arguments`. The checked-in `3.15×` figure is a Testenix 0.1.0 result for 100,000 generated no-op tests across 16 modules, four workers, disabled history (`--no-history`), and pytest-xdist's default `load` strategy. It is retained as transparent historical evidence; it is not a measurement of -Testenix 0.2.1. +Testenix 0.3.0. ![Historical Testenix 0.1.0 throughput ratios](../_static/benchmark-speedup.svg) diff --git a/docs/guides/fixtures.md b/docs/guides/fixtures.md index 144cbe1..01f4086 100644 --- a/docs/guides/fixtures.md +++ b/docs/guides/fixtures.md @@ -64,7 +64,7 @@ issues instead of hanging the run. ## Built-in fixtures -Testenix 0.2 provides two dependency-free, test-scoped built-ins by name: +Testenix 0.3 provides two dependency-free, test-scoped built-ins by name: ```python from pathlib import Path @@ -110,7 +110,7 @@ def worker_resource() -> Resource: return Resource() ``` -| Scope | Lifetime in Testenix 0.2 | +| Scope | Lifetime in Testenix 0.3 | | --- | --- | | `test` | One instance for one concrete test attempt. | | `module` | Shared by normal tests from the module inside one worker. | diff --git a/docs/guides/migration.md b/docs/guides/migration.md index 91c61a9..016c186 100644 --- a/docs/guides/migration.md +++ b/docs/guides/migration.md @@ -91,7 +91,7 @@ with external effects. ## Pytest conversion contract -The v0.2 converter supports the subset below: +The converter subset introduced in v0.2 supports the behavior below: - module-level pytest-default `test*` functions and normal Python `assert` statements; - simple `Test*` classes with a fresh zero-argument instance per test method, including ordinary diff --git a/docs/index.md b/docs/index.md index 3fd239e..c0cbffc 100644 --- a/docs/index.md +++ b/docs/index.md @@ -154,12 +154,12 @@ The checked-in development baseline measured **Testenix 0.1.0**, not the current `testenix run` completed 100,000 generated no-op tests across 16 modules on one Apple M4 Pro and CPython 3.11 machine in a median 8.04 seconds, compared with 25.33 seconds for pytest and 21.30 seconds for pytest-xdist. It used four workers, `--no-history`, and pytest-xdist's default `load` -scheduler. These measurements do not apply to Testenix 0.2.1, a real project, the default-history +scheduler. These measurements do not apply to Testenix 0.3.0, a real project, the default-history mode, or the delegated `testenix pytest` command.
This is historical synthetic evidence from one machine, not a promise that every project will be -3.15× faster. No clean Testenix 0.2.1 scaling matrix is checked in yet. The benchmark page publishes +3.15× faster. No clean Testenix 0.3.0 scaling matrix is checked in yet. The benchmark page publishes the raw samples, environment, variance, methodology, current matrix status, and limitations.
diff --git a/docs/llms-full.txt b/docs/llms-full.txt index 4513014..07d06eb 100644 --- a/docs/llms-full.txt +++ b/docs/llms-full.txt @@ -168,12 +168,12 @@ The checked-in development baseline measured **Testenix 0.1.0**, not the current `testenix run` completed 100,000 generated no-op tests across 16 modules on one Apple M4 Pro and CPython 3.11 machine in a median 8.04 seconds, compared with 25.33 seconds for pytest and 21.30 seconds for pytest-xdist. It used four workers, `--no-history`, and pytest-xdist's default `load` -scheduler. These measurements do not apply to Testenix 0.2.1, a real project, the default-history +scheduler. These measurements do not apply to Testenix 0.3.0, a real project, the default-history mode, or the delegated `testenix pytest` command.
This is historical synthetic evidence from one machine, not a promise that every project will be -3.15× faster. No clean Testenix 0.2.1 scaling matrix is checked in yet. The benchmark page publishes +3.15× faster. No clean Testenix 0.3.0 scaling matrix is checked in yet. The benchmark page publishes the raw samples, environment, variance, methodology, current matrix status, and limitations.
@@ -651,7 +651,7 @@ with external effects. ## Pytest conversion contract -The v0.2 converter supports the subset below: +The converter subset introduced in v0.2 supports the behavior below: - module-level pytest-default `test*` functions and normal Python `assert` statements; - simple `Test*` classes with a fresh zero-argument instance per test method, including ordinary @@ -1051,7 +1051,7 @@ issues instead of hanging the run. ## Built-in fixtures -Testenix 0.2 provides two dependency-free, test-scoped built-ins by name: +Testenix 0.3 provides two dependency-free, test-scoped built-ins by name: ```python from pathlib import Path @@ -1097,7 +1097,7 @@ def worker_resource() -> Resource: return Resource() ``` -| Scope | Lifetime in Testenix 0.2 | +| Scope | Lifetime in Testenix 0.3 | | --- | --- | | `test` | One instance for one concrete test attempt. | | `module` | Shared by normal tests from the module inside one worker. | @@ -2052,10 +2052,10 @@ evidence for specific synthetic workloads, not a universal claim that Testenix i than pytest. `Testenix` in these results means the native `testenix run` engine. The `testenix pytest` compatibility bridge delegates to pytest and is not represented here. -## Testenix 0.2.1 scaling matrix +## Testenix 0.3.0 scaling matrix No current-version matrix is checked in yet. The historical results below must therefore not be -described as Testenix 0.2.1 performance. The new provenance-gated harness covers +described as Testenix 0.3.0 performance. The new provenance-gated harness covers 100/500/1,000/3,000 tests, balanced/dominant/single-module layouts, 1/2/4/auto workers, and both default history and `--no-history`, plus explicit safe-module sharding. Its default design uses dimension sweeps; use @@ -2067,7 +2067,7 @@ logical CPU count. ```console $ uv run --no-editable python benchmarks/run_scaling_matrix.py \ - --output benchmarks/scaling_matrix_0_2_1.json + --output benchmarks/scaling_matrix_0_3_0.json ``` The command refuses a dirty worktree or an installed Testenix version that differs from @@ -2109,7 +2109,7 @@ sensitive argument indexes in `redact_arguments`. The checked-in `3.15×` figure is a Testenix 0.1.0 result for 100,000 generated no-op tests across 16 modules, four workers, disabled history (`--no-history`), and pytest-xdist's default `load` strategy. It is retained as transparent historical evidence; it is not a measurement of -Testenix 0.2.1. +Testenix 0.3.0. ![Historical Testenix 0.1.0 throughput ratios](https://polishdataengineer.github.io/testenix/_static/benchmark-speedup.svg) @@ -2375,7 +2375,7 @@ record throughput, mean, standard deviation, provenance, and raw samples. Pytest and its cache provider are disabled, pytest-xdist 3.8 is loaded explicitly with its default `load` distribution, and every tool runs from the generated suite directory so repository-level pytest configuration does not affect the comparison. Those historical records do not measure Testenix -0.2.1. +0.3.0. Schema-version 2 harness output additionally records the requested and resolved worker counts, balanced/dominant/single-module test distributions, default-history versus `--no-history`, the @@ -2471,7 +2471,7 @@ Generate the current-version dimension sweeps from a clean checkout with: ```bash uv run --no-editable python benchmarks/run_scaling_matrix.py \ - --output benchmarks/scaling_matrix_0_2_1.json + --output benchmarks/scaling_matrix_0_3_0.json ``` For an unpublished smoke test only, add `--quick --allow-dirty`. A publishable matrix must retain @@ -2509,7 +2509,7 @@ replacing the approved marketing baseline: their timing variance is outside this The checked-in baseline files are historical development evidence, not universal or current-version performance claims. See `docs/performance-analysis.md` for the large-suite results, optimization -profile, memory notes, and native-code decision. A clean Testenix 0.2.1 scaling matrix, publishable +profile, memory notes, and native-code decision. A clean Testenix 0.3.0 scaling matrix, publishable real-project suites, alternative pytest-xdist strategies, and cross-platform repetitions remain required before publishing broad comparative claims. @@ -2531,7 +2531,7 @@ Source: docs/performance-analysis.md ## Executive summary The checked-in headline numbers are a **historical Testenix 0.1.0 synthetic baseline**, not current -Testenix 0.2.1 results. In the largest recorded comparison, 100,000 generated no-op tests were spread +Testenix 0.3.0 results. In the largest recorded comparison, 100,000 generated no-op tests were spread evenly across 16 modules and run with four workers and `--no-history`. Testenix completed the suite in a median 8.038 seconds, pytest in 25.333 seconds, and pytest-xdist 3.8's default `load` scheduler in 21.300 seconds. Every command had to report the expected test count or the harness rejected the @@ -2541,8 +2541,8 @@ samples ranged from 7.912 to 8.096 seconds. This is evidence for the tested workload and machine, not a universal claim about every Python project. Import-heavy suites, fixture-heavy suites, slow tests, failure output, default history, alternative pytest-xdist schedulers, different operating systems, and real repositories still need -independent measurements. No clean Testenix 0.2.1 scaling matrix is checked in yet, so `3.15×` must -not be presented as a 0.2.1 speedup. +independent measurements. No clean Testenix 0.3.0 scaling matrix is checked in yet, so `3.15×` must +not be presented as a 0.3.0 speedup. These results do not apply to `testenix pytest`. The compatibility command delegates to pytest and has pytest execution performance plus launcher and adapter overhead, which has not yet been @@ -2593,7 +2593,7 @@ approximately 520 MiB on the same generated suite. These older macOS `time` figu the current baseline JSON and are process maxima, not aggregate memory across every xdist/Testenix child process. The console renderer changed substantially after these captures, and the historical harness did not record output byte counts. Current schema-version 2 runs do record stdout/stderr -sizes, but a clean 0.2.1 matrix is still pending. +sizes, but a clean 0.3.0 matrix is still pending. ### Migrated-suite measurements @@ -2743,7 +2743,7 @@ Relevant upstream constraints are documented in the ## Next measurement gates -- publish the clean Testenix 0.2.1 dimension-sweep matrix for 100/500/1,000/3,000 tests, +- publish the clean Testenix 0.3.0 dimension-sweep matrix for 100/500/1,000/3,000 tests, balanced/dominant/single-module layouts, 1/2/4/adaptive-auto workers, and both history modes; - compare pytest-xdist `load`, `loadfile`, `loadscope`, and `worksteal` where each strategy is valid; - collection, execution, IPC-byte, process-start, CPU, and aggregate-memory telemetry; @@ -2760,7 +2760,7 @@ Relevant upstream constraints are documented in the No universal “always faster than pytest” statement should be published until the current-version, real-project, and cross-platform gates pass. The supported claim today is historical and narrower: Testenix 0.1.0 was materially faster in the recorded native large passing-suite scenarios while -retaining supervised isolation and complete results. Testenix 0.2.1 has no checked-in speedup claim +retaining supervised isolation and complete results. Testenix 0.3.0 has no checked-in speedup claim yet. --- @@ -2790,7 +2790,7 @@ Windows: testenix pytest =========> pytest.console_main -> collector/plugins/exe ``` The bridge is a CLI infrastructure adapter, not a native collection adapter. It does not emit -Testenix events or construct a `RunResult` in version 0.2. +Testenix events or construct a `RunResult` in version 0.3. The migration adapter is separate from that handoff. It statically converts a deliberately small pytest subset or generates SHA-pinned wrappers around the standard unittest protocol. Its @@ -2848,7 +2848,7 @@ trusted manifest +-- roots/inventory/SHA-256 verify - migration analyzers depend on serializable migration contracts, while shadow execution and atomic publication remain application/infrastructure concerns. -## Version 0.2 scope +## Version 0.3 scope - explicit `@test` and `@fixture` authoring API, plus conventional `test_*` discovery; - sync functions, coroutines, generators, and async-generator fixture teardown; @@ -2871,7 +2871,7 @@ trusted manifest +-- roots/inventory/SHA-256 verify wrappers, plus simple pytest classes. Remote workers, distributed storage, result caching, automatic quarantine, and a stable third-party -plugin SDK are deliberately outside version 0.2. +plugin SDK are deliberately outside version 0.3. ## Fixture scopes and process isolation @@ -2886,7 +2886,7 @@ globals, and import-time lifecycle behavior. Function-scoped fixtures can be rec Because arbitrary dynamic calls and external effects cannot be proven safe, passing this policy is a caller trust decision; ineligible modules keep normal affinity. -Scope therefore has the following concrete meaning in version 0.2: +Scope therefore has the following concrete meaning in version 0.3: | Scope | Lifetime | | --- | --- | @@ -3014,6 +3014,8 @@ project intends to use Semantic Versioning once its public API reaches stability ## [Unreleased] +## [0.3.0] - 2026-07-21 + ### Added - `testenix tune` and its `testenix benchmark` alias for fresh-process, counterbalanced, diff --git a/docs/performance-analysis.md b/docs/performance-analysis.md index 8207dea..479322e 100644 --- a/docs/performance-analysis.md +++ b/docs/performance-analysis.md @@ -3,7 +3,7 @@ ## Executive summary The checked-in headline numbers are a **historical Testenix 0.1.0 synthetic baseline**, not current -Testenix 0.2.1 results. In the largest recorded comparison, 100,000 generated no-op tests were spread +Testenix 0.3.0 results. In the largest recorded comparison, 100,000 generated no-op tests were spread evenly across 16 modules and run with four workers and `--no-history`. Testenix completed the suite in a median 8.038 seconds, pytest in 25.333 seconds, and pytest-xdist 3.8's default `load` scheduler in 21.300 seconds. Every command had to report the expected test count or the harness rejected the @@ -13,8 +13,8 @@ samples ranged from 7.912 to 8.096 seconds. This is evidence for the tested workload and machine, not a universal claim about every Python project. Import-heavy suites, fixture-heavy suites, slow tests, failure output, default history, alternative pytest-xdist schedulers, different operating systems, and real repositories still need -independent measurements. No clean Testenix 0.2.1 scaling matrix is checked in yet, so `3.15×` must -not be presented as a 0.2.1 speedup. +independent measurements. No clean Testenix 0.3.0 scaling matrix is checked in yet, so `3.15×` must +not be presented as a 0.3.0 speedup. These results do not apply to `testenix pytest`. The compatibility command delegates to pytest and has pytest execution performance plus launcher and adapter overhead, which has not yet been @@ -65,7 +65,7 @@ approximately 520 MiB on the same generated suite. These older macOS `time` figu the current baseline JSON and are process maxima, not aggregate memory across every xdist/Testenix child process. The console renderer changed substantially after these captures, and the historical harness did not record output byte counts. Current schema-version 2 runs do record stdout/stderr -sizes, but a clean 0.2.1 matrix is still pending. +sizes, but a clean 0.3.0 matrix is still pending. ### Migrated-suite measurements @@ -215,7 +215,7 @@ Relevant upstream constraints are documented in the ## Next measurement gates -- publish the clean Testenix 0.2.1 dimension-sweep matrix for 100/500/1,000/3,000 tests, +- publish the clean Testenix 0.3.0 dimension-sweep matrix for 100/500/1,000/3,000 tests, balanced/dominant/single-module layouts, 1/2/4/adaptive-auto workers, and both history modes; - compare pytest-xdist `load`, `loadfile`, `loadscope`, and `worksteal` where each strategy is valid; - collection, execution, IPC-byte, process-start, CPU, and aggregate-memory telemetry; @@ -232,5 +232,5 @@ Relevant upstream constraints are documented in the No universal “always faster than pytest” statement should be published until the current-version, real-project, and cross-platform gates pass. The supported claim today is historical and narrower: Testenix 0.1.0 was materially faster in the recorded native large passing-suite scenarios while -retaining supervised isolation and complete results. Testenix 0.2.1 has no checked-in speedup claim +retaining supervised isolation and complete results. Testenix 0.3.0 has no checked-in speedup claim yet. diff --git a/llms-full.txt b/llms-full.txt index 4513014..07d06eb 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -168,12 +168,12 @@ The checked-in development baseline measured **Testenix 0.1.0**, not the current `testenix run` completed 100,000 generated no-op tests across 16 modules on one Apple M4 Pro and CPython 3.11 machine in a median 8.04 seconds, compared with 25.33 seconds for pytest and 21.30 seconds for pytest-xdist. It used four workers, `--no-history`, and pytest-xdist's default `load` -scheduler. These measurements do not apply to Testenix 0.2.1, a real project, the default-history +scheduler. These measurements do not apply to Testenix 0.3.0, a real project, the default-history mode, or the delegated `testenix pytest` command.
This is historical synthetic evidence from one machine, not a promise that every project will be -3.15× faster. No clean Testenix 0.2.1 scaling matrix is checked in yet. The benchmark page publishes +3.15× faster. No clean Testenix 0.3.0 scaling matrix is checked in yet. The benchmark page publishes the raw samples, environment, variance, methodology, current matrix status, and limitations.
@@ -651,7 +651,7 @@ with external effects. ## Pytest conversion contract -The v0.2 converter supports the subset below: +The converter subset introduced in v0.2 supports the behavior below: - module-level pytest-default `test*` functions and normal Python `assert` statements; - simple `Test*` classes with a fresh zero-argument instance per test method, including ordinary @@ -1051,7 +1051,7 @@ issues instead of hanging the run. ## Built-in fixtures -Testenix 0.2 provides two dependency-free, test-scoped built-ins by name: +Testenix 0.3 provides two dependency-free, test-scoped built-ins by name: ```python from pathlib import Path @@ -1097,7 +1097,7 @@ def worker_resource() -> Resource: return Resource() ``` -| Scope | Lifetime in Testenix 0.2 | +| Scope | Lifetime in Testenix 0.3 | | --- | --- | | `test` | One instance for one concrete test attempt. | | `module` | Shared by normal tests from the module inside one worker. | @@ -2052,10 +2052,10 @@ evidence for specific synthetic workloads, not a universal claim that Testenix i than pytest. `Testenix` in these results means the native `testenix run` engine. The `testenix pytest` compatibility bridge delegates to pytest and is not represented here. -## Testenix 0.2.1 scaling matrix +## Testenix 0.3.0 scaling matrix No current-version matrix is checked in yet. The historical results below must therefore not be -described as Testenix 0.2.1 performance. The new provenance-gated harness covers +described as Testenix 0.3.0 performance. The new provenance-gated harness covers 100/500/1,000/3,000 tests, balanced/dominant/single-module layouts, 1/2/4/auto workers, and both default history and `--no-history`, plus explicit safe-module sharding. Its default design uses dimension sweeps; use @@ -2067,7 +2067,7 @@ logical CPU count. ```console $ uv run --no-editable python benchmarks/run_scaling_matrix.py \ - --output benchmarks/scaling_matrix_0_2_1.json + --output benchmarks/scaling_matrix_0_3_0.json ``` The command refuses a dirty worktree or an installed Testenix version that differs from @@ -2109,7 +2109,7 @@ sensitive argument indexes in `redact_arguments`. The checked-in `3.15×` figure is a Testenix 0.1.0 result for 100,000 generated no-op tests across 16 modules, four workers, disabled history (`--no-history`), and pytest-xdist's default `load` strategy. It is retained as transparent historical evidence; it is not a measurement of -Testenix 0.2.1. +Testenix 0.3.0. ![Historical Testenix 0.1.0 throughput ratios](https://polishdataengineer.github.io/testenix/_static/benchmark-speedup.svg) @@ -2375,7 +2375,7 @@ record throughput, mean, standard deviation, provenance, and raw samples. Pytest and its cache provider are disabled, pytest-xdist 3.8 is loaded explicitly with its default `load` distribution, and every tool runs from the generated suite directory so repository-level pytest configuration does not affect the comparison. Those historical records do not measure Testenix -0.2.1. +0.3.0. Schema-version 2 harness output additionally records the requested and resolved worker counts, balanced/dominant/single-module test distributions, default-history versus `--no-history`, the @@ -2471,7 +2471,7 @@ Generate the current-version dimension sweeps from a clean checkout with: ```bash uv run --no-editable python benchmarks/run_scaling_matrix.py \ - --output benchmarks/scaling_matrix_0_2_1.json + --output benchmarks/scaling_matrix_0_3_0.json ``` For an unpublished smoke test only, add `--quick --allow-dirty`. A publishable matrix must retain @@ -2509,7 +2509,7 @@ replacing the approved marketing baseline: their timing variance is outside this The checked-in baseline files are historical development evidence, not universal or current-version performance claims. See `docs/performance-analysis.md` for the large-suite results, optimization -profile, memory notes, and native-code decision. A clean Testenix 0.2.1 scaling matrix, publishable +profile, memory notes, and native-code decision. A clean Testenix 0.3.0 scaling matrix, publishable real-project suites, alternative pytest-xdist strategies, and cross-platform repetitions remain required before publishing broad comparative claims. @@ -2531,7 +2531,7 @@ Source: docs/performance-analysis.md ## Executive summary The checked-in headline numbers are a **historical Testenix 0.1.0 synthetic baseline**, not current -Testenix 0.2.1 results. In the largest recorded comparison, 100,000 generated no-op tests were spread +Testenix 0.3.0 results. In the largest recorded comparison, 100,000 generated no-op tests were spread evenly across 16 modules and run with four workers and `--no-history`. Testenix completed the suite in a median 8.038 seconds, pytest in 25.333 seconds, and pytest-xdist 3.8's default `load` scheduler in 21.300 seconds. Every command had to report the expected test count or the harness rejected the @@ -2541,8 +2541,8 @@ samples ranged from 7.912 to 8.096 seconds. This is evidence for the tested workload and machine, not a universal claim about every Python project. Import-heavy suites, fixture-heavy suites, slow tests, failure output, default history, alternative pytest-xdist schedulers, different operating systems, and real repositories still need -independent measurements. No clean Testenix 0.2.1 scaling matrix is checked in yet, so `3.15×` must -not be presented as a 0.2.1 speedup. +independent measurements. No clean Testenix 0.3.0 scaling matrix is checked in yet, so `3.15×` must +not be presented as a 0.3.0 speedup. These results do not apply to `testenix pytest`. The compatibility command delegates to pytest and has pytest execution performance plus launcher and adapter overhead, which has not yet been @@ -2593,7 +2593,7 @@ approximately 520 MiB on the same generated suite. These older macOS `time` figu the current baseline JSON and are process maxima, not aggregate memory across every xdist/Testenix child process. The console renderer changed substantially after these captures, and the historical harness did not record output byte counts. Current schema-version 2 runs do record stdout/stderr -sizes, but a clean 0.2.1 matrix is still pending. +sizes, but a clean 0.3.0 matrix is still pending. ### Migrated-suite measurements @@ -2743,7 +2743,7 @@ Relevant upstream constraints are documented in the ## Next measurement gates -- publish the clean Testenix 0.2.1 dimension-sweep matrix for 100/500/1,000/3,000 tests, +- publish the clean Testenix 0.3.0 dimension-sweep matrix for 100/500/1,000/3,000 tests, balanced/dominant/single-module layouts, 1/2/4/adaptive-auto workers, and both history modes; - compare pytest-xdist `load`, `loadfile`, `loadscope`, and `worksteal` where each strategy is valid; - collection, execution, IPC-byte, process-start, CPU, and aggregate-memory telemetry; @@ -2760,7 +2760,7 @@ Relevant upstream constraints are documented in the No universal “always faster than pytest” statement should be published until the current-version, real-project, and cross-platform gates pass. The supported claim today is historical and narrower: Testenix 0.1.0 was materially faster in the recorded native large passing-suite scenarios while -retaining supervised isolation and complete results. Testenix 0.2.1 has no checked-in speedup claim +retaining supervised isolation and complete results. Testenix 0.3.0 has no checked-in speedup claim yet. --- @@ -2790,7 +2790,7 @@ Windows: testenix pytest =========> pytest.console_main -> collector/plugins/exe ``` The bridge is a CLI infrastructure adapter, not a native collection adapter. It does not emit -Testenix events or construct a `RunResult` in version 0.2. +Testenix events or construct a `RunResult` in version 0.3. The migration adapter is separate from that handoff. It statically converts a deliberately small pytest subset or generates SHA-pinned wrappers around the standard unittest protocol. Its @@ -2848,7 +2848,7 @@ trusted manifest +-- roots/inventory/SHA-256 verify - migration analyzers depend on serializable migration contracts, while shadow execution and atomic publication remain application/infrastructure concerns. -## Version 0.2 scope +## Version 0.3 scope - explicit `@test` and `@fixture` authoring API, plus conventional `test_*` discovery; - sync functions, coroutines, generators, and async-generator fixture teardown; @@ -2871,7 +2871,7 @@ trusted manifest +-- roots/inventory/SHA-256 verify wrappers, plus simple pytest classes. Remote workers, distributed storage, result caching, automatic quarantine, and a stable third-party -plugin SDK are deliberately outside version 0.2. +plugin SDK are deliberately outside version 0.3. ## Fixture scopes and process isolation @@ -2886,7 +2886,7 @@ globals, and import-time lifecycle behavior. Function-scoped fixtures can be rec Because arbitrary dynamic calls and external effects cannot be proven safe, passing this policy is a caller trust decision; ineligible modules keep normal affinity. -Scope therefore has the following concrete meaning in version 0.2: +Scope therefore has the following concrete meaning in version 0.3: | Scope | Lifetime | | --- | --- | @@ -3014,6 +3014,8 @@ project intends to use Semantic Versioning once its public API reaches stability ## [Unreleased] +## [0.3.0] - 2026-07-21 + ### Added - `testenix tune` and its `testenix benchmark` alias for fresh-process, counterbalanced, diff --git a/pyproject.toml b/pyproject.toml index 0413f8e..86771d9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "uv_build" [project] name = "testenix" -version = "0.2.1" +version = "0.3.0" description = "Fast, async-native, parallel-first testing for Python." readme = "README.md" requires-python = ">=3.11" diff --git a/scripts/generate_docs_assets.py b/scripts/generate_docs_assets.py index 644eee6..baaae39 100644 --- a/scripts/generate_docs_assets.py +++ b/scripts/generate_docs_assets.py @@ -34,7 +34,7 @@ ROOT / "benchmarks" / "migration_baseline_unittest_3000_delay_1ms.json", ) -SCALING_MATRIX = ROOT / "benchmarks" / "scaling_matrix_0_2_1.json" +SCALING_MATRIX = ROOT / "benchmarks" / "scaling_matrix_0_3_0.json" LLM_DOCUMENTS = ( ("Overview", Path("docs/index.md"), ""), @@ -463,7 +463,7 @@ def _render_current_matrix(matrix: dict[str, Any] | None, *, current_version: st ```console $ uv run --no-editable python benchmarks/run_scaling_matrix.py \\ - --output benchmarks/scaling_matrix_0_2_1.json + --output benchmarks/scaling_matrix_0_3_0.json ``` The command refuses a dirty worktree or an installed Testenix version that differs from diff --git a/src/testenix/__init__.py b/src/testenix/__init__.py index d6af89f..bca0bdc 100644 --- a/src/testenix/__init__.py +++ b/src/testenix/__init__.py @@ -58,4 +58,4 @@ try: __version__ = version("testenix") except PackageNotFoundError: # Source checkout without installed metadata. - __version__ = "0.2.1" + __version__ = "0.3.0" diff --git a/uv.lock b/uv.lock index ccd780d..632dbf7 100644 --- a/uv.lock +++ b/uv.lock @@ -1351,7 +1351,7 @@ wheels = [ [[package]] name = "testenix" -version = "0.2.1" +version = "0.3.0" source = { editable = "." } [package.optional-dependencies]