Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ project intends to use Semantic Versioning once its public API reaches stability

## [Unreleased]

## [0.3.0] - 2026-07-21

### Added

- `testenix tune` and its `testenix benchmark` alias for fresh-process, counterbalanced,
Expand Down
30 changes: 16 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,10 +119,11 @@ rename, Testenix warns without pretending the already published output was rolle
paths must be new, inside the project, and disjoint from both source and generated suites. There is
no `--force` option, and old tests are never deleted or renamed.

The converter stops on semantics it cannot preserve. The v0.2 pytest subset covers module
functions, one static parametrization, simple local/adjacent-conftest fixtures, statically declared
autouse fixtures, bare `@pytest.mark.asyncio` coroutine tests through fresh function-scoped loop
wrappers, and simple pytest classes. Native `tmp_path` and a dependency-free `monkeypatch`
The converter stops on semantics it cannot preserve. The pytest subset introduced in v0.2 covers
module functions, one static parametrization, simple local/adjacent-conftest fixtures, statically
declared autouse fixtures, bare `@pytest.mark.asyncio` coroutine tests through fresh
function-scoped loop wrappers, and simple pytest classes. Native `tmp_path` and a dependency-free
`monkeypatch`
implementation cover the common `setattr` and `setenv` forms with automatic per-test rollback,
including calls through statically provable module-local helpers. Complex class lifecycle, async
fixtures, unmarked async tests, configured async loop scopes or debug mode, custom
Expand Down Expand Up @@ -337,7 +338,7 @@ test.

## Where Testenix is deliberately different

The `testenix run` engine is not a native drop-in reimplementation of pytest. Its v0.2 value is a
The `testenix run` engine is not a native drop-in reimplementation of pytest. Its v0.3 value is a
smaller, coherent native stack: async tests and async fixtures need no plugin, parallel execution
and duration-aware scheduling need no xdist, every retry remains visible, and a worker crash cannot
silently erase tests that completed before it. The native runtime has no third-party dependencies.
Expand Down Expand Up @@ -368,19 +369,20 @@ can copy its own text or the complete project reference for an LLM.
## Benchmarks

The checked-in `3.15x` result is a **historical Testenix 0.1.0 synthetic baseline**, not a Testenix
0.2.1 measurement. On one M4 Pro/CPython 3.11 machine, native `testenix run` completed 100,000
0.3.0 measurement. On one M4 Pro/CPython 3.11 machine, native `testenix run` completed 100,000
generated no-op tests across 16 modules in a median 8.04 seconds, compared with 25.33 seconds for
pytest and 21.30 seconds for pytest-xdist. The run used four workers, `--no-history`, pytest-xdist
3.8's default `load` scheduler, one warm-up, and five counterbalanced measured rounds. It does not
describe `testenix pytest`, which executes through pytest, or promise the same ratio for a real
project.

No Testenix 0.2.1 scaling result is checked in yet. The provenance-gated matrix harness covers
No Testenix 0.3.0 scaling result is checked in yet. The provenance-gated matrix harness covers
100/500/1,000/3,000 tests, balanced/dominant/single-module layouts, 1/2/4/auto workers, and both
default history and `--no-history`, plus explicit safe-module sharding. Until that clean five-round
matrix is published, the project does not claim a 0.2.1 speedup. A separate manifest harness can measure a private real project
without copying its code, stdout, environment values, or absolute paths into the result. It marks a
result publishable only when a current successful migration report proves exact per-test
matrix is published, the project does not claim a 0.3.0 speedup. A separate manifest harness can
measure a private real project without copying its code, stdout, environment values, or absolute
paths into the result. It marks a result publishable only when a current successful migration
report proves exact per-test
inventory/outcome parity, complete source/generated Python-file inventories and hashes, and binds
canonical pytest and Testenix commands to the migrated source/output roots. Without that report,
the result is explicitly diagnostic-only. Publishable source roots must be directories so support
Expand All @@ -404,7 +406,7 @@ pytest-xdist, unittest, or real project suites.
```bash
# current-version synthetic matrix; refuses dirty/version-mismatched publication input
uv run --no-editable python benchmarks/run_scaling_matrix.py \
--output benchmarks/scaling_matrix_0_2_1.json
--output benchmarks/scaling_matrix_0_3_0.json

# redaction-safe real-project comparison driven by an argument-array manifest
cp benchmarks/real_project_manifest.example.json /tmp/testenix-project-benchmark.json
Expand Down Expand Up @@ -442,7 +444,7 @@ See the [generated results and chart](https://polishdataengineer.github.io/teste
JSON-safe representation when a value itself is not serializable.
- Synchronous test and fixture bodies run outside Testenix's internal asyncio loop. APIs restricted
to Python's main thread, such as installing signal handlers, are not supported inside those
bodies in v0.2. Migrated pytest-asyncio wrappers are synchronous from Testenix's perspective and
bodies in v0.3. Migrated pytest-asyncio wrappers are synchronous from Testenix's perspective and
therefore share this restriction while creating a fresh event loop for each test or case.
- On every supported platform, an executable script that calls the programmatic
`run()`/`run_async()` API must use the standard `if __name__ == "__main__":` multiprocessing
Expand All @@ -458,11 +460,11 @@ See the [generated results and chart](https://polishdataengineer.github.io/teste
checks pass. Spread independent tests across modules or validate opt-in sharding, then run
`testenix tune` before making a project-specific speed claim.
- Test impact analysis, result caching, remote workers, and deep pytest-result aggregation are not
part of version 0.2.
part of version 0.3.

## Project status

Testenix 0.2.1 is pre-1.0 software. The distribution, import package, CLI, configuration namespace,
Testenix 0.3.0 is pre-1.0 software. The distribution, import package, CLI, configuration namespace,
and state directory consistently use `testenix`. The project is licensed under MIT and releases
are published to PyPI through Trusted Publishing.

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/real_project_manifest.example.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"schema_version": 1,
"name": "private-project",
"testenix_version": "0.2.1",
"testenix_version": "0.3.0",
"expected_tests": 118,
"expected_passed": 118,
"warmups": 1,
Expand Down
8 changes: 4 additions & 4 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Windows: testenix pytest =========> pytest.console_main -> collector/plugins/exe
```

The bridge is a CLI infrastructure adapter, not a native collection adapter. It does not emit
Testenix events or construct a `RunResult` in version 0.2.
Testenix events or construct a `RunResult` in version 0.3.

The migration adapter is separate from that handoff. It statically converts a deliberately small
pytest subset or generates SHA-pinned wrappers around the standard unittest protocol. Its
Expand Down Expand Up @@ -76,7 +76,7 @@ trusted manifest +-- roots/inventory/SHA-256 verify
- migration analyzers depend on serializable migration contracts, while shadow execution and
atomic publication remain application/infrastructure concerns.

## Version 0.2 scope
## Version 0.3 scope

- explicit `@test` and `@fixture` authoring API, plus conventional `test_*` discovery;
- sync functions, coroutines, generators, and async-generator fixture teardown;
Expand All @@ -99,7 +99,7 @@ trusted manifest +-- roots/inventory/SHA-256 verify
wrappers, plus simple pytest classes.

Remote workers, distributed storage, result caching, automatic quarantine, and a stable third-party
plugin SDK are deliberately outside version 0.2.
plugin SDK are deliberately outside version 0.3.

## Fixture scopes and process isolation

Expand All @@ -114,7 +114,7 @@ globals, and import-time lifecycle behavior. Function-scoped fixtures can be rec
Because arbitrary dynamic calls and external effects cannot be proven safe, passing this policy is
a caller trust decision; ineligible modules keep normal affinity.

Scope therefore has the following concrete meaning in version 0.2:
Scope therefore has the following concrete meaning in version 0.3:

| Scope | Lifetime |
| --- | --- |
Expand Down
6 changes: 3 additions & 3 deletions docs/benchmarking.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ record throughput, mean, standard deviation, provenance, and raw samples. Pytest
and its cache provider are disabled, pytest-xdist 3.8 is loaded explicitly with its default `load`
distribution, and every tool runs from the generated suite directory so repository-level pytest
configuration does not affect the comparison. Those historical records do not measure Testenix
0.2.1.
0.3.0.

Schema-version 2 harness output additionally records the requested and resolved worker counts,
balanced/dominant/single-module test distributions, default-history versus `--no-history`, the
Expand Down Expand Up @@ -120,7 +120,7 @@ Generate the current-version dimension sweeps from a clean checkout with:

```bash
uv run --no-editable python benchmarks/run_scaling_matrix.py \
--output benchmarks/scaling_matrix_0_2_1.json
--output benchmarks/scaling_matrix_0_3_0.json
```

For an unpublished smoke test only, add `--quick --allow-dirty`. A publishable matrix must retain
Expand Down Expand Up @@ -158,7 +158,7 @@ replacing the approved marketing baseline: their timing variance is outside this

The checked-in baseline files are historical development evidence, not universal or current-version
performance claims. See `docs/performance-analysis.md` for the large-suite results, optimization
profile, memory notes, and native-code decision. A clean Testenix 0.2.1 scaling matrix, publishable
profile, memory notes, and native-code decision. A clean Testenix 0.3.0 scaling matrix, publishable
real-project suites, alternative pytest-xdist strategies, and cross-platform repetitions remain
required before publishing broad comparative claims.

Expand Down
8 changes: 4 additions & 4 deletions docs/benchmarks/results.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@ evidence for specific synthetic workloads, not a universal claim that Testenix i
than pytest. `Testenix` in these results means the native `testenix run` engine. The
`testenix pytest` compatibility bridge delegates to pytest and is not represented here.

## Testenix 0.2.1 scaling matrix
## Testenix 0.3.0 scaling matrix

No current-version matrix is checked in yet. The historical results below must therefore not be
described as Testenix 0.2.1 performance. The new provenance-gated harness covers
described as Testenix 0.3.0 performance. The new provenance-gated harness covers
100/500/1,000/3,000 tests, balanced/dominant/single-module layouts, 1/2/4/auto workers, and both
default history and `--no-history`, plus explicit safe-module sharding. Its default design uses
dimension sweeps; use
Expand All @@ -20,7 +20,7 @@ logical CPU count.

```console
$ uv run --no-editable python benchmarks/run_scaling_matrix.py \
--output benchmarks/scaling_matrix_0_2_1.json
--output benchmarks/scaling_matrix_0_3_0.json
```

The command refuses a dirty worktree or an installed Testenix version that differs from
Expand Down Expand Up @@ -62,7 +62,7 @@ sensitive argument indexes in `redact_arguments`.
The checked-in `3.15×` figure is a Testenix 0.1.0 result for 100,000 generated no-op
tests across 16 modules, four workers, disabled history (`--no-history`), and pytest-xdist's default
`load` strategy. It is retained as transparent historical evidence; it is not a measurement of
Testenix 0.2.1.
Testenix 0.3.0.

![Historical Testenix 0.1.0 throughput ratios](../_static/benchmark-speedup.svg)

Expand Down
4 changes: 2 additions & 2 deletions docs/guides/fixtures.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ issues instead of hanging the run.

## Built-in fixtures

Testenix 0.2 provides two dependency-free, test-scoped built-ins by name:
Testenix 0.3 provides two dependency-free, test-scoped built-ins by name:

```python
from pathlib import Path
Expand Down Expand Up @@ -110,7 +110,7 @@ def worker_resource() -> Resource:
return Resource()
```

| Scope | Lifetime in Testenix 0.2 |
| Scope | Lifetime in Testenix 0.3 |
| --- | --- |
| `test` | One instance for one concrete test attempt. |
| `module` | Shared by normal tests from the module inside one worker. |
Expand Down
2 changes: 1 addition & 1 deletion docs/guides/migration.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ with external effects.

## Pytest conversion contract

The v0.2 converter supports the subset below:
The converter subset introduced in v0.2 supports the behavior below:

- module-level pytest-default `test*` functions and normal Python `assert` statements;
- simple `Test*` classes with a fresh zero-argument instance per test method, including ordinary
Expand Down
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,12 +154,12 @@ The checked-in development baseline measured **Testenix 0.1.0**, not the current
`testenix run` completed 100,000 generated no-op tests across 16 modules on one Apple M4 Pro and
CPython 3.11 machine in a median 8.04 seconds, compared with 25.33 seconds for pytest and 21.30
seconds for pytest-xdist. It used four workers, `--no-history`, and pytest-xdist's default `load`
scheduler. These measurements do not apply to Testenix 0.2.1, a real project, the default-history
scheduler. These measurements do not apply to Testenix 0.3.0, a real project, the default-history
mode, or the delegated `testenix pytest` command.

<div class="benchmark-caveat">
This is historical synthetic evidence from one machine, not a promise that every project will be
3.15× faster. No clean Testenix 0.2.1 scaling matrix is checked in yet. The benchmark page publishes
3.15× faster. No clean Testenix 0.3.0 scaling matrix is checked in yet. The benchmark page publishes
the raw samples, environment, variance, methodology, current matrix status, and limitations.
</div>

Expand Down
Loading
Loading