fix: make the quantity-zero tolerance scale-relative

[brentianpalmer]

Fixes #66.

Base: 877332b04aa00691e372d5bd6ef251104f45c204 (tag v0.1.0b21). Draft because the base is the released tag rather than current main; happy to rebase onto main (883785e6) on request — no commit between the two touches any of the changed files.

The defect

FillExecutor._update_position zeroes a post-fill quantity against an absolute 1e-12. float64 spacing grows with magnitude, so that threshold only works over one binade: one ULP is 9.09e-13 just below 8192 units and 1.82e-12 from 8192 up. At and above 2^13 an exact close that lands one ULP off zero is not recognised, the key survives in broker.positions, and n_positions (a raw len() of that dict, engine.py:275-277) reports a position holding ~1e-12 units.

Observed on a short cover of 10,927.322882 units: residual exactly 2**-39, count high for 20 sessions. Issue #66 has a standalone reproducer.

The rule

core/shared.py gains one primitive next to the existing CASH_TOLERANCE:

QTY_ZERO_FLOOR: float = 1e-12
QTY_ZERO_ULPS: int = 16

def quantity_zero_tolerance(*operands: float) -> float:
    # tolerance = max(floor, 16 * ulp(largest finite operand))

Callers pass the quantities that produced the residual, never the residual itself — it is ~0 by construction and carries no scale, so a tolerance derived from it could never fire.

The floor keeps the previous absolute behaviour everywhere spacing is finer than 1e-12, which is every magnitude below 8192. Since 16 * ulp(x) < x for every normal x, a call with a single operand is exactly equivalent to the old rule — the "is this book quantity zero?" sites keep their behaviour, and only the closure sites, which pass both operands, become scale-aware.

Rationale for 16 ULP, the absent cap, subnormal/non-finite handling, and residue-versus-genuine-position separation is in docs/concepts/quantity-zero-tolerance.md.

Site dispositions

All 19 quantity-zero comparisons in the package are migrated onto the primitive; the two constants they used (FillExecutor._qty_zero_epsilon, OrderBook._QTY_EPS) are removed.

file	comparisons	scale passed
execution/fill_executor.py	1	(old_qty, ctx.signed_qty) — the closure site
core/order_book.py	10	(old_qty, size) on the closure/precheck paths, single operand on the book-is-empty filters
core/execution_engine.py	7	(current_qty, qty_delta) on the shadow-queue closure paths, single operand elsewhere
accounting/gatekeeper.py	1	single operand — position normalisation

Left unchanged: OrderBook._MIN_ORDER_SIZE (1e-8). It is an economic policy — the smallest order the engine accepts — not a statement about floating-point residue. A regression test pins it.

One deliberate consistency change. The sites previously mixed < (fill_executor, _simulate_position_update, gatekeeper) and <= (the shadow-book and shadow-queue paths), so they disagreed exactly at the threshold. They are unified on <=, the form the majority already used, so one predicate — |q| <= tol means zero — now holds everywhere. This was caught by the cross-site consistency test, not assumed.

Tests

tests/test_quantity_zero_tolerance.py, 66 tests: the binade boundary, ULP-graded residues on both sides of the bound, exact closes across several powers of two, genuine small long and short positions in the same numerical magnitude range, partial reductions and covers, repeated fills ending flat and not, position-key/count consistency, cross-site agreement between the executor and both shadow paths, long/short symmetry, zero scale, subnormals, non-finite inputs, and the retained sites.

Expected boundaries are reconstructed from math.frexp rather than obtained from the production helper, so a test cannot agree with a wrong rule.

Verification

check	base 877332b0	this branch
pytest tests/	1489 passed, 13 skipped	1555 passed, 13 skipped
ruff check --no-fix src/ tests/	35 errors	35 errors
ruff format --check src/ tests/	clean	clean
ty check	12 diagnostics	12 diagnostics
uv build	—	wheel byte-identical across two clean builds

The suite delta is exactly the 66 new tests. Lint and typecheck counts are unchanged; the pre-existing findings are in files this branch does not touch.

Note for anyone running lint locally: this repo sets fix = true under [tool.ruff], so a bare ruff check rewrites files. The numbers above use --no-fix.

uv sync --dev could not be used on the development machine — it fails building llvmlite 0.46.0 from source (spawn() got an unexpected keyword argument 'dry_run'), reproduced identically at the base commit and unrelated to this change. Environments were built with venv + pip instead.