[performance] Add bounded concurrent recognition for opt-in CPU throughput

[eric8810]

Why

Recognition calls and planned recognition batches currently execute serially within one engine, and the Node adapter intentionally uses one FIFO worker per engine. That is a safe baseline, but it leaves CPU throughput on the table for offline/bulk workloads.

This work must remain distinct from the interactive acceleration direction. CPU concurrency increases CPU pressure; it does not satisfy the Apple Silicon goal of freeing CPU for the user-facing application.

Evidence

A controlled probe against the current recognition session found:

• One shared ONNX Runtime recognition session supported concurrent Run calls in the tested configuration.
• Serial recognition with ORT intra-op=12 achieved about 104 calls/s on the sampled workload.
• 12 concurrent calls with ORT intra-op=1 achieved about 194 calls/s (~1.86× throughput).
• Multiple independent full engines can process pages faster, but memory grows quickly: 2 engines×6 threads ~10.9 s/~0.98 GiB; 3×4 ~9.95 s/~1.42 GiB; 4×3 ~8.54 s/~1.80 GiB; 6×2 ~8.87 s/~2.62 GiB on the 15-page OCR-only workload.
• Recognition output memory depends on width. At width 3200, one output is roughly 400×18,710 floats (~28.6 MiB); 12 such outputs alone approach ~343 MiB before other tensors/workspaces.

The promising unit of concurrency is therefore a bounded set of recognition runs on a shared session, not an unbounded pool of complete engines.

Goal

Add an opt-in CPU-throughput scheduler that can execute recognition batches concurrently while preserving deterministic result assembly, cancellation, backpressure, and strict memory admission.

Design constraints

• Keep detector scheduling and recognizer scheduling independently budgeted.
• Prefer one shared recognition session if supported and qualified; do not multiply model/session memory without evidence.
• Use a weighted semaphore/admission model based on input/output tensor size and reserved workspace, not only request count.
• Coordinate scheduler width with ORT intra-op threads to avoid oversubscription.
• Preserve input order and current reading-order/result semantics.
• Expose concurrency only through an explicit throughput profile; do not make it the interactive default.

Acceptance criteria

• TSAN/ASAN and stress tests qualify concurrent Run on every supported ORT version/platform, or the feature is disabled where not guaranteed.
• Scheduler concurrency and ORT thread budgets are explicit and bounded.
• Memory admission accounts for actual batch width/shape and prevents the wide-output amplification described above.
• Cancellation, close, queue-full behavior, and engine lifetime remain race-free and documented.
• Result order, text, confidence, and geometry match the serial baseline across the golden OCR corpus.
• Benchmarks report throughput, single-request latency, process CPU seconds, peak RSS, and foreground-load interference.
• The qualified configuration materially improves throughput over the current serial CPU-fast profile without exceeding the declared memory budget.
• Interactive/default behavior does not increase CPU usage as a side effect.

Non-goals

• Using page-level multi-engine fan-out as the default architecture.
• Claiming that CPU concurrency replaces CoreML/GPU offload.
• Unbounded task queues or worker counts derived only from logical CPU count.