Wavelength lookup tables as streamed context inputs (ADR 0010)

[SimonHeybrock]

Feeds the wavelength lookup table computed by wavelength_lut_workflow back into the backend workflows that need it, replacing the tables loaded from files at import time. Those files describe a nominal chopper configuration, so running in any other configuration is silently wrong; the LUT workflow already computes the right table from the live chopper cascade, but publishes it as a result nothing consumes. This PR closes that loop for DREAM and LOKI.

The new concept is a workflow output republished as an input stream for other workflows. The NICOS derived-device mirror (ADR 0006) is publish-only and nothing consumes a mirrored topic, so this is the first cross-service feedback edge. ADR 0010 records the design; the commits implement it end to end.

What it does

The LUT workflow keeps its single chopper_cascade source and now publishes one table per component, each covering only that component's flight-path range, declared as context outputs on a dedicated topic. Consumers bind their component's table as gated context. On DREAM each table spans 2–11 distance rows at the default 0.1 m resolution, against roughly 730 for one table covering source to detector — the cascade is still computed once per trigger, and only the polygon rasterization runs per component.

The user-facing distance-range parameter is gone. It defaulted to a span covering no instrument correctly, so an operator starting the workflow with defaults could silently blank every detector: a lookup outside the table's range yields NaN, those events fall outside every histogram bin, and the component renders empty with no error anywhere.

Decisions worth pulling out

Ranges are derived, not declared. Each component's range runs essreduce's own DetectorLtotal and MonitorLtotal providers — the very ones the consumer runs at lookup time — so the range and the lookup agree by construction rather than by review. An earlier draft of the ADR had the Ltotal rule declared per component, justified by indirect geometry putting a table "tens of metres" from where it is queried. That magnitude was wrong: the secondary flight path is metres against primary paths of tens to hundreds of metres, so padding absorbs it and the declaration buys exactness the table's distance resolution does not reward. Over-padding costs recompute in the LUT job and nothing else; under-padding is silent.

Motion needs two numbers, not one. The geometry artifact stores a live f144-driven transform as an empty NXlog, so it withholds the component's resting position as well as its travel. Instruments declare both, as one MotionEnvelope per moving axis keyed by NeXus transform path; which components ride an axis stays derived, since a component is affected precisely when the axis appears in its depends_on chain. LOKI's carriage rests at 0 and travels 15 m, placing the rear bank at 28.5 .. 43.9 m.

A component that cannot be placed gets no table. beam_monitor_m4 rides its own undeclared axis, so the LUT workflow cannot place it and publishes nothing for it. Views and reductions bind only placeable components, so no job waits forever on a stream that is never published; an I(Q) aux selection that picks such a monitor is rejected at job creation with an error naming the monitor, rather than surfacing as a gate that never opens.

Coordinate mode stays a parameter on one workflow. Gating every job of such a spec on the table would make time-of-arrival — the mode you fall back to when everything else is broken — depend on an operator-started job that re-emits only on chopper change. ADR 0003 prescribed splitting the spec for exactly this case. That was implemented here and then reverted: the dashboard data plane is keyed by (workflow_id, source_name, output_name), so two specs give the two modes different output identities, a plot cannot follow a mode switch, the operator runs two jobs, and every per-instrument params override is duplicated. Instead a ContextBinding carries a predicate over the job's validated params, so a TOA job resolves an empty gating set. This supersedes ADR 0003's param-dependent-context non-goal, which was a YAGNI call made when the only over-gated stream was an always-on control-system PV.

A reduction's monitor tables are selected by aux-templated stream names. I(Q) needs one table per sciline Component — the detector plus the incident and transmission monitor roles — and which physical monitor fills a role is a per-job aux selection an import-time binding cannot name. The binding names the aux field instead: its stream name carries a placeholder (wavelength_lut/{incident_monitor}) that gate resolution renders against the job's rendered aux selections. One binding per role, a gate covering exactly the selected monitors' tables, and no dead keys. Route derivation expands a template over the aux field's declared choices, so the statically derived subscriptions stay a superset of any rendered gate; a selection resolving one stream to two conflicting keys (the same monitor for both roles) is rejected at job creation. This supersedes an earlier plan to bind the default monitors and raise when the selection differed — making the aux selector a lie in wavelength mode — and an intermediate shape that bound every candidate monitor to its own synthesized key and had the factory map the chosen ones onto the roles, leaving every unselected key a dead parameter and restating the candidate list in the factory.

Gate resolution moved

Params validation moves up into job creation and WorkflowFactory.create takes the validated model, so the order is validate → resolve gate → build, with one validation site rather than two that can drift. Resolution splits in two rather than taking an optional argument: declared_context_keys ignores predicates and leaves templates unrendered, serving the static callers (route derivation, the workflow visualizer) that have no job and specifically need the superset, while resolve_context_keys filters by predicate and renders templates for the job path, which therefore also receives the job's rendered aux selections. A predicate can only remove bindings and rendering only substitutes declared choices, so the statically derived Kafka subscriptions stay a conservative superset of any resolved gate.

The stream-name-to-key mapping is declared exactly once, on the binding: consuming factories contribute only the unconditional reassembly provider, and the resolved bindings are injected after creation. The conflict check this enables surfaced that Instrument.load_factories was not idempotent — repeated loading appended duplicate bindings whose synthesized chopper-setpoint keys are fresh objects per run — so it now no-ops on a loaded instrument.

Scope

DREAM and LOKI migrate; they are the only instruments offering wavelength mode today. The detector- and monitor-view factories lose their lookup-table filename argument outright rather than defaulting it, so a view has no file path left to fall back to, and LOKI's I(Q) reduction migrates with them. This also removes a pooch download from both instruments' startup path. What remains of LookupTableFilename is set explicitly by the unmigrated BIFROST and ESTIA reduction pipelines at their own call sites — an input, not a fallback.

Wavelength mode is also no longer offered where it cannot work. Every logical view on every instrument previously offered it in the UI and raised at job start, since they run on InstrumentDetectorSource, which carries no geometry. Those specs now use a time-of-arrival-only params model, and a test pins that any spec accepting wavelength can actually build a workflow in that mode.

Known gaps

beam_monitor_m4 has no wavelength mode until its axis and travel are declared. The identity coord and consumer clearing described in ADR 0010 are not implemented yet (#1248), so a new table does not currently reset accumulated statistics. The standing limitations the ADR records — nothing guarantees the LUT job is running, a backend restart loses the table, and the gate protects startup rather than steady state — all still apply.

Test plan

• Full fast suite
• Full suite including slow tests
• End-to-end test running the real chain on DREAM: the LUT job computes per-component tables, they are extracted, serialized to da00, ingested back through the Kafka route, and reduced with by a wavelength-mode monitor job, with no file anywhere
• Service-level test covering LOKI I(Q) gating on its detector's table and exactly the selected monitors' tables, and rejecting a selection whose table is never published
• Dashboard launches against DREAM
• Start the lookup-table workflow and confirm one output per component appears
• Confirm a wavelength-mode view sits in WAITING FOR CONTEXT until the LUT job emits, then reduces
• Confirm a TOA-mode view starts immediately with no LUT job running