Enable droplet2Photons reconstruction for mfx102101026 (LUTE template fix + validation)

[lg345]

Context

Branch: mfx102101026

The mfx102101026 Fe XES analysis needs photon-counted droplet reconstruction (the droplet2Photons path), matching the working mfx101609126 RIXS workflow. XSpect already has everything on the analysis side — the missing piece is the LUTE smalldata config that produces the sparse photon arrays.

What is already done (this branch)

• XSpect native step exists and is correct. XSpect/analysis/droplet.py registers droplet_reconstruction, which reads <det>/droplet_droplet2phot_sparse_{row,col,data,tile} and scatters photons onto a dense image. ROI is applied on the sparse arrays before densifying (_scatter_roi — full 704×768 panel never allocated), and it is batch-aware. Downstream rotate_detector / patch_pixels / reduce_detector_spatial then run on the cropped image.
• Recon YAML added: experiments/mfx102101026/mfx102101026_droplet_recon_xes.yaml
  (reconstruction → union_shots → reduce_detector_shots → rotate_detector → reduce_detector_spatial). Parses cleanly.
• Notebook Section 7 (mfx102101026_droplet_visualization.ipynb) previews the reconstruction via the native step, and gracefully skips if the sparse keys are absent.
• LUTE YAML block added to both .../mfx102101026/results/lute_output/mfx_lute.yaml and mfx_lute_cmpars.yaml:

  getDroplet2Photons:
    epix100_0:
      droplet: {threshold: 5, thresholdLow: 5, thresADU: 60, useRms: true}
      aduspphot: 95     # single Fe Ka photon total ADU (measured from droplet spectrum)
      nData: 100000
      cputime: true
    epix100_1:
      droplet: {threshold: 5, thresholdLow: 5, thresADU: 60, useRms: true}
      aduspphot: 110    # single Fe Kb photon total ADU
      nData: 100000
      cputime: true

What still needs to be implemented in LUTE (BLOCKER)

The current smalldata run does not produce the droplet_droplet2phot_sparse_* keys because of a template bug. Our file only has droplet_sparse_* (plain droplet-finding: centroid + total ADU), not the photon-counted droplet_droplet2phot_sparse_*.

1. smd2_prod_config_template.py hardcodes droplet2photon parameters (must fix)

config/templates/smd2_prod_config_template.py renders get_droplet2photon() with hardcoded values that ignore the YAML:

d2p_dict["d2p"] = {
    "aduspphot": 20,          # <-- HARDCODED, ignores params.aduspphot (we set 95/110)
    "cputime": {{ params["cputime"] }},
}
d2p_dict["nData"] = None       # <-- HARDCODED, ignores params.nData (we set 100000)
d2p_dict["get_photon_img"] = False

Required changes:

• aduspphot must come from {{ params["aduspphot"] }} (per-detector; ours are 95 and 110 ADU, not 20).
• nData must come from {{ params["nData"] }} so the sparse arrays are wide enough (100000).
• Confirm get_photon_img: False is correct for our use (we want the sparse arrays, not the dense per-shot image written to HDF5 — reconstruction happens in XSpect). Keep False.

The LUTE Pydantic model (lute/io/models/smd.py Droplet2PhotonParams, lines ~220-250) already accepts aduspphot, nData, cputime, and the nested droplet params — so only the Jinja template needs updating to actually pass them through. Note the model default aduspphot: int = Field(162, ...) is also wrong for an ADU-calibrated ePix100 at Fe Kα/Kβ; our measured single-photon totals are ~95/110 ADU.

2. Verify the smalldata_tools droplet2photon writes fixed-length sparse keys

Confirm the producer emits epix100_0/droplet_droplet2phot_sparse_{row,col,data,tile} (fixed-length, zero-padded, shape (nshots, nData)) as the XSpect _reconstruct_fixed path expects. epix100_1 may come out variable-length (var_droplet_droplet2phot_sparse/* + _len) — XSpect handles both, but confirm which layout each detector produces.

After the LUTE fix — reprocessing + validation checklist

1. Re-run SubmitSMD for the beam-on runs (at least 9, 10) with the fixed template.
2. Confirm droplet_droplet2phot_sparse_data exists and data values are photon counts (1, 2, 3…), not ADU.
3. Run mfx102101026_droplet_recon_xes.yaml and Section 7 of the droplet notebook.
4. Refine these TODOs (currently estimated):
   • aduspphot (95/110 ADU) — refine against the single-photon peak in the droplet ADU spectrum.
   • rotate_detector angle in the recon YAML — currently 0.0; set from the rotation diagnostic.
   • Kα/Kβ band ROIs in the reduce step — verify against the reconstructed mean image.
5. Cross-check the photon-counted spectrum against the ADU ROI_area spectrum (should agree, with better S/N).

Notes on the plain droplet step (already working)

The existing droplet_sparse_* output (centroid + total ADU + npix) is retained as a diagnostic/calibration product (hit-rate QA, droplet-size studies, per-droplet ADU spectrum used to calibrate aduspphot). Sections 0–6 of the droplet notebook use it. It is not a substitute for photon reconstruction — each multi-pixel droplet is collapsed to one point.

Files touched (this branch)

• experiments/mfx102101026/mfx102101026_droplet_recon_xes.yaml (new)
• experiments/mfx102101026/mfx102101026_droplet_visualization.ipynb (Section 7 → native step)
• LUTE configs (outside repo): .../mfx102101026/results/lute_output/mfx_lute.yaml, mfx_lute_cmpars.yaml

[lg345]

The mfx102101026 branch (droplet reconstruction workflow) was merged into master and archived. Reconstruction pipeline configs are in master under experiments/mfx102101026/.