Flattening CLUBB's grid: allocations by any name, bounds by component, generics by arity

[chenyueqi]

Stacked on #25. What the flat plan, the gates and the emitter needed to take type(grid) :: gr and its neighbours through, found on the CLUBB extension's tier-1 units.

Flattening

• Allocations: allocate( gr%zm(ngrdcol,gr%nzm), gr%zt(...), … ) — one statement, many components, the object named by the setup routine's dummy rather than this.
• Bounds: an axis sized by a component of the same or another object (gr%nzm) is spelled by that component's flat name and the component is carried as an input; a dummy declared thvm(ngrdcol, gr%nzt) the same; an allocation sized by the allocating routine's dummy (1:nz) by the planned subprogram's dummy of that name; a bound from one is its upper bound alone.
• ngrdcol is declared once when the subprogram already takes it.
• Generics: a call through a generic is followed into the specifics its arity fits; the recursion guard is a stack, not a memo.
• Private module state (error_code % clubb_debug_level) is left to the module, under left_to_module.
• An intent(out) object's written components are read off the object the translation returned.

Read/write scope: a sibling's generics are procedures to it; a sibling's buffer-OUT actuals are read; a stub under control flow (if (stats%l_sample) then / call stats_update / end if) is a stub block.

Two silent translation defects, both fixed with executed tests

• A DO loop's index read after the loop: Fortran leaves it one step past the end on completion, for leaves the last value. lscale_width_vert_avg integrated one level short. A loop whose index a later statement reads before anything redefines it gets a for … else with the completion value; an EXIT (break) skips it, as Fortran keeps the exit value.
• ep = Rd / Rv over two real parameters was folded with integer division (the old "no real literal means integer arithmetic" rule) and came out zero; every unit reading ep1/ep2 computed NaN. The resolver records each constant's declared base type and the fold types names by it.

Bit parity: SUM accumulates in element order (_f_vsum beside _f_vdot); np.sum's pairwise rounding drifted 12 ULP in vertical_integral.

Gate: a subprogram the operator declared ungated is not sampled; a translated bare error stop (SystemExit) is a failed comparison, not the end of the run.

Tier-2 replay (recorded runs through dump-replay), found on CLUBB's advance_* solvers replayed on bomex recordings:

• The dump parser took only numbers: a logical T/F header (l_implemented) was no value and a zero-extent component (scalar tracers under sclr_dim = 0) vanished with its empty body. Both are values.
• The recorder wrote what the objects received; the plain OUT/INOUT dummies (wp2, wp3, mono_flux_limiter's low_lev_effect) are outputs too.
• Two specifics of one arity (tridiag_solve with its optional rcond and the multiple-rhs one without) were picked by count; the ranks of the bare-name actuals decide.
• A component name on an OUT actual (pdf_params%chi_1) was counted a read of a variable of that name — the documented disagreement inherited from the pipeline, resolved the tidier way now that a translation showed it (test flipped, deliberately).
• A call to a procedure of a stubbed module that no statement stub answers (lapack_band_solvex) deferred its whole block, if ( method == lapack ) included, so the candidate raised on the LU path the run takes. It raises on its own line.
• max(2, edsclr_dim) sizing a local is a bound Python can spell; a comma in a bound is legal inside max/min alone.
• An optional OUT handed on (rcond = rcond) carries the caller's presence (want_rcond=want_rcond, not True) — xm_wpxp_solve took the LAPACK diagnostic path on every call — and the read/write gate counts that hand-on as a read of the optional on both sides.
• A module-state object handed whole to a companion function (sponge_layer_damping's profile into sponge_damp_xm) is followed under the callee's dummy, so the plan carries the components the callee reads; a derived-type state variable is no longer listed as "left to the module".
• A local parameter initialized by a scalar times an array constructor is refused at emission #26: 100._core_rknd * (/ ... /) as a local parameter — each constructor rendered where it stands, the rest through the token pass, elementwise (saturation; pdf_closure's hybrid-PDF path).
• A subroutine with no OUT argument returns zero values, not one (finalize_tau_sponge_damp_api); a declared-ungated subprogram is not compared on a recording either, and the reason is reported.

Merged with main (2026-09-04, after PRs #29 and #30 landed there): main's comparison loop is kept as is — a translated ERROR STOP or an IndexError is a draw redrawn — with this branch's recorded-sample path, zero-value return and declared-ungated skip beside it; constant folding carries main's parameter aliases together with this branch's intrinsic calls; the appended tests of both sides are kept. Merged bottom-up through #22 and #25 so each PR stays mergeable.

Held to (self-check 2026-09-04 before tier 3): full suite and mypy green; conformance/ --plugin-set recast-clubb green; the corpus re-run on this branch regresses no unit (numfor's genrand 7→3 disagreeing blocks, slsqp 12→8, csplines/fitpack/slsqp_core as before); tools/emit_diff.py against the translator goes from 34 differing subprograms at this branch's base to 48, every added row an intended class — _f_vsum for SUM, want_x=want_x where an optional OUT is handed on (wvsat's qsat*: the translator's True is the defect fixed here), declared-initializer locals, and define_swept_areas where the translator drops the reshape around a constructor and this branch keeps it (#26). CLUBB tier 0 stays 12/14; tier 1 is 7/11 bit-exact with gr flattened (advance_helper_module: 25 subprograms, 7,770 points); tier 2 (summaries/tier2.json in clubb-jax): 10 of 12 units bit-exact on 20–40 recorded calls each from bomex and cgils_s6 under the hybrid PDF — advance_wp2_wp3, advance_xm_wpxp, advance_xp2_xpyp, advance_xp3, advance_windm_edsclrm, clip_explicit, mono_flux_limiter, mixing_length (incl. diagnose_lscale_from_tau), pdf_closure, new_hybrid_pdf_main — plus the sampled plain routines of clip_explicit, mixing_length and pdf_closure. sponge_layer_damping compares nothing (initializer's below-layer levels are undefined on both sides, declared; finalizer has no output; the sponge_damp_* functions take gr and have no flat plan); new_pdf_main has no recording (the new PDF needs LES input fields).

Tier 3 (2026-09-04): advance_clubb_core — the whole CLUBB time step, 135 dummies flattened to 476 arguments over 7 derived dummies and the 14 sponge state objects — is bit-exact on 40 recorded steps (20 bomex, 20 cgils_s6 under the hybrid PDF): 453,700 compared values, no rewrites to notarize. Two more fixes got it there: a probe spans blank and comment lines inside a continued call (cpp leaves them where an #ifdef stood), and a generic function reference is ranked by the specific it dispatches to (a nested sat_mixrat_liq_api( ..., thlm2T_in_K_api( ... ) ) deferred a block, and a deferred block took the subprogram out of the gate).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Cne4hrGgYqhTMzVgzJtXYd

[chenyueqi]

Review of #27 against two questions: generic or CLUBB-specific, and conflicts with existing rules on main (including #29/#30 merged today).

Genericity: every CLUBB mention in the engine (23 in transform, ~35 in fortran/oracle/verify) is a docstring or comment citing the motivating case; nothing branches on a CLUBB identifier. The rules (loop completion value, typed real/real fold, ordered SUM, max/min in bounds, optional presence forwarding, elementwise constructor arithmetic, allocation-by-any-name, generic-by-arity) are Fortran-semantics rules. Fine. (The KINDS_64 domain-name list this branch carries comes from #22; commented there.)

Existing rules: bitexact.py is main plus four additive hunks; #30's redraw loop, moved-extents rule and NaN rule are intact line for line. The typed fold lives in the tree evaluator, #30's packed constants and aliases in the token route; they never both decide a /. dump_replay keeps 9b2f515's int32-for-all-integer rule. No test deleted to hide a behavior change; the flipped rwset test (call refill(b%q, n) no longer reads q) is correct since q is a component, not a scope variable, and a real variable named q was never keyed on.

Three things to fix before merge, inline: the numba backend now emits an undefined _f_vsum; the stubbed-module call went from a recorded refusal to a runtime-only raise absent from deferred; left_to_module turns a refusal into an assumption that the recorded reference run can violate. Two smaller ones: record.py silently narrowed which dummies are recorded as outputs, and _allocation_bounds keys on component name alone across the whole file.

One note for the ledger rather than this PR: the for...else completion-value rule is line-order based (a redefinition in one if-arm hides a read in the sibling arm; a read before the loop inside an enclosing loop's next iteration; an index that is a dummy read by the caller) and falls back to Python's last-value semantics in those cases without refusing. That is main's behavior too, and it is documented nowhere (no hit for "completion value" / "one past" in docs). Worth a line in disclosure-ledger.md.

Also: test_loud_refusals.py swaps max(n, 2) for mod(n, 3) + 2 in the fixture because max in a bound is now accepted; the refusal test still exercises an unresolvable bound, but no test now asserts that a max bound is accepted.

[chenyueqi]

The numba emitter reuses numpy.expressions.Expressions (numba/emitter.py:47, NumbaExpressions(Expressions)), so it picks up this mapping too, but numba/runtime.py defines and exports _f_vdot only, no _f_vsum. Any SUM in a numba candidate is now a NameError where main emitted np.sum (njit-supported). No numba test covers SUM. Either add _f_vsum to the numba runtime or let NumbaExpressions keep np.sum.

[chenyueqi]

Keeping the surrounding branch is the right call, but this line is returned as a rendered statement, so the block's status is mechanical and it never reaches candidate.deferred (translate.py collects only agent_queue); bitexact then judges the subprogram as if fully ported. On main the block was NoRule -> deferred -> counted in the evidence. git grep 'not ported' finds only this raise. Suggest recording it, e.g. a notes entry like the JAX side's host_calls in tree.py, so a reader of the verdict sees that a procedure was left unported rather than discovering it as a runtime failure on another input.

[chenyueqi]

The comment says "both sides run with the module's own default", but the recorded reference run may have set this variable through a public setter before the step: CLUBB's set_clubb_debug_level sets exactly error_code%clubb_debug_level. The plan records it under left_to_module, so it is not silent, but this turns what was a refusal into an assumption about the reference run. At minimum the flat plan could check the module for a public procedure that writes the variable and refuse when one exists.

[chenyueqi]

Before this change every non-derived array dummy with intent other than IN was recorded as an output, including undeclared intent (which extract spells as UNKNOWN). Now only OUT/INOUT are, with scalars added. An undeclared-intent array the routine writes is no longer captured, so the gate has no reference for it. The direction is fail-closed, but the narrowing is not in the description and no test covers the UNKNOWN case.

[chenyueqi]

_allocation_bounds now scans every allocate( in the file and keys by component name alone, so two types in one file with a same-named component get the first one's bounds. The previous this%-only scan bounded that collision. Keying on the object's dummy name (which the setup routine gives) as well would keep it bounded.

[chenyueqi]

The places where this branch turns a refusal on main into a silent fallback are recorded apart from the review, in #32 (rows 7-10), so each is ruled on rather than merged by omission. A row there closes when the rule refuses again, records what it assumed in the evidence, or takes the assumption from the extension's configuration.

[chenyueqi]

Second sweep, this pattern only: rows 15-21 of #32 are this branch's (the early-return flag in a loop body, the constructor token pass without integer division, the character component and the attribute-declared public variable both left to the module, the static-test and host-call assumptions in the JAX port, the stub-module names recorded as procedures).