Skip to content

fix(n_prov_destiny): make the processed-item substitution conserve N - #443

Merged
lbm364dl merged 1 commit into
alice/typologiesfrom
fix/conserve-processing-n
Jul 31, 2026
Merged

fix(n_prov_destiny): make the processed-item substitution conserve N#443
lbm364dl merged 1 commit into
alice/typologiesfrom
fix/conserve-processing-n

Conversation

@lbm364dl

Copy link
Copy Markdown
Collaborator

Summary

Targets alice/typologies so it lands inside #416. Three files, one concern: the processed-item substitution added in 96bcbe9 does not conserve nitrogen.

The commit message for 96bcbe9 says it "correctly conserve[s] production N". Measured against the real pins, it does not — and it fails in both directions, not only the zero-N one the PR already flags as an open question.

The measurement

Running the PR's own .prepare_prod_data() + .convert_fm_dm_n() against a no-processing baseline, total production N moves by:

year 1860 1930 1961 1980 2000 2020 2022
ΔN −1.10% +2.27% +3.77% +4.31% +3.69% −1.56% 0.00%

Worst single year +7.84%, net +1.875 Mt N (+1.56%) over 1860–2023.

Why

.expand_processed_items() applies every output row's cf to the same processed_fm. Summed per input item for Spain in 2000, those factors reach:

input item Σ cf n outputs
Maize and products 5.22 3
Wheat and products 5.11 4
Barley and products 4.90 3
Oilcrops, Other 2.38 2
Olives (including preserved) 0.24 1

So one tonne of maize sent to processing yielded 5.22 t of product. This is not a defect in the pin: R/supply_use.R:33-39 already documents these as the outputs of a virtual process ("a virtual process like 'Wheat and products processing', giving all its possible outputs"), i.e. not co-product shares of one physical process. Beer and starch are water-diluted, which is why the factors are large. Meanwhile grapes → wine, olives → oil and sugar beet → sugar destroy N, because those outputs are nearly N-free.

The fix

Nitrogen is an element: processing moves it between products but cannot create or destroy it. .processing_n_scaling() computes two factors per province-year and primary item:

  • output_scale caps the outputs at the N actually available in the input, scaling all of an input's outputs by the same factor so their relative mix is untouched.
  • removal_scale removes from the primary item only the N its named outputs actually account for.

Verified on the real pins — N removed from primaries now equals N credited to processed items to 2.1e-16 relative error, every year 1860–2023:

year 1860 1961 2000 2021
N removed (MgN) 1293.00179877 16183.23353365 48626.28109086 41638.54279214
N added (MgN) 1293.00179877 16183.23353365 48626.28109086 41638.54279214

A missing product N coefficient now drops that substitution and warns via cli, instead of silently pricing a real flow at zero. The special_items list shared with .convert_fm_dm_n() is extracted to .special_biomass_items() so the conservation pricing and the final FM→N conversion cannot drift apart.

What this deliberately does not decide

removal_scale is the conservative half. When outputs are N-poor the unaccounted N stays with the primary item rather than disappearing. Physically that residue is by-product (grape pomace, olive cake, beet pulp) or an agro-industry loss, so attributing it to the primary item's destinies is an approximation — it keeps the balance honest without choosing where the residue belongs. Routing it to explicit by-product items is a methodological decision for the model owners, tracked in #432.

Also not addressed here, by design: processing_coefs ends in 2021 while production runs to 2023, so 2022–23 still get no processing at all (#433).

Test plan

  • 5 new unit tests: N-poor outputs conserve exactly; N-rich outputs are capped and never create N; the output mix is preserved under scaling; a missing coefficient drops the substitution and warns; an exactly balanced substitution is a no-op.
  • R CMD check --as-cran — 0 errors, 0 warnings, 1 pre-existing NOTE
  • devtools::test() — 4575 passing, 0 failures
  • lintr — 0
  • air format . — clean

Related

Found while reviewing #416. Separate follow-ups, none of them blocking this PR:

The processing substitution added in 96bcbe9 was not N-conserving. The
conversion factors in the `processing_coefs` pin are per-output factors of
a *virtual* process (documented in R/supply_use.R:33-39), not co-product
shares of one physical process, so they do not sum to a mass yield: for
Spain in 2000 they sum to 5.22 for maize, 5.11 for wheat and 4.90 for
barley (beer and starch are water-diluted). Applying every factor to the
same input mass over-produced, while items whose outputs are nearly N-free
(wine, olive oil, sugar) lost N outright.

Measured with the pipeline's own .prepare_prod_data() + .convert_fm_dm_n()
against a no-processing baseline, total production N moved by -2.75% to
+7.84% depending on the year: -1.10% (1860), +3.77% (1961), +4.31% (1980),
+3.69% (2000), -1.56% (2020), for a net +1.875 Mt N (+1.56%) over
1860-2023.

Nitrogen is an element, so processing can move it between products but
cannot create or destroy it. .processing_n_scaling() now computes, per
province-year and primary item, the two factors that close the balance:

- output_scale caps the outputs at the N available in the input, scaling
  all of an input's outputs equally so their relative mix is untouched.
- removal_scale removes from the primary item only the N that the named
  outputs actually account for.

Verified on the real pins: N removed from the primary now equals N
credited to the processed items to 2.1e-16 relative error, for every year
1860-2023.

The conservative half is removal_scale. When outputs are N-poor, the
unaccounted N stays with the primary item instead of vanishing. Physically
that residue is by-product (grape pomace, olive cake, beet pulp) or an
agro-industry loss, so attributing it to the primary item's destinies is
an approximation; routing it to explicit by-product items needs a decision
from the model owners and is called out in the helper's documentation.

A missing product N coefficient now drops that substitution and warns via
cli, rather than silently pricing a flow at zero.

Also extracts the special_items list shared by .convert_fm_dm_n() and the
new N pricing into .special_biomass_items(), so the conservation check and
the final FM->N conversion cannot drift apart.

Not addressed here: processing_coefs ends in 2021 while production runs to
2023, so 2022-23 still get no processing at all. Tracked separately, as is
the duplication with build_supply_use() (#428).

Refs #428
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant