fix(n_prov_destiny): make the processed-item substitution conserve N - #443
Merged
Conversation
The processing substitution added in 96bcbe9 was not N-conserving. The conversion factors in the `processing_coefs` pin are per-output factors of a *virtual* process (documented in R/supply_use.R:33-39), not co-product shares of one physical process, so they do not sum to a mass yield: for Spain in 2000 they sum to 5.22 for maize, 5.11 for wheat and 4.90 for barley (beer and starch are water-diluted). Applying every factor to the same input mass over-produced, while items whose outputs are nearly N-free (wine, olive oil, sugar) lost N outright. Measured with the pipeline's own .prepare_prod_data() + .convert_fm_dm_n() against a no-processing baseline, total production N moved by -2.75% to +7.84% depending on the year: -1.10% (1860), +3.77% (1961), +4.31% (1980), +3.69% (2000), -1.56% (2020), for a net +1.875 Mt N (+1.56%) over 1860-2023. Nitrogen is an element, so processing can move it between products but cannot create or destroy it. .processing_n_scaling() now computes, per province-year and primary item, the two factors that close the balance: - output_scale caps the outputs at the N available in the input, scaling all of an input's outputs equally so their relative mix is untouched. - removal_scale removes from the primary item only the N that the named outputs actually account for. Verified on the real pins: N removed from the primary now equals N credited to the processed items to 2.1e-16 relative error, for every year 1860-2023. The conservative half is removal_scale. When outputs are N-poor, the unaccounted N stays with the primary item instead of vanishing. Physically that residue is by-product (grape pomace, olive cake, beet pulp) or an agro-industry loss, so attributing it to the primary item's destinies is an approximation; routing it to explicit by-product items needs a decision from the model owners and is called out in the helper's documentation. A missing product N coefficient now drops that substitution and warns via cli, rather than silently pricing a flow at zero. Also extracts the special_items list shared by .convert_fm_dm_n() and the new N pricing into .special_biomass_items(), so the conservation check and the final FM->N conversion cannot drift apart. Not addressed here: processing_coefs ends in 2021 while production runs to 2023, so 2022-23 still get no processing at all. Tracked separately, as is the duplication with build_supply_use() (#428). Refs #428
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Targets
alice/typologiesso it lands inside #416. Three files, one concern: the processed-item substitution added in 96bcbe9 does not conserve nitrogen.The commit message for 96bcbe9 says it "correctly conserve[s] production N". Measured against the real pins, it does not — and it fails in both directions, not only the zero-N one the PR already flags as an open question.
The measurement
Running the PR's own
.prepare_prod_data()+.convert_fm_dm_n()against a no-processing baseline, total production N moves by:Worst single year +7.84%, net +1.875 Mt N (+1.56%) over 1860–2023.
Why
.expand_processed_items()applies every output row'scfto the sameprocessed_fm. Summed per input item for Spain in 2000, those factors reach:So one tonne of maize sent to processing yielded 5.22 t of product. This is not a defect in the pin:
R/supply_use.R:33-39already documents these as the outputs of a virtual process ("a virtual process like 'Wheat and products processing', giving all its possible outputs"), i.e. not co-product shares of one physical process. Beer and starch are water-diluted, which is why the factors are large. Meanwhile grapes → wine, olives → oil and sugar beet → sugar destroy N, because those outputs are nearly N-free.The fix
Nitrogen is an element: processing moves it between products but cannot create or destroy it.
.processing_n_scaling()computes two factors per province-year and primary item:output_scalecaps the outputs at the N actually available in the input, scaling all of an input's outputs by the same factor so their relative mix is untouched.removal_scaleremoves from the primary item only the N its named outputs actually account for.Verified on the real pins — N removed from primaries now equals N credited to processed items to 2.1e-16 relative error, every year 1860–2023:
A missing product N coefficient now drops that substitution and warns via
cli, instead of silently pricing a real flow at zero. Thespecial_itemslist shared with.convert_fm_dm_n()is extracted to.special_biomass_items()so the conservation pricing and the final FM→N conversion cannot drift apart.What this deliberately does not decide
removal_scaleis the conservative half. When outputs are N-poor the unaccounted N stays with the primary item rather than disappearing. Physically that residue is by-product (grape pomace, olive cake, beet pulp) or an agro-industry loss, so attributing it to the primary item's destinies is an approximation — it keeps the balance honest without choosing where the residue belongs. Routing it to explicit by-product items is a methodological decision for the model owners, tracked in #432.Also not addressed here, by design:
processing_coefsends in 2021 while production runs to 2023, so 2022–23 still get no processing at all (#433).Test plan
R CMD check --as-cran— 0 errors, 0 warnings, 1 pre-existing NOTEdevtools::test()— 4575 passing, 0 failureslintr— 0air format .— cleanRelated
Found while reviewing #416. Separate follow-ups, none of them blocking this PR:
build_supply_use()'s national process definition, so this class of bug cannot recurplot_typology_indicators_panel()fails unless patchwork is attached (same)processing_coefsends 2021, so 2022–23 get no processing_oldExcreta graft, which this branch's pin bump widens