Reconcile with main, add typology sensitivity/decomposition analysis, processed-item accounting change - #416
Reconcile with main, add typology sensitivity/decomposition analysis, processed-item accounting change#416AliceBeckmann wants to merge 12 commits into
Conversation
…ly conserve production N create_n_prov_destiny() added processed items (wine, oil, etc.) as new production rows without subtracting the corresponding share from the primary crop's own production, double-counting nitrogen for any heavily-processed item (wheat, barley, maize, oilseeds, sugar crops). Replaces the manually-fixed processed_prov_fixed pin with the package's own calibrated get_processing_coefs() pin. Adds .spain_processing_coefs(), .national_item_production(), .calculate_processing_shares(), .calculate_processed_amounts(), .expand_processed_items(), .backfill_processing_shares(), and .backfill_processing_cf() (the last two reproducing the original workflow's pre-1961 backfill, reusing 1961 shares/cf for 1860-1960). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Restores work-in-progress that was stashed ("WIP before checking
Josette bug") and never returned to: a full LMDI-based decomposition
analysis (decompose_cropland_surplus, decompose_semi_natural_surplus,
decompose_manure_losses, decompose_urban_losses,
decompose_territorial_n_losses, decompose_specialization_covariance,
decompose_crop_livestock_connectivity, decompose_destiny_mix, and
their corresponding plot_* functions), plus circularity_index.R,
typologies_sensitivity.R, typology_panel_plot.R, and
validate_national_trade.R.
Fixed a pre-existing double-escaped '%' in typologies_sensitivity.R's
roxygen docs (invalid Rd syntax under markdown = TRUE).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The processing substitution added in 96bcbe9 was not N-conserving. The conversion factors in the `processing_coefs` pin are per-output factors of a *virtual* process (documented in R/supply_use.R:33-39), not co-product shares of one physical process, so they do not sum to a mass yield: for Spain in 2000 they sum to 5.22 for maize, 5.11 for wheat and 4.90 for barley (beer and starch are water-diluted). Applying every factor to the same input mass over-produced, while items whose outputs are nearly N-free (wine, olive oil, sugar) lost N outright. Measured with the pipeline's own .prepare_prod_data() + .convert_fm_dm_n() against a no-processing baseline, total production N moved by -2.75% to +7.84% depending on the year: -1.10% (1860), +3.77% (1961), +4.31% (1980), +3.69% (2000), -1.56% (2020), for a net +1.875 Mt N (+1.56%) over 1860-2023. Nitrogen is an element, so processing can move it between products but cannot create or destroy it. .processing_n_scaling() now computes, per province-year and primary item, the two factors that close the balance: - output_scale caps the outputs at the N available in the input, scaling all of an input's outputs equally so their relative mix is untouched. - removal_scale removes from the primary item only the N that the named outputs actually account for. Verified on the real pins: N removed from the primary now equals N credited to the processed items to 2.1e-16 relative error, for every year 1860-2023. The conservative half is removal_scale. When outputs are N-poor, the unaccounted N stays with the primary item instead of vanishing. Physically that residue is by-product (grape pomace, olive cake, beet pulp) or an agro-industry loss, so attributing it to the primary item's destinies is an approximation; routing it to explicit by-product items needs a decision from the model owners and is called out in the helper's documentation. A missing product N coefficient now drops that substitution and warns via cli, rather than silently pricing a flow at zero. Also extracts the special_items list shared by .convert_fm_dm_n() and the new N pricing into .special_biomass_items(), so the conservation check and the final FM->N conversion cannot drift apart. Not addressed here: processing_coefs ends in 2021 while production runs to 2023, so 2022-23 still get no processing at all. Tracked separately, as is the duplication with build_supply_use() (#428). Refs #428
…ocessing-n-conservation
…dedupe the typology classifier
Five independent fixes found while reviewing the typology/decomposition work.
1. .finn_flow_matrix() wrote flows transposed. It computed a column-major
linear index by hand, (from - 1) * n + to, which lands on [to, from] --
the transpose of the documented "entry [i, j] is the flow from
compartment i to j". That is not cosmetic: .calculate_finn() then reads
colSums() as each compartment's outflow instead of its inflow,
contradicting its own comment ("external input plus its total inflow"),
and because exports and food leave the four compartments those two
quantities differ everywhere. Now indexed with an explicit row/column
matrix. On real data mean FCI moves 0.1105 -> 0.1102, but individual
province-years shift by up to 0.035 absolute (about half the typical
modern value) and the decadal pattern tilts (1860s -3.9%, 1920s +1.3%),
which feeds plot_finn_circularity()'s 1860/1920/1960/2020 comparison.
2. inst/extdata/Europe_FAO_completed.xlsx (16.5 MB) is no longer shipped.
It is a 19-sheet, 35-country workbook of which the package read two
sheets and only the Spanish rows -- 253 of 9113. The 6202 rows actually
consumed are now a 199 KB CSV, regenerated by
data-raw/europe_fao_spain_trade.R from a local copy via
WHEP_EUROPE_FAO_XLSX. Output verified identical. Installed package size
drops from 24.2 Mb to 8.6 Mb (extdata 18.7 -> 3.1 Mb). Also corrects the
documented coverage: the series starts in 1849, not 1860.
3. The typology decision rules existed twice, in create_typologies_spain()
and in .reclassify_typology() in typologies_sensitivity.R. They agreed,
but this branch had just moved two thresholds in one copy and mirrored
them by hand in the other. Now one .typology_thresholds() and one
.classify_typology_base(). Proven equivalent to both previous blocks over
2,239,488 grid cells covering every exact boundary value, 0 mismatches,
all 9 labels exercised.
4. The three scattered `<= 2021` cutoffs go through .grafs_last_year(),
which documents why the cap exists (the n_balance pin ends in 2021 and
processing_coefs covers 1961-2021) and that raising it needs the inputs
extended first. .warn_processing_coverage_gap() now warns when production
outruns the processing coefficients instead of silently emitting years
with no processing at all.
5. readxl moves from Imports to Suggests. It is used only in data-raw/ and
inst/scripts/, so removing the one R/ call in (2) exposed an "unused
Import" NOTE that main already latently carried. Matches how here,
jsonlite and zip are already declared.
Adds 92 tests across three previously untested modules (circularity_index,
typologies_sensitivity, validate_national_trade). Two pin assumptions found
during the review: that no flow carries destiny == "export" with
origin == "Outside", so .sum_provincial_net()'s case_when is safe; and that
a province-year drops to NA if any of the four compartments has no inflow.
One documents a gap in the typology rules -- the two specialised-cropping
branches are not complementary, so a crop-dominated province with high
synthetic share and low productivity matches neither.
Also replaces 13 commented-out @examples blocks in the plot functions with
runnable ones built from small inline fixtures, per CLAUDE.md.
R CMD check --as-cran: 0 errors, 0 warnings, 1 pre-existing NOTE.
devtools::test(): 4667 passing, 0 failures.
Refs #428
… bug it exposed
CLAUDE.md forbids \dontrun{} and requires every exported function to have an
example that runs during R CMD check. All 33 functions added by this branch
had a commented-out example (`#' # decompose_cropland_surplus()`) or none at
all, which passes the check while exercising nothing.
Wiring them up exposed a real bug. plot_typology_indicators_panel() composed
its four panels with bare `(p_ext + p_fci) / (p_pol + p_int)`. Those are
patchwork operators, and patchwork exports them so they *mask* ggplot2's --
they only take effect once library(patchwork) has been run. patchwork is a
Suggests and whep never attaches it, so the function failed for any ordinary
user with "Can't add `p_fci` to a <ggplot> object". The sibling helper
.n_losses_two_panel() already did this correctly with namespaced
patchwork::wrap_plots(), which is exactly why the plot_loss_decomp_periods_panel
example passed and this one did not. Now routed through .wrap_two_by_two(),
which uses wrap_plots() and applies the shared legend theme via ggplot2's own
`+`. Only bare `ggplot + ggplot` was affected; compositions on objects already
of class `patchwork` dispatch fine unattached, so the other panel functions
were never broken.
The examples themselves, by tier:
- 16 pure plot functions get inline tribble() fixtures using the real
category vocabularies (cropland/semi_natural/manure/urban, Size/
Intensification/Inefficiency, the destiny-mix groups), so the rendered doc
pages show a realistic call rather than a toy one. Where a rolling window
is involved the example passes `window` explicitly, since zoo::rollmean()
requires k <= n and the default of 10 would fail on a small fixture.
- 13 remote-data functions get the documented `example = FALSE` pattern, with
helpers in R/toy_examples.R sampled from real output: each function was run
once against the cached pins and representative rows kept.
- 4 that read pins with no override (plot_finn_circularity,
plot_typology_indicators_panel, plot_typology_periods_panel,
run_typology_sensitivity) instead gain optional arguments for the frames
they were fetching internally -- typologies, area_df/typo_df, panel_data,
baseline -- each defaulting to the previous reader, so existing calls are
unchanged. Preferred over an example=TRUE shim because those bodies
orchestrate several private panel builders, and duplicating that
orchestration inside toy_examples.R would recreate the same
duplication-drift problem just fixed in the typology classifier.
Also fixes a WARNING introduced along the way: the new @PARAM docs linked
[create_typo_ts_plot()] and [create_typologies_spain()], neither of which is
exported, so they became broken Rd cross-references. Since
r-lib/actions/check-r-package defaults to error-on: warning, that would have
failed CI. Every \link{} target in man/ is now checked against NAMESPACE.
New .example_* helper names are abbreviated to stay inside the 30-character
limit, matching the existing convention (.example_build_proc_coefs for
build_processing_coefs).
R CMD check --as-cran: 0 errors, 0 warnings, examples OK in 67s.
devtools::test(): 4667 passing, 0 failures. lintr: 0.
Refs #428
fix(n_prov_destiny): make the processed-item substitution conserve N
…fs-followup-fixes
|
Pushed three fixes onto this branch (plus a sync with 1. The processed-item substitution now conserves N (#443, merged)
This is not a defect in the pin: What moved. Totals vs. before: 1860 +11.1%, 1900 +5.8%, 1950 +5.7%, 2000 −0.35%, 2020 −1.11%. In 2000, Cropland-origin N −7.7% and imports +3.5%; exports −9.4%. Any figures already generated from this branch — the LMDI panels and typology plots especially — are stale and want regenerating. The residual N that near-zero-N outputs cannot account for now stays with the primary crop rather than vanishing. That keeps the balance exact but attributes grape pomace and olive cake to table-grape destinies. Where it should really go is #432, which is the issue your to-do list asked for. 2.
|
|
Correction to my comment above. I listed #440 (a stale Concretely, this means the thing I flagged as a concern is not one: the manure compartment of the LMDI decomposition is computed on the correct Everything else in the comment above stands — the measurements, the three fixes, and the remaining open items (#428, #432, #433, and the second half of #438). |
|
Thank you @lbm364dl! For #428 and #438 I was writing a comment in the respective issue. #432 (unaccounted N from wine/olive oil/sugar). For now, the current behaviour is kept as it is. When grapes, olives, or sugar beet get processed into wine, oil, or sugar, most of the crop's N doesn't show up in the output (those products are nearly N-free), so that leftover N stays attached to the primary crop instead of disappearing. The alternative would be creating separate items for the by-products (grape pomace, olive cake, beet pulp) so that leftover N has its own place to go. We can't do that here now, since it would mean adding new items that don't exist in our current item list, and we're not in a position to introduce new ones for the first paper. Since there's no better destination available, I'm treating the current behaviour as it is. For the next paper (nutrient coefs etc.), we should think about adding these by-product items. It is explained and tracked as future work in #445. #433 (processing_coefs stops at 2021, but our data now runs to 2023). Fixed by copying the last known value from 2021 forward into 2022 and 2023, for both processing_coefs and population_yg. I opened issue #449 to discuss whether this is good enough, or whether it's worth it to look for source data for 2022-2023 instead. |
2 participants
Summary
Data
inst/extdata/whep_inputs.csv keeps newer versions of 4 pins reflecting this branch's own work: n_balance_ygpit_all, n_excretion_ygs, npp_ygpit, intake_ygiac. All other pin entries match main.
Test plan
To do