Skip to content

chore(regions): re-sync the polities snapshot to whep-polities 2830fb7 - #767

Draft
lbm364dl wants to merge 1 commit into
mainfrom
polity/745
Draft

chore(regions): re-sync the polities snapshot to whep-polities 2830fb7#767
lbm364dl wants to merge 1 commit into
mainfrom
polity/745

Conversation

@lbm364dl

Copy link
Copy Markdown
Collaborator

What was wrong

Not what #745 says, and that is the main finding.

#745 compares the shipped crosswalk's upstream_map rows (245 over 197 areas)
against eduaguilera/whep-polities@main's published map (281 over 228) and
concludes the snapshot is stale, so ~31 areas resolve through whep's
Rest-of-World bucket instead of upstream's mapping.

The map is not stale. Over the areas the crosswalk actually consumes, the
committed rows and the current upstream map agree on every
(area_code, year_start, year_end, polity_code) tuple: 245 against 245, zero
rows on either side alone. The 31 areas without an upstream_map row are the
FABIO Rest-of-World fold, which outranks the map deliberately. It is
documented at data-raw/table_mappings.R:242 ("ONE DELIBERATE EXCEPTION … 31
map-covered areas carry fabio_code == 999 … Letting the map win there would
move every Rest-of-World figure"), pinned by
test_polity_faostat_map.R ("the FABIO Rest-of-World fold still outranks the
map"), and printed by the build itself:

! 31 reporting area codes are named by the upstream FAOSTAT map but kept on the
  FABIO Rest-of-World fold.
i Areas: 5, 6, 17, 42, 47, 61, 64, 65, 69, 85, 87, 88, 125, 135, 140, 142, 153,
  154, …, 240, and 299.
i Lifting the fold is tracked separately and is out of scope here.

Those are exactly the 31 areas #745 lists. Lifting the fold is #717/#740, not a
re-sync, and it is a science decision with large value movement — the opposite
of what this PR does.

What is real is drift in the two other upstream artifacts. The snapshot was
taken at 09d7eaf; upstream main is 22 commits ahead, and the polities
database and the label alias map both moved.

Evidence it reproduced BEFORE the change

At the base commit (ee4ded6d), against upstream main at 2830fb7:

rows areas
pinned faostat_area_polity_map.csv 281 228
polity_area_crosswalk, mapping_source == "upstream_map" 245 197
tuple-level disagreement over the 197 consumed areas 0 0
the 31 remaining areas, all mapping_source == "fabio_row_fold", all fabio_code == 999, all polity_code == "ROW-1850-2025" 31

So the reported gap reproduces exactly, and its cause is the fold, not staleness.

The drift that is real, 09d7eaf2830fb7:

  • faostat_area_polity_map.csv — 16 registry spans corrected over 15 areas.
    All 15 are Rest-of-World folded, so none reaches the crosswalk join.
  • polities_database.gpkg — 4 new polities, 4 wrong cow_code values, polygon
    metadata for 8 polities, 6 predecessor/successor edges.
  • label_alias_map.csv — 1 new alias, 3 corrected year_start bounds.

What I changed

Regenerated data-raw/table_mappings.R against a pinned upstream revision,
not the live checkout (which sits on a feature branch with a dirty tree):

whep-polities main @ 2830fb735bcfaaead5e9465324dec9ce9f9b2fba
  "Merge pull request #205 from eduaguilera/fix/aliases-before-target"
git -C ~/whep-polities cat-file -p <rev>:data/final/polities_database.gpkg   > /tmp/v.gpkg
git -C ~/whep-polities cat-file -p <rev>:data/final/faostat_area_polity_map.csv > /tmp/v_map.csv
git -C ~/whep-polities cat-file -p <rev>:data/final/label_alias_map.csv      > /tmp/v_alias.csv
WHEP_POLITIES_GPKG=/tmp/v.gpkg WHEP_POLITIES_FAOSTAT_MAP=/tmp/v_map.csv \
  WHEP_POLITIES_LABEL_ALIAS_MAP=/tmp/v_alias.csv \
  Rscript --vanilla data-raw/table_mappings.R
dataset before after
polities 753 rows, 715 geometries 757 rows, 725 geometries
polity_label_aliases 902 rows 903 rows
polity_area_crosswalk 595 rows 595 rows, every routing column identical
gleam_geographic_hierarchy 3 of 204 unresolved 204 of 204 resolved
items_cbs, items_prod value-identical, git checkout --ed

data/livestock_coefs.rda had to be rebuilt too, and that was not a guess —
test_territorial_identity.R:196 failed the moment polities refreshed,
which is precisely what its "FRESHNESS BACKSTOP FOR THE GLEAM TABLE" comment
says it exists to do. Rebuilt from the same GLEAM workbook: 45 objects, one
changed, three cells in it (ATF, SGS, WLF from NA to
ATF-1800-2025 / SGS-1800-2025 / WLF-1800-2025, upstream
whep-polities#187). Its sibling assertion at :244 pinned those three as
permanently unresolvable and becomes the empty set.

Three comments that the re-sync makes factually wrong are corrected in place
(the retained-polity count, the map's two year-boundary exceptions upstream has
fixed, and the GLEAM NA note).

How I verified

  • Both generators re-run from the pinned revision, every .rda diffed with
    all.equal() per dataset; the two that came back value-identical were
    reverted so the diff shows no spurious binaries.
  • .area_year_polity_conflicts(): 0 rows after the re-sync (0 before too).
  • fix(regions): stop the prefix expansion awarding a polity outside the area's fold #743's invariant holds: 0 crosswalk rows award a polity another area owns
    outside its fold. Its guard still passes.
  • polity_mapping_provenance() is bit-identical before and after —
    46,816 (area, year) pairs, 34,065 upstream / 10,912 whep_bucket /
    1,622 whep_prefix / 217 unresolved. That is the direct measurement that
    the re-sync moves nothing from a guess to upstream_map, i.e. polity_area_crosswalk is built from a stale upstream map: 31 areas upstream now maps resolve through whep's Rest-of-World bucket instead #745's
    predicted 3.37% shift does not happen.
  • Prefix fallback still exactly 351, 901-906.
  • The GLEAM backstop is load-bearing, proved by reintroducing the defect:
    restore the stale livestock_coefs.rda next to the fresh polities
    test_territorial_identity.R:196 fails; restore the rebuilt one → passes.
  • Gates: air format . clean · devtools::document() · devtools::test()
    FAIL 0 | PASS 7190 · lintr::lint_package(...) no lints ·
    rcmdcheck 0 errors, 0 warnings, 0 notes (needs
    _R_CHECK_FORCE_SUGGESTS_=false locally: archive and RSQLite are not
    installed on this machine) · pkgdown: every man/*.Rd topic present in
    _pkgdown.yml.

Moves published values

It does not, and that is measured, not assumed.

A full real get_primary_production() (1850-2023, warm pins) was driven before
and after:

before after
rows 6,310,390 6,310,390
all 12 published columns identical, every one
keys added / removed 0 / 0
(area, year) pairs whose polity_area_code, reporting_polity_code or reporting_polity_has_geometry moved 0
Σ value, all 8 units (tonnes, ha, heads, LU, slaughtered_heads, t_ha, t_head, t_LU) delta 0.0 in each

all.equal() on the whole object reports one difference, in the non-published
.cb_extracts attribute. That is run-to-run nondeterminism in the pin read,
not this change
: a third control build with the same data reproduces the
same attribute difference, while the published columns stay identical, and the
sorted contents hash equal across all three runs.

Why it cannot move: the crosswalk's routing columns (area_code,
polity_area_code, reporting_polity_code, polity_code,
polity_start_year, polity_end_year, mapping_source, mapping_status,
map_year_*, fabio_code, polity_type, has_geometry, …) are all
bit-identical; only cow_code (3 cells) and polygon_status (1 cell) changed,
and neither is emitted anywhere. The corrected map spans are all on ROW-folded
areas. The alias changes are on the fao1952, mitchell and unscoped sources,
while the only in-package caller of resolve_polity_label() asks for
source = "trade-sources".

The user-visible change is therefore confined to the tables themselves:
polities gains 4 rows and 10 geometries, 4 cow_code values are corrected,
and gleam_geographic_hierarchy resolves 3 territories it previously left
NA. NEWS.md records it.

Classification: mechanical. It adopts upstream's corrections verbatim and
changes no whep decision. The science decision in this area — lifting the
Rest-of-World fold — is deliberately not taken here.

What I deliberately did not do

Closes #745.
Part of the polity migration epic #458.

🤖 Generated with Claude Code

Regenerated data-raw/table_mappings.R and data-raw/livestock_coefficients.R
against whep-polities main at 2830fb735bcfaaead5e9465324dec9ce9f9b2fba, pinned
explicitly with WHEP_POLITIES_GPKG / WHEP_POLITIES_FAOSTAT_MAP /
WHEP_POLITIES_LABEL_ALIAS_MAP so the run cannot pick up whatever the local
checkout happens to be on. The previous snapshot was 09d7eaf. Neither generator
changed; only their input moved.

    polities                    753 -> 757 rows, 715 -> 725 geometries
    polity_label_aliases        902 -> 903 rows
    polity_area_crosswalk       595 rows, every routing column IDENTICAL
    gleam_geographic_hierarchy  204 rows, 3 cells resolved
    items_cbs / items_prod      byte-identical, reverted

#745 ASKED FOR THIS RE-SYNC ON A PREMISE THAT DOES NOT HOLD. It reads the
crosswalk's 245 upstream_map rows over 197 areas against upstream's 281 over
228 and concludes the map is stale. It is not: over the areas the crosswalk
actually consumes, the committed rows and the pinned upstream map agree on
every (area, span, polity) tuple -- 245 against 245, zero on either side alone.
The 31 areas with no upstream_map row are the FABIO Rest-of-World fold, which
outranks the map on purpose (documented at data-raw/table_mappings.R:242, and
the build itself prints "31 reporting area codes are named by the upstream
FAOSTAT map but kept on the FABIO Rest-of-World fold"). Re-syncing does not
move them, measured: polity_mapping_provenance() is bit-identical before and
after at 34,065 upstream / 10,912 whep_bucket / 1,622 whep_prefix / 217
unresolved over 46,816 (area, year) pairs.

NO PUBLISHED VALUE MOVES, measured rather than assumed. A full real
get_primary_production() (1850-2023) before and after: 6,310,390 rows both
sides, all twelve columns identical, 0 keys added, 0 removed, 0 (area, year)
re-attributed, 0.0 delta in each of the eight units. The one attribute that
differs, .cb_extracts, differs between two runs of the SAME data too -- that is
pin read-order nondeterminism, shown with a third control run.

WHAT DID MOVE IS UPSTREAM CORRECTIONS. Four wrong cow_code values (Albania
400->339, Comoros 403->581, Sao Tome and Principe 411->403, Sardinia 338->325);
four new polities (ATF-1800-2025, SGS-1800-2025, WLF-1800-2025 and
FEZ-1943-1951); polygon sources, statuses and areas for eight polities; six
predecessor/successor edges; one new label alias and three corrected alias
year_start bounds.

THE GLEAM FRESHNESS BACKSTOP EARNED ITS PLACE. gleam_geographic_hierarchy
carries a reporting_polity_code recomputed from the snapshot, and
test_territorial_identity.R:196 failed the moment polities refreshed and that
table had not been rebuilt -- exactly what its comment says it is for. Rebuilt
data/livestock_coefs.rda from the same GLEAM workbook: 45 objects, one changed,
three cells in it, ATF/SGS/WLF from NA to a real polity (upstream
whep-polities#187). Its sibling assertion at :244 pinned those three as
permanently unresolvable and is now the empty set.

Guards after the re-sync: .area_year_polity_conflicts() 0, the #743 invariant
(no crosswalk row awards a polity another area owns outside its fold) 0
violations, prefix fallback still exactly 351 and 901-906.

Closes #745.
Part of the polity migration epic #458.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQsqidk2V7jDfFfwh5qomP
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant