chore(regions): re-sync the polities snapshot to whep-polities 2830fb7 - #767
Draft
lbm364dl wants to merge 1 commit into
Draft
chore(regions): re-sync the polities snapshot to whep-polities 2830fb7#767lbm364dl wants to merge 1 commit into
lbm364dl wants to merge 1 commit into
Conversation
Regenerated data-raw/table_mappings.R and data-raw/livestock_coefficients.R
against whep-polities main at 2830fb735bcfaaead5e9465324dec9ce9f9b2fba, pinned
explicitly with WHEP_POLITIES_GPKG / WHEP_POLITIES_FAOSTAT_MAP /
WHEP_POLITIES_LABEL_ALIAS_MAP so the run cannot pick up whatever the local
checkout happens to be on. The previous snapshot was 09d7eaf. Neither generator
changed; only their input moved.
polities 753 -> 757 rows, 715 -> 725 geometries
polity_label_aliases 902 -> 903 rows
polity_area_crosswalk 595 rows, every routing column IDENTICAL
gleam_geographic_hierarchy 204 rows, 3 cells resolved
items_cbs / items_prod byte-identical, reverted
#745 ASKED FOR THIS RE-SYNC ON A PREMISE THAT DOES NOT HOLD. It reads the
crosswalk's 245 upstream_map rows over 197 areas against upstream's 281 over
228 and concludes the map is stale. It is not: over the areas the crosswalk
actually consumes, the committed rows and the pinned upstream map agree on
every (area, span, polity) tuple -- 245 against 245, zero on either side alone.
The 31 areas with no upstream_map row are the FABIO Rest-of-World fold, which
outranks the map on purpose (documented at data-raw/table_mappings.R:242, and
the build itself prints "31 reporting area codes are named by the upstream
FAOSTAT map but kept on the FABIO Rest-of-World fold"). Re-syncing does not
move them, measured: polity_mapping_provenance() is bit-identical before and
after at 34,065 upstream / 10,912 whep_bucket / 1,622 whep_prefix / 217
unresolved over 46,816 (area, year) pairs.
NO PUBLISHED VALUE MOVES, measured rather than assumed. A full real
get_primary_production() (1850-2023) before and after: 6,310,390 rows both
sides, all twelve columns identical, 0 keys added, 0 removed, 0 (area, year)
re-attributed, 0.0 delta in each of the eight units. The one attribute that
differs, .cb_extracts, differs between two runs of the SAME data too -- that is
pin read-order nondeterminism, shown with a third control run.
WHAT DID MOVE IS UPSTREAM CORRECTIONS. Four wrong cow_code values (Albania
400->339, Comoros 403->581, Sao Tome and Principe 411->403, Sardinia 338->325);
four new polities (ATF-1800-2025, SGS-1800-2025, WLF-1800-2025 and
FEZ-1943-1951); polygon sources, statuses and areas for eight polities; six
predecessor/successor edges; one new label alias and three corrected alias
year_start bounds.
THE GLEAM FRESHNESS BACKSTOP EARNED ITS PLACE. gleam_geographic_hierarchy
carries a reporting_polity_code recomputed from the snapshot, and
test_territorial_identity.R:196 failed the moment polities refreshed and that
table had not been rebuilt -- exactly what its comment says it is for. Rebuilt
data/livestock_coefs.rda from the same GLEAM workbook: 45 objects, one changed,
three cells in it, ATF/SGS/WLF from NA to a real polity (upstream
whep-polities#187). Its sibling assertion at :244 pinned those three as
permanently unresolvable and is now the empty set.
Guards after the re-sync: .area_year_polity_conflicts() 0, the #743 invariant
(no crosswalk row awards a polity another area owns outside its fold) 0
violations, prefix fallback still exactly 351 and 901-906.
Closes #745.
Part of the polity migration epic #458.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQsqidk2V7jDfFfwh5qomP
This was referenced Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
Not what #745 says, and that is the main finding.
#745 compares the shipped crosswalk's
upstream_maprows (245 over 197 areas)against
eduaguilera/whep-polities@main's published map (281 over 228) andconcludes the snapshot is stale, so ~31 areas resolve through whep's
Rest-of-World bucket instead of upstream's mapping.
The map is not stale. Over the areas the crosswalk actually consumes, the
committed rows and the current upstream map agree on every
(area_code, year_start, year_end, polity_code)tuple: 245 against 245, zerorows on either side alone. The 31 areas without an
upstream_maprow are theFABIO Rest-of-World fold, which outranks the map deliberately. It is
documented at
data-raw/table_mappings.R:242("ONE DELIBERATE EXCEPTION … 31map-covered areas carry
fabio_code == 999… Letting the map win there wouldmove every Rest-of-World figure"), pinned by
test_polity_faostat_map.R("the FABIO Rest-of-World fold still outranks themap"), and printed by the build itself:
Those are exactly the 31 areas #745 lists. Lifting the fold is #717/#740, not a
re-sync, and it is a science decision with large value movement — the opposite
of what this PR does.
What is real is drift in the two other upstream artifacts. The snapshot was
taken at
09d7eaf; upstreammainis 22 commits ahead, and the politiesdatabase and the label alias map both moved.
Evidence it reproduced BEFORE the change
At the base commit (
ee4ded6d), against upstreammainat2830fb7:faostat_area_polity_map.csvpolity_area_crosswalk,mapping_source == "upstream_map"mapping_source == "fabio_row_fold", allfabio_code == 999, allpolity_code == "ROW-1850-2025"So the reported gap reproduces exactly, and its cause is the fold, not staleness.
The drift that is real,
09d7eaf→2830fb7:faostat_area_polity_map.csv— 16registryspans corrected over 15 areas.All 15 are Rest-of-World folded, so none reaches the crosswalk join.
polities_database.gpkg— 4 new polities, 4 wrongcow_codevalues, polygonmetadata for 8 polities, 6 predecessor/successor edges.
label_alias_map.csv— 1 new alias, 3 correctedyear_startbounds.What I changed
Regenerated
data-raw/table_mappings.Ragainst a pinned upstream revision,not the live checkout (which sits on a feature branch with a dirty tree):
politiespolity_label_aliasespolity_area_crosswalkgleam_geographic_hierarchyitems_cbs,items_prodgit checkout --eddata/livestock_coefs.rdahad to be rebuilt too, and that was not a guess —test_territorial_identity.R:196failed the momentpolitiesrefreshed,which is precisely what its "FRESHNESS BACKSTOP FOR THE GLEAM TABLE" comment
says it exists to do. Rebuilt from the same GLEAM workbook: 45 objects, one
changed, three cells in it (
ATF,SGS,WLFfromNAtoATF-1800-2025/SGS-1800-2025/WLF-1800-2025, upstreamwhep-polities#187). Its sibling assertion at
:244pinned those three aspermanently unresolvable and becomes the empty set.
Three comments that the re-sync makes factually wrong are corrected in place
(the retained-polity count, the map's two year-boundary exceptions upstream has
fixed, and the GLEAM
NAnote).How I verified
.rdadiffed withall.equal()per dataset; the two that came back value-identical werereverted so the diff shows no spurious binaries.
.area_year_polity_conflicts(): 0 rows after the re-sync (0 before too).outside its fold. Its guard still passes.
polity_mapping_provenance()is bit-identical before and after —46,816
(area, year)pairs, 34,065upstream/ 10,912whep_bucket/1,622
whep_prefix/ 217unresolved. That is the direct measurement thatthe re-sync moves nothing from a guess to
upstream_map, i.e. polity_area_crosswalk is built from a stale upstream map: 31 areas upstream now maps resolve through whep's Rest-of-World bucket instead #745'spredicted 3.37% shift does not happen.
351, 901-906.restore the stale
livestock_coefs.rdanext to the freshpolities→test_territorial_identity.R:196fails; restore the rebuilt one → passes.air format .clean ·devtools::document()·devtools::test()FAIL 0 | PASS 7190 ·
lintr::lint_package(...)no lints ·rcmdcheck0 errors, 0 warnings, 0 notes (needs_R_CHECK_FORCE_SUGGESTS_=falselocally:archiveandRSQLiteare notinstalled on this machine) · pkgdown: every
man/*.Rdtopic present in_pkgdown.yml.Moves published values
It does not, and that is measured, not assumed.
A full real
get_primary_production()(1850-2023, warm pins) was driven beforeand after:
(area, year)pairs whosepolity_area_code,reporting_polity_codeorreporting_polity_has_geometrymovedvalue, all 8 units (tonnes,ha,heads,LU,slaughtered_heads,t_ha,t_head,t_LU)all.equal()on the whole object reports one difference, in the non-published.cb_extractsattribute. That is run-to-run nondeterminism in the pin read,not this change: a third control build with the same data reproduces the
same attribute difference, while the published columns stay identical, and the
sorted contents hash equal across all three runs.
Why it cannot move: the crosswalk's routing columns (
area_code,polity_area_code,reporting_polity_code,polity_code,polity_start_year,polity_end_year,mapping_source,mapping_status,map_year_*,fabio_code,polity_type,has_geometry, …) are allbit-identical; only
cow_code(3 cells) andpolygon_status(1 cell) changed,and neither is emitted anywhere. The corrected map spans are all on ROW-folded
areas. The alias changes are on the
fao1952,mitchelland unscoped sources,while the only in-package caller of
resolve_polity_label()asks forsource = "trade-sources".The user-visible change is therefore confined to the tables themselves:
politiesgains 4 rows and 10 geometries, 4cow_codevalues are corrected,and
gleam_geographic_hierarchyresolves 3 territories it previously leftNA.NEWS.mdrecords it.Classification: mechanical. It adopts upstream's corrections verbatim and
changes no whep decision. The science decision in this area — lifting the
Rest-of-World fold — is deliberately not taken here.
What I deliberately did not do
polity_area_crosswalk is built from a stale upstream map: 31 areas upstream now maps resolve through whep's Rest-of-World bucket instead #745's 31 areas, it moves every Rest-of-World figure, and it belongs to
A promoted Rest-of-World member still carries the ROW aggregate polity: polity_type = "aggregate", continent = "World" #717/288 of 595 polity_area_crosswalk rows have no upstream authority: whep invents them by ISO3-prefix match #740 with its own before/after. Left standing, and the comment saying so
is still accurate.
R/polities.RorR/polity_folds.R(owned by the A back-cast row reports mapping_status = "matched" for a polity that was not alive that year, in 41.5% of cells — and the gap diagnostic cannot see it #763worker) or
R/build_production.R(The pre-1962 back-cast measures hectares on present-day borders: make the area half historical, as the yield half already is #761). One consequence worth handing over:resolve_polity_label()'s roxygen inR/polities.Rstill says the upstreamdatabase "is ahead of the
politiessnapshot this package ships (740 rowsupstream against 603 here)" and that "225 of the 869 published aliases …
point at one of 115 codes
get_polity_geometries()cannot yet return a rowfor". Measured after this re-sync: 903 aliases, 757 polities, 0 aliases
pointing at a missing code. That paragraph was already stale before this PR
(the build has a hard guard making it 0 by construction) and needs one edit
in a file I do not own.
data-raw/harmonization_tables.R. Noregions.csvoritems_*.csvchanged.both fired: the fold pin refutes polity_area_crosswalk is built from a stale upstream map: 31 areas upstream now maps resolve through whep's Rest-of-World bucket instead #745's premise on its own, and the GLEAM
freshness backstop caught the one genuinely stale table. Adding a third
restatement would not catch anything they miss.
A promoted Rest-of-World member still carries the ROW aggregate polity: polity_type = "aggregate", continent = "World" #717 is the fold, which is untouched.
Closes #745.
Part of the polity migration epic #458.
🤖 Generated with Claude Code