Skip to content

Correct the dcast comment about which CBS buckets duplicate - #774

Open
lbm364dl wants to merge 1 commit into
mainfrom
claude/select-best-source-stale-comment-557
Open

Correct the dcast comment about which CBS buckets duplicate#774
lbm364dl wants to merge 1 commit into
mainfrom
claude/select-best-source-stale-comment-557

Conversation

@lbm364dl

Copy link
Copy Markdown
Collaborator

What was wrong

.select_best_source() in R/build_cbs.R justifies its fun.aggregate
argument with a measurement from #429 — 31,642 duplicated (key, source)
combinations at full range "in areas 206 and 999", plus a dump of bucket 999 in
2010 for wheat holding four distinct territories. That is the recorded evidence
for why sum is the right aggregate, and it is also what a reader uses to
reason about the Rest-of-World bucket. Neither bucket duplicates any more:

  • 999: all 62 areas with fabio_code == 999 resolve to the single aggregate
    polity ROW-1850-2025, so even with the fold applied the bucket emits one
    row; and since the un-fold became the default (.unfold_rest_of_world(),
    "all") the members are not folded into it at all.
  • 206: The polity-level ROW un-fold splits one bucket into several area labels, so .aggregate_to_polities() stops summing it #563 removed polity_name from .aggregate_to_polities()'s
    grouping keys (b2526238), so a bucket folding several live territories now
    emits exactly one row per key. The issue measured 206 at 60fc6610, which is
    not a descendant of that fix — verified with git merge-base --is-ancestor — so the 206 half of the claim was already stale when filed
    too, not only the 999 half.

Measured on this branch, off polity_area_crosswalk alone:

real crosswalk    -> buckets folding >1 live polity in some year 1850-2023: 206
                     (2 polities, from 2011: Sudan, South Sudan)
fabio_code == 999 -> polity_code: "ROW-1850-2025" (single value, 62 areas)

What changed

Comment only, no behaviour change. The measurement is kept but marked as
history, with the two reasons it ended, and the argument for fun.aggregate
restated so it no longer depends on a live duplicate: dcast()'s fallback is
length() applied to every cell, so any future duplicate anywhere — a
bucket, a re-mapped item code, one key reported in two units, since key_cols
excludes unit — would silently turn the whole table into row counts.
fun.aggregate is untouched.

Two tests pin the property instead of a measurement:

  • test_build_cbs.R — a duplicated (key, source) pair comes back summed,
    and an unduplicated key in the same table keeps its tonnes.
  • test_polity_folds.R — 206 is the only polity_area_code folding more than
    one live polity in any year, and every fabio_code == 999 area resolves to
    ROW-1850-2025. Crosswalk-derived: no pins, no network.

How it was verified

A comment fix cannot have a test that fails before it, so both tests are shown
to be sensitive by mutation instead.

fun.aggregate deleted from the dcast() call:

Failure ('test_build_cbs.R'): a duplicated (key, source) pair is summed, not counted
Failure ('test_build_cbs.R'): a duplicated (key, source) pair is summed, not counted
[ FAIL 2 | WARN 9 | SKIP 0 | PASS 154 ]

(the wheat cell becomes the count 2 instead of 3,347,000, and the unduplicated
maize cell becomes 1 instead of 500 — the global-fallback bug itself)

Crosswalk assertion, with one ROW member given back a live national polity
inside bucket 999:

real crosswalk:    206
mutated crosswalk: 206 999

Restored, both files green:

devtools::test(filter = "build_cbs|polity_folds")
[ FAIL 0 | WARN 19 | SKIP 0 | PASS 274 ]

air format . run; lintr::lint() clean on all three changed files (the repo
.lintr keeps commented_code_linter on, which the first draft of both test
comments tripped). No roxygen, no exports, no NSE symbols, so no
document() / _pkgdown.yml / globalVariables() change. No NEWS.md entry:
no user-visible behaviour changes.

Classification: mechanical — documentation accuracy plus two assertions; no
number moves.

Closes #557.

🤖 Generated with Claude Code

`.select_best_source()`'s justification for `fun.aggregate` recorded a
measurement from #429: 31,642 duplicated (key, source) combinations at
full range, in buckets 206 and 999, with a dump of bucket 999 in 2010
holding four distinct territories. Neither bucket duplicates any more:

- 999 resolves all 62 `fabio_code == 999` areas to the one aggregate
  polity `ROW-1850-2025`, and since the un-fold became the default they
  are not folded into the bucket at all.
- 206 stopped when #563 removed `polity_name` from
  `.aggregate_to_polities()`'s grouping keys, so a bucket folding several
  live territories now emits exactly one row per key. The issue measured
  206 at 60fc661, which predates that fix.

The comment is the recorded evidence for why `sum` is the right
aggregate, and a reader reasoning about the Rest-of-World bucket from it
was being told the duplication is still there — which cost a full
re-measure in #555. Narrowed to history plus the two reasons it ended,
and restated why `fun.aggregate` must stay: dcast's fallback is applied
to every cell, so any future duplicate silently turns the whole table
into row counts.

Two tests pin what the comment now claims instead of a stale
measurement: a duplicated (key, source) pair must come back summed
(fails with `fun.aggregate` removed — the wheat cell becomes 2 and the
unduplicated maize cell becomes 1), and 206 must be the only
`polity_area_code` that folds more than one live polity in any year,
derived from `polity_area_crosswalk` alone with no pins or network.

Closes #557

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lbm364dl lbm364dl self-assigned this Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

.select_best_source()'s dcast comment is stale about bucket 999: it no longer duplicates, only 206 does

1 participant