feat: lintle dedup — latest-re-issue-only import list (#142)#160
Merged
Conversation
…dedup These two helpers define "the orbital identity of a record" and "which re-issue is latest" — exactly what `dedup` (#142) needs, and it must agree with `verify` byte-for-byte on both. Promote them from `_orbital_state`/`_element_set` to public `orbital_state`/`element_set` so dedup shares the one definition rather than re-implementing a divergent orbital mask. Pure rename; the two call sites in find_conflicts follow. No behaviour change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add `lintle dedup <out-dir>`: reads a clean run's cleaned/ (never mutating it) and writes a single ingest-ready import list under <out-dir>/dedup — one card per (catalog, epoch), keeping the latest re-issue (highest element-set number). - **Collapse** benign re-issues: same (catalog, epoch), a bumped element-set or revolution number but an identical parsed orbital state → keep the highest element-set, drop the rest. Shares verify's orbital_state/element_set so the two passes agree on "same orbit" and "which is latest". - **Never resolve a real conflict in silence**: a group with >1 distinct orbital state (same satellite+epoch, different orbit) keeps the latest but is flagged in notes.jsonl, and the run exits 1 so a human reviews it. - **Exclude hard suspects**: when <out-dir>/verify/suspects.jsonl exists, every hard suspect is dropped from the import list by (src_file, index) first; dedup still runs (collapse only) when no verify run exists. - **Constant memory**: records stream through verify's ExternalSorter; only one (catalog, epoch) group is held at a time. import.txt streams out sorted by (catalog, epoch). Outputs (import.txt, notes.jsonl, summary.json) are byte-deterministic. cleaned/ is immutable — the archive is a reproducible function of (source, clean-code); dedup is a separate, re-derivable artifact. Exit codes: 0 clean, 1 genuine contradiction(s) arbitrated (review), 2 no cleaned output. Verified on a 20,000-record sample of the real tle2006 cleaned output: 30 (catalog, epoch) groups collapsed, 0 conflicts — matching the issue's finding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#142) README: a "latest only import list" usage subsection (the collapse/conflict/ suspect-exclusion behaviour, `<out-dir>/dedup/` outputs, exit codes 0/1/2) plus `lintle dedup` in the command block; states the source/ → cleaned/ → import.txt output tiering. ARCHITECTURE §2: `dedup.py` in the module tree and table, and the acyclic-import paragraph — `cli → dedup → verify.{checks,grouping,records}`, one-way, barred from the clean path, reusing verify's one definition of "same orbit"/"latest". CHANGELOG: an unreleased `lintle dedup` bullet. CLAUDE.md: the layout line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lintle dedup— latest-re-issue-only import list (#142)Adds
lintle dedup <out-dir>: reads a clean run'scleaned/(never mutating it) and writes asingle ingest-ready import list under
<out-dir>/dedup— one card per(catalog, epoch),keeping the latest re-issue (highest element-set number).
Closes #142.
Behaviour
(catalog, epoch), a bumped element-set or revolutionnumber but an identical parsed orbital state → keep the highest element-set, drop the rest.
satellite + epoch, different orbit) keeps the latest but is flagged in
notes.jsonl, and therun exits 1 so a human reviews it.
<out-dir>/verify/suspects.jsonlexists, every hard suspectis dropped from the import list by
(src_file, index)first; dedup still runs (collapse only)when no verify run exists.
verify'sExternalSorter; onlyone
(catalog, epoch)group is held at a time;import.txtstreams out sorted by(catalog, epoch). Outputs (import.txt,notes.jsonl,summary.json) are byte-deterministic.Realises the output tiering
source/→cleaned/(immutable archive) →import.txt(verified + deduped, ready to ingest). Exit codes: 0 clean, 1 genuine contradiction arbitrated
(review), 2 no cleaned output.
One definition of "same orbit" / "latest"
Promotes
verify.checks._orbital_state/_element_setto publicorbital_state/element_setso
dedupshares them — not a divergent second orbital mask.dedupis a second read-onlyconsumer of the
verify/package (cli → dedup → verify.{checks,grouping,records}), one-way andequally barred from the clean path.
Verification
sorting/cli);
ruff checkclean;ruff format --checkclean.tle2006cleaned output, dedup collapsed30
(catalog, epoch)groups, 0 conflicts — matching the issue's documented finding(catalog 2864 kept element-set 956 over 955), and the cleaned slice was byte-identical after.
🤖 Generated with Claude Code