Wikipedia/Wikimedia import pipeline — seed the KNOWLEDGE_COMMONS corpus as derivative attestations (NOT canonical authority)

[emooreatx]

Tracking issue for the CIRISProxy → KNOWLEDGE_COMMONS import pipeline. Design authority: FSD-005 §0.1 (CIRISConstitution/FSD/FSD-005_KNOWLEDGE_COMMONS.md).

The one rule that governs everything

Import Wikipedia's STATEMENTS, never Wikipedia's AUTHORITY. The accord/canonical trust root is NOT involved — blessing the import as canonical truth would rebuild the appeal-to-authority the whole system replaces, make one import key a corpus-wide capture point, and kill contestability (you can't render "Contradicted by: 12" against root-canonical content). CIRISProxy is just a registered federation key — an institutional attester, not a root.

What the trust root needs: nothing beyond CIRISProxy being a registered peer

• No canonical/anchor ceremony. register_federation_key for CIRISProxy, done.
• The import's trustworthiness rides on provenance + transparency-log, not root blessing.

Import shape (per imported claim)

A scores attestation on truth_grounding:{subject}:

• epistemic_mode: derivative (relaying another source — NOT a direct witness)
• witness_relation: external
• evidence_refs: [enwiki://{title}?oldid={R}, sha256:{content_hash}]
• context: "enwiki '{title}' rev {R} imported {T}"
• score/confidence mean "Wikipedia asserts this," NOT "this is true"
• a transparency_log:inclusion anchor (tamper-evident, auditable)

A freshly-imported-only claim therefore reads "attested by CIRISProxy (Wikipedia import), 1 source, derivative" — modest, single-source, honestly labelled. Real diversity accrues only as independent attesters engage; the UI must never render an import-only claim as high confidence.

Channel-trust, not content-trust (the only place trust attaches)

Optionally give CIRISProxy a partner_role / "faithful-relay" credential: consumers weight "faithfully relays a known source," never "the source is true."

Pipeline tasks

• Register CIRISProxy federation key (institutional attester); optional faithful-relay credential.
• Fetcher: MediaWiki API/dumps → (title, oldid, wikitext/HTML, content_hash, fetched_at).
• Granularity split: hand-decompose the ~5 FSD flagship subjects into sentence-level claims (so the Evidence Panel + "show me why" demo lands); bulk-import everything else as article-blobs (holds_bytes) with lazy claim-extraction later. Do NOT block the corpus on an NLP extraction pipeline. (Extraction, when it runs, is itself a derivative attestation "article X contains claim Y".)
• Revision-as-supersession: a later Wikipedia revision is a supersedes on the import claim → the CEG timeline natively renders edit history. Imports MUST carry oldid so supersession has a key.
• Emit via the persist attestation write path; anchor each in the transparency log.
• Licensing: CC BY-SA — attribution satisfied by construction (evidence_refs + provenance chain ARE the attribution); carry the SA notice on any derived/summary layer.
• Scale = the substrate's first real load test. ~7M enwiki articles → that many Contributions + holds_bytes blobs + persist V106 projection rows. Start with a bounded slice (the flagship subjects + one category) before the full dump; coordinate with the persist v17.4.0 read-surface + commons-scale fountain/blob wiring.

Dependencies

• Persist v17.4.0 (the resolve_scores/list_scores read surface + V106 subject projection) — in flight.
• FSD-005 §0.1 / Appendix C (the read contract) — CIRISConstitution/FSD.

What NOT to do (explicit)

• Do NOT route imports through the accord/canonical trust root.
• Do NOT score imports as "true" — they are derivative "Wikipedia asserts."
• Do NOT render import-only claims as high-confidence.
• Do NOT make imported claims un-contestable.

🤖 Generated with Claude Code

[emooreatx]

Model correction (operator) — articles as the unit, claims as editorial

Backing off the sentence-decomposition requirement. The bootstrap model is simpler and better:

1. Article-granularity, not claim-decomposition. An encyclopedia article is its OWN object — a Contribution whose content is the article text (holds_bytes), stable article-id, under an article content_class (candidate content_class:encyclopedia_article or reuse educational — additive CC editorial call, no new family). Articles and claims reference each other bidirectionally. No NLP claim-extraction in the bootstrap path — the granularity fork dissolves.

2. Display gate = two concurring proxy occurrences. An imported article is "good enough to DISPLAY" when two independent CIRISProxy occurrences agree on (article-id, oldid, content_hash) — a faithful-relay cross-attestation (two independent fetches confirm this is Wikipedia revision R). Renders as "Wikipedia revision R — 2 concurring relays" — a provenance statement, never a truth claim.

3. Claims are editorial, layered on the article. Any proposition in a displayed article can be broken out: a real attester extracts the specific claim → creates a truth_grounding attestation ON it → cites it back into a revision of the article. Claim-creation moves from generation (exhaustive/NLP-blocked) to editorial (targeted, real, attributed). Epistemic weight accrues exactly where someone asserts or contests.

Revised pipeline tasks (supersedes the granularity task above)

• Register CIRISProxy federation key + ≥2 occurrences (the concurrence needs two independent import instances); optional faithful-relay credential.
• Fetcher: MediaWiki API/dumps → (article-id, title, oldid, content, content_hash, fetched_at).
• Import each article as a Contribution (holds_bytes article text) + a derivative scores attestation carrying oldid + content_hash + transparency_log:inclusion. NO sentence decomposition.
• Two-occurrence concurrence: the display gate fires when 2 CIRISProxy occurrences independently emit matching (article-id, oldid, content_hash). Wire this as the "displayable" predicate the reader checks.
• Editorial claim break-out (the reader/editor path, coordinate with GUI): UI to select a proposition in an article → mint a truth_grounding claim → cite it into an article revision. (This is the write path, layered on the imported articles.)
• Revision-as-supersession: later revision = supersedes on the article import; carry oldid.
• Licensing CC BY-SA (attribution by construction); scale = bounded slice first (flagship + one category), then wider.

Design authority: FSD-005 §0.1 (updated), CIRISConstitution/FSD.

[emooreatx]

Namespaces confirmed — all pre-existing (CEG 0.3/0.4), zero new families

The article model maps 1:1 onto spec already:

• encyclopedia_article (external_content sub_kind, CEG 0.3) — verbatim: "Wikipedia-shape; editor-consensus + revision chain via supersedes; indefinite valid_until." The 2-proxy-concurrence display gate IS the "editor-consensus"; revision-as-supersession IS the "revision chain via supersedes." Import each article as this sub_kind.
• encyclopedia:* (dimension family) — the claim plane for broken-out encyclopedia-content claims (signed, editor-consensus + revision chain).
• topical_relation:{kind} (open-vocab, no amendment) — the bidirectional article↔claim edges: cites_source (claim → article revision), references, corrects, supersedes_article, see_also. New kinds are documentation-only.

So the earlier "candidate content_class / editorial call" is moot — use encyclopedia_article + encyclopedia:* + topical_relation:*. No new family, no CC amendment. The pipeline's "import as a Contribution" = import as an encyclopedia_article external_content Contribution; "cite claim back into revision" = a topical_relation:cites_source edge on an article supersedes revision.