Skip to content

[OMEGA-294] Independent empirical audit: doc-vs-engine discrepancies, provenance blindness, belief poisoning under revision (all reproducible) #302

Description

@NullLabTests

Title

Independent empirical audit of OmegaClaw-Core: documentation-vs-engine discrepancies, provenance blindness, and belief poisoning under revision (all reproducible)

Body

Hi maintainers — first, thank you for open-sourcing OmegaClaw-Core; the design intent around auditable inference and transparent implementation made it an ideal target for an independent empirical audit. I ran a fully contained, reproducible audit against the real engine (PyMeTA hyperon==0.2.10) over two days and found several concrete discrepancies between the documentation and live behavior. Everything is reproducible in seconds; all artifacts are in the linked repository.

Documentation vs engine (measured, not read):

  • Contradiction example: docs (0.395, 0.875) — engine (stv 0.34 0.833)
  • NAL analogy asymmetry: docs 0.36 — engine (stv 0.25 0.09) (argument-order effects)
  • PLN abduction orientation: docs (robin bird) (0.767, 0.422) — engine (Inheritance bird robin) (1.0, 0.4475)
  • Confidence decay (4-hop): docs quote <0.5 (hop 3) / ~0.25 (hop 4) — engine 0.81 → 0.729 → 0.656 → 0.590 (premise-dependent)

Adversarial findings (31-case battery, 24 PASS / 3 DISCREPANCY / 4 INFO):

  • Premise swap is silently accepted (8/8) — term-order swaps are invisible to the reasoner; no polarity check in truth propagation
  • Provenance blindness — a confident lie and a verified claim get identical numeric treatment from the truth function
  • Revision absorbs poison — four revisions from a 0.6/0.5 counter-source drift truth 0.915 → 0.842 while confidence grows 0.909 → 0.929
  • Documented counter-example — a persistent AtomSpace does accumulate and revise across turns, contradicting the "fresh AtomSpace per (metta |-) call" claim

What validated well: deterministic NAL/PLN truth calculus, reversible self-modification (18→19→18 with proof invariants), auth-gated channel access (real channels/irc.py against a local server, 13/13 scenarios), and the triage/novelty-modulation extension surface.

Suggestions (open to maintainers' judgment):

  1. Re-run the tutorial numeric examples and update the docs (or pin the interpreter version they target)
  2. Consider a polarity/provenance check in truth propagation — the 8/8 swap acceptance is the single highest-impact finding
  3. Consider documenting revision's confidence-growth-under-poison property, and whether a triage gate (like the pure-MeTTa lib_triage.metta in the audit) should be a default layer

Reproduce everything (~20 s on a 2-core sandbox):

python research/final_metrics.py
python research/harness.py
PYTHONPATH=$PWD python research/irc_integration.py --mode all

Full audit: https://github.com/NullLabTests/OmegaClawCAMPAIGN_SUMMARY.md, RESEARCH_LOG.md, research/ARCHITECTURE_FINAL.md, and every raw measurement under research/measurements/.

Happy to open this as separate issues per finding, or turn any of it into a PR (e.g., doc fixes) if useful.


Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    in-jiraThe issue has been accepted for fixing

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions