Skip to content

Present MiniCode-Python reimplementation with LiteCodeBench - #1

Merged
Dopetaiga merged 3 commits into
mainfrom
agent/litecodebench
Aug 15, 2026
Merged

Present MiniCode-Python reimplementation with LiteCodeBench#1
Dopetaiga merged 3 commits into
mainfrom
agent/litecodebench

Conversation

@Dopetaiga

Copy link
Copy Markdown
Owner

What changed

  • rename the current MiniCode-Python evaluation profile to LiteCodeBench v1.0
  • publish the raw 14/15 DSV4 run, the improved 15/15 rerun, and a human-readable analysis
  • strengthen implement_deep_merge with explicit alias-freedom wording and hidden mutation checks for both base and override inputs
  • add Wilson intervals, latency distribution, average API/token metrics, and cache-hit rate to reports
  • make offline readiness honor explicit settings.provider without a model-catalog network probe
  • update CI and repository documentation to use the LiteCodeBench entry point

Why

The original full run scored 14/15. Its only failure used dict(base), which copied only the outer mapping and left nested mutable objects aliased to the input. The hidden verifier detected the issue by mutating the returned object.

The audit also found a specification problem: “do not mutate either input” does not explicitly state the stronger postcondition that the returned object must retain no mutable aliases to either input. The updated task names that contract directly and expands the verifier rather than relaxing it.

The improved 15/15 run is therefore reported as a contract-alignment rerun, not as a directly comparable 6.67-point model improvement.

Validation

  • LiteCodeBench offline baseline/oracle gate: 15/15
  • DSV4 improved live run: 15/15
  • targeted config and benchmark tests: 28 passed
  • relevant Ruff E/F checks: passed
  • structure compliance: 0 findings
  • repository regression excluding 8 existing Windows-only /tmp cases: 1311 passed, 2 skipped, 8 deselected
  • committed result JSON secret scan: passed

Scope and limitations

  • project-internal executable benchmark, not SWE-bench or an industry leaderboard
  • single run per task; 15/15 Wilson 95% interval is 79.61%–100%
  • per-episode process/workspace isolation is not a container security boundary
  • subagent reports currently expose parent traces only

@Dopetaiga Dopetaiga added the documentation Improvements or additions to documentation label Aug 15, 2026
@Dopetaiga Dopetaiga changed the title Add LiteCodeBench evaluation and DSV4 analysis Present MiniCode-Python reimplementation with LiteCodeBench Aug 15, 2026
@Dopetaiga
Dopetaiga marked this pull request as ready for review August 15, 2026 07:16
@Dopetaiga
Dopetaiga merged commit f98f0af into main Aug 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant