Skip to content

Add mcoplib mcoplib benchmark compare JSON#50

Open
ghangz wants to merge 2 commits into
MetaX-MACA:mainfrom
ghangz:mengz/mcoplib-benchmark-compare-json
Open

Add mcoplib mcoplib benchmark compare JSON#50
ghangz wants to merge 2 commits into
MetaX-MACA:mainfrom
ghangz:mengz/mcoplib-benchmark-compare-json

Conversation

@ghangz

@ghangz ghangz commented Jul 1, 2026

Copy link
Copy Markdown

Summary

  • Adds a focused mcoplib benchmark compare JSON improvement for MetaX-MACA/mcoplib.
  • The change targets MetaX MACA development and validation workflows, with emphasis on earlier diagnostics, reproducible logs, or safer benchmark tooling.
  • Existing default behavior is kept compatible; the new logic is scoped to explicit checks, helper tools, or validation metadata.

Validation

  • Verified on Gitee.AI MetaX GPU resources: mcoplib_McFlashInfer_TileLang_20260701, mcoplib and McFlashInfer batch 18/18 PASS.
  • Branch validation command: python tools/compare_benchmark_json.py --self-test x y
  • Pull request text is intentionally ASCII-only to avoid encoding issues on web forms and API clients.

Review notes

  • Source branch: ghangz:mengz/mcoplib-benchmark-compare-json
  • Target branch: MetaX-MACA/mcoplib:main
  • Maintainers can modify this branch if follow-up adjustments are needed.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new Python script, tools/compare_benchmark_json.py, which compares baseline and current performance JSON files to detect regressions. The review feedback highlights two important improvements: first, modifying the command-line argument parsing so that positional arguments are not required when running the --self-test flag, and second, replacing the assert statement in the self-test function with an explicit exception to ensure reliability when Python is run with optimization flags.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread tools/compare_benchmark_json.py Outdated
Comment on lines +45 to +52
parser.add_argument("baseline")
parser.add_argument("current")
parser.add_argument("--self-test", action="store_true")
args = parser.parse_args()
if args.self_test:
self_test()
return 0
result = compare(load(Path(args.baseline)), load(Path(args.current)))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

When running with --self-test, the positional arguments baseline and current are still required by argparse, forcing the user to provide dummy values (e.g., python tools/compare_benchmark_json.py --self-test x y). Making these positional arguments optional (nargs='?') and validating their presence manually when --self-test is not set resolves this usability issue.

Suggested change
parser.add_argument("baseline")
parser.add_argument("current")
parser.add_argument("--self-test", action="store_true")
args = parser.parse_args()
if args.self_test:
self_test()
return 0
result = compare(load(Path(args.baseline)), load(Path(args.current)))
parser.add_argument("baseline", nargs="?")
parser.add_argument("current", nargs="?")
parser.add_argument("--self-test", action="store_true")
args = parser.parse_args()
if args.self_test:
self_test()
return 0
if not args.baseline or not args.current:
parser.error("the following arguments are required: baseline, current")
result = compare(load(Path(args.baseline)), load(Path(args.current)))

Comment thread tools/compare_benchmark_json.py Outdated
Comment on lines +38 to +39
data = compare({"case": 100.0}, {"case": 99.0})
assert data["ok"]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using assert statements for validation or control flow in production/CLI code is discouraged because they are ignored when Python is run with optimization flags (e.g., python -O). It is safer to raise an explicit exception like RuntimeError.

Suggested change
data = compare({"case": 100.0}, {"case": 99.0})
assert data["ok"]
data = compare({"case": 100.0}, {"case": 99.0})
if not data["ok"]:
raise RuntimeError("Self-test failed: expected comparison to succeed.")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant