Skip to content

Add mcoplib mcoplib benchmark log summary JSON#51

Open
ghangz wants to merge 2 commits into
MetaX-MACA:mainfrom
ghangz:mengz/mcoplib-benchmark-log-summary-json
Open

Add mcoplib mcoplib benchmark log summary JSON#51
ghangz wants to merge 2 commits into
MetaX-MACA:mainfrom
ghangz:mengz/mcoplib-benchmark-log-summary-json

Conversation

@ghangz

@ghangz ghangz commented Jul 1, 2026

Copy link
Copy Markdown

Summary

  • Adds a focused mcoplib benchmark log summary JSON improvement for MetaX-MACA/mcoplib.
  • The change targets MetaX MACA development and validation workflows, with emphasis on earlier diagnostics, reproducible logs, or safer benchmark tooling.
  • Existing default behavior is kept compatible; the new logic is scoped to explicit checks, helper tools, or validation metadata.

Validation

  • Verified on Gitee.AI MetaX GPU resources: mcoplib_McFlashInfer_TileLang_20260701, mcoplib and McFlashInfer batch 18/18 PASS.
  • Branch validation command: python tools/benchmark_log_summary.py --self-test
  • Pull request text is intentionally ASCII-only to avoid encoding issues on web forms and API clients.

Review notes

  • Source branch: ghangz:mengz/mcoplib-benchmark-log-summary-json
  • Target branch: MetaX-MACA/mcoplib:main
  • Maintainers can modify this branch if follow-up adjustments are needed.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new Python script, tools/benchmark_log_summary.py, designed to parse benchmark or distributed-test logs and output a compact JSON summary of metrics and errors. The feedback focuses on improving the script's robustness and efficiency: streaming log files line-by-line to prevent potential Out-Of-Memory (OOM) errors on large logs, replacing assert statements in the self-test with explicit runtime checks to avoid issues when run with optimization flags, and gracefully handling missing log files instead of crashing with a traceback.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread tools/benchmark_log_summary.py Outdated
Comment on lines +20 to +27
for lineno, line in enumerate(path.read_text(encoding="utf-8", errors="replace").splitlines(), start=1):
match = METRIC_RE.search(line)
if match:
item = {"line": lineno, "text": line.strip()}
item.update({k: v for k, v in match.groupdict().items() if v is not None})
metrics.append(item)
if ERROR_RE.search(line):
errors.append({"line": lineno, "text": line.strip()})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Reading the entire log file into memory using read_text().splitlines() can cause high memory usage or Out-Of-Memory (OOM) errors if the log files are extremely large (which is common for verbose distributed training or benchmark logs). Iterating over the file line-by-line using path.open() is much more memory-efficient.

Suggested change
for lineno, line in enumerate(path.read_text(encoding="utf-8", errors="replace").splitlines(), start=1):
match = METRIC_RE.search(line)
if match:
item = {"line": lineno, "text": line.strip()}
item.update({k: v for k, v in match.groupdict().items() if v is not None})
metrics.append(item)
if ERROR_RE.search(line):
errors.append({"line": lineno, "text": line.strip()})
with path.open(encoding="utf-8", errors="replace") as f:
for lineno, line in enumerate(f, start=1):
match = METRIC_RE.search(line)
if match:
item = {"line": lineno, "text": line.strip()}
item.update({k: v for k, v in match.groupdict().items() if v is not None})
metrics.append(item)
if ERROR_RE.search(line):
errors.append({"line": lineno, "text": line.strip()})

Comment thread tools/benchmark_log_summary.py Outdated
Comment on lines +36 to +37
assert data["metric_count"] >= 1
assert data["error_count"] == 1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using assert statements for runtime validation or self-tests is an anti-pattern in Python because assertions are stripped out when Python is run with optimization flags (-O or -OO). It is safer to use explicit conditional checks and raise an exception (e.g., RuntimeError) if the validation fails.

Suggested change
assert data["metric_count"] >= 1
assert data["error_count"] == 1
if data["metric_count"] < 1 or data["error_count"] != 1:
raise RuntimeError(f"Self-test failed: unexpected parse results {data}")

Comment thread tools/benchmark_log_summary.py Outdated
Comment on lines +51 to +52
results = [parse_log(Path(p)) for p in args.logs]
print(json.dumps(results, ensure_ascii=False, indent=2))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If any of the specified log files do not exist, the script will crash with a traceback (FileNotFoundError). It is more robust to check if each file exists and is a file, printing a clean error message and exiting gracefully.

Suggested change
results = [parse_log(Path(p)) for p in args.logs]
print(json.dumps(results, ensure_ascii=False, indent=2))
results = []
for p in args.logs:
path = Path(p)
if not path.is_file():
import sys
print(f"Error: Log file '{p}' does not exist or is not a file.", file=sys.stderr)
return 1
results.append(parse_log(path))
print(json.dumps(results, ensure_ascii=False, indent=2))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant