Add mcoplib mcoplib benchmark log summary JSON#51
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a new Python script, tools/benchmark_log_summary.py, designed to parse benchmark or distributed-test logs and output a compact JSON summary of metrics and errors. The feedback focuses on improving the script's robustness and efficiency: streaming log files line-by-line to prevent potential Out-Of-Memory (OOM) errors on large logs, replacing assert statements in the self-test with explicit runtime checks to avoid issues when run with optimization flags, and gracefully handling missing log files instead of crashing with a traceback.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| for lineno, line in enumerate(path.read_text(encoding="utf-8", errors="replace").splitlines(), start=1): | ||
| match = METRIC_RE.search(line) | ||
| if match: | ||
| item = {"line": lineno, "text": line.strip()} | ||
| item.update({k: v for k, v in match.groupdict().items() if v is not None}) | ||
| metrics.append(item) | ||
| if ERROR_RE.search(line): | ||
| errors.append({"line": lineno, "text": line.strip()}) |
There was a problem hiding this comment.
Reading the entire log file into memory using read_text().splitlines() can cause high memory usage or Out-Of-Memory (OOM) errors if the log files are extremely large (which is common for verbose distributed training or benchmark logs). Iterating over the file line-by-line using path.open() is much more memory-efficient.
| for lineno, line in enumerate(path.read_text(encoding="utf-8", errors="replace").splitlines(), start=1): | |
| match = METRIC_RE.search(line) | |
| if match: | |
| item = {"line": lineno, "text": line.strip()} | |
| item.update({k: v for k, v in match.groupdict().items() if v is not None}) | |
| metrics.append(item) | |
| if ERROR_RE.search(line): | |
| errors.append({"line": lineno, "text": line.strip()}) | |
| with path.open(encoding="utf-8", errors="replace") as f: | |
| for lineno, line in enumerate(f, start=1): | |
| match = METRIC_RE.search(line) | |
| if match: | |
| item = {"line": lineno, "text": line.strip()} | |
| item.update({k: v for k, v in match.groupdict().items() if v is not None}) | |
| metrics.append(item) | |
| if ERROR_RE.search(line): | |
| errors.append({"line": lineno, "text": line.strip()}) |
| assert data["metric_count"] >= 1 | ||
| assert data["error_count"] == 1 |
There was a problem hiding this comment.
Using assert statements for runtime validation or self-tests is an anti-pattern in Python because assertions are stripped out when Python is run with optimization flags (-O or -OO). It is safer to use explicit conditional checks and raise an exception (e.g., RuntimeError) if the validation fails.
| assert data["metric_count"] >= 1 | |
| assert data["error_count"] == 1 | |
| if data["metric_count"] < 1 or data["error_count"] != 1: | |
| raise RuntimeError(f"Self-test failed: unexpected parse results {data}") |
| results = [parse_log(Path(p)) for p in args.logs] | ||
| print(json.dumps(results, ensure_ascii=False, indent=2)) |
There was a problem hiding this comment.
If any of the specified log files do not exist, the script will crash with a traceback (FileNotFoundError). It is more robust to check if each file exists and is a file, printing a clean error message and exiting gracefully.
| results = [parse_log(Path(p)) for p in args.logs] | |
| print(json.dumps(results, ensure_ascii=False, indent=2)) | |
| results = [] | |
| for p in args.logs: | |
| path = Path(p) | |
| if not path.is_file(): | |
| import sys | |
| print(f"Error: Log file '{p}' does not exist or is not a file.", file=sys.stderr) | |
| return 1 | |
| results.append(parse_log(path)) | |
| print(json.dumps(results, ensure_ascii=False, indent=2)) |
Summary
Validation
Review notes