Add mcoplib mcoplib benchmark compare JSON#50
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a new Python script, tools/compare_benchmark_json.py, which compares baseline and current performance JSON files to detect regressions. The review feedback highlights two important improvements: first, modifying the command-line argument parsing so that positional arguments are not required when running the --self-test flag, and second, replacing the assert statement in the self-test function with an explicit exception to ensure reliability when Python is run with optimization flags.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| parser.add_argument("baseline") | ||
| parser.add_argument("current") | ||
| parser.add_argument("--self-test", action="store_true") | ||
| args = parser.parse_args() | ||
| if args.self_test: | ||
| self_test() | ||
| return 0 | ||
| result = compare(load(Path(args.baseline)), load(Path(args.current))) |
There was a problem hiding this comment.
When running with --self-test, the positional arguments baseline and current are still required by argparse, forcing the user to provide dummy values (e.g., python tools/compare_benchmark_json.py --self-test x y). Making these positional arguments optional (nargs='?') and validating their presence manually when --self-test is not set resolves this usability issue.
| parser.add_argument("baseline") | |
| parser.add_argument("current") | |
| parser.add_argument("--self-test", action="store_true") | |
| args = parser.parse_args() | |
| if args.self_test: | |
| self_test() | |
| return 0 | |
| result = compare(load(Path(args.baseline)), load(Path(args.current))) | |
| parser.add_argument("baseline", nargs="?") | |
| parser.add_argument("current", nargs="?") | |
| parser.add_argument("--self-test", action="store_true") | |
| args = parser.parse_args() | |
| if args.self_test: | |
| self_test() | |
| return 0 | |
| if not args.baseline or not args.current: | |
| parser.error("the following arguments are required: baseline, current") | |
| result = compare(load(Path(args.baseline)), load(Path(args.current))) |
| data = compare({"case": 100.0}, {"case": 99.0}) | ||
| assert data["ok"] |
There was a problem hiding this comment.
Using assert statements for validation or control flow in production/CLI code is discouraged because they are ignored when Python is run with optimization flags (e.g., python -O). It is safer to raise an explicit exception like RuntimeError.
| data = compare({"case": 100.0}, {"case": 99.0}) | |
| assert data["ok"] | |
| data = compare({"case": 100.0}, {"case": 99.0}) | |
| if not data["ok"]: | |
| raise RuntimeError("Self-test failed: expected comparison to succeed.") |
Summary
Validation
Review notes