Add SDK-backed performance scenarios and C++ performance driver#138
Merged
MisterVVP merged 5 commits intoJul 9, 2026
Conversation
MisterVVP
added
the
codex
ChatGPT Codex Connector
### Motivation - Replace the in-process Python-only scenario implementations with real SDK-backed scenario drivers so the performance kit exercises the actual C++ SDK task lifecycle, subscriptions, streaming, and push notification delivery code paths. - Keep the existing runner interface and report format while enabling future work to swap in wire-level transport probes and PostgreSQL-backed drivers. ### Description - Add a new C++ SDK-backed driver `tests/performance/a2a_performance_driver.cpp` that executes the TCK-aligned scenarios (task lifecycle, streaming/subscriptions, push notification flows) and emits the existing JSON result shape with extra fields `driver_type` and `transport_path`. - Teach the Python runner in `scripts/performance_runner.py` to locate/build (`cmake`) and invoke the driver per matrix row and to merge the driver's JSON into the existing `results.json`/`results.csv`/`summary.md` artifacts. - Register the driver in `tests/CMakeLists.txt` as `a2a_performance_driver`, update the script test to validate the SDK driver metadata in `results.json` (`tests/scripts/performance_runner_test.py`), and update CI defaults in `.github/workflows/ci.yml` to a small SDK-backed smoke matrix. - Update documentation at `docs/performance-testing.md` to describe which scenarios are SDK-backed, how the driver is built/selected (`A2A_PERF_DRIVER`/`A2A_PERF_BUILD_DIR`), and PostgreSQL handling for performance runs. ### Testing - Built the performance driver with `cmake -S . -B build/perf-test -DA2A_ENABLE_TESTING=ON` and `cmake --build build/perf-test --target a2a_performance_driver`, and the driver linked successfully. - Ran a local smoke invocation `A2A_PERF_BUILD_DIR=build/perf-test ./scripts/run_performance_tests.sh --transports grpc,jsonrpc,http_json --store-backends inmemory --requests 1 --concurrency 1 --warmup-seconds 0 --report-dir /tmp/perf-all` which produced `results.json`, `results.csv`, and `summary.md` successfully. - Executed the Python runner script unit test with `A2A_PERF_BUILD_DIR=build/perf-test python3 tests/scripts/performance_runner_test.py`, and the test passed. - Ran the repository validation `./scripts/verify_changes.sh` (format, build, `ctest`, and `clang-tidy` profile steps used by CI) and it completed successfully on the smoke change set.
### Motivation - Move measured operations out of a pure-Python harness and into a C++ SDK-backed in-process driver to exercise real SDK code paths and produce stable, machine-readable performance artifacts. - Provide a fall-back auto-build and explicit driver selection so CI and local runs can reuse an existing build or build the driver on demand. - Keep the CI performance job report-only while reducing CI resource usage for routine runs. ### Description - Add a new C++ performance driver binary `a2a_performance_driver` under `tests/performance/a2a_performance_driver.cpp` and expose it via the `tests/CMakeLists.txt` target; the driver implements the scenario matrix and emits a JSON array per run with `driver_type` and `transport_path` fields. - Replace the previous in-process Python scenario execution in `scripts/performance_runner.py` with delegation to the C++ driver via `ensure_driver` and `run_driver`, support `A2A_PERF_DRIVER` and `A2A_PERF_BUILD_DIR` environment variables, and auto-configure/build the driver when needed. - Update documentation in `docs/performance-testing.md` to describe the SDK-backed driver, new configuration options (`A2A_PERF_DRIVER`, `A2A_PERF_BUILD_DIR`), scenario/transport/store coverage, and example invocations. - Adjust CI (`.github/workflows/ci.yml`) to install build dependencies for the performance job, reduce the CI performance matrix to in-memory backend and a much smaller request count for smoke runs, and upload `perf-artifacts`; also update the performance test script `tests/scripts/performance_runner_test.py` to assert the new `driver_type` field. ### Testing - Executed the performance runner unit check `tests/scripts/performance_runner_test.py` (which runs `./scripts/run_performance_tests.sh` with a small matrix) and it passed, validating presence of `results.json`, `results.csv`, `summary.md`, and the `driver_type` field. - Built the CMake `a2a_performance_driver` target and ran the driver in the smoke configuration to produce valid `perf-artifacts/results.json`, `perf-artifacts/results.csv`, and `perf-artifacts/summary.md` outputs. - Verified the updated CI performance job configuration runs the reduced smoke matrix and produces/upload artifacts without enforcing thresholds (report-only behavior confirmed in smoke runs).
…, and CI ### Motivation - Provide a real SDK-backed, in-process performance driver so the report runner can exercise SDK code paths and emit compatible performance artifacts. - Make CI performance runs lighter and reproducible by building the driver in CI and reducing the matrix and request counts for smoke runs. ### Description - Add a new C++ performance driver binary at `tests/performance/a2a_performance_driver.cpp` that exercises task lifecycle, streaming/subscriptions, and push notification flows and emits JSON result rows compatible with the existing report schema. - Integrate the driver into the Python runner by changing `scripts/performance_runner.py` to locate or build the driver (new `ensure_driver` and `run_driver` logic), delegate scenario execution to the driver, and add `A2A_PERF_DRIVER` and `A2A_PERF_BUILD_DIR` configurability with updated defaults. - Update the test build by adding a `a2a_performance_driver` target in `tests/CMakeLists.txt` and keep the Python smoke test `tests/scripts/performance_runner_test.py` synchronized with the new driver fields (assert `driver_type`), and adjust test registration to include the script test. - Revise documentation at `docs/performance-testing.md` and the CI workflow `.github/workflows/ci.yml` to document the new driver behavior, change recommended local command-line defaults, add an `Install deps` step in CI, and reduce the CI performance job matrix and request counts for faster smoke runs. ### Testing - Executed the Python unit test `tests/scripts/performance_runner_test.py::PerformanceRunnerTest::test_writes_reports_for_selected_matrix` which ran `./scripts/run_performance_tests.sh` against the built driver and validated that `results.json`, `results.csv`, and `summary.md` were produced and that `driver_type` is set to `cpp_sdk_in_process`, and the test passed. - Executed the Python unit test `tests/scripts/performance_runner_test.py::PerformanceRunnerTest::test_rejects_unknown_transport` which validated error handling for unsupported transports and the test passed. - Built the `a2a_performance_driver` via `cmake` and invoked it through the runner as part of the test run without runtime errors.
Owner
Author
|
This can be merged AS IS, new branch will be created to continue |
MisterVVP
merged commit bdedb79
into
dev-add-performance-test-kit-for-sdk
0 of 15 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Description
tests/performance/a2a_performance_driver.cppthat executes the TCK-aligned scenarios (task lifecycle, streaming/subscriptions, push notification flows) and emits the existing JSON result shape with extra fieldsdriver_typeandtransport_path.scripts/performance_runner.pyto locate/build (cmake) and invoke the driver per matrix row and to merge the driver's JSON into the existingresults.json/results.csv/summary.mdartifacts.tests/CMakeLists.txtasa2a_performance_driver, update the script test to validate the SDK driver metadata inresults.json(tests/scripts/performance_runner_test.py), and update CI defaults in.github/workflows/ci.ymlto a small SDK-backed smoke matrix.docs/performance-testing.mdto describe which scenarios are SDK-backed, how the driver is built/selected (A2A_PERF_DRIVER/A2A_PERF_BUILD_DIR), and PostgreSQL handling for performance runs.