Add report-only performance test kit#136
Draft
MisterVVP wants to merge 7 commits into
Draft
Conversation
MisterVVP
added
the
codex
ChatGPT Codex Connector
Owner
Author
|
Consider k6 tests here. Also we should show test results table in CI, so it's easier to observe the impact of changes on performance. |
### Motivation - Provide a repeatable, report-only performance test kit that exercises TCK-aligned SDK flows (task lifecycle, streaming/subscriptions, push notifications) across transport and store labels so developers and CI can collect baseline metrics without gating on thresholds. - Make lightweight, deterministic scenarios available locally and in CI so we can track latency/throughput/operation counts and extend to wire-level probes later. - Emit machine-readable artifacts to enable automated ingestion, reporting, and future baseline comparisons. ### Description - Add a Python-based runner `scripts/performance_runner.py` and a small wrapper `scripts/run_performance_tests.sh` that support CLI/env matrix selection (transports, store backends, requests, concurrency, warmup, report dir) and produce `results.json`, `results.csv`, and `summary.md`. - Implement a set of TCK-aligned in-process scenarios (task lifecycle, streaming/subscription, push config CRUD/notify) and record throughput and latency percentiles (p50/p90/p95/p99/max) along with SDK commit and host metadata. - Add a smoke CI job in `.github/workflows/ci.yml` that runs the suite in report-only mode, validates outputs, and uploads `perf-artifacts`; add docs at `docs/performance-testing.md` describing usage and configuration. - Add a Python unit test `tests/scripts/performance_runner_test.py` and wire it into CTest via `tests/CMakeLists.txt` so the runner is exercised in the test matrix. ### Testing - Ran the Python unit tests `python3 tests/scripts/performance_runner_test.py`, which passed. - Ran the smoke runner locally with a matrix (`A2A_PERF_TRANSPORTS=grpc,jsonrpc,http_json A2A_PERF_STORE_BACKENDS=inmemory,postgres A2A_PERF_REQUESTS=25 A2A_PERF_CONCURRENCY=1,4 A2A_PERF_WARMUP_SECONDS=0 A2A_PERF_REPORT_DIR=/tmp/a2a-perf-ci ./scripts/run_performance_tests.sh`) and validated `results.json`, `results.csv`, and `summary.md` were produced and JSON-parsable. - Ran the repository validation `./scripts/verify_changes.sh` (format, build, `ctest`, `clang-tidy` profile) and it completed successfully in the local environment; all automated tests passed.
### Motivation - Add a repeatable, report-only performance test kit to produce deterministic local and CI artifacts for TCK-aligned scenarios without introducing threshold gates. ### Description - Add `scripts/performance_runner.py` which implements an in-process, matrix-driven performance runner and report generation (`results.json`, `results.csv`, `summary.md`). - Add `scripts/run_performance_tests.sh` as a thin wrapper to invoke the runner and add `docs/performance-testing.md` documenting usage and configuration. - Add a new GitHub Actions job `performance-tests` in `.github/workflows/ci.yml` to run a smoke matrix, validate generated artifacts, append `summary.md` to the Actions step summary, and upload `perf-artifacts` as an artifact. - Integrate a CTest entry and a unit test `tests/scripts/performance_runner_test.py` that executes the runner for a small matrix and validates the produced reports. ### Testing - Added `performance_runner_script_test` (via CTest) which runs `tests/scripts/performance_runner_test.py` and asserts the runner writes `results.json`, `results.csv`, and `summary.md`, and this test was executed and passed in the test run. - The unit test invokes `./scripts/run_performance_tests.sh` with a small smoke matrix (3 requests, single concurrency) and verifies the expected row count and summary contents, and it succeeded.
Owner
Author
|
This is just a skeleton, more improvements are going to be developed in separate branches and merged into The current scenarios are still simulated in Python, and grpc / jsonrpc / http_json plus inmemory / postgres are effectively labels rather than real SDK-backed transport/store executions. |
### Motivation - Replace the in-process Python-only scenario implementations with real SDK-backed scenario drivers so the performance kit exercises the actual C++ SDK task lifecycle, subscriptions, streaming, and push notification delivery code paths. - Keep the existing runner interface and report format while enabling future work to swap in wire-level transport probes and PostgreSQL-backed drivers. ### Description - Add a new C++ SDK-backed driver `tests/performance/a2a_performance_driver.cpp` that executes the TCK-aligned scenarios (task lifecycle, streaming/subscriptions, push notification flows) and emits the existing JSON result shape with extra fields `driver_type` and `transport_path`. - Teach the Python runner in `scripts/performance_runner.py` to locate/build (`cmake`) and invoke the driver per matrix row and to merge the driver's JSON into the existing `results.json`/`results.csv`/`summary.md` artifacts. - Register the driver in `tests/CMakeLists.txt` as `a2a_performance_driver`, update the script test to validate the SDK driver metadata in `results.json` (`tests/scripts/performance_runner_test.py`), and update CI defaults in `.github/workflows/ci.yml` to a small SDK-backed smoke matrix. - Update documentation at `docs/performance-testing.md` to describe which scenarios are SDK-backed, how the driver is built/selected (`A2A_PERF_DRIVER`/`A2A_PERF_BUILD_DIR`), and PostgreSQL handling for performance runs. --------- Co-authored-by: Vladimir Pavlov <mistervvp@outlook.com>
### Motivation - Make the in-process C++ performance driver easier to maintain by centralizing constants, scenario names, and common types. - Ensure measured latency does not include driver bookkeeping or contention from result collection. - Improve concurrency primitives and reuse SDK helpers instead of duplicating logic. ### Description - Added `tests/performance/a2a_performance_driver.h` to centralize constants, scenario names/order, `Options`/`ScenarioResult` types, and function declarations. - Refactored `tests/performance/a2a_performance_driver.cpp` to use the new header, replaced string literals with named `constexpr` values, and moved shared helpers into the `a2a::tests::performance` namespace. - Replaced the push-delivery mutex and integer with a relaxed `std::atomic<int>` counter to avoid unnecessary locking on hot paths. - Replaced `std::async` + shared-result mutex with a fixed worker pool that uses `std::atomic<int>` work distribution and per-thread result buffers so aggregation happens after measured operations finish, avoiding measurement bias from collection locks. - Added `tests/performance/README.md` documenting the driver, measured fields, store backends, and example invocations. --------- Co-authored-by: Vladimir Pavlov <mistervvp@outlook.com>
### Motivation - Reuse the existing SUT for both TCK conformance and wire-level performance tests to avoid duplicated server setup and keep endpoint behavior consistent. - Centralize SUT configuration constants and avoid scattering magic literals across the implementation. - Harden startup/shutdown semantics to make failures and CI diagnosis clearer and to avoid leaking sockets or leaving threads/sockets live after termination. ### Description - Rename the test target from `tck_http_sut` to `tck_sut`, move implementation into `tests/sut/tck_sut.cpp`, and add `tests/sut/tck_sut.h` to centralize defaults, env names, paths, ports, and config types. - Harden SUT parsing and startup with `ParseEndpoint`, clearer port/endpoint validation, socket creation/bind/listen error messages, non-blocking listener setup, accept error handling with retry delays, gRPC startup checks, PostgreSQL store error messages, deterministic SIGINT/SIGTERM shutdown, active-HTTP-connection tracking and shutdown, and graceful gRPC shutdown. - Update test build and scripts: `tests/CMakeLists.txt` builds `tck_sut`; `scripts/run_tck_sut.sh` defaults to `tck_sut` and preserves store-backend behavior; docs updated to reference `tck_sut`. - Extend the performance runner (`scripts/performance_runner.py`) to optionally build/start/stop `tck_sut` per wire matrix entry, wait for HTTP and gRPC ports to be ready, capture SUT logs in the report dir, and mark wire rows with `driver_type=wire_tck_sut` and transport paths (`wire_http_json`, `wire_jsonrpc`, `wire_grpc`). The in-process driver rows remain `driver_type=cpp_sdk_in_process` with `transport_path=in_process`. - Adjust the in-process performance driver metadata (`tests/performance/a2a_performance_driver.*`) and update the performance test script and unit test (`tests/scripts/performance_runner_test.py`) and docs to reflect the new report fields and initial wire-level coverage set.
MisterVVP
added
performance
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Description
scripts/performance_runner.pyand a small wrapperscripts/run_performance_tests.shthat support CLI/env matrix selection (transports, store backends, requests, concurrency, warmup, report dir) and produceresults.json,results.csv, andsummary.md..github/workflows/ci.ymlthat runs the suite in report-only mode, validates outputs, and uploadsperf-artifacts; add docs atdocs/performance-testing.mddescribing usage and configuration.tests/scripts/performance_runner_test.pyand wire it into CTest viatests/CMakeLists.txtso the runner is exercised in the test matrix.