Refactor shared TCK SUT and wire performance runner#140
Merged
MisterVVP merged 4 commits intoJul 9, 2026
Merged
Conversation
MisterVVP
added
the
codex
ChatGPT Codex Connector
Owner
Author
|
More refactoring incoming |
### Motivation - Reuse the existing SUT for both TCK conformance and wire-level performance tests to avoid duplicated server setup and keep endpoint behavior consistent. - Centralize SUT configuration constants and avoid scattering magic literals across the implementation. - Harden startup/shutdown semantics to make failures and CI diagnosis clearer and to avoid leaking sockets or leaving threads/sockets live after termination. ### Description - Rename the test target from `tck_http_sut` to `tck_sut`, move implementation into `tests/sut/tck_sut.cpp`, and add `tests/sut/tck_sut.h` to centralize defaults, env names, paths, ports, and config types. - Harden SUT parsing and startup with `ParseEndpoint`, clearer port/endpoint validation, socket creation/bind/listen error messages, non-blocking listener setup, accept error handling with retry delays, gRPC startup checks, PostgreSQL store error messages, deterministic SIGINT/SIGTERM shutdown, active-HTTP-connection tracking and shutdown, and graceful gRPC shutdown. - Update test build and scripts: `tests/CMakeLists.txt` builds `tck_sut`; `scripts/run_tck_sut.sh` defaults to `tck_sut` and preserves store-backend behavior; docs updated to reference `tck_sut`. - Extend the performance runner (`scripts/performance_runner.py`) to optionally build/start/stop `tck_sut` per wire matrix entry, wait for HTTP and gRPC ports to be ready, capture SUT logs in the report dir, and mark wire rows with `driver_type=wire_tck_sut` and transport paths (`wire_http_json`, `wire_jsonrpc`, `wire_grpc`). The in-process driver rows remain `driver_type=cpp_sdk_in_process` with `transport_path=in_process`. - Adjust the in-process performance driver metadata (`tests/performance/a2a_performance_driver.*`) and update the performance test script and unit test (`tests/scripts/performance_runner_test.py`) and docs to reflect the new report fields and initial wire-level coverage set. ### Testing - Ran formatting checks with `./scripts/run_clang_format.sh` and `clang-format --dry-run --Werror` on repository C++ files (passed). - Built and exercised targets with CMake: configured and built `tck_sut` and `a2a_performance_driver` (build succeeded). - Ran the performance runner unit script `python3 tests/scripts/performance_runner_test.py` (tests passed). - Verified `./scripts/run_tck_sut.sh` can build and start the `tck_sut` binary and reports readiness (succeeded), and `./scripts/stop_tck_sut.sh` stops it cleanly. - Ran `./scripts/verify_changes.sh` which performed format, full build, and CTest (371 tests) and completed the test run (371/371 passed); clang-tidy initially flagged two magic-number issues in the new SUT code which were fixed and the touched-file clang-tidy was rerun successfully. - Attempted the local mandatory TCK flow but `run_tck_mandatory.sh` could not locate a pinned TCK entrypoint in the external TCK repo (the script requests `TCK_RUN_CMD`), so the mandatory-suite execution was not completed locally.
### Motivation - Provide a single reusable `tck_sut` SUT binary to unify TCK conformance and wire-level performance coverage and keep endpoint layouts consistent across flows. - Add robust socket utilities and parsing utilities to safely parse endpoints, build HTTP URLs, and configure non-blocking listeners. - Extend the performance runner to exercise both in-process SDK paths and real wire transports against the shared SUT for initial wire coverage. ### Description - Added a shared test SUT target `tck_sut` with a new header `tests/sut/tck_sut.h` and reorganized the previous `tck_http_sut` into `tests/sut/tck_sut.cpp`, improving endpoint parsing, error handling, non-blocking accept, and startup/shutdown behavior. - Introduced `a2a::server::HostPortEndpoint`, `BuildHttpUrl`, `ParseHostPortEndpoint`, and `SetSocketNonBlocking` in `include/a2a/server/socket_utils.h` and implemented them in `src/server/socket_utils.cpp` with validation and POSIX/Windows handling. - Extended the Python `scripts/performance_runner.py` to build/locate and start `tck_sut`, wait for HTTP and gRPC readiness, capture `tck_sut_<store>_<port>.log`, annotate wire-level results (`driver_type=wire_tck_sut` and `transport_path=*`), and include both in-process and wire rows in reports and CSV/markdown output. - Updated CMake to add the `tck_sut` executable and wire-related test targets, renamed/added tests and test sources, added `socket_utils_test` unit test, and adjusted the in-process performance driver to emit `transport_path="in_process"` consistently. - Documentation updates in `docs/compliance/README.md` and `docs/performance-testing.md` describing the shared `tck_sut` binary, endpoints, environment variables, and how wire-level coverage is reported. ### Testing - Built the project and CMake targets and produced `a2a_performance_driver` and `tck_sut` binaries successfully using the CMake build flow; build succeeded. - Ran unit tests including the new `socket_utils_test` (via the gtest/CTest integration) and the existing server unit tests; all gtests passed. - Executed the Python integration test `tests/scripts/performance_runner_test.py` which runs the runner script and validates `results.json`, `results.csv`, and `summary.md`; the test passed and verified combined in-process and wire rows were present.
…performance runner integration ### Motivation - Provide a single SUT binary reused by TCK conformance and wire-level performance tests so endpoint behavior is consistent across validation flows. - Add wire-level coverage to the performance matrix by driving the shared `tck_sut` over real HTTP/JSON, JSON-RPC and gRPC transports. - Replace the small ad-hoc socket helper with a richer, cross-platform `network_utils` module for host:port parsing, URL building, and socket helpers used by the SUT. ### Description - Added `include/a2a/server/network_utils.h` and `src/server/network_utils.cpp` and removed the old `socket_utils` files, providing `CloseSocketCrossPlatform`, `BuildHttpUrl`, `ParseHostPortEndpoint`, and `SetSocketNonBlocking`. - Reworked the TCK SUT: renamed/moved `tck_http_sut` to `tests/sut/tck_sut.cpp`, introduced `tests/sut/tck_sut.h`, improved endpoint parsing, error reporting, non-blocking HTTP listener setup, robust accept loop, and a `RunTckSut` wrapper with exception-safe `main`. - Extended the performance runner `scripts/performance_runner.py` to build/run a shared `tck_sut` (`SutProcess`), wait for readiness, capture startup logs, emit wire-level results (`driver_type=wire_tck_sut` and `transport_path`), and include both in-process and wire rows in the reports and CSV/summary output. - CMake and tests updates: build target `tck_sut` added and linked, `src/CMakeLists.txt` switched to `network_utils.cpp`, `tests/CMakeLists.txt` registers `network_utils_test`, `tests/unit/network_utils_test.cpp` added, and `tests/scripts/performance_runner_test.py` updated to assert the new wire rows and CSV/summary layout. - Documentation and helper scripts updated to describe the shared SUT: `docs/compliance/README.md`, `docs/performance-testing.md`, and `scripts/run_tck_sut.sh` default target changed to `tck_sut`. ### Testing - Ran the Python performance runner unit test `tests/scripts/performance_runner_test.py` (which invokes the runner script) and it passed, validating the combined in-process and wire result rows. - Ran the C++ unit test `network_utils_test` via the gtest/CTest discovery and it passed, validating `BuildHttpUrl` and `ParseHostPortEndpoint` behaviors. - Verified the build of `tck_sut` and `a2a_performance_driver` targets during test runs succeeded and produced the expected artifacts (log and `results.json`).
MisterVVP
merged commit 9a5799d
into
dev-add-performance-test-kit-for-sdk
16 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Description
tck_http_suttotck_sut, move implementation intotests/sut/tck_sut.cpp, and addtests/sut/tck_sut.hto centralize defaults, env names, paths, ports, and config types.ParseEndpoint, clearer port/endpoint validation, socket creation/bind/listen error messages, non-blocking listener setup, accept error handling with retry delays, gRPC startup checks, PostgreSQL store error messages, deterministic SIGINT/SIGTERM shutdown, active-HTTP-connection tracking and shutdown, and graceful gRPC shutdown.tests/CMakeLists.txtbuildstck_sut;scripts/run_tck_sut.shdefaults totck_sutand preserves store-backend behavior; docs updated to referencetck_sut.scripts/performance_runner.py) to optionally build/start/stoptck_sutper wire matrix entry, wait for HTTP and gRPC ports to be ready, capture SUT logs in the report dir, and mark wire rows withdriver_type=wire_tck_sutand transport paths (wire_http_json,wire_jsonrpc,wire_grpc). The in-process driver rows remaindriver_type=cpp_sdk_in_processwithtransport_path=in_process.tests/performance/a2a_performance_driver.*) and update the performance test script and unit test (tests/scripts/performance_runner_test.py) and docs to reflect the new report fields and initial wire-level coverage set.