Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ jobs:
env:
A2A_PERF_TRANSPORTS: grpc,jsonrpc,http_json
A2A_PERF_STORE_BACKENDS: inmemory,postgres
A2A_PERF_REQUESTS: 1000
A2A_PERF_REQUESTS: 2000
A2A_PERF_CONCURRENCY: 1,4
A2A_PERF_WARMUP_SECONDS: 0
A2A_PERF_REPORT_DIR: perf-artifacts
Expand Down
4 changes: 2 additions & 2 deletions docs/performance-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ matching environment variables.
| --- | --- | --- |
| Transports (`grpc`, `jsonrpc`, `http_json`, or `all`) | `A2A_PERF_TRANSPORTS` | `grpc,jsonrpc,http_json` |
| Store backends (`inmemory`, `postgres`, or `all`) | `A2A_PERF_STORE_BACKENDS` | `inmemory,postgres` |
| Operations per result row | `A2A_PERF_REQUESTS` | `1000` |
| Operations per result row | `A2A_PERF_REQUESTS` | `2000` |
| Concurrency levels | `A2A_PERF_CONCURRENCY` | `1,4` |
| Warmup seconds | `A2A_PERF_WARMUP_SECONDS` | `1` |
| Duration seconds metadata | `A2A_PERF_DURATION_SECONDS` | `0` |
Expand Down Expand Up @@ -88,7 +88,7 @@ enforce latency or throughput thresholds.
```bash
A2A_PERF_TRANSPORTS=grpc,jsonrpc,http_json \
A2A_PERF_STORE_BACKENDS=inmemory,postgres \
A2A_PERF_REQUESTS=1000 \
A2A_PERF_REQUESTS=2000 \
A2A_PERF_CONCURRENCY=1,4,16,64 \
A2A_PERF_WARMUP_SECONDS=5 \
A2A_PERF_REPORT_DIR=perf-artifacts \
Expand Down
2 changes: 1 addition & 1 deletion scripts/performance_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
"PushNotify_ManyConfigsOneTaskUpdate",
"PushDelivery_CallbackLatency",
)
DEFAULT_REQUESTS = 10_000
DEFAULT_REQUESTS = 2_000
DEFAULT_CONCURRENCY = (1, 4)
DEFAULT_BUILD_DIR = "build/performance"
DRIVER_NAME = "a2a_performance_driver"
Expand Down
48 changes: 48 additions & 0 deletions tests/performance/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# A2A performance driver

`a2a_performance_driver` is the in-process C++ SDK performance driver used by
`scripts/run_performance_tests.sh` and `scripts/performance_runner.py`. It runs a
fixed set of report-only scenarios against the SDK's example executor and prints
a JSON array of per-scenario measurements.

## What it measures

The driver covers task, streaming, subscription, push-configuration, and push
notification paths. Each scenario reports:

- operation counts and success/error totals;
- throughput in operations per second;
- latency percentiles (`p50`, `p90`, `p95`, `p99`) and maximum latency in
milliseconds;
- metadata identifying the transport label, store backend, and SDK dispatch
path.

Warmup operations run before timing starts, so setup and warmup work are excluded
from performance calculations. During measured execution, each worker records its
own counters and latency samples, then the driver aggregates those samples after
workers finish. This avoids adding result-collection mutex contention to the
measured operation latency.

## Store backends

By default, scenarios use the in-memory stores owned by the example executor. To
exercise PostgreSQL-backed stores, run with `--store-backend postgres` and set
`A2A_TEST_POSTGRES_DSN` to a valid connection string. The driver creates a unique
schema name for each PostgreSQL run.

## Example

```bash
./build/tests/a2a_performance_driver \
--transport grpc \
--store-backend inmemory \
--requests 1000 \
--concurrency 4 \
--warmup-seconds 1
```

For the canonical wrapper and generated summary artifacts, prefer:

```bash
./scripts/run_performance_tests.sh --store-backends inmemory --requests 1000 --concurrency 1,4
```
Loading
Loading