[Feature]: Add SDK performance test kit covering core A2A and TCK functionality

[MisterVVP]

Proposal

Background

The SDK now has broad functional and TCK coverage across gRPC, JSON-RPC, and HTTP+JSON transports. We also have targeted microbenchmarks, but we do not yet have a structured performance test kit that measures realistic SDK behavior across the same operation paths covered by the A2A TCK.

We should add a repeatable performance test kit that can be run locally and in CI/reporting mode to track latency, throughput, allocation pressure, and scalability regressions for core SDK functionality.

Goals

Build a performance test kit that covers:

• Core A2A task lifecycle operations.
• Streaming and subscription behavior.
• Push notification storage and delivery paths.
• Transport-level behavior across gRPC, JSON-RPC, and HTTP+JSON.
• Store-backed behavior for both in-memory and PostgreSQL stores.
• Representative operation flows used by the A2A TCK.

The first version should report results without blocking CI. Threshold-based gating can be added later after we collect enough baseline data.

Scope

1. Performance test runner

Add a dedicated performance runner script, for example:

./scripts/run_performance_tests.sh

The runner should support:

• local execution;
• CI execution;
• selecting transport: grpc, jsonrpc, http_json, or all;
• selecting store backend: inmemory, postgres, or all;
• configurable request count, concurrency, warmup, and duration;
• output directory for machine-readable results.

Suggested environment variables:

A2A_PERF_TRANSPORTS=grpc,jsonrpc,http_json
A2A_PERF_STORE_BACKENDS=inmemory,postgres
A2A_PERF_REQUESTS=10000
A2A_PERF_CONCURRENCY=1,4,16,64
A2A_PERF_WARMUP_SECONDS=5
A2A_PERF_DURATION_SECONDS=60
A2A_PERF_REPORT_DIR=perf-artifacts

2. TCK-aligned performance scenarios

Add performance scenarios that map to the same functionality expected by the TCK.

Core task lifecycle:

• SendMessage creates a new task.
• GetTask retrieves an existing task.
• CancelTask cancels a working task.
• ListTasks lists tasks with and without pagination.
• SendMessage follow-up on existing task.
• error path for missing task lookup.

Streaming:

• SendStreamingMessage finite stream.
• SubscribeToTask first-event latency for an existing non-terminal task.
• multiple concurrent subscribers on the same task.
• terminal update delivery and stream completion latency.
• cancellation/disconnect of one subscriber while other subscribers remain active.

Push notifications:

• create push notification config.
• get push notification config.
• list push notification configs.
• delete push notification config.
• notify many configs for one task update.
• push delivery callback latency with a local fake delivery endpoint/client.

Transport coverage:

• run the same logical scenarios across:
  • gRPC;
  • JSON-RPC;
  • HTTP+JSON.

Store coverage:

• run applicable scenarios with:
  • InMemoryTaskStore / InMemoryPushNotificationStore;
  • PostgreSQL-backed task and push notification stores.

3. Metrics

Capture at least:

• total operations;
• success count;
• error count;
• throughput: requests/second or operations/second;
• latency percentiles: p50, p90, p95, p99;
• max latency;
• warmup-adjusted timing;
• concurrency level;
• transport;
• store backend;
• scenario name;
• SDK commit SHA;
• runner host OS and CPU metadata.

Nice-to-have:

• allocations per operation for microbenchmarks where practical;
• RSS memory before/after long-running streaming tests;
• number of active subscribers;
• publish-to-delivery latency for subscriptions;
• push notification delivery latency.

4. Output format

Write machine-readable output to:

perf-artifacts/results.json
perf-artifacts/results.csv
perf-artifacts/summary.md

summary.md should be human-readable and suitable for CI artifacts.

The JSON result should include one object per scenario/transport/backend/concurrency combination.

Example shape:

{
  "scenario": "SubscribeToTask_MultiSubscriber",
  "transport": "http_json",
  "store_backend": "inmemory",
  "concurrency": 16,
  "operations": 10000,
  "success": 10000,
  "errors": 0,
  "throughput_ops_per_sec": 1234.5,
  "latency_ms": {
    "p50": 2.1,
    "p90": 4.8,
    "p95": 6.2,
    "p99": 12.7,
    "max": 30.4
  }
}

5. CI integration

Add a CI workflow/job that runs the performance test kit in report-only mode.

Initial CI behavior:

• run a short smoke-sized performance suite;
• upload perf-artifacts;
• do not fail on performance thresholds yet;
• fail only on functional errors, crashes, timeouts, or malformed reports.

Future behavior:

• add baseline comparison;
• add broad rounded thresholds;
• fail only on meaningful regressions.

6. Benchmark alignment

Review existing benchmark targets and align naming/results with the new performance kit.

Existing microbenchmarks should stay useful for isolated components. The new kit should focus on realistic SDK operation flows and transport/store integration behavior.

Non-goals

• Do not replace TCK functional compatibility tests.
• Do not add strict performance gates in the first version.
• Do not depend on external network services.
• Do not benchmark third-party infrastructure instability as SDK performance.

Suggested implementation order

1. Add performance test runner skeleton and report format.
2. Add in-memory core task lifecycle scenarios.
3. Add transport matrix for gRPC, JSON-RPC, and HTTP+JSON.
4. Add streaming/subscription scenarios.
5. Add push notification scenarios.
6. Add PostgreSQL backend coverage.
7. Add CI artifact upload in report-only mode.
8. Add documentation with local usage examples.
9. Add optional baseline/threshold support later.

Acceptance criteria

• A developer can run performance tests locally with one command.
• CI runs a short report-only performance suite and uploads artifacts.
• Results are emitted as JSON, CSV, and Markdown.
• The suite covers TCK-relevant functionality:
  • task send/get/cancel/list;
  • streaming send;
  • task subscription;
  • multi-subscriber behavior;
  • push notification config CRUD;
  • push update delivery path;
  • gRPC, JSON-RPC, and HTTP+JSON transports.
• Both in-memory and PostgreSQL store backends are covered where applicable.
• The initial CI job does not fail because of missing thresholds or normal performance variance.
• The implementation is documented in docs/performance-testing.md.