Skip to content

Add core query APIs (listTransactions/listEvents/listCheckpoints) backed by new gRPC List RPCs#1137

Open
hayes-mysten wants to merge 3 commits into
mainfrom
core-query-apis
Open

Add core query APIs (listTransactions/listEvents/listCheckpoints) backed by new gRPC List RPCs#1137
hayes-mysten wants to merge 3 commits into
mainfrom
core-query-apis

Conversation

@hayes-mysten

@hayes-mysten hayes-mysten commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Description

Adds support for the new filtered query APIs from sui-apis#29, which close the gap on JSON-RPC query functionality (queryTransactionBlocks, queryEvents, getCheckpoints) ahead of the JSON-RPC shutdown.

Core API query methods

New listTransactions and listEvents core API methods that behave identically on all three transports, enforced by parity e2e tests using the allClients helpers. The core surface is intentionally restricted to what every transport can support exactly:

  • Filters take exactly one predicate: sender/function for transactions, sender/emitModule/eventType for events (emitModule and eventType are validated to be module-qualified everywhere, since JSON-RPC requires a module).
  • Pagination uses explicit after/before position bounds (matching the gRPC QueryOptions and Relay semantics) with startCursor/endCursor returned for both ends of each page. This makes activity feeds natural: read recent items with order: 'descending', page older with before: endCursor, and poll for new items with after: startCursor — verified by a cross-client polling e2e test. Until JSON-RPC is gone, a query accepts one bound matching its traversal direction (validated identically on all transports); windowed after+before queries are tracked in the follow-ups doc.
  • Filter validation and MVR-name resolution run through shared helpers (src/client/query-filters.ts) so errors and resolution behave identically on every transport (including client-side on gRPC).
  • EventEntry.checkpoint is string | nullnull on JSON-RPC, whose queryEvents responses carry no checkpoint info.

Capabilities that only some transports can support (affectedAddress, affectedObject — rejected as unsupported by JSON-RPC nodes — checkpoint ranges, combined/negated/DNF predicates) are not part of the API surface, per review discussion. They remain reachable through the raw gRPC ledgerService RPCs, and are tracked in packages/sui/JSON_RPC_SUNSET_FOLLOW_UPS.md for promotion into core once JSON-RPC is removed (with notes on which items are additionally gated on GraphQL support). A listCheckpoints core method was prototyped and dropped per review discussion — checkpoint summaries alone aren't very useful, and transactions-per-checkpoint is better served by listTransactions with checkpoint ranges once those are promoted; it's tracked in the follow-ups doc, and getCheckpoints maps to the raw ledgerService.listCheckpoints RPC in the migration guide.

gRPC client

  • Regenerated protos from sui-apis#29: streaming ListCheckpoints/ListTransactions/ListEvents, new SubscribeTransactions/SubscribeEvents (plus filter on SubscribeCheckpoints), ledger-position fields on Event/ExecutedTransaction, and the lossless ProtocolConfig.configs map. All available on the raw service clients.
  • The gRPC core implementations are backed by the new streaming List RPCs; pagination is driven by Watermark cursors with hasNextPage derived from the QueryEnd reason.
  • The repeated transaction read-mask construction across getTransaction/executeTransaction/simulateTransaction/listTransactions is extracted into a shared transactionReadMaskPaths helper.

BCS: system transaction kinds

Unfiltered listTransactions surfaced that the SDK's BCS TransactionKind defined system-transaction variants as payload-less, so parsing any system transaction (starting with genesis) misaligned and threw. The complete definitions (genesis, change epoch, consensus commit prologues V1–V4, authenticator/randomness state updates, end-of-epoch kinds, programmable system transactions) now live in a new src/bcs/transactions.ts (mirroring how effects are handled), along with the TransactionKind/TransactionData/SenderSignedData definitions that depend on them — bcs.ts only has those definitions removed. This also fixes getTransaction on system transactions via JSON-RPC. The JSON-RPC transport additionally stops reporting the genesis transaction's placeholder signature, matching the other transports (verified empirically: GraphQL reports the placeholder for consensus prologues but not genesis).

Docs

  • Migration guide maps queryTransactionBlockslistTransactions, queryEventslistEvents, getCheckpointslistCheckpoints, documents the JSON-RPC filter mapping, and adds a websocket-subscription → gRPC subscriptionService section.
  • Core API docs gained a "Query methods" section; gRPC docs cover the raw List RPCs' DNF filters and the subscription service.
  • Refreshed the GraphQL schema (no new filter capabilities were available).

Test plan

  • Parity e2e (test/e2e/clients/core/queries.test.ts): every option and combination — no filter, each predicate at each specificity, all includes, ascending/descending, multi-page pagination, event ledger positions, and filter-validation errors — compared across clients with expectAllClientsReturnSameData (which gained an exclude option) and testWithAllClients. 31 passing against the docker localnet; the gRPC server-dependent variants are excluded via a single EXCLUDE constant until SUI_TOOLS_TAG points at an image that serves the new List RPCs.
  • Unit tests for the shared filter validation/resolution helpers and the gRPC filter conversions.
  • pnpm turbo build (full workspace), pnpm --filter @mysten/sui test, the full test/e2e/clients/core/ suite, wallet-sdk tests, oxlint, prettier, and docs validation all pass.

Known parity caveat (documented in the follow-ups doc)

  • MoveEventType generic-type matching semantics may differ between JSON-RPC and gRPC/GraphQL; parity tests cover non-generic event types.

AI Assistance Notice

Please disclose the usage of AI. This is primarily to help inform reviewers of how careful they need to review PRs, and to keep track of AI usage across our team. Please fill this out accurately, and do not modify the content or heading for this section!

  • This PR was primarily written by AI.
  • I used AI for docs / tests, but manually wrote the source code.
  • I used AI to understand the problem space / repository.
  • I did not use AI for this PR.

🤖 Generated with Claude Code

@hayes-mysten
hayes-mysten requested a review from a team as a code owner July 13, 2026 19:12
@hayes-mysten
hayes-mysten temporarily deployed to sui-typescript-aws-kms-test-env July 13, 2026 19:12 — with GitHub Actions Inactive
@vercel

vercel Bot commented Jul 13, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
sui-typescript-docs Ready Ready Preview, Comment Jul 15, 2026 12:11am

Request Review

@jessiemongeon1

jessiemongeon1 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Style Guide Audit

All 3 file(s) pass the style guide audit.

@hayes-mysten

Copy link
Copy Markdown
Contributor Author

@clud-bot review

@clud-bot

clud-bot Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Review sub-agent spawned. I'll post the verdict once it reports back.

Adds core API query methods backed by the new gRPC filtered List
streaming RPCs from sui-apis#29, with identical behavior across the
gRPC, GraphQL, and JSON-RPC transports enforced by cross-client parity
e2e tests. Filters take a single predicate resolved through shared
validation and MVR-resolution helpers, and pagination uses explicit
after/before position bounds with startCursor/endCursor returned for
both ends of each page, enabling activity feeds that page older items
and poll for new ones. Capabilities not expressible on every transport
are tracked in JSON_RPC_SUNSET_FOLLOW_UPS.md for promotion once
JSON-RPC is removed.

Also regenerates the gRPC protos (adding SubscribeTransactions and
SubscribeEvents), completes the BCS TransactionKind definitions so
system transactions parse correctly (in a new bcs/transactions.ts), and
updates the JSON-RPC migration guide to cover the queries that
previously required raw GraphQL.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n filters

The queries e2e file now publishes its own copy of the test package so
package-scoped filters only match its own transactions, since CI runs
all e2e files in parallel against one localnet. This surfaced two
transport findings, both documented in JSON_RPC_SUNSET_FOLLOW_UPS.md:
the JSON-RPC MoveFunction filter matches aborted transactions while the
GraphQL function filter does not, and the JSON-RPC server rejects
cursors on package- or module-level MoveFunction filters. The latter is
now validated identically on every transport: pagination bounds with a
function filter require a fully qualified package::module::function.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@clud-bot clud-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — Core query APIs (listTransactions/listEvents)

Verdict: COMMENT — no blockers. @mysten/sui typechecks clean; JSON-RPC and GraphQL parity logic is correct on the paths CI actually exercises. The real risk is concentrated in the gRPC path, which is excluded from the e2e parity matrix, plus one genuine default-limit parity gap. Shippable with follow-ups, but the gRPC parity claim is currently unproven.

Major

  • GraphQL default page-size asymmetry (graphql/core.ts, txns + events): omitted limit yields different page sizes ascending vs descending, and GraphQL vs gRPC/JSON-RPC — violates the identical-across-transports invariant. Never fires in CI because tests always pass a limit. Inline comment has detail.

Minor

  • gRPC endCursor tracks the scan frontier rather than the last item on non-empty final pages (grpc/core.ts); also a null-startCursor robustness edge. Cursors are opaque so continuation still works.
  • GraphQL event mapper skips normalizeSuiAddress on packageId/sender (defensive parity).
  • JSON-RPC JSON.parse(cursor) is unguarded (robustness vs gRPC's fromBase64).

Test coverage gap (the thing to close before trusting parity)

  • queries.test.ts: EXCLUDE = ['grpc'] leaves the most complex new code (streaming frame accumulation, watermark/cursor tracking, QueryEnd-derived hasNextPage) uncovered by the parity matrix. Both gRPC findings above live in this untested path. Enabling gRPC here is the gating follow-up.

Positives

  • Shared filter validation + MVR resolution centralized in query-filters.ts; split('::') edge cases and assertSinglePredicate are sound.
  • BCS TransactionKind/EndOfEpochTransactionKind variant ordering matches the regenerated proto enums; moving them to transactions.ts with full payloads (vs the old payload-less null variants) is correct and necessary, and index.ts re-exports are intact.
  • Docs, migration guide, and the minor changeset accurately match the real public API. No leaked stream iterators on abort.

Confidence on the two highest-risk areas

  • Cross-transport parity: HIGH for JSON-RPC + GraphQL (covered by e2e), LOW–MEDIUM for gRPC (CI-excluded, both gRPC findings unguarded).
  • BCS layout: MEDIUM. Top-level enum orders are strongly corroborated by the regenerated protos, but a human should verify against Rust (crates/sui-types) / sui-apis#29 for the nested/newtype details: ConsensusDeterminedVersionAssignments tuple nesting, StoredExecutionTimeObservations wrapper + AuthorityName encoding, ConsensusCommitPrologueV4 subDagIndex position, GenesisObject variant completeness, and digest-newtype BCS equivalence for BridgeStateCreate/consensusCommitDigest/additionalStateDigest. A misordered field here corrupts system-transaction parsing silently.

filter.$kind === 'function'
? [filter.package, filter.module, filter.function].filter(Boolean).join('::')
: undefined,
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major — default page-size asymmetry breaks cross-transport parity.

When limit is omitted, ascending sends first: undefined (→ GraphQL server default) but descending sends last: options.limit ?? 50. So on GraphQL, ascending and descending return different page sizes when no limit is given, and GraphQL diverges from gRPC/JSON-RPC too — a direct violation of the "identical across transports" invariant.

It never trips CI because every parity test passes an explicit limit, but the docs' "poll for new items" example omits it. Same pattern at listEvents (:676). Suggest a single shared default limit applied for both directions across all three transports.


const transactions: SuiClientTypes.TransactionResult<Include>[] = [];
let startCursor: string | null = null;
let endCursor: string | null = null;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor — endCursor tracks the scan frontier, not the last item.

endCursor is overwritten on every watermark-bearing frame, including terminal non-item frames (LEDGER_TIP / CURSOR_BOUND). On a non-empty final page it ends up as the tip cursor rather than the last returned item's cursor, diverging semantically from JSON-RPC/GraphQL which report the last item's position. Continuation still works (cursors are opaque), so no user-visible break — but it's exactly the gRPC-only detail that no test guards since gRPC is excluded from the parity matrix.

Suggest only updating endCursor from a watermark on frames that also carry an item, mirroring the startCursor ??= placement. Same at listEvents (:625).

Separately (minor/robustness): startCursor ??= endCursor on a data frame will capture null if a data frame ever arrives with a falsy watermark.cursor. The proto says a watermark is present on every frame, so this only fires on a server contract violation — worth treating a data frame without a resume cursor as an error rather than silently emitting a null start cursor.

// Backwards pagination returns nodes in ascending order, so reverse them for descending reads
const nodes = descending ? [...events.nodes].reverse() : events.nodes;

return {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor — event mapper skips normalizeSuiAddress.

packageId (:689) and sender (:691) are taken raw from ...address, while the gRPC and JSON-RPC mappers normalize both. Sui GraphQL returns canonical addresses so results match in practice; a short/system address like 0x2 is the only way to surface a diff. Wrap both in normalizeSuiAddress for defensive parity.

async listTransactions<Include extends SuiClientTypes.TransactionInclude = {}>(
options: SuiClientTypes.ListTransactionsOptions<Include>,
): Promise<SuiClientTypes.ListTransactionsResponse<Include>> {
const filter = options.filter

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor — unguarded JSON.parse(cursor).

A malformed cursor throws an unhandled SyntaxError, and the as EventId cast has no runtime validation. Not exploitable, but a robustness gap versus gRPC's fromBase64. Consider a try/catch that surfaces a clear "invalid cursor" error.

// TODO: Set this to [] once SUI_TOOLS_TAG points at an image that serves the
// gRPC ListCheckpoints/ListTransactions/ListEvents RPCs, so these tests cover
// all three transports.
const EXCLUDE: Array<'jsonrpc' | 'grpc' | 'graphql'> = ['grpc'];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test coverage gap — gRPC is excluded from the parity matrix.

EXCLUDE = ['grpc'] means the streaming gRPC implementation — the most complex new code (frame accumulation, watermark/cursor tracking, QueryEnd-derived hasNextPage) — isn't covered by the parity tests. Both gRPC findings above live entirely in this untested path, and the changeset/docs parity claim is currently unverified for gRPC. The TODO correctly notes this is blocked on a tools image; flagging it as the gating follow-up before the gRPC parity claim can be trusted.

Applies a shared default page size (50) through resolvePagination so
every transport requests the same page size instead of falling back to
differing server defaults, fixing the ascending/descending asymmetry on
GraphQL. gRPC endCursor now reports the last item's position (matching
the other transports) and only falls back to the scan frontier for
item-less in-progress scans where it's required for continuation. Also
normalizes event addresses in the GraphQL mapper, guards event cursor
parsing on JSON-RPC with a clear invalid-cursor error, and fixes the
doc style-guide violations flagged in review.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hayes-mysten
hayes-mysten temporarily deployed to sui-typescript-aws-kms-test-env July 15, 2026 00:09 — with GitHub Actions Inactive
@hayes-mysten

Copy link
Copy Markdown
Contributor Author

Addressed the review feedback in 1444068:

  • Default page-size asymmetry (major): fixed. resolvePagination now applies a shared DEFAULT_PAGE_SIZE = 50, so every transport always sends an explicit, identical page size instead of falling back to differing server defaults (and GraphQL's ascending/descending paths now agree). Documented in the core API options table and covered by unit tests.
  • gRPC endCursor frontier drift: fixed. endCursor now tracks the last item's frame (matching JSON-RPC/GraphQL semantics), and the scan frontier is only used for item-less in-progress scans (SCAN_LIMIT), where it's required for correct continuation. Since ITEM_LIMIT pages end on the final item's frame, this loses no resume efficiency.
  • GraphQL event address normalization: packageId and sender are now wrapped in normalizeSuiAddress for defensive parity.
  • JSON-RPC cursor parsing: parseEventCursor validates shape and surfaces a clear Invalid event cursor error instead of an unhandled SyntaxError.
  • gRPC coverage gap: acknowledged — this is the gating follow-up, tracked by the EXCLUDE constant TODO and in JSON_RPC_SUNSET_FOLLOW_UPS.md. Flipping EXCLUDE to [] once SUI_TOOLS_TAG serves the List RPCs puts the streaming path (frame accumulation, watermark cursors, QueryEnd-derived hasNextPage) under the full parity matrix.
  • BCS layout verification: the definitions were transcribed from crates/sui-types in a local sui checkout (transaction.rs, messages_consensus.rs, execution.rs, authenticator_state.rs), and genesis + consensus-prologue parsing is exercised end-to-end by the unfiltered parity e2e (JSON-RPC parses real system transactions from the localnet). The end-of-epoch variants and StoredExecutionTimeObservations payloads aren't exercised by localnet, so a human double-check of those nested layouts is still worthwhile before merge.

Style-guide audit violations are also fixed (all 12).

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants