Add comprehensive RAM billing and KV coverage tests by heifner · Pull Request #270 · Wire-Network/wire-sysio

heifner · 2026-03-23T23:31:56Z

Summary

Retargeted onto feature/kv-secondary-primary-id; secondary index uses the int64_t primary_id signature.
44 new tests covering RAM billing, iterator invalidation, cross-contract reads, key/value boundaries, and primary_id validation/lifecycle.
New test contract actions for parameterized billing, notification handlers, boundary testing, and primary_id edge cases.
ROA limit and privileged contract bypass tests in protocol_feature_tests.
Uses test_table_id constexpr (not raw literals). All new table_ids live in the uint16_t namespace (300-308) so checked_table_id never rejects them.
Shared ram_idx_val / ram_idx_val_size constants replace the duplicated "v\0" payload magic on both contract and test sides.

Test Coverage

Category	Tests	What it validates
Payer change billing	4	Correct refund/charge on payer change (same size, grow, shrink, back-to-self)
Erase refund	1	Refunds stored payer, not receiver
Secondary index billing	4	Correct amounts for idx store/remove/update
Cross-account idx billing	3	Idx bills payer (not contract), refunds correctly
Read-only rejection	4	erase, idx_store/remove/update blocked (table_operation_not_permitted)
Payer auth edge cases	3	Old payer unauth ok, missing auth fails, active-only (no sysio.payer) fails
Notification billing	2	Self-pay ok, third-party blocked in notify
Mixed-payer	2	Independent deltas, net zero
ROA limit	1	Quota exceeded -> ram_usage_exceeded
Privileged bypass	1	sysio.* bypasses payer auth, non-sysio.* does not
Key/value boundaries	6	Max and oversize on both create and update
Iterator invalidation	6	Primary + secondary mutation during iteration
Cross-contract reads	1	Secondary index read via kv_idx_primary_key + kv_get across contracts
primary_id validation	3	Negative pid, nonexistent pid, cross-contract pid all rejected
primary_id lifecycle	3	Orphan after primary erase, id stable on update, id fresh after recreate

38 new tests covering: - Payer change billing (create, grow, shrink, back-to-self) - Erase refunds correct stored payer - Secondary index billing (store, remove, update amounts) - Cross-account secondary index billing (payer, not receiver) - Read-only transaction rejection (erase, idx_store/remove/update) - Payer authorization edge cases (unauth, active-only, old payer unauth) - Notification context billing (self-pay, third-party blocked) - Mixed-payer transactions (independent deltas, net zero) - ROA limit enforcement on KV operations - Privileged sysio.* contract payer bypass - Key/value size boundaries on both create and update - Iterator invalidation under mutation (primary + secondary) - Cross-contract secondary index reads Also use key_format constexpr consistently in test_kv_api contract.

Copilot

Pull request overview

This PR expands KV/RAM-related unit coverage by adding new contract actions and host-side tests that exercise RAM billing edge cases (payer changes, secondary index billing), iterator mutation semantics, cross-contract secondary-index reads, ROA quota enforcement, and privileged payer-authorization bypass behavior.

Changes:

Added new test_kv_api contract actions to parameterize payer-billing scenarios, read-only rejection paths, notification-context billing, iterator invalidation under mutation, and key/value boundary cases (create + update).
Extended kv_api_tests.cpp with comprehensive RAM billing tests (primary + secondary, cross-account), read-only transaction write rejection tests, notification billing tests, iterator mutation tests, and cross-contract read tests.
Added protocol feature tests for ROA RAM ceiling enforcement via KV paths and for privileged sysio.* payer-authorization bypass behavior.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 3 comments.

File	Description
unittests/test-contracts/test_kv_api/test_kv_api.cpp	Adds new test actions for boundary enforcement, payer-parameterized billing, read-only rejection, notification billing, iterator mutation, and cross-contract reads; updates callers to use `key_format`.
unittests/test-contracts/test_kv_api/test_kv_api.abi	Updates ABI to include the newly added `test_kv_api` actions/structs used by host-side tests.
unittests/protocol_feature_tests.cpp	Adds ROA-limit enforcement test for KV paths and privileged payer-auth bypass tests for `sysio.` vs non-`sysio.` privileged contracts.
unittests/kv_api_tests.cpp	Adds host-side Boost tests covering KV RAM billing, payer changes, secondary index billing, read-only rejection, notification billing, iterator mutation, and cross-contract reads.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-02T16:19:26Z

+      memset(key, 0xB0, 256);
+      kv_set(key_format, 0, key, 256, "v1", 2);  // create
+      kv_set(key_format, 0, key, 256, "v2", 2);  // update — same key, different value
+      char buf[4]; uint32_t actual = 0;


tstmaxkeyupd declares uint32_t actual but never uses it, which can trigger -Wunused-variable (and potentially fail builds when warnings are treated as errors). Remove the variable, or use it (e.g., by switching to an API that returns an actual-size outparam if intended).

Suggested change

char buf[4]; uint32_t actual = 0;

char buf[4];

Copilot · 2026-04-02T16:19:26Z

+
+   // Send notification — receiver's on_notify handler does self-pay kv_set
+   [[sysio::action]]
+   void ramnotify(uint32_t key_id, uint32_t val_size) {


ramnotify doesn't use key_id or val_size, which can trigger -Wunused-parameter warnings in contract builds. Consider marking them [[maybe_unused]] or explicitly casting to void to keep builds warning-clean while still keeping the parameters in the action data for the notification handler.

Suggested change

void ramnotify(uint32_t key_id, uint32_t val_size) {

void ramnotify([[maybe_unused]] uint32_t key_id, [[maybe_unused]] uint32_t val_size) {

Copilot · 2026-04-02T16:19:26Z

+
+   // Send notification — receiver's handler tries third-party payer
+   [[sysio::action]]
+   void ramnotiferr(uint32_t key_id, uint32_t val_size, sysio::name payer) {


ramnotiferr doesn't use its parameters (key_id, val_size, payer), which can trigger -Wunused-parameter warnings in contract builds. Mark unused parameters [[maybe_unused]] or cast them to void so warnings don’t become errors under stricter build settings.

Suggested change

void ramnotiferr(uint32_t key_id, uint32_t val_size, sysio::name payer) {

void ramnotiferr(uint32_t key_id, uint32_t val_size, sysio::name payer) {

(void)key_id;

(void)val_size;

(void)payer;

Retarget PR #270 onto the primary_id branch so kv_idx_* signatures match. test_kv_api.cpp: take target's refactor as base, add branch-unique actions adapted for the int64_t primary_id signature (capture pid from kv_set first, pass to kv_idx_store/remove/update). Added ramidxrmpyr for payer-preserving removes. All new actions use small uint32_t table_ids (300-305) within the uint16_t checked_table_id cap. kv_api_tests.cpp: keep target's idx billing tests; retarget this branch's idx billing formulas for the primary_id model -- kv_idx_store now bills only sec_size + KV_INDEX_OVERHEAD, and ramidxstore creates a primary row alongside. idx_update_payer_change reflects the full payer transfer (primary + secondary). Dropped duplicate KV_IDX_OVERHEAD; added no-arg get_ram_usage() overload; fixed read-only tests to use t. prefix. test_kv_api.wasm/.abi regenerated with feature/kv-secondary-primary-id CDT. All 80 kv_api_tests pass under --sys-vm and --sys-vm-jit.

Six new kv_api_tests exercising the surface introduced when secondary index entries switched from storing primary key bytes to storing a primary_id reference: * idx_store_negative_primary_id_rejected -- kv_key_not_found when pid < 0 * idx_store_nonexistent_primary_id_rejected -- kv_key_not_found when the referenced row does not exist * idx_store_cross_contract_primary_rejected -- table_operation_not_permitted when one contract references another contract's primary row * idx_orphan_after_primary_erase -- kv_idx_primary_key returns iterator_erased (status 2) after the primary is deleted out from under a live secondary iterator * primary_id_stable_on_update -- kv_set on an existing key returns the same chainbase id (secondary refs remain valid across value updates) * primary_id_differs_after_recreate -- erase + recreate yields a new id (chainbase ids are monotonic, never reused) Adds supporting actions in test_kv_api.cpp: tstidxbadpid, tstxcprep, tstxcbadidx, tstorphansec, tstpidstable, tstpidrecyc. The cross-contract test uses the existing kv_notify_billing_tester fixture (same WASM deployed on kvtest and kvnotify).

* xcidxread now materializes the foreign primary key via kv_idx_primary_key instead of only kv_get on a hardcoded literal -- exercises the lazy cross-code primary resolution path. * Iterator-mutation actions use private primary-key prefixes so they can't alias each other at test_table_id. * idx_store_nonexistent_primary_id_rejected uses INT64_MAX (was 999999999) so fixture growth can't turn it into a false pass. * ram_idx_val / ram_idx_val_size named constants replace the duplicated "v\0" magic on both sides. * get_ram_usage marked const; auto -> int64_t on RAM-usage locals; UPPER_CASE locals renamed to snake_case; kv_idx_key status values now captured; int32_t handles cast to uint32_t on CDT calls; char buffers value-initialized. * ram_pyr_set drops the unreachable test_account branch and documents name{0} semantics; payer-auth comments describe validate_account_ram_deltas directly instead of referencing "Path 1/Path 2". * tstrdoidxst/rm/up pass primary_id = -1 so an assertion reorder surfaces. * Dedicated table_ids: orphan_tbl_id, xc_bad_tbl_id, pid_life_tbl_id. * on_ramnotiferr documents its fresh-fixture assumption.

payer_choice_test/kv_idx_payer_billing and protocol_feature_tests/kv_roa_limit_enforcement + protocol_feature_tests/privileged_kv_payer_bypass began failing with ram_usage_exceeded at set_code because their contract-deploy ROA allocations were tuned to wasms that have since grown: - test_kv_api.wasm: 122042 -> 122517 bytes in 05509af (pr-review coverage additions). kvtest needs 1226398 bytes; 1.1000 SYS gave 1145144. Bump to 1.2000 SYS (~1249248 bytes, ~22KB headroom). - ram_restrictions_test.wasm: 9720 -> 10000 bytes since the 0.0941 SYS tuning on 2026-04-01 (37729e7 CDT/intrinsic refresh and earlier). tester1/privtest need 101228 bytes; 0.0941 SYS gave 99008. Bump to 0.1000 SYS (~105220 bytes, ~4KB headroom). No test logic change. kv_roa_limit_enforcement still exercises the RAM-exceeded assertion via alice's separate 0.0100 SYS quota; the tester1 bump only covers contract deployment.

Background ---------- PR #270 covers RAM billing and boundary cases through CDT's kv_multi_index / kv::table wrappers, but nothing in-tree drives the 22 kv_* host intrinsics with inputs that CDT cannot emit. A malicious contract can skip CDT and call the raw host ABI with zero-length spans, forged iterator handles, cross-pool handles, negative primary_ids, table_ids above the uint16 namespace, oversize buffers, etc. This suite exercises each defensive check at the host-intrinsic boundary and pins several behaviors that a future refactor could silently change. Layout ------ unittests/test-contracts/kv_intrinsic_probe/kv_intrinsic_probe.cpp declares all 22 intrinsics via extern "C" + __attribute__((sysio_wasm_import)). Listing the ABI at the call site (not via <sysio/kv.h>) makes every adversarial input explicit and trivial to mutate. One [[sysio::action]] per probe, each using a distinct table_id so the shared tester is safe. unittests/kv_intrinsic_probe_tests.cpp deploys the contract to kvprobe (and kvprobe2 for the cross-contract case) and runs each action. Accepted-behavior actions use BOOST_CHECK_NO_THROW; rejection probes use BOOST_CHECK_THROW with the specific exception type the host is expected to raise. Coverage -------- Every SYS_ASSERT in apply_context's kv paths has a dedicated rejection probe, including the three "missing entry" guards (kv_erase, kv_idx_remove, kv_idx_update), the three negative-primary_id guards, and the cross-contract primary->code == receiver guard. The full cross-pool handle matrix is covered for every rejecting intrinsic. Accepted boundaries pinned: max_kv_key_size = 256 and max_kv_value_size = 256 KiB both round-trip; zero-length value and zero-length secondary key are legal; empty iterator prefix is legal; payer = 0 bills receiver; overlapping key/value spans deep-copied. Dual-dispatch finding --------------------- kv_it_value and kv_it_status accept both primary AND secondary handles by design - their implementations dispatch on the secondary-tag bit rather than asserting on it. An early draft of the suite assumed all primary kv_it_* reject secondary handles; the suite caught the wrong assumption. A positive behavior test (itvalsec) now pins the dual-dispatch so a future "normalization" that locks either intrinsic to primary-only would fail here. Host behaviors pinned for regression ------------------------------------ - Dangling secondary: erasing a primary does NOT auto-remove secondary entries referencing it. The entries remain findable and updatable; kv_idx_primary_key reports iterator_erased. Cleanup is the contract's responsibility. Auto-cascade would be a protocol change. - Cross-table primary_id: kv_idx_store checks only primary->code == receiver, NOT the primary's table_id. A secondary entry at table S may reference a primary row in an unrelated primary table P (same receiver). Convention, not enforcement. - Inline-action iterator isolation: inline actions receive a fresh apply_context with an empty iterator pool; a handle allocated by the parent is meaningless to the child. - Iterator pool limits: the primary pool and the independent secondary pool each cap at 1024 slots; the 1025th allocation raises kv_iterator_limit_exceeded. Chain-takedown classes considered and ruled out ----------------------------------------------- Explicit audit recorded so future probes do not re-investigate: host buffer overflow (legacy_span bounds-checked at the WASM-host boundary); host heap exhaustion (all pools and buffers bounded; apply_context destroyed on action exit); CPU starvation (trx_context.checktime during the _account_ram_deltas loop); divergent consensus (no floats / time / randomness in kv_*); integer overflow in offset/size math (copy_value_window gates with offset < src_size; checked_table_id caps at uint16); notify handler reading parent's iterators (shared apply_context but parent's iterators are dead state by then); payer = nonexistent account (has_authorization rejects any payer not in act->authorization; auth matrix owned by PR #270); kv_idx_update with cross-contract primary_id (composite lookup cannot find an entry kv_idx_store refused to create).

heifner · 2026-05-12T20:01:08Z

Replaced by #331

heifner force-pushed the feature/db-kv branch 6 times, most recently from 14b9eb6 to f9d1cd5 Compare March 28, 2026 05:22

heifner force-pushed the feature/db-kv branch from f9d1cd5 to 83a195d Compare March 31, 2026 21:28

Base automatically changed from feature/db-kv to master April 1, 2026 01:10

heifner requested a review from brianjohnson5972 April 1, 2026 13:10

heifner force-pushed the feature/ram-billing-tests branch from 19cd888 to 3579049 Compare April 1, 2026 14:53

brianjohnson5972 requested a review from Copilot April 2, 2026 16:14

Copilot started reviewing on behalf of brianjohnson5972 April 2, 2026 16:15 View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

heifner added 4 commits April 2, 2026 13:48

Fix unused variable and parameter warnings in test_kv_api contract

c906a7a

heifner changed the base branch from master to feature/kv-secondary-primary-id April 21, 2026 15:52

heifner mentioned this pull request Apr 23, 2026

Test: add adversarial kv_* host intrinsic probes #308

Open

heifner closed this May 12, 2026

heifner deleted the feature/ram-billing-tests branch May 12, 2026 20:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add comprehensive RAM billing and KV coverage tests#270

Add comprehensive RAM billing and KV coverage tests#270
heifner wants to merge 6 commits into
feature/kv-secondary-primary-idfrom
feature/ram-billing-tests

heifner commented Mar 23, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

heifner Apr 2, 2026

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

heifner Apr 2, 2026

Uh oh!

Copilot AI Apr 2, 2026

Uh oh!

heifner Apr 2, 2026

Uh oh!

heifner commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	void ramnotify(uint32_t key_id, uint32_t val_size) {
	void ramnotify([[maybe_unused]] uint32_t key_id, [[maybe_unused]] uint32_t val_size) {

-   void ramnotiferr(uint32_t key_id, uint32_t val_size, sysio::name payer) {
+   void ramnotiferr(uint32_t key_id, uint32_t val_size, sysio::name payer) {
+      (void)key_id;
+      (void)val_size;
+      (void)payer;

Conversation

heifner commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

heifner Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

heifner Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

heifner Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

heifner commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

heifner commented Mar 23, 2026 •

edited

Loading