Emissions: configurable cadence + epoch-frequency-agnostic gate#281
Open
heifner wants to merge 67 commits into
Open
Emissions: configurable cadence + epoch-frequency-agnostic gate#281heifner wants to merge 67 commits into
heifner wants to merge 67 commits into
Conversation
Port emissions from develop branch with key improvement: all emission parameters are configurable via setemitcfg action instead of hard-coded constants. Includes node owner tiered distributions, epoch-based T5 treasury emissions with decay, performance-based producer pay, and category splits (compute, capital, capex, governance). New actions: setemitcfg, setinittime, addnodeowner, claimnodedis, viewnodedist, initt5, processepoch, viewepoch, viewemitcfg. Producer round tracking added to onblock for performance-based rewards. ROA contract updated to call addnodeowner on node registration. 75 emission tests plus ROA test updates for emission config setup.
The emission config singleton must exist before regnodeowner can call addnodeowner. Insert setemitcfg with default parameters into the Cluster.bootstrap() sequence before activateroa/forcereg.
undo_index::post_modify handles AVL tree rebalancing when composite index key fields change. This avoids node deallocation, fresh node allocation, and reinsertion into all 3 AVL trees per secondary index update. Added test proving modify correctly rekeys across indices.
92-commit merge bringing in the kv migration of sysio.system tables (PR #291) and the OPP contract family -- sysio.opreg, sysio.epoch, sysio.msgch, sysio.chalg, sysio.uwrit (PR #303). Conflicts resolved: - contracts/sysio.system/src/producer_pay.cpp: layer this branch's eligible_rounds tracking onto master's kv::table API (producer_key_t, contains/modify(same_payer, key, ...)) - contracts/sysio.system/sysio.system.abi, contracts/sysio.system/sysio.system.wasm, contracts/sysio.roa/sysio.roa.wasm: take master's versions; will be regenerated by CDT after subsequent commits migrate emissions to kv. emissions.hpp/emissions.cpp/emissions_tests.cpp retained on this branch and will be rewritten on kv in the following commits.
Rewrite contracts/sysio.system/include/sysio.system/emissions.hpp and contracts/sysio.system/src/emissions.cpp to use sysio::kv::table / sysio::kv::global in place of multi_index / singleton.
Behavior changes:
- processepoch is permissionless and decoupled from sysio.epoch::advance. It reads epoch_state.current_epoch_index and distributes exactly one epoch per call (target_epoch = last_epoch_index + 1). Safe gap-catch-up by repeated invocation; idempotent via the last_epoch_index guard.
- Recipients filtered by sysio.opreg status (skip non-OPERATOR_STATUS_ACTIVE); slashed / terminated share stays in treasury -- total_distributed is only incremented for amounts actually paid.
- Batch-op pay routed to the current rotation group of 7 from epoch_state.batch_op_groups[current_batch_op_group]; fixed 1/7 slice per member; unfilled slices (slashed / terminated / unknown) stay in treasury.
- Producer pay proportional to eligible_rounds with the existing rank-weight formula; reset to 0 after distribution. Rank resolved at processepoch time. Standby producers (rank 22..cfg.standby_end_rank) paid via the existing rank weight with opreg filter, no eligible_rounds required.
- Verify sysio's WIRE balance is sufficient before transferring (via sysio::token::accounts kv::scoped_table).
Node-owner distributions (addnodeowner, claimnodedis, viewnodedist) ported straight to the kv API with unchanged semantics. Direct sysio.token::transfer remains the payment mechanism; opreg stake deposits intentionally not used -- those represent slashable collateral and must not conflate with rewards.
Cross-contract state reads use the authex_readonly pattern: opreg::operators and epoch::epochstate are mirrored inside emissions.cpp (implementation detail, not public API) so that emissions.hpp stays free of opp / protobuf dependencies and sysio.roa can include it to guard its inline addnodeowner call.
CMakeLists.txt adds ${CMAKE_BINARY_DIR}/libraries/opp/generated-cdt and contracts/sysio.opp.common/include to sysio.system's include path so emissions.cpp can pull the generated protobuf types and shared DataStream operator overloads.
sysio.roa::regnodeowner calls sysio.system::addnodeowner inline so the emissions node-owner distribution row is created in the same transaction as ROA registration. Guarded on sysiosystem::emissions::emitcfg_t::exists("sysio"_n) so bootstrap paths that do not deploy sysio.system (loadSystemContract=False in Cluster.py) or that run forcereg before setemitcfg remain functional -- the inline action is simply skipped when emissions is not configured.
Include sysio.system/emissions.hpp; the header is intentionally free of opp / protobuf deps so this does not widen sysio.roa's dependency surface.
Fixture rewrite: - Deploy sysio.opreg and sysio.epoch as real contracts (not mocks) so processepoch's cross-contract reads exercise the same kv::table code paths as production. - Add ROA RAM policies for sysio.opreg / sysio.epoch / sysio.chalg / sysio.msgch so set_code succeeds on the ~40 KB contract WASMs. - bootstrap_epoch() in the constructor configures sysio.epoch with a short 5-second epoch duration and advances genesis to index 1, leaving the fixture in a state where a single processepoch call can consume one epoch. - setup_producers() now auto-registers each producer as a bootstrapped opreg operator (OPERATOR_STATUS_ACTIVE) so existing producer-pay tests pass through the new opreg filter without test-level churn. - New helpers: register_operator, slash_operator, init_epoch_state, advance_epoch_state, get_opreg_operator, get_epoch_state_row. - Swap <test_contracts.hpp> for "contracts.hpp" and contracts:: accessors for consistency with opreg/epoch tests. - get_t5_state / get_emission_state keyed on the table name (kv::global primary-key convention) instead of the legacy 0 sentinel. Test semantic updates: - Update 75 ported test cases for the new processepoch behavior: recipients filtered by opreg status, batch-op share goes to the current rotation group (or stays in treasury when empty), total_distributed reflects only amounts actually paid. Tests using the "no operators" baseline now assert against compute_undistributed_if_no_operators (producer_pool + batch_pool) instead of producer_pool alone. - Multi-epoch tests call advance_epoch_state between processepoch calls so sysio.epoch's current_epoch_index stays ahead of t5state.last_epoch_index. - processepoch_fails_when_caught_up_to_epoch replaces processepoch_fails_before_epoch_elapsed (the failure condition changed from wall-clock to index comparison). - t5_epoch_info FC_REFLECT gains last_epoch_index, matching the new t5_state field. New integration tests: - opreg_slashed_producer_excluded_from_pay: slashed producer gets 0, remaining producers keep their share (no redistribution). - opreg_unregistered_producer_excluded_from_pay: sysio.system-registered but not opreg-registered producers are filtered out. - processepoch_gap_catchup_one_epoch_per_call: three sysio.epoch advances require three separate processepoch calls (no catch-up batching). - processepoch_fails_on_insufficient_treasury_balance: balance check fires before the first transfer when sysio is underfunded. - roa_forcereg_skips_addnodeowner_when_emitcfg_absent: happy-path coverage of the guard landing an inline sysio::addnodeowner when emitcfg is set.
Artifacts produced by wire-cdt-master (matches wire-sysio master's kv host function signatures). Contains the kv-migrated emissions tables plus the sysio.roa regnodeowner guard that inline-calls sysio::addnodeowner only when the emissions config singleton exists.
Cross-contract mirrors - Move opp/epoch read-only mirrors out of emissions.cpp into public headers owned by sysio.opreg and sysio.epoch. Co-locating the mirrors with the canonical structs makes the invariant visible to maintainers of those contracts. setemitcfg hardening - Cap standby_end_rank at 100 to bound processepoch inline-action count. - After initt5, reject setemitcfg changes that would brick emissions: t5_distributable < floor + total_distributed, or epoch_min_emission above remaining distributable. - Reject node_rewards_start = epoch 0 in setinittime; it silently blocks all future claims via compute_node_claim's guard. - Drop unreachable default case in addnodeowner switch. - Extract BPS_DENOMINATOR constant; no more bare 10000 literals. processepoch - Read operators_per_epoch from sysio.epoch::epochcfg at distribution time instead of hardcoding 7; tracks any upstream group-size change. - Fuse the two prod_by_rank passes into a single walk. - Rename block_seq -> prod_counter_stamp in onblock with invariant comment; the counter is NOT a monotonic block height, the gap check only works because processepoch also resets last_block_num. Audit log and types - Rename CAPEX_ACCOUNT -> CAPEX_OPERATIONS_ACCOUNT with a clarifying comment (capex bucket lives on sysio.ops). - Key epoch_log on sysio.epoch's current_epoch_index so forensics speak the protocol's language; keep the internal epoch_count field alongside. - Memo strings constexpr auto -> constexpr std::string_view; update send_wire_transfer to accept string_view. Read-only view actions - Mark viewnodedist / viewepoch / viewemitcfg [[sysio::read_only]]. sysio.roa finalizereg - Add require_auth(get_self()). - Fix status check to 2 || 3 (was 3 || 4, with 2 as dead code in the confirm branch). - Confirm branch now actually calls regnodeowner. Misc cleanup - Replace Unicode em-dashes with ASCII double-hyphen in three doc comments. - Remove unused <unordered_set> include from sysio.system.hpp. - Rename roa_forcereg_skips_addnodeowner_when_emitcfg_absent to roa_forcereg_inlines_addnodeowner_happy_path, matching actual coverage. Tests - Add setemitcfg_rejects_standby_rank_over_cap. - Add setinittime_rejects_epoch_zero. - Add setemitcfg_post_initt5_rejects_brick_reduce and setemitcfg_post_initt5_rejects_unreachable_min_emission. - Update epoch_log_records_all_fields for new schema. Build - Fix nested-project OPP include path: CMAKE_BINARY_DIR doesn't refer to the host build root inside the contracts ExternalProject; use CMAKE_CURRENT_BINARY_DIR navigation matching sysio.opreg/sysio.epoch. - Add sysio.opreg/include and sysio.epoch/include to sysio.system's include dirs for the new readonly mirror headers.
Security fix: processepoch previously read sysio.epoch's CURRENT batch-op group on every call, so when processepoch lagged sysio.epoch by one or more advances, the current group collected emissions owed to the epochs it processed -- even though those epochs had different groups active. Since processepoch is permissionless, any batch operator could time calls to land during their group's rotation window and harvest catch-up emissions belonging to other groups. Fix uses a per-epoch snapshot written at advance() time and consumed (then pruned) by sysio.system at payout time: sysio.epoch - New batch_snapshot row keyed by epoch_index, capturing the active rotation group's members as they stood when that epoch began. - advance() writes the snapshot after rotating current_batch_op_group. - New prunesnap(epoch_index) action, auth-gated to sysio, erases a consumed row. sysio.system emissions - processepoch reads the snapshot for target_epoch (not sysio.epoch's live state) to build the batch-op recipient list. - After payout, inline-calls sysio.epoch::prunesnap(target_epoch). A contains() guard tolerates missing snapshots (defensive for replays of epochs that predate the snapshot table). readonly mirror - sysio.epoch_readonly.hpp adds mirrors of batchsnap_key / batch_snapshot so sysio.system can read the table. Tests - batch_snapshot_written_on_advance: verifies each advance writes a row. - processepoch_prunes_only_consumed_snapshot: N pending snapshots, each processepoch prunes exactly its target epoch's row. - batch_snapshot_captures_rotation_index: multiple advances capture distinct active_group_index values; epoch 1 and epoch 4 (same rotation slot) are SEPARATE snapshot rows. - prunesnap_requires_sysio_system_auth: non-sysio callers rejected. - prunesnap_rejects_unknown_epoch: unknown index fails cleanly.
operator+/-/*// for fc::variant were declared and defined but never called from anywhere in the tree (verified by grep + a clean full build of unit_test, plugin_test, contracts_unit_test, nodeop, and test_fc with the definitions removed and `= delete` declarations in their place). operator- additionally contained an unreachable bug: the array-walk loop counter decremented (`--i`) instead of incrementing, mirroring the otherwise-identical operator+. Rather than fix dead code, the four operators are marked `= delete` so any future caller fails at compile time with a clear message instead of silently invoking surprising multi-type coercion.
Adds libraries/libfc/benchmark/variant_bench.cpp, modeled on libraries/chain/benchmark/abi_serializer_bench.cpp -- plain chrono timing, no external benchmark dependency, EXCLUDE_FROM_ALL standalone target run manually. Each scenario warms up, then takes the median of 10 runs of N iterations and prints median/min/max ns/op. Scenario coverage targets the paths the fc::variant performance follow-on series will touch: ctor / copy / find / as_* / json round- trip on small and 50-key workloads. The 50-key shape mirrors an ABI-decoded get_table_rows row so deltas reflect a realistic caller. BASELINES.md documents how to build (Release is required) and which scenarios watch which catalogued perf items (lazy variant_object alloc, find_or, from_chars, hash side-table). The baseline-numbers row is a placeholder; the next commit captures it from a Release build. Build / run: ninja -C cmake-build-release -j8 variant_bench ./cmake-build-release/libraries/libfc/benchmark/variant_bench
variant_bench numbers from cmake-build-relwithdebinfo (-O2 -g -DNDEBUG) on 12th Gen Intel Core i9-12900K, clang 18.1.8. Headline observations that motivate Phase A: - as_enum_string_invalid: ~4 us per call. All of it is stoll throwing and the catch(...) unwinding. Replacing with from_chars (Phase A item 3) should drop this by 1-2 orders of magnitude. - find_hit_50key_last: 51 ns vs find_hit_50key_first 2.9 ns -- a 17x ratio. That's the linear scan over 50 entries. Phase B item 4 watches this; find_hit_4key at 4.1 ns is the regression watch (a hash index that hurts small objects is not a win). - contains_then_op_50key: 36.6 ns -- two scans of the same vector. Phase A item 2 (find_or) collapses this to one. - ctor_empty_mvo / ctor_empty_vo: 7.5 / 8.6 ns -- the make_shared<vector<entry>> allocation cost on every default ctor. Phase A item 1 (lazy-allocate) targets this. The series is pinned to RelWithDebInfo so deltas remain comparable and the binary stays debuggable; a re-baseline at -O3 can happen once at the end if absolute numbers need to leave the project.
Adds seven new test files under libraries/libfc/test/variant/ to lock
down current behaviour before the perf series starts touching it.
102 new test cases across:
- test_variant_ctor.cpp: every ctor (primitives, int128/256, char*,
wchar_t*, string, blob, variant_object,
mutable_variant_object, variants, copy,
move, optional<T>) plus deep-copy and
move-leaves-source-null invariants.
- test_variant_assign.cpp: self-assign (copy + move), cross-type
and same-type assignment, template
assignment.
- test_variant_as.cpp: conversion matrix for as_int64 /
as_uint64 / as_int128 / as_uint128 /
as_int256 / as_uint256 / as_double /
as_bool / as_string / as_blob, including
bad-cast paths.
- test_variant_enum.cpp: as_enum_value<E> for int / uint / bool /
double / valid string / invalid string /
object / array / blob sources. Watches
the upcoming stoll -> from_chars switch
(Phase A item 3) for behaviour parity.
- test_variant_visitor.cpp: visit() dispatch coverage of all 13 type
tags. Documents that int128 / uint128 /
int256 / uint256 dispatch through
handle(string), not the typed handle()
overloads -- those are unreachable in
the current visit() implementation.
- test_variant_object_misc.cpp: insertion-order preservation,
contains/find/operator[] consistency,
mvo set vs operator() (variant overload
appends, T&& template overload dedups
via set), erase, merge, op[] insert-
default-on-miss.
- test_variant_operators.cpp: ==, !=, <, > for primitives + arrays;
cross-type string coercion behaviour;
operator! truthiness; documents that
arithmetic operators are = delete.
Suite names use distinct prefixes (variant_ctor_suite,
variant_assign_suite, ...) to coexist with the pre-existing
variant_test_suite / variant_estimated_size_suite /
json_variant_test_suite without filter ambiguity.
The string-source path of variant::as_enum_value<E> previously used
std::stoll wrapped in `try { ... } catch (...) {}`. When the input
was non-numeric, stoll threw std::invalid_argument; the inner catch
swallowed it and the function then threw std::runtime_error to the
caller. Two stack unwinds per bad input.
std::from_chars is non-throwing and parses the same numeric grammar
(leading minus accepted, leading whitespace and leading '+' rejected,
suffix garbage silently ignored), so the swap is observably
equivalent for the cases the ABI serializer can produce -- the
existing test_variant_enum.cpp suite (added in the prior commit)
covers all of them.
Bench delta (libraries/libfc/benchmark/variant_bench, RelWithDebInfo,
12th Gen i9-12900K, ns/op median):
as_enum_string_valid 11.6 -> 4.6 (-60%, 2.5x)
as_enum_string_invalid 3976.4 -> 2965.0 (-25%, the outer
throw still dominates
the bad-input path)
as_enum_int 1.8 -> 1.5 (noise)
Other scenarios are within run-to-run noise (no shared code path).
Default-constructed variant_object and mutable_variant_object no
longer call make_shared<vector<entry>> / make_unique<...> at
construction time. _key_value stays null until the first mutating
operation (op[] / op() / set / reserve, or assignment from a
populated source). Read paths (begin/end/find/contains/op[]/size/
estimated_size) route through a non-const empty-vector singleton in
the null case so iterators are well-formed and comparable.
The mutable singleton is never written to: every write path
allocates a fresh vector first. As a side effect this fixes a
latent aliasing bug in `variant_object::operator=(const
mutable_variant_object&)` where the old `*_key_value = *obj._key_value`
write-through path would mutate any sibling variant_object that had
been copy-shared from the assignee. The new path always
make_shared<vector<entry>>(*obj._key_value), detaching cleanly.
find() in both vo and mvo bypasses the public begin()/end() to keep
the hot loop free of per-iteration null branches: one upfront
null-check, then iterate directly on the vector.
Bench delta (libraries/libfc/benchmark/variant_bench, RelWithDebInfo,
12th Gen i9-12900K, ns/op median):
ctor_empty_mvo 7.5 -> 1.4 (-81%, 5.4x)
ctor_empty_vo 8.6 -> 1.2 (-86%, 7.2x)
json_parse_50key 9760.6 -> ~9500 (~3%, low signal)
find_hit_50key_last 51.0 -> 55.1 (+8%, within run-to-run noise
band of +/-10% measured
across 3 runs)
walk_50key_by_name 997.4 -> 972.2 (within noise)
Adds variant_object::find_or(key, default_value) -> const variant& (plus a string-key overload). Returns the value for `key` if present, otherwise a reference to `default_value`. Caller is responsible for the default's lifetime. Replaces the common `obj.contains(k) ? obj[k] : default_v` pattern, which scans the entry vector twice on hit and throws+catches a key_not_found_exception on miss when the alternate branch is taken via op[]. Bench delta (libraries/libfc/benchmark/variant_bench, RelWithDebInfo, 12th Gen i9-12900K, ns/op median): contains_then_op_50key 38.9 ns (existing double-scan) find_or_50key_hit 19.9 ns (49% faster, single scan) find_or_50key_miss 15.1 ns (no throw on miss) Tests in test_variant_object_misc.cpp cover the hit / miss / empty- object / string-key-overload paths and assert that the returned reference is to the matching entry on hit and to the supplied default on miss.
…allers Step 1 of small-string-optimisation prep: change the public string accessor so it can return a view of inline bytes without requiring a heap std::string object. Pure API + caller migration in this commit; no SSO storage yet, all variant strings still go through `new std::string`. Migrated APIs (signatures changed; existing callers either keep working via implicit conversion or were updated): - fc::variant::get_string() now returns std::string_view (was const std::string&). as_string() (returns std::string by value) is unchanged for callers that need an owning copy. - fc::variant gains a std::string_view ctor. - fc::throw_bad_enum_cast(k, e) takes std::string_view k. - fc::reflector<E>::from_string collapsed into one std::string_view overload (was const char* + const std::string&). FC_REFLECT_ENUM_FROM_STRING uses string_view operator==; the WITH_STRIP variant replaces the `str = b + s` allocation with a starts_with + substr check on the elem name (allocation only happens when strip_base_enum=true and only for the prefix `b`). - fc::from_hex(const std::string&, ...) takes std::string_view (the body only walked iterators). - sysio::chain::asset::from_string and symbol::from_string take std::string_view; both also use std::string_view internally end- to-end via boost::algorithm::trim_copy(string_view) which returns string_view. to_int64 / sysio::chain::to_string still need owning strings, materialised at the call site. - sysio::chain::symbol's `string_to_symbol` / `string_to_symbol_c` helpers and the `symbol(uint8_t, ...)` ctor take std::string_view. - chain_plugin's `string_to_symbol` callers drop now-redundant `.c_str()` calls. Caller migrations: - fc::reflect/variant.hpp's enum from_variant adapter passes v.get_string() through to fc::reflector<T>::from_string without materialising. - bitset's from_variant passes v.get_string() into fc::bitset (which already had a string_view ctor). - variant.cpp's as_blob / from_variant<vector<char>> use string_view locally; base64_decode still gets a materialised std::string because its signature is out of scope. - sysio::chain::symbol_code's from_variant adapter constructs the symbol directly from v.get_string(). The next commit adds the actual SSO storage and reaps the construction-cost win on short strings.
Strings up to 14 bytes are now stored inline in the variant's 16-byte buffer instead of going through `new std::string`. Layout: bytes 0..13 : string content byte 14 : length (0..14) byte 15 : type tag = string_sso_type (13) Heap-allocated strings keep the existing layout (pointer in bytes 0..7, type tag string_type in byte 15) for any input longer than 14 bytes; the storage choice is made by every string-constructing ctor (const char*, char*, wchar_t*, const wchar_t*, std::string, std::string_view) based on the source length. Affected paths: - variant ctors for string-shaped sources route through a single make_string_inline_or_heap helper. std::string ctor short-circuits the move when the source fits inline, so the source's heap buffer (if any) drops on the way out. - clear() / copy ctor / copy assignment / move-related paths handle the new tag: SSO bytes are byte-copied via the existing _data array assignment with no heap involvement. - get_string() returns a view of inline bytes when the tag is string_sso_type, of the heap-resident string when string_type. - as_string() materialises a fresh std::string from inline bytes for SSO; existing heap path unchanged. - as_int64 / as_uint64 / as_int128 / as_uint128 / as_int256 / as_uint256 / as_double / as_bool / as_blob each gained an SSO branch parallel to the existing string_type branch. fc::to_int64 / to_uint64 / to_double now take std::string_view so the SSO branch can pass the view through without materialising. - is_string() returns true for both encodings. - visitor::handle(const std::string&) is now handle(std::string_view) -- visit() dispatches with a view of either inline or heap bytes for string_type / string_sso_type, and the visitor's similarly- unreachable handle(string_view) path is also used for the int128 / uint128 / int256 / uint256 cases (they have always packed a decimal-string view, not the typed handler). raw::variant_packer is the only production override; it now writes the bytes directly rather than calling raw::pack(string_view) (overload resolution picks the generic pack<Stream, T> over a string_view-typed pack via partial-ordering rules; inlining sidesteps that). - raw::pack(variant) translates string_sso_type to string_type at the wire boundary so existing peers and chainbase-resident buffers deserialize unchanged. Payload bytes are identical. - estimated_size() routes string_sso_type through the same formula as string_type (allowed to over-report; keeps tests stable). - json::to_stream handles the new tag alongside string_type. The single non-libfc caller checking `get_type() == string_type` (tests/trx_generator/trx_generator.cpp) is migrated to is_string(). Bench delta (libraries/libfc/benchmark/variant_bench, RelWithDebInfo, 12th Gen i9-12900K, ns/op median): ctor_short_string 14.2 -> 3.6 (-75%, 3.9x) ctor_sso_boundary_14 -- -> 3.4 (new row, max inline) ctor_just_over_sso_15 -- -> 14.3 (new row, just heap) ctor_long_string 19.6 -> 19.7 (unchanged, heap path) copy_short_string 11.0 -> 2.2 (-80%, 5.0x) copy_long_string 16.5 -> 18.8 (unchanged within noise) Workload-shaped scenarios (json_parse / walk) are within run-to-run noise: short-string allocation savings exist but are diluted by non-string parser cost in the 50-key fixture. New tests in test_variant_ctor.cpp cover the SSO/heap boundary, both encoding paths, copy independence, move-leaves-source-null for SSO, and SSO/heap equality.
variant::operator=(const variant&) previously did clear() (which delete's the existing heap object for heap-backed variants) followed by a fresh `new` of the rhs's underlying type. When the lhs already holds a heap object of the same type as the rhs, the dealloc+alloc pair is wasted -- the existing object can absorb the rhs's value directly via its own assignment operator. This commit checks for the same-type case at the top of op=(const&) and routes through the existing heap object for object_type, array_type, blob_type, string_type, and the std::string-backed multi-precision integer encodings (int128/uint128/int256/uint256). Cross-type assignment, and assignment between inline encodings (null/int/uint/double/bool/string_sso), continue to use the existing clear+memcpy path. Bench delta (libraries/libfc/benchmark/variant_bench, RelWithDebInfo, 12th Gen i9-12900K, ns/op median; pre/post captured by stashing the variant.cpp change and rerunning): assign_long_string_to_long 17.0 -> 3.3 (-81%, 5.2x) assign_object_to_object 10.3 -> 2.0 (-81%, 5.1x) assign_array_to_array 32.9 -> 12.1 (-63%, 2.7x) The non-assign rows in the log are unchanged within run-to-run noise (B5 only affects the same-type op=(const&) path).
The per-commit log in BASELINES.md tracks RelWithDebInfo (-O2 -g) so commits stay directly comparable. This commit captures the same post-B5 scenarios at Release (-O3 -DNDEBUG) on the same host so an external reader can see the absolute numbers Wire-Sysio hits in a deployment build. -O3 is 2-15% faster than -O2 on the inlinable paths (find loops, array copies, find_or, contains_then_op). A handful of rows (copy_short_string, copy_object_50key, as_enum_string_invalid) are slower at -O3, all within run-to-run variance for those particular scenarios. The relative ordering across scenarios is preserved, so the per-commit deltas remain meaningful. No code change in this commit -- BASELINES.md only.
`read_table_rows` posts onto the executor's read_only queue and blocks the caller on the future. From a main-thread context (e.g. a plugin's `plugin_startup`), the main thread is the only drain for that queue during write window, so the post sits until the deadline and returns empty rows. This silently broke `batch_operator_plugin::refresh_outposts` at startup -- the cron pool came up sized for zero OPP jobs. When `std::this_thread::get_id() == executor().get_main_thread_id()`, run the scan inline; the read-window discipline is already satisfied on the main thread. Off-thread callers keep the post + wait path so cron threads still iterate chainbase during the read window. Scan body is factored into a `run_scan` lambda shared by both paths.
…42hrs+`) with integrated pruning, cross-chain data consistency & chain-agnostic orchestration layer
## Overview
Achieved memory & on-chain storage stability (`>2400` epochs over `42hrs+`) with integrated pruning, cross-chain data consistency & chain-agnostic orchestration layer.
This commit implements a comprehensive architectural refactoring to achieve cross-chain data-state consistency between WIRE and external blockchains. It introduces a chain-agnostic orchestration layer (`depot_ops` + `outpost_opp_job`) that cleanly separates WIRE table operations from chain-specific client implementations, fixing critical bugs in secondary index queries, row unwrapping, and EIP-1559 RLP signature encoding that previously broke batch operator envelope delivery.
Additionally, the commit removes obsolete auto-generated contract files, adds devcontainer awareness to build tooling, eliminates deprecated macOS signing infrastructure, and standardizes CMake module naming conventions.
---
## Detailed Summary of Changes
### **Removed Deprecated Auto-Generated Contract Files**
Cleaned up stale build artifacts that should never have been committed to version control:
- **Deleted** `contracts/sysio.chalg/sysio.chalg.cpp.actions.cpp` (138 lines of codegen action wrappers)
- **Deleted** `contracts/sysio.chalg/sysio.chalg.dispatch.cpp` (40 lines of codegen dispatch handlers)
- **Deleted** `contracts/sysio.chalg/sysio.chalg.sysio.chalg.cpp.desc` (264 lines of ABI metadata)
These files are regenerated during the build process and do not belong in the repository.
---
### **Chain-Agnostic Orchestration Layer**
Introduced two new abstractions to decouple WIRE operations from outpost-specific client logic:
#### **New Interface: `depot_ops` (`batch_operator_plugin/depot_ops.hpp`)**
Provides chain-agnostic WIRE table/action operations:
- `read_pending_outbound()`: Queries `sysio.msgch::outenvelopes` for pending delivery
- `has_delivered_envelope()`: Checks `sysio.msgch::envelopes` secondary index for delivery status
- `deliver_to_depot()`: Pushes inbound messages via `sysio.msgch::deliver` action
- `emit_debug_envelope()`: Publishes debugging events to `external_debugging_plugin`
- `within_epoch_window()`, `is_elected()`, `current_epoch()`: Epoch state accessors
Concrete implementation (`depot_ops_impl_t`) translates high-level requests into `read_table()` / `push_action()` calls with row unwrapping and batch operator authorization checks.
#### **New Orchestrator: `outpost_opp_job` (`batch_operator_plugin/outpost_opp_job.hpp`)**
Per-outpost job lifecycle manager:
- Owns a single `outpost_client` instance (Ethereum or Solana)
- Runs on dedicated cron thread to avoid blocking other outposts
- **Outbound Flow (WIRE → L1):**
Queries pending envelope → submits to outpost → waits for confirmation → emits debug event
- **Inbound Flow (L1 → WIRE):**
Polls L1 events → checks delivery status → submits to WIRE → prevents duplicates
#### **Refactored `batch_operator_plugin::impl`**
Replaced monolithic `run_epoch_cycle()` with job-based architecture:
- Removed direct outpost client management (`opp_client`, `sol_outpost_client`, transient state tracking)
- Introduced `std::map<uint64_t, std::shared_ptr<outpost_opp_job>> opp_jobs` (one job per outpost)
- Added `build_opp_jobs()` factory using chain plugin `make_outpost_client()` APIs
- Centralized WIRE contract constants (`msgch` and `epoch` namespaces) to prevent string literal duplication
---
### **Critical Bug Fixes**
#### **1. Secondary Index Query Fix (`chain_plugin.cpp`)**
**Problem:** Secondary indexes on `multi_index` tables store keys as `[scope:8B][value:N]`, but `json=true` queries only provided the value portion in bounds, causing incorrect row lookups.
**Solution:** Prepend `scope_prefix_bytes` to lower/upper bounds when:
- Query uses secondary index (`!resolved_index_name.empty()`)
- Scope is set (`!scope_prefix_bytes.empty()`)
- JSON mode is enabled (`p.json == true`)
**Test Coverage:**
- `(sec-10)`: `find` on `multi_index` secondary index returns exact match
- `(sec-11)`: `find` miss returns zero rows (not lexicographically-next row)
#### **2. Row Unwrapping Fix (`batch_operator_plugin.cpp`)**
**Problem:** `chain_plugin::get_table_rows` wraps rows as `{"key", "value", "payer"}`, breaking direct field access in plugin code.
**Solution:** Post-process `combined.rows` vector to extract `"value"` object using safe pattern:
```cpp
fc::variant value{row_obj["value"]}; // Copy to temporary
row = std::move(value); // Move-assign to avoid self-destruction
```
**Critical Pattern:** Direct assignment `row = row["value"]` causes undefined behavior because `variant::operator=` calls `clear()` before reading the source.
#### **3. EIP-1559 RLP Signature Encoding Fix (`ethereum_rlp_encoder.cpp`)**
**Problem:** Signature components `r` and `s` were encoded as fixed 32-byte strings. When the MSB was `0x00`, strict RLP decoders (alloy-rs in reth/anvil) rejected transactions with "leading zero" errors (~1/256 signatures affected).
**Solution:** Implemented `encode_sig_scalar()` to strip leading zeros and emit minimal big-endian integers per EIP-2718.
**Test Coverage:**
- `signed_rlp_strips_leading_zeros_in_r_and_s`: Verifies 31-byte encoding when MSB is `0x00`
- `eip1559_signed_rlp_strips_leading_zero_in_s`: Regression test for production failure case (612-byte payload, s=`0x00 9b bd d7 ...`)
---
### **Plugin API Enhancements**
#### **Outpost Client Factory Methods**
**Ethereum:** `outpost_ethereum_client_plugin::make_outpost_client()`
Creates `outpost_ethereum_client` from ETH client ID, outpost ID, chain ID, contract addresses. Validates client existence and contract address configuration.
**Solana:** `outpost_solana_client_plugin::make_outpost_client()`
Creates `outpost_solana_client` from SOL client ID, outpost ID, chain ID, program ID. Filters loaded IDL set to match `OPP_SOLANA_OUTPOST_PROGRAM_NAME` and asserts IDL availability.
#### **Debug Event Type Relocation**
Moved `DebugEnvelopeEvent` to dedicated header `batch_operator_plugin/debug_envelope_event.hpp` to break circular dependency between `batch_operator_plugin.hpp` and `depot_ops.hpp`. Remains in `sysio::opp::debugging` namespace.
---
### **Build System Improvements**
#### **CMake Module Standardization**
- Renamed `VersionUtils.cmake` → `version-tools.cmake`
- Renamed `AddTestHelpers.cmake` → `test-helpers.cmake`
- Follows lowercase-hyphenated naming convention
#### **Devcontainer Detection in Version Generation**
Enhanced `version-tools.cmake` to gracefully handle non-git environments:
```cmake
if(IS_DIRECTORY ${CMAKE_SOURCE_DIR}/.git AND NOT "$ENV{IN_DEVCONTAINER}" STREQUAL "1")
```
Sets `V_HASH="unknown"` and `V_DIRTY="true"` in devcontainers/tarballs to prevent build failures.
#### **Removed Deprecated MAS Signing Infrastructure**
Deleted `cmake/MASSigning.cmake` (22 lines of unused App Store code signing macros) and removed `mas_sign(${KEY_STORE_EXECUTABLE_NAME})` invocation from `programs/kiod/CMakeLists.txt`.
#### **OPP Bundle Generation Enhancements**
**`generate-opp-bundles.fish`:**
- Added `
Avoid materialising a std::string in variant::as_blob's base64 fallback path, where the source is already a string_view from variant::get_string(). The string_view overload is added alongside the existing const std::string& one - keeping both is necessary because the template fc::base64_decode is also visible at call sites and would silently win for std::string callers (returning std::string instead of std::vector<char>) if only string_view were left. detail::decode's remove_linebreaks branch is also routed through std::string regardless of the input String type - it needs erase(), which std::string_view does not provide.
…nto `wire-sysio` removing a potential circular dep with `wire-libraries-ts` ## Overview Moved `protoc-gen-<solana|solidity>` plugins and protobuf-bundler into `wire-sysio` removing a potential circular dep with `wire-libraries-ts` Additionally, it adds a build options `BUILD_OPP_BUNDLES` (default is `ON`), which allows gating of the OPP bundler The output by default is `<wire-sysio>/build/opp/<solidity|typescript|solana>` and can be link with either `pnpm` or `npm` package managers from there in the case you'd like to use them for local development.
- get_string() doc: add explicit lifetime contract - find_or() doc: add on-hit reference invalidation note - mutable_variant_object(variant) / (T&&): drop wasted entry-vector alloc (operator= replaces _key_value, so the initializer was overwritten) - mutable_variant_object: add const contains() overloads to keep the empty-state lookup allocation-free for default-constructed mvos - estimated_size: use [] instead of at() inside size-bounded loop - write_sso: static_assert(sso_max_length < 128) so a future bump can't silently corrupt the signed-byte length round-trip - test: variant_sso_wire_roundtrip pins the pack/unpack normalisation invariant (string_sso_type tag 13 must wire as legacy string_type 9) - test: self_assign_aliased_subvariant_same_type locks the spicy `v = v.get_object()["k"]` path under the same-type op= heap reuse
… assignment ASAN heap-use-after-free in self_assign_aliased_subvariant_same_type: when v aliases storage owned by *this (e.g. v = v.get_object()["k"] where lhs is object_type and rhs is string_type), the different-type path called clear() before reading from v in the new switch -- and clear() destroys the heap object that v references. Deep-copy v into a temporary BEFORE clear(), then take the temp's data and disarm its destructor. Same observable effect on lhs as the inline new-per-type switch the path used to do (the copy ctor does the same per-type allocation), just safe under aliasing. This is a latent pre-existing bug surfaced by the new test, not a regression introduced by the same-type op= heap reuse refactor.
… storage The int128/uint128/int256/uint256 variant types store their value via new std::string(...) (variant.cpp:134-155), but neither clear() nor the copy ctor handled these type tags: - clear(): leaks the heap std::string for these types on every destructor / assignment that should free them. - copy ctor: falls through to the inline _data byte-copy default arm, which copies the std::string* pointer rather than deep-copying. Two variants then share the same pointer; whichever clear() runs first (after the matched fix below) double-frees. Fix both atomically: - clear() adds the four type tags alongside string_type, all of which use the same std::string* storage shape. - copy ctor adds the same fan-out, deep-copying via new std::string(*v.string_ptr). These are pre-existing latent bugs (the leak was silent, the share was masked by the missing clear() handler so no double-free fired). Surfaced while reviewing the same-type op= refactor.
## Overview
Added chunking in order to support large envelopes on SOLANA
Enforce `MAX_ENVELOPE_BYTES` cap on outbound envelope packing loop; add chunked Solana `epoch_in` delivery; fix secondary index queries, row unwrapping, and EIP-1559 RLP signature encoding
This commit introduces a hard 65 536-byte (64 KiB) envelope size cap validated in both the WIRE contract's `buildenv` packing loop and the Solana client's `deliver_outbound_envelope` handler. It replaces the single-call Solana `epoch_in` flow with a chunked streaming protocol that fits within the 1 232-byte transaction MTU, eliminating the need for separate `emit_outbound_envelope` and `cleanup_envelope_chunks` transactions by inlining finalization. Additionally, it fixes critical bugs in secondary index queries (scope-prefix handling), row unwrapping (self-destruction during assignment), and EIP-1559 signature encoding (leading-zero stripping in `r`/`s` components).
The changes deliver cross-chain data-state consistency, demonstrated memory stability over 2 400+ epochs (42+ hours), and end-to-end envelope-cap enforcement from contract emission through Solana program ingestion.
---
## Detailed Summary of Changes
### **Contract-Side Envelope Cap Enforcement**
#### **contracts/sysio.msgch/sysio.msgch.abi**
- **Removed trailing newline** (cosmetic whitespace fix).
#### **contracts/tests/sysio.msgch_tests.cpp**
1. **New Test Helper: `queueout_with_data()`** (Lines 235-246):
- Variable-payload `queueout` action wrapper for envelope-cap boundary tests.
- Accepts arbitrary `std::vector<char> data` to control per-attestation size.
2. **New Test Helper: `count_ready_attestations()`** (Lines 251-264):
- Counts READY-status (queued-but-not-yet-bundled) attestations for a given `outpost_id`.
- Scans table by primary ID (avoids needing secondary index ABI binding).
- Used to verify packing loop leaves remainder attestations queued after hitting cap.
3. **New Test Case: `buildenv_packs_until_cap_then_leaves_remainder`** (Lines 447-541):
- **Setup:** Queues 12 attestations × 8 KiB each (≈96 KiB cumulative, 50% over cap).
- **First `buildenv` Call:**
- Asserts emitted envelope ≤ 65 536 bytes.
- Asserts a non-trivial subset of READY attestations remained queued (pack loop stopped before exhausting queue).
- **Second `buildenv` Call:**
- Drains remainder under the same cap.
- Asserts no READY attestations remain (full queue processed across two epochs).
- **Purpose:** Demonstrates §6c boundary condition—contract enforces "analytical max" as "demonstrated max" in unit test.
---
### **Solana Client Chunked Delivery Protocol**
#### **plugins/outpost_solana_client_plugin/include/sysio/outpost_solana_client_plugin/outpost_solana_client.hpp**
1. **New Constant: `SOLANA_MAX_ENVELOPE_BYTES = 65'536`** (Line 22):
- Hard cap on assembled OPP envelope.
- Mirrors `MAX_ENVELOPE_BYTES` in Solana program (`programs/opp-outpost/src/state/envelope_chunks.rs`).
- **Rationale (Comment Lines 15-21):**
- 64 KiB is the e2e-supported max across WIRE/Ethereum/Solana.
- Binding constraint: Solana's 256 KB BPF heap ÷ 3.3× peak heap usage (envelope decode + keccak hash + assembled buffer + clone during finalization).
- Kept in sync by hand (no shared C++/Rust constant header).
2. **New Constant: `SOLANA_MAX_CHUNK_BYTES = 704`** (Line 32):
- Per-`epoch_in` chunk payload limit.
- Mirrors `MAX_CHUNK_BYTES` in Solana program.
- **Rationale (Comment Lines 24-31):**
- Chosen to fit a single `epoch_in` chunk transaction inside Solana's 1 232-byte tx packet MTU after header/signature/account-meta overhead.
- **Historical Context:** Bumped down from 768 → 704 when `EpochIn` grew from 7 → 10 accounts (added `outbound_message_buffer`, `outbound_envelopes`, `latest_outbound_envelope` for inline-emit-on-finalize path).
- The extra 99 raw bytes of account keys/indices consumed the previous margin and pushed 768-byte chunks past MTU.
#### **plugins/outpost_solana_client_plugin/src/outpost_solana_client.cpp**
1. **Refactored `deliver_outbound_envelope()` (Lines 78-121):**
- **Removed Single-Call `epoch_in` + Separate `emit_outbound_envelope` Pattern:**
- Old flow: (1) submit full envelope via `epoch_in(bytes)`, (2) submit `emit_outbound_envelope(epoch)` to trigger program-side emission, (3) assume external cleanup call for chunk buffer.
- **New Chunked Streaming Protocol:**
- **Upfront Validation (Lines 78-85):**
- Asserts envelope is non-empty.
- **Enforces Cap:** `FC_ASSERT(total <= SOLANA_MAX_ENVELOPE_BYTES)` with error message indicating Solana program will reject oversized envelopes.
- **Chunk Count Calculation (Lines 87-88):**
- `total_chunks = ceil(total / SOLANA_MAX_CHUNK_BYTES)`.
- **Sequential Chunk Submission Loop (Lines 98-120):**
- Iterates `i ∈ [0, total_chunks)`.
- For each chunk:
- Calculates `offset` and `length` (clamped to remainder on last chunk).
- Constructs `std::vector<uint8_t> chunk` from slice of `envelope_bytes`.
- Calls `_program_client->epoch_in(epoch_index, chunk_index, total_chunks, total_bytes, chunk)`.
- **Synchronous Confirmation:** Each call goes through `solana_program_client::execute_tx_and_confirm`, which serializes submission + waits for `processed`-commitment before returning.
- **Inline Finalization (Comment Lines 100-108):**
- The **last chunk** triggers `finalize_envelope` in the program, which:
- (a) Records the operator's delivery.
- (b) On consensus reach, fires `emit_outbound_inner` inline (packs queued outbound attestations, writes to `latest_outbound_envelope` PDA).
- (c) Self-closes the operator's chunk_buffer (rent reclaimed).
- **No Separate Transactions Needed:** The batch operator's **only Solana-side tx type is `epoch_in`**—no separate `emit_outbound_envelope` or `cleanup_envelope_chunks` calls.
- **Deadline Check:** `throw_if_past_deadline()` before each chunk submission.
- **Logging:** Per-chunk log includes `chunk={i}/{total_chunks}` and `bytes={len}`.
- **Return Value:** Transaction signature of the **last chunk** (the one that finalized the envelope).
2. **Removed Operation Labels (Lines 21-22):**
- Deleted `OP_EMIT_OUTBOUND_ENVELOPE` (no longer applicable).
- Kept `OP_EPOCH_IN` and `OP_READ_LATEST`.
---
### **Solana IDL Update (Test Fixture)**
#### **tests/fixtures/solana-idl-opp-outpost-stub.json**
1. **`epoch_in` Instruction Signature Change (Lines 127-202):**
- **Old Arguments:** `envelope_data: bytes` (single-call full envelope).
- **New Arguments (5 total):**
- `epoch_index:
Brings in opp-part2's chunking support for large Solana envelopes (commit 5363460) plus the related sysio.msgch contract update, libfc Solana client / json_rpc_client tweaks, and outpost_solana_client_plugin updates.
Replace epoch_duration_secs / decay_numerator+denominator / epoch_initial+max+min_emission with a canonical sysio.epoch::epoch_duration_sec and annual config values; per-epoch quantities are derived inside the contract from the canonical epoch length so the wall-clock emission curve is invariant under epoch-frequency retunes. - sysio.epoch::setconfig now requires epoch_duration_sec in [60, 30 days]. - emission_config drops epoch_duration_secs (canonical on sysio.epoch); payepoch and viewepoch read sysio.epoch::epochcfg cross-contract. - emission_config replaces decay_numerator/denominator with target_annual_decay_bps; per-epoch decay derived as (target/10000)^(epoch_secs/year_secs) via Q32.32 fixed-point pow in the new fp_math.hpp helper (Taylor series + halving range reduction, ~2.3e-10 relative precision). - emission_config replaces epoch_initial/max/min_emission with annual_initial/max/min_emission; per-epoch values derived linearly as annual * epoch_secs / SECONDS_PER_YEAR. - emission_config gains epoch_log_retention_count; payepoch head-prunes epochlog after each insert (mirrors envelope_log pattern). Governance can change epoch cadence (e.g. 6h to 6min) without re-deriving decay or per-epoch emission caps. The TestHarness/Cluster.py setemitcfg payload and the contract test suites are updated to the new annual schema, including a Q32.32 mirror in emissions_tests.cpp so per-epoch decay expectations are bit-exact.
Conflict resolutions:
- contracts/sysio.epoch/src/sysio.epoch.cpp: kept HEAD (the inline payepoch action block from this branch's emissions feature; master never had it).
- plugins/outpost_solana_client_plugin/{include/.../outpost_solana_client_plugin.hpp, include/.../outpost_solana_client.hpp, test/test_outpost_solana_client_plugin.cpp}: took master's PR #326 OOM fix at epoch 13 (chunk size 704 to 672 + 256 KiB BPF heap pre-ix on the final chunk).
- contracts/sysio.epoch/sysio.epoch.wasm: rebuilt from the resolved source.
db.modify preserves the kv_index_object's chainbase id, so a live secondary iterator that cached the id would advance from the post-modify position and silently skip entries when sec_key changed. The prior remove+create path worked because the new object had a fresh id, so the cached id no longer resolved and iteration fell back to the slow re-seek using stored key bytes. Add kv_iterator_pool::invalidate_secondary_cache that clears cached_id only on matching secondary slots, leaving stored key bytes and status intact so the next op resumes from the old position. kv_idx_update calls it before db.modify.
- setemitcfg: reject configs where any nonzero annual_* value scales to 0 at the canonical epoch_duration_sec; without this, a tiny annual value silently disables emissions because the gate sees emission_amount = 0 and blocks every advance. Skipped pre-bootstrap (sysio.epoch not yet configured); fires on the next setemitcfg call. - payepoch epochlog prune: drop up to two oldest rows per call instead of one. Steady state is unchanged (only the just-added row needs eviction); the second erase only fires when a recent retention-cap shrink left the table over cap, draining it twice as fast without unbounded CPU. - fp_math.hpp: precision comment now accounts for compounding across mul/ln/exp; worst-case relative error on a single pow_frac is ~2e-9 (per-op Q32 epsilon ~2.3e-10 compounds across exp's back-squaring), cumulative over 88k epochs/year is ~2e-4.
…ssert Aliased self-assignment (rhs refers to storage owned by lhs) was UB in the previous fc::variant via the clear()-then-new pattern. Earlier in this branch e6e198a added a `variant tmp(v); take(tmp)` deep-copy in the different-type path that turned the UB into well-defined behaviour, but only there -- the same-type fast path added in 3e4e69a still UAFs on aliased rhs (vector and variant_object copy assignments read element-by-element from rhs while writing through lhs). Restore the previous UB contract symmetrically: - different-type path: switch back to clear()-then-new, removing the variant tmp(v) deep-copy from e6e198a. - same-type fast path: unchanged (the optimisation is preserved). - add a debug-only direct-aliasing detector (rhs_not_aliased) called from assert() so the body and call site compile out under NDEBUG. Catches v = v.get_array()[i] v = v.get_object()["k"] Deeper nesting (rhs reachable through inner array/object owned by lhs) remains UB and may slip past undetected. Drop the two tests that exercised aliased self-assign (a test cannot pin UB); keep coverage of the std::string-backed multi-precision op= paths via a new non-aliased int128_op_assign_same_and_cross_type test.
4252bc3 (this branch) replaced std::stoll with std::from_chars to avoid the exception round-trip. The two parsers agree on the inputs the ABI serializer actually emits but disagree on rejected forms: - " 1" (leading whitespace): stoll accepted, from_chars rejects. - "+1" (leading plus): stoll accepted, from_chars rejects. - "-1" / "1abc": both accept identically. The earlier comment claimed the behaviour matched stoll which was inaccurate. Update the comment to spell out the differences and pin them with new tests (string_source_with_leading_whitespace_throws, string_source_with_leading_plus_throws). ABI-emitted enum strings never contain whitespace or sign prefixes so the stricter contract surfaces malformed input as an exception rather than silently parsing it.
heifner
changed the title
- setemitcfg: hoist sysio.epoch::epochcfg read so the round-to-zero guard and the post-init guard share a single read. - epoch::advance: pass cfg.epoch_duration_sec into check_emissions_ready instead of having the gate re-read epochcfg. Renames the local emitcfg/epochcfg vars to emit_cfg_tbl / epoch_cfg_tbl for clarity. - payepoch expected_rounds: inline comment explaining the fallback at the 60s minimum (coarse-grained pay is the documented price of allowing sub-rotation epoch durations). - fp_math: rename exp_nonpositive -> exp_neg; lift Taylor term counts to constexpr (LN_TAYLOR_TERMS = 35, EXP_TAYLOR_TERMS = 20); add early-break when truncated terms hit zero; tighten LN2 comment. - emissions.hpp: rename compute_per_epoch_decay_q32 -> compute_per_epoch_decay (the _q32 suffix duplicated info already in the return type); express SECONDS_PER_YEAR as a compiler-evaluated 365 * 24 * 60 * 60. - Drop an orphaned comment block in emissions.cpp.
…ptimizations Resolve conflict in libraries/chain/apply_context.cpp at the kv_idx_update db.modify lambda. Master's remove+create added o.table_id = table_id; the merged db.modify path drops the assignment since idx.find already constrains table_id on the located row. Adapt HEAD's invalidate_secondary_cache (added in 1146c38) to master's uint16_t table_id kv API: - kv_iterator_slot/invalidate_secondary_cache use (code, table_id) instead of (code, table, index_id) - kv_idx_update call site updated accordingly - kv_tests kv_index_modify_rekeys_correctly and kv_iterator_pool_invalidate_secondary_cache rewritten against the new schema; the by_code_table_idx_prikey lookup is replaced with a 4-tuple find on by_code_table_id_seckey since master has only the seckey index.
…for unit testing The < / >= comparisons in the aliasing detector between &v and dst_vec's [begin, end) range are unspecified per [expr.rel] when &v does not point into the same array as begin. std::less<T*> is required by [comparisons]/2 to provide a strict total order over all object pointers, so it gives a defined result regardless of whether &v aliases lhs's storage. Rename the inline helper to a static member variant::_rhs_not_aliased and declare it in variant.hpp so test_variant_assign can verify the detector directly: the helper only reads pointer addresses (no deref, no write), so calling it on intentionally-aliased pointers is well-defined. The UB lives in the full operator= flow that follows the assert, which the tests do not invoke. Pointed out by huangminghuang on PR #315.
The merge from master in cd02d6d brought commit 30b1697, which collapsed kv_index_object's (table, index_id) into a single uint16_t table_id and reduced kv_iterator_pool::{allocate_secondary, invalidate_secondary_cache} to 2/3 args. apply_context.cpp::kv_idx_update merged correctly; the two test cases added in 4fe6dfd and 1146c38 still referenced the old fields and signatures. - kv_index_modify_rekeys_correctly: rewritten against table_id and by_code_table_id_seckey. Replaces the by_code_table_idx_prikey lookup (no longer a separate index) with a composite find on (sec_key + pri_key) plus an explicit "old key no longer resolves" check. - kv_iterator_pool_invalidate_secondary_cache: 2-arg allocate_secondary, 3-arg invalidate_secondary_cache. Old (different table) vs (different index) slots collapse to "different table_id"; kept as two distinct table_ids so the loop preserves more than one non-matching slot.
…ions Optimize kv_idx_update: use modify instead of remove+create
At a 6-minute epoch the per-epoch payepoch was firing 100+ inline
send_wire_transfer calls (~21 active producers + ~79 standbys + 3
batch ops) every 6 minutes -- ~1000 transfers/hour for tiny per-
recipient amounts. The frequency is the cost driver, not the math.
Add a single `uint16_t pay_cadence_epochs` knob on emit_cfg
controlling how many epochs accumulate before payepoch fires.
Recommended production value 100; setemitcfg rejects 0.
Schema:
- emit_cfg gains pay_cadence_epochs.
- t5_state gains pending_emission_amount (period accumulator),
period_start_epoch (period anchor), and batch_group_epochs
(per-batch-op-group active-epoch counter).
Gate (sysio.epoch::check_emissions_ready):
- Decides is_pay_epoch via `target >= period_start + cadence - 1`.
- TREASURY_EXHAUSTED still gates every epoch so the chain cannot
roll forward into a depleted treasury.
- Balance-coverage check fires only on pay-epochs against the
period total (pending + this-epoch's per-epoch share).
Advance (sysio.epoch::advance):
- Always queues accrueepoch.
- Conditionally queues payepoch on pay-epochs.
- Inline FIFO ordering means payepoch reads the post-accrue state.
accrueepoch (new sysio.system action, auth sysio.epoch):
- Owns per-epoch state advancement (last_epoch_index /
last_epoch_time / last_epoch_emission, decay continuity).
- Increments pending_emission_amount and bumps
batch_group_epochs[batch_group_index].
payepoch (rewritten):
- Takes batch_op_groups and period_emission instead of single-
epoch active_batch_group + emission_amount.
- Producer pool: expected_rounds scales by pay_cadence_epochs;
eligible_rounds accumulates across non-pay epochs and resets
only here.
- Batch pool: each group's slice is weighted by its active-epoch
count (state.batch_group_epochs[g] / pay_cadence_epochs);
groups active in zero epochs are skipped (only possible when
cadence < batch_op_groups.size()), their slice stays in
treasury, same convention as slashed members.
- On completion drains pending_emission_amount, zeros
batch_group_epochs, advances period_start_epoch.
- epochlog gains one row per pay-epoch; non-pay epochs add none.
Tests (t5_emissions_tests):
- pay_cadence_2_pays_every_other_epoch
- pay_cadence_pending_accumulates_then_drains (cadence=3)
- pay_cadence_epochlog_only_on_pay_epoch
- pay_cadence_treasury_exhausted_gates_non_pay_epoch
- pay_cadence_change_via_setemitcfg_takes_effect
setemitcfg_defaults stays at cadence=1 (preserves existing test
semantics bit-for-bit). New setemitcfg_with_cadence helper is used
by the cadence>1 cases. tests/TestHarness/Cluster.py bootstraps
local soak clusters at cadence=2.
Regenerates contracts/sysio.{system,epoch}/sysio.{system,epoch}.{wasm,abi}.
heifner
changed the title
…d names Wire CDT emits `first`/`second` for `std::pair` and `std::map` fields in generated ABIs (see tests/toolchain/abigen-pass/nested_container.abi in wire-cdt). cc1dde4 previously swapped in a Leap-derived ABI using `key`/`value` as a workaround, but it regresses every time the WASM is regenerated and diverges from Wire CDT's own toolchain tests. Remove the workaround. The contract ABI now uses `first`/`second` matching CDT output, and the Python test inputs and assertions are updated to match. The `pvo` case (`pair<uint16_t, vec_op_uint16>`) was already on `first`/`second` and is unchanged. `transaction_json[...]['value']` row accessors are untouched - that is the clio RPC envelope field, not a pair field name. WASM is unchanged: kv_multi_index binary serialization is positional, so the field rename is purely a JSON-side concern.
…st-cdt-abi-names Test: align nested_container_multi_index with CDT pair/map field names
Reverse the on-disk byte order of the snapshot v1 header magic so a hex dump of a snapshot file reads 'W','I','R','E' instead of 'E','R','I','W'. Stored little-endian as 0x45524957 -> bytes on disk 57 49 52 45. Pre-launch; no backward compatibility. Snapshots written with the old magic cannot be read after this change.
The snapshot magic flip changes the on-disk header bytes. Regenerated
via:
unit_test --run_test='snapshot_part2_tests/*' -- --sys-vm \
--save-snapshot --generate-snapshot-log
head_block_id in sysio_util_snapshot_info_test.py updated to match the
new fixture; blocks.log / blocks.index regenerated as a byproduct of
--generate-snapshot-log.
The snapshot regen run on this branch also drifted the action_mroot
that savanna_misc_tests/verify_block_compatibitity compares against.
Regenerated via:
unit_test -t "savanna_misc_tests/verify_block_compatibitity" \
-- --sys-vm --save-blockchain
jglanz
requested changes
May 12, 2026
Collaborator
jglanz
left a comment
jglanz
left a comment
There was a problem hiding this comment.
I left a few questions - but I'm happy with the PR once you go thru them
Comment on lines
+54
to
+55
| // whether emissions can pay this epoch. If not, advance emits an | ||
| // EmissionsBlocked attestation per outpost (deduped by the local blocklog |
Collaborator
There was a problem hiding this comment.
why would an emissions crank fail?
Contributor
Author
There was a problem hiding this comment.
The gate returns a non-ready result for four reasons, enumerated by EmissionsBlockReason and visible cross-chain via the EmissionsBlocked attestation:
CONFIG_MISSING--sysio.system::emitcfghas never been set.STATE_UNINITIALIZED--sysio.system::t5statehas not been initialized (setinittimenever ran).TREASURY_EXHAUSTED--t5_distributable - t5_floor - total_distributed <= 0. Computed every advance (pay or non-pay) so we never roll forward into a depleted treasury.BALANCE_INSUFFICIENT-- on a pay-epoch only,sysio.tokenWIRE balance can't coverpending + this-epoch's share. Non-pay epochs don't transfer, so no balance check fires.
All four should not fail in practice -- the gate is defense-in-depth so we report a clean cross-chain signal and hold the epoch index, rather than silently corrupting state.
| }; | ||
|
|
||
| void roa::finalizereg(const name& owner,const uint8_t& status) { | ||
| void roa::finalizereg(const name& owner, const uint8_t& status) { |
Contributor
Author
There was a problem hiding this comment.
This was created before enum support. We could change it but I'm not really sure how many clients that would affect.
Comment on lines
+283
to
+287
| int64_t total_emission = 0; | ||
| int64_t compute_amount = 0; | ||
| int64_t capital_amount = 0; | ||
| int64_t capex_amount = 0; | ||
| int64_t governance_amount = 0; |
Collaborator
There was a problem hiding this comment.
is there any value in using std::vector<TokenAmount> to potentially support other token emissions considering our place in the market?
Contributor
Author
There was a problem hiding this comment.
I'm not sure I'm following -- the four fields here (compute/capital/capex/governance_amount) are category splits of a single WIRE emission per the bps fields on emit_cfg, not different tokens. Did you mean the treasury could emit in tokens other than WIRE (BTC/USDC/etc.), or something else?
…nfigurable Conflict resolution: - types.proto: opp-part3 added 60953-60956 (UNDERWRITE_INTENT_COMMIT, UNDERWRITE_INTENT_REJECT, SWAP_REVERT, DEPOSIT_REVERT). Renumbered ATTESTATION_TYPE_EMISSIONS_BLOCKED 60953 -> 60957 to avoid collision. EmissionsBlockReason enum preserved unchanged. - contracts/sysio.epoch/sysio.epoch.wasm: ours (emissions branch has the contract source changes). - contracts/sysio.msgch/sysio.msgch.abi: theirs (emissions does not touch msgch; opp-part3 added new attestation lifecycle). - contracts/sysio.uwrit/sysio.uwrit.abi: theirs (same). WASM/ABI artifacts will be regenerated in a follow-up commit so all contracts pick up the renumbered EMISSIONS_BLOCKED.
Picks up: - types.proto renumber (EMISSIONS_BLOCKED 60953 -> 60957) - opp-part3 lifecycle additions (UNDERWRITE_INTENT_COMMIT/REJECT, SWAP_REVERT, DEPOSIT_REVERT) - new sysio.reserv contract from opp-part3
heifner
changed the base branch from
master
to
feature/opp-part3-operator-management
Fc: variant performance improvements
…swap Flip WIRE snapshot magic for hex-dump readability
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
T5 treasury emissions, triggered inline from
sysio.epoch::advancevia a cross-contract readiness gate. Three composable pieces:sysio.epoch::check_emissions_readyreadssysio.system::emitcfg,sysio.system::t5state, andsysio.token's WIRE balance to compute the next epoch's emission. If emissions cannot pay, the gate writes ablocklogrow onsysio.epochand emits anEmissionsBlockedOPP attestation per registered outpost. The chain stays at the prior epoch index until conditions allow a successful pass; the wall-clock duration of the current epoch effectively extends.emit_cfg; per-epoch values are derived insidecompute_epoch_emissionfrom the canonicalepoch_duration_seconsysio.epoch::epochcfg. Governance can retune the epoch frequency without distorting the wall-clock emission curve.emit_cfg.pay_cadence_epochscontrols how many epochs accumulate before payepoch fires. At a 6-min epoch the per-epoch payepoch was firing ~31 inlinesend_wire_transfercalls (21 active + 7 standbys at defaultstandby_end_rank=28+ 3 batch ops) every 6 min, ~310/hr -- the frequency was the cost driver, not the math. Recommended production value is 100 (≈10 h between pays at 6-min epochs);setemitcfgrejects 0.Cadence-aware lifecycle
Every successful
advancequeues two emission-side inline actions in FIFO order:accrueepoch(epoch_index, batch_group_index, per_epoch_emission)-- always. Owns per-epoch state advancement (last_epoch_index,last_epoch_time,last_epoch_emissionfor decay continuity) plus the period accumulators (pending_emission_amount,batch_group_epochs[]).payepoch(epoch_index, batch_op_groups, period_emission)-- pay-epochs only. Drainspending_emission_amountandbatch_group_epochs, distributes the period total across producer / batch / capital / capex / governance pools, resets the accumulator, and advancesperiod_start_epoch.Pay-epoch decision:
target_epoch >= period_start_epoch + pay_cadence_epochs - 1. Withpay_cadence_epochs=1this collapses to "every advance is a pay-epoch" -- the legacy per-epoch behavior, which existing tests cover bit-for-bit.TREASURY_EXHAUSTEDgates every epoch (pay or non-pay) so the chain cannot silently roll forward into a depleted treasury. The balance-coverage check fires only on pay-epochs against the period total (pending + this-epoch's per-epoch share), so a non-pay epoch never readssysio.token::accounts.Producer / batch pool math under cadence > 1
eligible_rounds / expected_rounds_period, whereexpected_rounds_period = epoch_duration_sec * pay_cadence_epochs * 2 / TOTAL_BLOCKS_PER_ROUND. Counters accumulate across non-pay epochs (accrueepochdoes not reset them); onlypayepochclears them. Below the truncate-to-zero floor (~126 s effective period duration, e.g. 60 s epoch with cadence=1),expected_roundsclamps to 1 and pay collapses to "elig_rounds==0 ? 0 : full_share" -- same fallback as before.cfg.standby_end_rank). Paid every pay-epoch unconditionally by rank-decreasing weight; noeligible_roundsrequirement. Defaultstandby_end_rank=28gives 7 standbys; safety cap is 100.batch_pooldivides into per-group slices weighted by each group's active-epoch count over the period:group_pool = batch_pool * batch_group_epochs[g] / pay_cadence_epochs. Members not registered as ACTIVE insysio.opreg(slashed / terminated / unknown) are skipped; their slice stays in the treasury. Groups that were active in zero epochs (only possible whenpay_cadence_epochs < batch_op_groups.size()) are skipped the same way.epochloggains one row per pay-epoch; non-pay epochs add nothing. Retention pruning is unchanged (head-first, capped atcfg.epoch_log_retention_count).emission_config (governance-tunable via
setemitcfg)Node-owner allocations
t1_allocation,t2_allocation,t3_allocation(int64, WIRE subunits) -- total vesting amount per tier registration.t1_duration,t2_duration,t3_duration(uint32, seconds) -- linear vesting period per tier.min_claimable(int64) -- minimum claim amount; smaller claims are deferred until vested or threshold is met.T5 treasury (annual semantics)
t5_distributable(int64) -- total amount available for T5 emissions across the schedule.t5_floor(int64) -- treasury reserve. Emissions stop oncet5_distributable - t5_floor - total_distributed <= 0.target_annual_decay_bps(uint16,(0, 10000]) -- surviving fraction per year;compute_per_epoch_decayderives the per-epoch factor from this andepoch_duration_sec.annual_initial_emission(int64) -- E_0 expressed per year.annual_max_emission,annual_min_emission(int64) -- per-year ceiling / floor; per-epoch values derive viascale_annual_to_epoch.Pay cadence
pay_cadence_epochs(uint16,> 0) -- how many epochs accumulate before payepoch fires. Default helper sets 1; recommended production value 100; soak cluster bootstrap uses 2.Audit-log retention
epoch_log_retention_count(uint32) -- capsepochlogrows. payepoch prunes head-first up to two rows per call to drain after a retention-cap shrink.Category splits (basis points; must sum to 10000)
compute_bps,capital_bps,capex_bps,governance_bps.Compute sub-split (basis points; must sum to 10000)
producer_bps,batch_op_bps.Producer config
standby_end_rank(uint32,[22, 100]) -- last standby rank that receives pay.Hardcoded economic constants (in
emissions.hpp)T1_MAX_NODE_OWNERS = 21,T2_MAX_NODE_OWNERS = 84,T3_MAX_NODE_OWNERS = 1000-- per-tier caps; same constants used bysysio.roa::activateroafor pool seeding.ACTIVE_PRODUCER_COUNT = 21,STANDBY_START_RANK = 22,MAX_STANDBY_END_RANK = 100.BPS_DENOMINATOR = 10000.TOTAL_BLOCKS_PER_ROUND = 21 * 12 = 252(per-rotation block count).New / changed actions
payepoch(epoch_index, batch_op_groups, period_emission)(sysio.system,require_auth(sysio.epoch)) -- pays one pay period; called inline from advance. Signature changed from the per-epoch shape: takes the fullbatch_op_groupsvector (so it can pay any group that was active in the period) and the period total.accrueepoch(epoch_index, batch_group_index, per_epoch_emission)(sysio.system,require_auth(sysio.epoch)) -- new. Per-epoch state advancement + accumulator update; runs every successful advance.setemitcfg,setinittime,addnodeowner,claimnodedis,viewnodedist,initt5,viewepoch,viewemitcfg-- emissions configuration and queries.sysio.epoch::advance-- runs the readiness gate before any state mutation; on block, writesblocklogrow and queuesEmissionsBlockedattestations. On gate-pass, queuesaccrueepoch(always) pluspayepoch(pay-epochs only).sysio.roa::finalizereg-- gainsrequire_auth(get_self())(was missing in master, allowing any caller to bypass registration); corrects status validation so confirmation is reachable.New protobuf types
ATTESTATION_TYPE_EMISSIONS_BLOCKED = 60953.EmissionsBlockedmessage (epoch_index,reason,attempted_emission,treasury_remaining,sysio_balance,first_blocked_at).EmissionsBlockReasonenum (UNSPECIFIED, CONFIG_MISSING, STATE_UNINITIALIZED, TREASURY_EXHAUSTED, BALANCE_INSUFFICIENT).Removed
processepochaction (cranker entry; no caller exists in this design).prunesnapaction andbatchsnaptable on sysio.epoch.[[sysio::contract(\"...\")]]attribute on each table prevents ABI pollution).