agent: document multicast device capacity and add capacity eval#654
Open
armcconnell wants to merge 1 commit into
Open
agent: document multicast device capacity and add capacity eval#654armcconnell wants to merge 1 commit into
armcconnell wants to merge 1 commit into
Conversation
The agent had no guidance on device-level multicast capacity and would reason over the stale on-chain multicast_subscribers_count / multicast_publishers_count columns on dz_devices_current, producing nonsensical answers to questions like "which DZDs have capacity for new subscribers" (#651). Add a "Device multicast capacity" section to SQL_CONTEXT.md that: - marks the on-chain device count columns as stale and not to be used for counts - defines current subscribers/publishers as activated multicast users on the device with a non-empty subscribers/publishers array (from dz_users_current) - defines available capacity as max_multicast_subscribers - current, only when max_multicast_subscribers > 0 (0 means unknown, not full) - includes a canonical "which devices have capacity" query Add the MulticastDeviceCapacity eval, which seeds devices whose stale on-chain counts deliberately disagree with their attached users, so the eval fails if the agent reads the device count columns instead of counting live users.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves: #651
Summary of Changes
SQL_CONTEXT.mdso it can answer questions like "which DZDs have capacity for new subscribers". Previously the agent had no definition of device capacity and would reason over the stale on-chainmulticast_subscribers_count/multicast_publishers_countcolumns ondz_devices_current, producing nonsensical answers.subscribers/publishersarray (fromdz_users_current); defines available capacity asmax_multicast_subscribers - current, only meaningful whenmax_multicast_subscribers > 0(0 means unknown, not full); and includes a canonical "which devices have capacity" query.MulticastDeviceCapacityeval. It seeds devices whose stale on-chain counts deliberately disagree with their attached users (a: on-chain 0 vs live 2; c: on-chain 8 vs live 0), so the eval fails if the agent reads the device count columns instead of counting live users.This is the agent-facing half of the multicast count problem. The data/display half is #650 (PR #653).
Diff Breakdown
Prompt guidance plus a new eval; no production code paths change.
Testing Verification
MulticastDeviceCapacityeval passes against the live agent (claude-haiku-4-5): the agent generated a query counting live subscribers viacountIf(JSONLength(subscribers) > 0)and computingmax - current, reportednyc-dzd-c(10 available) andnyc-dzd-a(3 available), correctly excluded the fullnyc-dzd-b, and used live counts (a=2, c=0) rather than the stale on-chain values (a=0, c=8). The eval's LLM judge passed all three expectations.