Hunt-page ruleset tracking: favorites, rule counts, hunt provenance

specs/02-resources.md

@@ -36,7 +36,9 @@ BaseResource                          # holds .api, ._content, .parse_result cla
     ├── HistoricalHunt                # /hunt/historical
     ├── HistoricalHuntResult / List   # /hunt/historical/results (+ /results/list)
     ├── LiveHuntResult / List         # /hunt/live (+ /hunt/live/list)
+    ├── LiveHuntResultCounts          # /hunt/live/results/count
     ├── YaraRuleset                   # /hunt/rule
+    ├── YaraRulesetFavorite           # /hunt/rule/favorite
     ├── Tag, MalwareFamily, TagLink   # /tags/tag, /tags/family, /tags/link
     ├── AssertionsJob, VotesJob       # /consumer/assertions-job, /consumer/votes-job
     ├── SandboxTask, SandboxProvider  # /sandbox/sandboxtask, /sandbox/provider
@@ -343,7 +345,9 @@ Holds `handle`, `artifact_name`, `artifact_type`, `sha256`, `sha1`, `md5`. Also
 Several resources add domain-specific classmethods on top of the standard CRUD set:
 
 - `IOC` — `iocs_by_hash`, `ioc_search`, `check_known_hosts`, `create_known_good`, `create_known_bad`, `update_known_good`, `delete_known_good`.
-- `LiveYaraRuleset` / `HistoricalHunt` / `YaraRuleset` — standard CRUD plus list/delete-batch variants.
+- `LiveYaraRuleset` / `HistoricalHunt` / `YaraRuleset` — standard CRUD plus list/delete-batch variants. `YaraRuleset` also parses the hunt-page tracking fields (`favorite`, `favorited_at`, `rule_count` — `None` means the server had no answer, distinct from 0 — `historical_hunt_count`, `new_results_count`), and `HistoricalHunt` the source-rule provenance (`rule_id`, `rule_modified`, and the tri-state `source_rule_changed` — `None` is "unknown", never "unchanged"). All additive `.get()` parses; an older server leaves them `None`.
+- `YaraRulesetFavorite` — `RESOURCE_ENDPOINT = '/hunt/rule/favorite'`, **`RESOURCE_ID_KEYS = []`**: a deliberate deviation from the default `['id']`, because the server reads the toggle exclusively from the PUT **body** — emptying the key list is the only thing routing `id` (plus `favorite`/`community`) into `json` instead of the query string. `favorite` serialises as `1`/`0` (the `_params` bool→int coercion), which the server's boolean parser accepts. Response carries the star state plus the team's `favorites_used`/`favorites_limit`. Pinned by `test/hunt_tracking_builder_test.py`.
+- `LiveHuntResultCounts` — `RESOURCE_ENDPOINT = '/hunt/live/results/count'`, GET-only; `since` (seconds) rides the query and is omitted when unset (server default window). `counts` is a list of `{livescan_id, count}` with `livescan_id` as a digit string — the same join key `YaraRuleset.livescan_id` carries; a hunt absent from `counts` collected 0.
 - `SandboxTask` — `create_file`, `update_file`, `latest`, `my_tasks` for the various sandbox-submission shapes. **No `upload_file` instance method** in 4.0.
 - `Sample` — `create` with `endpoint_fmt={'sha256': sha256}` for the URL-parametrised path.
 - `Webhook` — `test(api, webhook_id)` for the test-payload endpoint.

specs/03-endpoints.md

@@ -66,6 +66,7 @@ Internal-only CRUD for the `/known-good` binary resource (distinct from the IOC
 | `live_stop(rule_id)` | `LiveYaraRuleset.delete` |
 | `live_feed_delete(result_ids)` | `LiveHuntResultList.delete` (catches `NoResultsException`) |
 | `live_result(result_id)` | `LiveHuntResult.get` |
+| `live_results_count(since=None)` | `LiveHuntResultCounts.get` — per-live-hunt result counts in a window, grouped by `livescan_id` (digit strings, the same join key `YaraRuleset.livescan_id` carries); a hunt absent from `counts` collected 0 |
 
 ### Historical hunts
 
@@ -86,6 +87,7 @@ Internal-only CRUD for the `/known-good` binary resource (distinct from the IOC
 | `ruleset_create(name, rules, description=None)` | `YaraRuleset.create` |
 | `ruleset_get(ruleset_id=None)` | `YaraRuleset.get` |
 | `ruleset_update(ruleset_id, name=None, rules=None, description=None)` | `YaraRuleset.update` |
+| `ruleset_favorite(ruleset_id, favorite=True)` | `YaraRulesetFavorite.update` — idempotent star/unstar; the response carries the team's `favorites_used`/`favorites_limit`, and an over-budget star is refused with a machine-readable `FAVORITE_LIMIT` error. The id rides the PUT **body**, not the query (see the `RESOURCE_ID_KEYS = []` note in specs/02) |
 | `ruleset_delete(ruleset_id)` | `YaraRuleset.delete` |
 | `tag_link_get(sha256)` | `TagLink.get` |
 | `tag_link_update(sha256, tags=None, families=None, emerging=None, remove=False)` | `TagLink.update` |
@@ -186,10 +188,10 @@ refusal.
 | `iocs_by_hash(hash_type, hash_value, hide_known_good=False, beta=False)` | `IOC.iocs_by_hash` |
 | `search_by_ioc(ip=None, domain=None, ttp=None, imphash=None)` | `IOC.ioc_search` |
 | `check_known_hosts(ips=[], domains=[])` | `IOC.check_known_hosts` |
-| `live_feed(since=None, …)` | `LiveHuntResult.list` |
+| `live_feed(since=None, …, livescan_id=None)` | `LiveHuntResult.list` — `livescan_id` scopes the feed to one live hunt (the hunt-page per-ruleset feed) |
 | `historical_list(since=None)` | `HistoricalHunt.list` |
 | `historical_results(hunt=None, …)` | `HistoricalHuntResultList.get` |
-| `ruleset_list()` | `YaraRuleset.list` |
+| `ruleset_list(name=None, status=None, favorites_only=None, has_new_results=None, since=None, include_counts=None)` | `YaraRuleset.list` — the hunt-page filters, conjunctive and optional; unset filters are omitted from the query so the no-filter request is byte-compatible with the old contract |
 | `tag_list()` | `Tag.list` |
 | `family_list()` | `MalwareFamily.list` |
 | `assertions_list(engine_id)` | `AssertionsJob.list` |

specs/04-testing.md

@@ -26,7 +26,8 @@ How the test suite is organised. Three layers: pure unit tests (no HTTP at all 
 - `test/async_client_test.py` — async, VCR-backed integration tests (not yet on the parametrised harness — follow-up work).
 - `test/jmespath_test.py` — unit tests for `BaseJsonResource.jmespath`.
 - `test/vcr/*.vcr` — recorded cassettes.
-- `test/eicar.yara`, `test/malicious` — fixture files for upload tests.
+- `test/malicious` — fixture file for upload tests (`test/eicar.yara` was retired when the rules tests moved to per-test `uid_yara` bodies).
+- `test/hunt_tracking_builder_test.py` — pure-unit request-shape and parse tests for the hunt-page tracking builders/resources.
 
 ## Three test layers
 

specs/05-downstream-contract.md

@@ -147,9 +147,9 @@ ArtifactInstance, LocalArtifact, Hash
 Engine
 Metadata, MetadataMapping, MetadataFieldProperties
 IOC
-LiveYaraRuleset, LiveHuntResult, LiveHuntResultList
+LiveYaraRuleset, LiveHuntResult, LiveHuntResultList, LiveHuntResultCounts
 HistoricalHunt, HistoricalHuntResult, HistoricalHuntResultList, HistoricalHuntList
-YaraRuleset
+YaraRuleset, YaraRulesetFavorite
 Tag, MalwareFamily, TagLink
 AssertionsJob, VotesJob
 SandboxTask, SandboxProvider
@@ -180,6 +180,8 @@ class TimeoutException(PolyswarmException): ...
 
 `KnownGoodWithheldException` is the 404 raised when a download is refused because the artifact is a known-good binary — the platform never stores or serves those bytes. It **subclasses `NotFoundException`** precisely so invariant 3 holds for existing consumers: code that already does `except NotFoundException:` keeps catching the refusal with no change, and only callers that want to distinguish "withheld by design" from a plain miss catch the subclass. It adds one attribute, `.sources` — the known-good feeds that flagged the hash (e.g. `['nsrl']`), `[]` when the server named none. The contract is only this: **always a list of strings**, whatever the envelope carried, so `for feed in exc.sources` needs no shape check. Which wire shapes are coerced, and which are dropped and logged, is `exceptions._normalise_sources`' business rather than a promise to consumers — the server sends a list of strings today. Note it is **not** normalised the same way as `ArtifactInstance.known_good_sources`, which is the same concept reached from the instance response: that one is sorted and de-duplicated, while `.sources` preserves the order the envelope carried and can repeat a feed. Don't assume parity between the two. The raw envelope stays reachable at `exc.request.errors` (`{'code': 'KNOWN_GOOD', 'known_good': True, 'sources': [...]}`). The artifact's metadata — the flagging feeds plus any scan data already collected — remains readable through the search / instance endpoints; only the bytes are withheld, and the instance's `KNOWN_GOOD` state/status is the signal for the typed refusal (there is no separate "withheld" field; a `NOT_STORED` instance has no bytes either, but 404s plainly — see below).
 
+`FAVORITE_LIMIT` (a refused ruleset star: the team's favorite budget is spent) has **no typed exception** — deliberately, since no pre-existing consumer needs re-routing the way `NotFoundException` did. It surfaces as the generic 400 `RequestException`, and the machine-readable path is the raw envelope: `exc.request.errors == {'code': 'FAVORITE_LIMIT', 'favorites_used': N, 'favorites_limit': M}`. The counters are the same pair a successful toggle returns, so a caller can render the "budget full" state from either outcome. Pinned by the respx refusal test in `client_scan_test.py`.
+
 **What "known-good" means on the server, as of artifact-index's two-predicate model** (its `specs/05`): the refusal fires on the server's *current understanding* — a catalogue entry exists for the sha256 **and** that entry's extension passes an executable allow-list — evaluated live on every request. Two consequences worth knowing as a consumer: the same download can start working again with no action on your part (the entry is deleted, or the policy narrows), and `ArtifactInstance.state` can report the new value **`NOT_STORED`** — a submission the server declined as known-good at the time whose hash is no longer currently known-good, so nothing was ever stored for it and a fresh submit of the same file works. `state` is a plain string here; the SDK does not enumerate it, so a new member needs no SDK release.
 
 Each `RequestException` subclass carries a `.request` attribute holding the originating `PolyswarmRequest` (set by `RequestException.__init__`). Callers can read `exc.request.status_code`, `exc.request.json` (the parsed response body after execution), `exc.request.input_json` (the body that was sent), `exc.request.request_parameters` (the request kwargs that built the call), etc. `InvalidValueException` and `TimeoutException` are client-side errors and don't carry a request descriptor.

src/polyswarm_api/aio/api.py

@@ -502,7 +502,8 @@ async def live_stop(self, rule_id):
         return await self._single(resources.LiveYaraRuleset.delete(self, rule_id=rule_id))
 
     async def live_feed(self, since=None, rule_name=None, family=None,
-                           polyscore_lower=None, polyscore_upper=None, community=None):
+                           polyscore_lower=None, polyscore_upper=None, community=None,
+                           livescan_id=None):
         """
         Get live hunts feed
 
@@ -512,14 +513,29 @@ async def live_feed(self, since=None, rule_name=None, family=None,
         :param polyscore_lower: Polyscore lower bound for the hunt results.
         :param polyscore_upper: Polyscore upper bound for the hunt results.
         :param community: Community to retrieve live results from, or public/private.
+        :param livescan_id: Scope the feed to one live hunt's results.
         :return: Generator of HuntResult resources
         """
         async for item in self._paginate(resources.LiveHuntResult.list(
             self, since=since, rule_name=rule_name, family=family,
             polyscore_lower=polyscore_lower, polyscore_upper=polyscore_upper,
+            livescan_id=livescan_id,
             community=community or self.community)):
             yield item
 
+    async def live_results_count(self, since=None):
+        """
+        Per-live-hunt result counts for the current account, grouped by
+        livescan_id. One request answers every "new results in the window"
+        badge; a hunt absent from counts collected 0.
+
+        :param since: Window in seconds (server default: 86400 — 24 hours)
+        :return: A LiveHuntResultCounts resource
+        """
+        logger.info('Live results count since %s', since)
+        return await self._single(
+            resources.LiveHuntResultCounts.get(self, since=since, community=self.community))
+
     async def live_feed_delete(self, result_ids):
         """
         Delete live feed results
@@ -684,15 +700,47 @@ async def ruleset_delete(self, ruleset_id):
         logger.info('Delete ruleset %s', ruleset_id)
         return await self._single(resources.YaraRuleset.delete(self, id=ruleset_id, community=self.community))
 
-    async def ruleset_list(self):
+    async def ruleset_list(self, name=None, status=None, favorites_only=None,
+                           has_new_results=None, since=None, include_counts=None):
         """
         List all YaraRulesets for the current account.
+
+        All filters are optional and conjunctive:
+        :param name: Case-insensitive substring match on the ruleset name.
+        :param status: 'active' returns only rulesets whose live hunt is
+            currently running.
+        :param favorites_only: True returns only favorited rulesets.
+        :param has_new_results: True returns only rulesets whose live hunt
+            collected results inside the window.
+        :param since: Window in seconds for has_new_results/include_counts
+            (server default: 86400).
+        :param include_counts: True attaches new_results_count to each
+            ruleset that has a live hunt.
         :return: A generator of YaraRuleset resources
         """
         logger.info('List rulesets')
-        async for item in self._paginate(resources.YaraRuleset.list(self, community=self.community)):
+        async for item in self._paginate(resources.YaraRuleset.list(
+                self, name=name, status=status, favorites_only=favorites_only,
+                has_new_results=has_new_results, since=since,
+                include_counts=include_counts, community=self.community)):
             yield item
 
+    async def ruleset_favorite(self, ruleset_id, favorite=True):
+        """
+        Favorite or unfavorite a YaraRuleset. Idempotent; works while a live
+        hunt is running. Favorites are shared by the whole team and capped
+        (the response carries favorites_used / favorites_limit); when the
+        budget is exhausted the server refuses with a machine-readable
+        FAVORITE_LIMIT error.
+
+        :param ruleset_id: Id of the ruleset
+        :param favorite: True to star, False to unstar
+        :return: A YaraRulesetFavorite resource
+        """
+        logger.info('%s ruleset %s', 'Favorite' if favorite else 'Unfavorite', ruleset_id)
+        return await self._single(resources.YaraRulesetFavorite.update(
+            self, id=ruleset_id, favorite=favorite, community=self.community))
+
     async def tag_link_get(self, sha256):
         """
         Fetch the Tags and Families associated with the given sha256.

src/polyswarm_api/api.py

@@ -568,6 +568,7 @@ def live_feed(
         polyscore_lower=None,
         polyscore_upper=None,
         community=None,
+        livescan_id=None,
     ):
         """
         Get live hunts feed
@@ -578,6 +579,7 @@ def live_feed(
         :param polyscore_lower: Polyscore lower bound for the hunt results.
         :param polyscore_upper: Polyscore upper bound for the hunt results.
         :param community: Community to retrieve live results from, or public/private.
+        :param livescan_id: Scope the feed to one live hunt's results.
         :return: Generator of HuntResult resources
         """
         for item in self._paginate(
@@ -588,11 +590,28 @@ def live_feed(
                 family=family,
                 polyscore_lower=polyscore_lower,
                 polyscore_upper=polyscore_upper,
+                livescan_id=livescan_id,
                 community=community or self.community,
             )
         ):
             yield item
 
+    def live_results_count(self, since=None):
+        """
+        Per-live-hunt result counts for the current account, grouped by
+        livescan_id. One request answers every "new results in the window"
+        badge; a hunt absent from counts collected 0.
+
+        :param since: Window in seconds (server default: 86400 — 24 hours)
+        :return: A LiveHuntResultCounts resource
+        """
+        logger.info("Live results count since %s", since)
+        return self._single(
+            resources.LiveHuntResultCounts.get(
+                self, since=since, community=self.community
+            )
+        )
+
     def live_feed_delete(self, result_ids):
         """
         Delete live feed results
@@ -823,17 +842,67 @@ def ruleset_delete(self, ruleset_id):
             resources.YaraRuleset.delete(self, id=ruleset_id, community=self.community)
         )
 
-    def ruleset_list(self):
+    def ruleset_list(
+        self,
+        name=None,
+        status=None,
+        favorites_only=None,
+        has_new_results=None,
+        since=None,
+        include_counts=None,
+    ):
         """
         List all YaraRulesets for the current account.
+
+        All filters are optional and conjunctive:
+        :param name: Case-insensitive substring match on the ruleset name.
+        :param status: 'active' returns only rulesets whose live hunt is
+            currently running.
+        :param favorites_only: True returns only favorited rulesets.
+        :param has_new_results: True returns only rulesets whose live hunt
+            collected results inside the window.
+        :param since: Window in seconds for has_new_results/include_counts
+            (server default: 86400).
+        :param include_counts: True attaches new_results_count to each
+            ruleset that has a live hunt.
         :return: A generator of YaraRuleset resources
         """
         logger.info("List rulesets")
         for item in self._paginate(
-            resources.YaraRuleset.list(self, community=self.community)
+            resources.YaraRuleset.list(
+                self,
+                name=name,
+                status=status,
+                favorites_only=favorites_only,
+                has_new_results=has_new_results,
+                since=since,
+                include_counts=include_counts,
+                community=self.community,
+            )
         ):
             yield item
 
+    def ruleset_favorite(self, ruleset_id, favorite=True):
+        """
+        Favorite or unfavorite a YaraRuleset. Idempotent; works while a live
+        hunt is running. Favorites are shared by the whole team and capped
+        (the response carries favorites_used / favorites_limit); when the
+        budget is exhausted the server refuses with a machine-readable
+        FAVORITE_LIMIT error.
+
+        :param ruleset_id: Id of the ruleset
+        :param favorite: True to star, False to unstar
+        :return: A YaraRulesetFavorite resource
+        """
+        logger.info(
+            "%s ruleset %s", "Favorite" if favorite else "Unfavorite", ruleset_id
+        )
+        return self._single(
+            resources.YaraRulesetFavorite.update(
+                self, id=ruleset_id, favorite=favorite, community=self.community
+            )
+        )
+
     def tag_link_get(self, sha256):
         """
         Fetch the Tags and Families associated with the given sha256.