fix(viper): use Asset.modified as webhook sync anchor (#158)#167
Conversation
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (10)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ViperWebhookResponseList.from_request filtered on Asset.last_pinged, but the TapirXL upsert path (PUT /api/assets/upsert/) never stamps that field. First sync after a fresh boot therefore returned an empty queryset, the webhook reported 202, and no POST reached Viper. Switch Asset to django-extensions' TimeStampedModel and use the new modified field (auto-stamped on every save, including upserts) as the sync anchor. last_pinged is retained but its meaning is narrowed to "last observed on the wire" -- only ping/fingerprint code paths set it. The accompanying data migration backfills: created = date_added modified = COALESCE(last_pinged, date_added, NOW()) so the webhook keeps returning the same set it would have if last_pinged had been correctly populated, and the demo workaround (init/backfill-last-pinged.sh) is no longer needed. date_added is dropped from Asset entirely; AssetFilter and the CSV exporter switch the public field name to "created". Other models that still use date_added are out of scope for this bug fix. Closes #158 Refs #161
44387eb to
f311712
Compare
Summary
Assetto inherit from django-extensions'TimeStampedModel, giving uscreated+modifiedfields.ViperWebhookResponseList.from_request) to filter and order onmodifiedrather thanlast_pinged, so TapirXL-upserted assets (which never setlast_pinged) are included on the first sync.Asset.date_added(its semantic role is nowcreated); renames the publicAssetFilterfield and CSV exporter column tocreated.last_pingedas "last observed on the wire" — only ping/fingerprint code paths still touch it.Why
last_pingedwas the wrong sync anchor. It is only set by code paths that observe an asset on the wire (ping, fingerprint), not by the upsert endpoint that scanning tools like TapirXL use. On a fresh boot, no asset hadlast_pingedyet, so the webhook'slast_pinged__gtefilter returned empty and no POST reached Viper. The bug surfaced asTask viper_webhook succeeded in 0.01s: Nonewith zero downstream effect.modifiedis auto-stamped on every save — including upserts — so it is a reliable "this row recently changed" anchor.Migrations
Two new migrations, applied in order:
0009_asset_time_aware— addscreated/modifiedwithdefault=timezone.nowso existing rows are populated, thenRunPythonbackfills the real values (created = date_added,modified = COALESCE(last_pinged, date_added, NOW())) usingupdate()to bypass theauto_now/auto_now_addpre_save hooks, thenRemoveField(date_added). Reverse is symmetric.0010_asset_time_aware_finalize— tightens the field definitions to match the model (removes the bootstrap defaults) and folds in a pre-existingviperwebhookjob.callbackAlterFieldthat develop was missing a migration for.Test plan
uv run pytest blueflow/celery/tests/test_viper.py— 6/6 pass (5 existing + 1 new regression test)test_viper_webhook_includes_assets_without_last_pingedcreates an asset withlast_pinged=None(mirroring the TapirXL upsert path) and asserts the webhook still forwards ituv run python project/manage.py makemigrations --check --dry-run— "No changes detected"init/backfill-last-pinged.shfrominit/integrate.shand confirmingjust integratepropagates all assets to Viper (this is the literal acceptance gate from release: shipvirtalabsinc/blueflow:demo-0.3.xfor ARPA-H hackathon #161 before taggingdemo-0.3.1)Breaking changes
/api/assets/no longer returnsdate_added; returnscreatedandmodifiedinstead.?date_added=query parameter is removed;?created=replaces it (DateRangeFilterandDateTimeFromToRangeFilter).date_addedforcreated.demo-0.3.xtrain per release: shipvirtalabsinc/blueflow:demo-0.3.xfor ARPA-H hackathon #161's milestone scope ("Full REST audit and cleanup are deferred to post-hackathon").Out of scope
Other models that still carry their own
date_added(Network,Vulnerability,Group,Tag,Scan,Attachment,AssetCustomField{Name}) are deliberately untouched. This PR sets theTimeStampedModelprecedent; their migration is a follow-up under a non-hotfix milestone.E2E test
Closes #158
Refs #161