Skip to content

fix(viper): use Asset.modified as webhook sync anchor (#158)#167

Merged
t-a-y-l-o-r merged 1 commit into
developfrom
bug/158-time-aware
May 20, 2026
Merged

fix(viper): use Asset.modified as webhook sync anchor (#158)#167
t-a-y-l-o-r merged 1 commit into
developfrom
bug/158-time-aware

Conversation

@t-a-y-l-o-r
Copy link
Copy Markdown
Collaborator

@t-a-y-l-o-r t-a-y-l-o-r commented May 20, 2026

Summary

  • Switches Asset to inherit from django-extensions' TimeStampedModel, giving us created + modified fields.
  • Repoints the Viper webhook (ViperWebhookResponseList.from_request) to filter and order on modified rather than last_pinged, so TapirXL-upserted assets (which never set last_pinged) are included on the first sync.
  • Drops Asset.date_added (its semantic role is now created); renames the public AssetFilter field and CSV exporter column to created.
  • Re-documents last_pinged as "last observed on the wire" — only ping/fingerprint code paths still touch it.

Why

last_pinged was the wrong sync anchor. It is only set by code paths that observe an asset on the wire (ping, fingerprint), not by the upsert endpoint that scanning tools like TapirXL use. On a fresh boot, no asset had last_pinged yet, so the webhook's last_pinged__gte filter returned empty and no POST reached Viper. The bug surfaced as Task viper_webhook succeeded in 0.01s: None with zero downstream effect.

modified is auto-stamped on every save — including upserts — so it is a reliable "this row recently changed" anchor.

Migrations

Two new migrations, applied in order:

  1. 0009_asset_time_aware — adds created/modified with default=timezone.now so existing rows are populated, then RunPython backfills the real values (created = date_added, modified = COALESCE(last_pinged, date_added, NOW())) using update() to bypass the auto_now/auto_now_add pre_save hooks, then RemoveField(date_added). Reverse is symmetric.
  2. 0010_asset_time_aware_finalize — tightens the field definitions to match the model (removes the bootstrap defaults) and folds in a pre-existing viperwebhookjob.callback AlterField that develop was missing a migration for.

Test plan

  • uv run pytest blueflow/celery/tests/test_viper.py — 6/6 pass (5 existing + 1 new regression test)
  • New test test_viper_webhook_includes_assets_without_last_pinged creates an asset with last_pinged=None (mirroring the TapirXL upsert path) and asserts the webhook still forwards it
  • Asset/relations/history/usage/vulnerability test suite: 87 pass / 52 fail / 9 errors — exactly one more passing than the develop baseline (52/86/9); no new failures
  • uv run python project/manage.py makemigrations --check --dry-run — "No changes detected"
  • Validate end-to-end against the PATCH VMP demo repo by removing init/backfill-last-pinged.sh from init/integrate.sh and confirming just integrate propagates all assets to Viper (this is the literal acceptance gate from release: ship virtalabsinc/blueflow:demo-0.3.x for ARPA-H hackathon #161 before tagging demo-0.3.1)

Breaking changes

  • Public API: /api/assets/ no longer returns date_added; returns created and modified instead.
  • ?date_added= query parameter is removed; ?created= replaces it (DateRangeFilter and DateTimeFromToRangeFilter).
  • The default CSV export header swaps date_added for created.
  • Acceptable for the demo-0.3.x train per release: ship virtalabsinc/blueflow:demo-0.3.x for ARPA-H hackathon #161's milestone scope ("Full REST audit and cleanup are deferred to post-hackathon").

Out of scope

Other models that still carry their own date_added (Network, Vulnerability, Group, Tag, Scan, Attachment, AssetCustomField{Name}) are deliberately untouched. This PR sets the TimeStampedModel precedent; their migration is a follow-up under a non-hotfix milestone.

E2E test

Screenshot 2026-05-20 at 11 37 13

Closes #158
Refs #161

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 20, 2026

Warning

Rate limit exceeded

@t-a-y-l-o-r has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 1 minute and 11 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6631be85-393d-46c4-bc47-d397f13f4acf

📥 Commits

Reviewing files that changed from the base of the PR and between ad7b396 and f311712.

⛔ Files ignored due to path filters (1)
  • uv.lock is excluded by !**/*.lock
📒 Files selected for processing (10)
  • blueflow/celery/tests/test_viper.py
  • blueflow/management/commands/create_assets.py
  • blueflow/migrations/0009_asset_time_aware.py
  • blueflow/migrations/0010_asset_time_aware_finalize.py
  • blueflow/models/asset.py
  • blueflow/models/viper.py
  • blueflow/views/asset.py
  • data/assets.json
  • project/settings/base.py
  • pyproject.toml
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch bug/158-time-aware

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@t-a-y-l-o-r t-a-y-l-o-r self-assigned this May 20, 2026
@t-a-y-l-o-r t-a-y-l-o-r marked this pull request as ready for review May 20, 2026 18:38
ViperWebhookResponseList.from_request filtered on Asset.last_pinged, but
the TapirXL upsert path (PUT /api/assets/upsert/) never stamps that field.
First sync after a fresh boot therefore returned an empty queryset, the
webhook reported 202, and no POST reached Viper.

Switch Asset to django-extensions' TimeStampedModel and use the new
modified field (auto-stamped on every save, including upserts) as the
sync anchor. last_pinged is retained but its meaning is narrowed to
"last observed on the wire" -- only ping/fingerprint code paths set it.

The accompanying data migration backfills:
  created  = date_added
  modified = COALESCE(last_pinged, date_added, NOW())
so the webhook keeps returning the same set it would have if last_pinged
had been correctly populated, and the demo workaround
(init/backfill-last-pinged.sh) is no longer needed.

date_added is dropped from Asset entirely; AssetFilter and the CSV
exporter switch the public field name to "created". Other models that
still use date_added are out of scope for this bug fix.

Closes #158
Refs #161
@t-a-y-l-o-r t-a-y-l-o-r force-pushed the bug/158-time-aware branch from 44387eb to f311712 Compare May 20, 2026 18:43
Copy link
Copy Markdown
Contributor

@xnorkl xnorkl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lg2m

@t-a-y-l-o-r t-a-y-l-o-r merged commit f65496b into develop May 20, 2026
3 checks passed
@t-a-y-l-o-r t-a-y-l-o-r deleted the bug/158-time-aware branch May 20, 2026 18:56
@xnorkl xnorkl added this to the v0.3.0-Hackathon milestone May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Asset.last_pinged filter in Viper webhook produces empty queryset

2 participants