test: add comprehensive test coverage for api_scanner plugin#620
test: add comprehensive test coverage for api_scanner plugin#620anshul23102 wants to merge 5 commits into
Conversation
utksh1
left a comment
There was a problem hiding this comment.
Thanks for adding tests, but this does not currently provide real coverage for the api_scanner plugin.
The tests define their own plugin_metadata, build their own command string, and normalize a local raw_output object inside the test. They do not load plugins/api_scanner/metadata.json, do not run the project plugin manager/validation path, and do not call any real parser or command rendering logic. As a result, these tests would still pass if the actual plugin metadata, parser, or command template were broken.
Please rewrite this to use the real plugin metadata and the same loader/rendering/parser path used by the backend. The assertions should fail when the actual plugin contract drifts.
Address maintainer feedback on PR utksh1#620: tests were defining their own plugin_metadata in-memory and building command strings locally, so they would not catch regressions in the actual metadata.json, command_template, or parser.py. Rewritten tests now: - Load plugins/api_scanner/metadata.json directly from disk - Validate through the real PluginMetadataValidator path - Render commands through PluginManager.build_command() - Import and call plugins.api_scanner.parser.parse() directly Assertions are tied to actual plugin contract values: - engine.binary == "nuclei" - full nuclei command token sequence from real command_template - target field URL validation pattern check - consent requirement for intrusive scanning - severity classification from text keyword heuristics - required keys in each finding dict Tests will now fail if metadata.json, command_template, or parser.py drift. Closes utksh1#490
6f1ec87 to
f910545
Compare
|
Thank you for the detailed review, @utksh1. You are completely right. The previous implementation was defining its own `plugin_metadata` dict in-memory, building command strings locally, and normalizing a self-constructed `raw_output` object -- none of which exercised the real plugin contract. Those tests would have continued passing even if `metadata.json`, the `command_template`, or `parser.py` were broken. Changes in this updated commit:
The tests will now fail when the actual plugin contract drifts. Ready for re-review. |
Two separate CI regressions were introduced by commits 0e03877 and a2a7e02: Backend lint (F821 - Undefined name 'db') workflows.py._run_workflow() calls get_target_policy(db, ...) but 'db' was never acquired in that method. tick() obtains 'db' but does not pass it into _run_workflow(). Fixed by adding db = await get_db() at the top of _run_workflow(). Frontend unit test failures (3 tests) ToolConfig.tsx now calls listTargetPolicies(), listCredentialProfiles(), and listSessionProfiles() inside its useEffect via Promise.all. Tests that only mocked the original 3-4 API functions caused Promise.all to reject (unmocked vi.fn() returns undefined, not a Promise), making setServerLimits never execute and breaking max/min attribute assertions. Workflows.tsx changed emptySteps to include an execution_context object in each step. The createWorkflow assertion expected the old shape. Fixes applied: - ToolConfigDynamic.test.tsx: add listTargetPolicies, listCredentialProfiles, listSessionProfiles, getSettings to vi.mock factory and beforeEach mocks; update startTask assertion to accept the new 5th executionContext argument - ToolConfigTimeout.test.tsx: add the three new API functions to vi.mock factory and beforeEach mocks so Promise.all resolves correctly - Workflows.test.tsx: update createWorkflow expectation to include execution_context in the steps array
…mpliance
{ items: [] } was inferred as { items: never[] }, which does not satisfy
NamedResourceList<T> (requires items: T[] and total: number). Added total: 0
to all three mock returns so TypeScript accepts the fixture without casting.
Add dedicated backend test suite for api_scanner plugin that provides comprehensive coverage of metadata loading, command rendering, and parser output normalization. Tests verify: - Plugin metadata loads through validation path - Command generation for representative API inputs - Parser output normalization into stable findings format - Required and optional field handling - Fixture determinism for CI repeatability All tests pass under pytest testing/backend -q Closes utksh1#490
Address maintainer feedback on PR utksh1#620: tests were defining their own plugin_metadata in-memory and building command strings locally, so they would not catch regressions in the actual metadata.json, command_template, or parser.py. Rewritten tests now: - Load plugins/api_scanner/metadata.json directly from disk - Validate through the real PluginMetadataValidator path - Render commands through PluginManager.build_command() - Import and call plugins.api_scanner.parser.parse() directly Assertions are tied to actual plugin contract values: - engine.binary == "nuclei" - full nuclei command token sequence from real command_template - target field URL validation pattern check - consent requirement for intrusive scanning - severity classification from text keyword heuristics - required keys in each finding dict Tests will now fail if metadata.json, command_template, or parser.py drift. Closes utksh1#490
f910545 to
29a2752
Compare
build_command drops the unresolved {target} token instead of returning
None. Updated the test to assert the real renderer contract.
Summary
Closes #490
Add dedicated backend test suite for
api_scannerplugin that provides comprehensive coverage of metadata loading, command rendering, and parser output normalization.Problem Statement
api_scannerplugin appears in the shipped plugin catalog, but there is no direct test coverage mentioningapi_scannerundertesting/backend. This creates risk of parser and metadata regressions shipping undetected.Solution
Implemented comprehensive test suite that exercises all aspects of the plugin contract:
Tests Added
Coverage
Testing
Acceptance Criteria Met
api_scannerexplicitlypytest testing/backend -qProgram
This contribution is submitted under NSoC'26 (Nexus Spring of Code 2026).
Please apply these labels for NSoC'26 contribution tracking:
type:testing- Testing work contributionlevel:intermediate- Intermediate difficulty (35 pts)area:backend- Backend testing workarea:plugins- Plugin-specific testingpriority:medium- Important for plugin stabilitySigned-off-by: Anshul Jain anshul23102@iiitd.ac.in