An AI-augmented web crawler, state-machine explorer, and security abuse analyzer engineered specifically for web penetration testing, API route discovery, and business logic threat modeling.

Crawler4Pentest combines deterministic browser automation (via Playwright) with LLM visual-semantic planning (via Google Gemini / Stagehand) to systematically map single-page applications (SPAs), multi-step wizard flows, hidden API endpoints, and complex state transitions.

---

🌟 Key Capabilities

1. 5-Tier Interaction Prioritization

A disciplined queue model ensuring full in-page exploration before navigating away:

• Tier 1 (In-Page Local Forms): Discovers, fills, and submits all in-page forms, search inputs, and filter controls that operate without changing the page.
• Tier 2 (In-Page Standalone Controls): Exercises in-page buttons, tabs, dropdown toggles, details buttons, and sub-form openers that do not navigate to external URLs.
• Tier 3 (Reactive DOM Progression): Evaluates DOM mutations after clicks and form submits. Any newly revealed elements, modal dialogs, popup cards, or multi-step wizard forms (e.g. Step 1 $\rightarrow$ Step 2) immediately jump to the top of the queue to complete relational flows while active on screen.
• Tier 4 (Navigation Forms): Executes forms that trigger full-page submissions or route transitions to new pages.
• Tier 5 (Navigation Links & Routes): Explores topbar/sidebar navigation links and route transitions after all in-page actions are completely exhausted.

2. Dual-Source Entity Context Pool (EntityContextPool)

• Seed Entity Injection: Supply real testing credentials, customer IDs, and business keys via seed_entities.json (e.g. customerId, orderCode, promoCode).
• Dynamic Runtime Harvesting: Automatically extracts newly created IDs, tokens, and entities from network JSON responses and DOM attributes to fulfill foreign-key dependencies in downstream forms.

3. Multi-Step Wizards & Universal Modal Detection

• Detects standard dialogs (<dialog>, [role="dialog"]) and custom fixed/absolute floating overlays (z-index >= 10, #policyModal, #wizardModal).
• Automatically links opener buttons as triggerSelector with built-in state-recovery replay (ensureRevealed) if intermediate steps close the modal.

4. Deep DOM & Modern Web Support

• Shadow DOM Piercing: Discovers and interacts with web components inside shadow roots (>>>).
• Infinite & Virtualized Scroll Crawling: Dynamically scrolls feeds and virtualized lists (e.g. React-Window) to discover lazy-loaded buttons and forms while preventing unwanted URL drift.
• Hover Flyouts: Discovers zero-overhead hover dropdowns and flyout menus.

5. Network Traffic Interception & Security Attribution

• Passive interception of XHR, Fetch, and WebSocket traffic.
• Automatic authentication context extraction: Session Cookies, JWT Authorization: Bearer headers, CSRF tokens, and custom API keys.
• AI-driven API classification (feature_api, background_api, telemetry_noise).
• Built-in noise learning store to suppress analytics, beacons, and telemetry polls.

6. Security Use Case & Abuse Analyzer

• Evaluates the crawler output in output/features.json to generate actionable security assessment reports covering:
  • BFLA (Broken Function Level Authorization)
  • BOLA / IDOR (Insecure Direct Object References)
  • State-Machine & Wizard Sequence Bypasses
  • Form Validation & Rate-Limiting Abuse Vectors

---

🚀 Quick Start

Prerequisites

• Node.js: v18.0.0 or higher
• Chromium / Playwright: Installed automatically or via npx playwright install chromium
• Google Gemini API Key: For AI visual planning and API attribution

1. Installation

# Clone the repository
git clone https://github.com/your-org/Crawler4Pentest.git
cd Crawler4Pentest

# Install dependencies
npm install

# Install Playwright browser binaries
npx playwright install chromium

2. Environment Setup

Create a .env file in the root directory:

# Gemini API Key for AI visual planning
GEMINI_API_KEY=your_gemini_api_key_here
GOOGLE_GENERATIVE_AI_API_KEY=your_gemini_api_key_here

# Optional: HTTP/HTTPS Proxy for upstream traffic interception (e.g. Burp Suite / Caido)
# HTTP_PROXY=http://127.0.0.1:8080
# HTTPS_PROXY=http://127.0.0.1:8080

---

⚙️ Configuration

The crawler is configured via crawl_config.json. See crawl_config.example.json and CrawlConfigGuide.md for complete documentation.

Example crawl_config.json

{
  "start_url": "https://app.example.com/dashboard",
  "allowed_domains": [
    "*.example.com",
    "localhost:4173"
  ],
  "exclusion_domains": [
    "*.analytics.example.com",
    "sentry.io"
  ],
  "crawl": {
    "resume_from_checkpoint": false,
    "skip_initial_noise_learning": false,
    "wait_for_manual_login": false
  },
  "limits": {
    "maxSteps": 150,
    "networkQuietMs": 500,
    "networkSettleTimeoutMs": 8000,
    "maxRequestsPerFeature": 20
  },
  "paths": {
    "outputFile": "output/features.json",
    "seedEntitiesFile": "seed_entities.json"
  }
}

Tip

Wildcard Domain Scope: You can use *.domain.com (or domain.com) in allowed_domains and exclusion_domains. It automatically matches the root apex domain (domain.com) and all subdomains (api.domain.com, auth.staging.domain.com). Specific exclusions (*.analytics.domain.com) take precedence over allowed domains.

Custom Seed Entities (seed_entities.json)

To pre-seed the crawler with valid relational IDs, create a seed_entities.json (see seed_entities.example.json):

{
  "customerId": "701",
  "orderCode": "SFO-TYO-JUL",
  "invoiceId": "INV-24001",
  "promoCode": "SAVE10",
  "email": "test-user@example.com"
}

---

🎯 Running the Crawler

1. Launch the Crawler

npm start

The crawler will execute the 5-tier crawl loop, explore the target application, and output structured feature records to output/features.json.

---

🔐 Authenticated Crawling & Manual Login

For applications protected by multi-factor authentication (MFA), CAPTCHA, OAuth/SSO, or complex login flows, you can perform manual authentication before the automated crawl starts:

1. Configure crawl_config.json: Set "wait_for_manual_login": true and specify your "login_url" (optional if same as start_url):

   {
     "start_url": "https://staging.app.example.com/en/dashboard",
     "crawl": {
       "wait_for_manual_login": true,
       "login_url": "https://staging.app.example.com/en/login"
     }
   }

2. Launch the crawler:

   npm start

3. Complete login in the browser: The crawler will open a visible browser window, navigate to the login page, and pause execution with a prompt in the terminal:

   [BOOTSTRAP] Waiting for manual login. Please log in through the opened browser window and press ENTER to continue...
   

4. Press ENTER to resume: Perform login, complete any MFA/CAPTCHA verification, and verify you are in the logged-in area. Return to the terminal and press ENTER. The crawler will inherit the authenticated session cookies, local storage, and auth headers, and begin the automated crawl.

---

2. Run Built-in Test Applications (Optional)

To test and benchmark the crawler locally, launch any of the included mock applications:

# Multi-page e-commerce & campaign mock app (Port 4173)
node mock-app/server.js

# Single-Page App with Shadow DOM & Relational Modals (Port 4183)
node crawler-test-spa/server.js

# Infinite & Virtualized Scroll Test SPA (Port 4184)
node crawler-scroll-test-spa/server.js

3. Generate Security Assessment Reports

Once output/features.json is generated, run the Security Abuse Analyzer:

# Run analysis from project root
npm run analyze:security

This generates:

• JSON Report: security-usecase-abuse-analyzer/reports/security-assessment.json
• Markdown Report: security-usecase-abuse-analyzer/reports/security-assessment.md

---

📊 Output Schema (features.json)

Each entry in output/features.json captures the full interaction and network attribution trace:

{
  "feature_name": "Finish Package (Wizard Step 2)",
  "trigger_action": "revealed: Finish Package",
  "url_context": "http://localhost:4173/en/campaigns",
  "feature_status": "completed",
  "execution_trace": {
    "handler": "form-filler",
    "actionKind": "form",
    "selector": "#btnWizFinish",
    "containerSelector": "#wizardStep2Form",
    "submitLabel": "Finish & Create Package",
    "filledFields": [
      { "name": "seats", "type": "number", "value": "10" }
    ]
  },
  "apis_triggered": [
    {
      "url": "http://localhost:4173/api/packages",
      "method": "POST",
      "type": "feature_api",
      "status_class": "success",
      "request_sample": {
        "headers": { "content-type": "application/json" },
        "auth_context": {
          "hasCookieHeader": true,
          "cookieNames": ["session_id"]
        },
        "body": "{\"seats\":10,\"code\":\"PKG-2026\"}"
      },
      "response_sample": {
        "status": 201,
        "body": "{\"ok\":true,\"packageId\":\"PKG-901\"}"
      }
    }
  ]
}

---

🧪 Testing & Validation

The codebase follows a specification-driven architecture governed by 23 formal specifications in specs/.

# Run typechecking and all unit/DOM test suites (235+ tests)
npm run check

# Verify specification traceability matrix & linting (23/23 specs)
npm run spec:trace && npm run spec:lint

# Run end-to-end integration test suites
node tools/run-integration.mjs

---

📁 Repository Structure

├── app/                              # Core Crawler Engine
│   ├── adapters/                     # Browser, Playwright, DOM, Network, Logging
│   │   ├── browser/dom/scripts/      # In-page bundled DOM scripts (form-fill, click, view-state)
│   │   └── network/                  # Passive network interception & epoch isolation
│   ├── domain/                       # Pure domain entities, models, and services
│   │   ├── model/                    # Feature, ViewState, QueuedFeature, CapturedRequest
│   │   └── services/                 # Scope, Deduplication, EntityContextPool, Classification
│   ├── orchestration/                # Crawl loop, stages, discovery, and pipeline
│   │   ├── discovery/                # Observed & LLM Planned discovery sources
│   │   ├── loop/stages/              # Stage pipeline (CaptureState, Discovery, Select, Guard, Execute)
│   │   └── scenario/handlers/        # Action pipeline (FormFiller, ClickTarget, AiFallback)
│   └── config/                       # Typed AppConfig schemas and loader
├── mock-app/                         # Full multi-page mock testing application
├── crawler-test-spa/                 # SPA testbed with Shadow DOM & Relational Modals
├── crawler-scroll-test-spa/          # Infinite & Virtualized Scroll testbed
├── security-usecase-abuse-analyzer/  # Security threat modeling & abuse case analyzer
├── specs/                            # 23 Formal subsystem specifications & traceability
├── tests/                            # Comprehensive unit, DOM (jsdom), and integration tests
├── CrawlConfigGuide.md               # Complete configuration field reference
└── README.md                         # Project documentation

---

📄 License

This project is licensed under the MIT License.