[Feature] Active Exploit Testing — Security as a Functional Requirement

[ArthurCRodrigues]

Feature Overview

A new security_testing template that transforms the Autograder into an active adversary against student-submitted APIs. Instead of teaching security as theory, the grader fires real injection payloads (SQL injection, path traversal, command injection, header injection) at student endpoints through the existing sandbox infrastructure. If exploits succeed — meaning the grader extracts unauthorized data, corrupts state, or escapes intended boundaries — the penalty category is applied automatically.

The student's API runs in its sandbox as usual. Alongside it, the system spins up vulnerable mock services (a database with known injectable entry points, a mock internal service) that simulate a real production backend. The student's code is expected to interact with these services safely. The grader then acts as a malicious client, probing the student's API for weaknesses.

This trains developers to treat security as a functional requirement evaluated continuously in the CI pipeline, not a checkbox reviewed once during code review.

---

Motivation

1. Security is taught reactively — Students learn about SQLi and XSS from slides, then write vulnerable code in assignments because nothing enforces secure patterns during development.

2. The infrastructure already supports this — sandbox.make_request() already fires HTTP requests at student APIs (api_testing template). The penalty category already deducts points. Asset injection already delivers config/data into sandboxes. All the primitives exist.

3. Shift-left security for education — The same way the autograder enforces correct output and API contracts, it should enforce that GET /users?id=1 OR 1=1 does NOT return all users.

4. Real-world alignment — Production CI pipelines increasingly include DAST (Dynamic Application Security Testing). Students who experience automated exploit testing during coursework arrive job-ready.

---

Architecture Mapping

This feature maps cleanly onto existing autograder primitives:

Existing Primitive	Security Testing Usage
TestFunction abstract class	New security test functions (sql_injection, path_traversal, command_injection, etc.)
sandbox.make_request(method, endpoint, **kwargs)	Fire malicious HTTP payloads at student API
sandbox.run_command()	Verify backend state after attack (did data leak?)
CriteriaTree.penalty category	Apply score deductions when exploits succeed
Asset injection (inject_assets → /tmp/)	Deliver vulnerable mock DB configs, seed data, attack payload dictionaries
sandbox.extract_file()	Extract logs/DB state to verify if corruption occurred
Docker container networking	Connect student sandbox to vulnerable mock services

---

Proposed Test Functions

sql_injection

Fires parameterized SQLi payloads at specified endpoints and checks if the response leaks unauthorized data or database errors.

Parameter	Type	Description
endpoint	string	Target API endpoint (e.g., /users)
method	string	HTTP method (GET, POST, etc.)
injection_point	string	Where to inject: query_param, body_field, path_param
target_field	string	Field name to inject into (e.g., id, username)
payloads	list[string]	Override default payload set (optional)
leak_indicator	string	Regex pattern indicating data was leaked (e.g., multiple rows returned when one expected)
error_indicator	string	Regex pattern indicating raw DB errors exposed (e.g., SQL syntax, pg_catalog)

Default payload set: ' OR 1=1 --, ''; DROP TABLE users; --, 1 UNION SELECT * FROM information_schema.tables, ' OR ''=', 1; SELECT pg_sleep(5)--

Scoring: Score 100 = no payload succeeded. Score 0 = at least one payload leaked data or exposed errors.

---

path_traversal

Tests if the student's file-serving or file-reading endpoints can be tricked into accessing files outside the intended directory.

Parameter	Type	Description
endpoint	string	File-serving endpoint (e.g., /files/{filename})
base_path	string	Expected confined directory (e.g., /app/uploads)
payloads	list[string]	Override default traversal set (optional)
sensitive_files	list[string]	Files that should NEVER be readable (e.g., /etc/passwd, .env)

Default payload set: ../../../etc/passwd, ....//....//etc/passwd, %2e%2e%2f%2e%2e%2fetc/passwd, ..%252f..%252f..%252fetc/passwd, /app/../../../etc/shadow

Scoring: Score 100 = all traversal attempts blocked. Score 0 = sensitive file content was returned.

---

command_injection

Tests if user-supplied input to the student's API can escape into shell execution.

Parameter	Type	Description
endpoint	string	Target endpoint that processes user input
method	string	HTTP method
injection_point	string	Where to inject (body_field, query_param, header)
target_field	string	Field to inject into
payloads	list[string]	Override defaults (optional)
canary_command	string	Command whose output proves injection worked (default: echo PWNED-{uuid})

Default payload set: ; echo PWNED, `echo PWNED`, $(echo PWNED), | cat /etc/passwd, \n/bin/sh -c "echo PWNED"

Scoring: Score 100 = no injection indicators in response. Score 0 = canary output detected in response body.

---

header_injection

Tests if the student's API reflects user input into HTTP response headers without sanitization (enabling response splitting, XSS via headers).

Parameter	Type	Description
endpoint	string	Target endpoint
injection_point	string	Which header or parameter to inject through
payloads	list[string]	Override defaults (optional)

Default payload set: value\r\nX-Injected: true, value\r\n\r\n<script>alert(1)</script>, value%0d%0aSet-Cookie:hacked=true

Scoring: Score 100 = no injected headers in response. Score 0 = response contains injected header or body split.

---

mass_assignment

Tests if the student's API accepts and persists fields that should be protected (e.g., role, isAdmin, price).

Parameter	Type	Description
endpoint	string	Resource creation/update endpoint
method	string	POST or PUT
base_body	dict	Valid request body
forbidden_fields	dict	Fields + values that should NOT be accepted (e.g., {"role": "admin", "isAdmin": true})
verify_endpoint	string	Endpoint to check if forbidden field was persisted
verify_field	string	Field to check in verification response

Scoring: Score 100 = forbidden fields rejected or ignored. Score 0 = forbidden field was persisted.

---

Vulnerable Mock Services

The system needs to provide backend services that the student's code is expected to query safely. These mock services are intentionally vulnerable — they will execute whatever SQL/commands they receive — to test whether the student's code properly sanitizes before forwarding.

Mock Vulnerable Database

A lightweight SQLite or PostgreSQL container pre-seeded with test data:

• Users table with known records
• Responds to raw SQL (no ORM, no parameterization)
• Student's API is expected to use parameterized queries when talking to it
• If the student's code passes user input directly into SQL strings, the grader's payloads will succeed through the student's API

Delivery mechanism: Docker Compose sidecar or in-sandbox process started via pre_flight setup commands. Connection string injected via asset injection to /tmp/db_config.json.

Mock Internal Service

A simple HTTP service that simulates an internal microservice:

• Executes commands or returns files based on input (intentionally unsafe)
• Student's API is expected to validate/sanitize before forwarding requests to it
• Tests whether students implement proper input validation at service boundaries

---

Criteria Tree Integration

Security tests naturally belong in the penalty category. A teacher configures which exploit vectors to test:

{
  "penalty": {
    "weight": 30,
    "subjects": [
      {
        "name": "Security Vulnerabilities",
        "weight": 100,
        "tests": [
          {
            "name": "sql_injection",
            "params": {
              "endpoint": "/api/users",
              "method": "GET",
              "injection_point": "query_param",
              "target_field": "id",
              "leak_indicator": ".*admin.*password.*"
            },
            "weight": 40
          },
          {
            "name": "path_traversal",
            "params": {
              "endpoint": "/api/files/{filename}",
              "sensitive_files": ["/etc/passwd", "/app/.env"]
            },
            "weight": 30
          },
          {
            "name": "command_injection",
            "params": {
              "endpoint": "/api/process",
              "method": "POST",
              "injection_point": "body_field",
              "target_field": "filename"
            },
            "weight": 30
          }
        ]
      }
    ]
  }
}

When a student's API is vulnerable:

• Penalty applied → score deducted from final grade
• Feedback generated → explains what payload succeeded and why (educational)
• Iterative improvement → student fixes vulnerability, resubmits, penalty disappears

---

Implementation Plan

Phase 1: Core Test Functions

• Implement SecurityTestingTemplate class extending Template
• Implement sql_injection test function
• Implement path_traversal test function
• Implement command_injection test function
• Implement header_injection test function
• Implement mass_assignment test function
• Default payload dictionaries (injectable via asset override)
• Unit tests for each test function with mock sandbox

Phase 2: Mock Service Infrastructure

• Vulnerable mock database Docker image (SQLite-based, lightweight)
• Vulnerable mock internal service image
• Sidecar orchestration: start mock services alongside student sandbox
• Connection config injection via asset system (/tmp/db_config.json)
• Health check: wait for mock services before running exploit tests

Phase 3: Feedback & Reporting

• Security-specific feedback generation (explains vulnerability, shows payload that worked)
• Educational remediation hints in feedback (e.g., "Use parameterized queries instead of string concatenation")
• Translation keys for en and pt_br locales
• Integration with existing penalty category scoring

Phase 4: Documentation & Examples

• Template documentation page (docs/templates/security_testing.md)
• Example grading config with security penalties
• Example vulnerable student submission (for testing)
• Example secure student submission (passes all checks)
• GitHub Action configuration example

---

Technical Considerations

1. Sandbox networking: Mock services need to be reachable from the student's container. Options: shared Docker network, or start mock processes inside the same container via setup commands.

2. Payload safety: All payloads execute within the isolated sandbox environment. The Docker container has no access to host filesystem, network, or other submissions. Existing sandbox security controls (resource limits, timeouts, user isolation) apply.

3. False positives: Payload sets should be curated to minimize false positives. A response containing "syntax error" from legitimate validation is not the same as leaking pg_catalog table names. The leak_indicator and error_indicator regex patterns let teachers tune sensitivity.

4. Performance: Security tests add HTTP round-trips. With warm containers and the existing timeout mechanisms (120s running timeout), even 20+ payload attempts per test function should complete within the grading window.

5. Extensibility: Teachers can override default payload sets via test parameters, allowing course-specific attack vectors (e.g., NoSQL injection for MongoDB courses, LDAP injection for enterprise courses).

---

References

• Existing api_testing template: autograder/template_library/api_testing.py
• SandboxContainer.make_request(): sandbox_manager/sandbox_container.py
• Penalty category scoring: autograder/models/criteria_tree.py → CriteriaTree.penalty
• Asset injection: sandbox_manager/sandbox_container.py → inject_assets()
• OWASP Testing Guide: https://owasp.org/www-project-web-security-testing-guide/
• OWASP Top 10 (2021): https://owasp.org/Top10/