Skip to content

Replace regex-based pytest parsing with a structured Python test runner #21

@NickK21

Description

@NickK21

Summary

Python evaluation currently parses pytest's human-readable terminal output with regex. This is brittle and makes error handling unreliable.

Current Problem

The Python path in codeval/executor.js uses parsePytestOutput() to scrape pytest terminal text.

This is fragile for cases like:

  • assertion failures
  • import errors
  • syntax errors
  • missing libraries
  • no tests collected

Planned Changes

  • Add a dedicated Python test runner that executes pytest and returns structured JSON.
  • Update the Python execution path to consume that structured result instead of regex-parsing terminal output.
  • Add regression coverage for Python evaluation edge cases.

Regression Cases

Add automated coverage for:

  • normal pass
  • assertion failure
  • import error in program.py
  • missing dependency
  • syntax error
  • no tests collected

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request
No fields configured for Feature.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions