Skip to content

TaskProtocol.max_retries and TimeoutAction.RETRY/EXTEND are declared but never consumed #62

Description

@cemde

Summary

Two retry-related fields on TaskProtocol are exposed in the public API and documented as functional, but the execution loop never reads them. They are silent no-ops for users.

Affected fields

  1. TaskProtocol.max_retriesmaseval/core/task.py:45-52. Docstring says "Maximum retry attempts for transient failures (not timeouts)." Grepping maseval/core/ shows the only references are the dataclass field itself and its to_dict() serialization. Benchmark._execute_task_repetition (maseval/core/benchmark.py:1069-1255) never reads it.

  2. TimeoutAction.RETRY and TimeoutAction.EXTENDmaseval/core/task.py:21-26. The TaskTimeoutError handler at benchmark.py:1139-1159 records the timeout and returns. It does not branch on task.protocol.timeout_action. So SKIP, RETRY, and EXTEND all behave identically (skip).

Reproduction

```python
from maseval import Task
from maseval.core.task import TaskProtocol, TimeoutAction

task = Task(
query="...",
protocol=TaskProtocol(max_retries=5, timeout_action=TimeoutAction.RETRY, timeout_seconds=1.0),
)

Run a benchmark where this task raises a transient error or times out.

Observed: a single failed report is produced, max_retries / RETRY are ignored.

Expected: up to 5 retries on transient errors; one retry on timeout.

```

Suggested resolution

Either:

  • Wire the fields into `_execute_task_repetition` (retry loop around the execution `try` block; branch on `timeout_action` in the timeout handler), or
  • Mark the fields as "not implemented" in their docstrings and raise on non-default values, so downstream users do not silently rely on them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcoreIn regards to the core package `maseval/core`

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions