[codex] add adaptive inference rate limiting

[Hynek Kydlíček (hynky1999)]

Summary

Makes mdr.inference.generate(...) rate-limit aware by default, with a single rate_limit parameter for choosing adaptive or fixed request concurrency.

Changes

• Adds mdr.inference.AdaptiveRateLimit, used by default when rate_limit is omitted or None.
• Adds mdr.inference.StaticRateLimit for the old fixed-concurrency behavior.
• Removes max_concurrent_requests from generate(...); concurrency caps now live on the rate-limit object.
• Preserves Retry-After from OpenAI-compatible endpoints through a typed GenerationRateLimitError.
• Adds focused tests, updates docs, and updates the LeRobot inference example.

Validation

• uv run ruff check --force-exclude --fix src/refiner/inference tests/test_inference.py tests/test_inference_rate_limit.py examples/lerobot/sarm_annotation.py
• uv run ruff format --force-exclude src/refiner/inference tests/test_inference.py tests/test_inference_rate_limit.py examples/lerobot/sarm_annotation.py
• uv run ty check
• uv run pytest

Full suite: 627 passed.

[gemini-code-assist]

Code Review

This pull request implements an adaptive request limiting mechanism for inference providers, allowing the system to dynamically adjust concurrency based on HTTP 429 responses and successful request windows. The changes include the AdaptiveRateLimit configuration, the AdaptiveRateLimiter logic, and integration into the generation pipeline with enhanced metrics and documentation. Reviewer feedback identifies opportunities to improve documentation self-containment, eliminate redundant variable updates in the generation loop, and simplify logic in the rate limiter by leveraging class invariants.

[gemini-code-assist]

The summarize function is used here but is not defined in this code example. To make the example self-contained and easier for users to understand and copy, please define the summarize function within the python code block, similar to how it's done in the preceding example.

[gemini-code-assist]

The running_requests variable is updated on every request, even when an adaptive_limiter is in use. In the adaptive case, the running_requests metric already uses adaptive_limiter.running as the source of truth, making the updates to the local running_requests variable redundant and potentially confusing. For better clarity and maintainability, these updates should only happen when using the fixed-concurrency request_semaphore.

        if adaptive_limiter is None:
            running_requests += 1
        try:
            if permit is not None:
                async with permit:
                    response = await client.generate(request_payload)
            else:
                response = await client.generate(request_payload)
        except GenerationRateLimitError as err:
            if adaptive_limiter is not None:
                await adaptive_limiter.record_rate_limit(err.retry_after_seconds)
            row.log_throughput("rate_limited_requests", 1, unit="requests")
            row.log_throughput("failed_requests", 1, unit="requests")
            raise
        except Exception:
            row.log_throughput("failed_requests", 1, unit="requests")
            raise
        finally:
            if adaptive_limiter is None:
                running_requests -= 1
            if request_semaphore is not None:
                request_semaphore.release()

[gemini-code-assist]

Since __init__ ensures self._config.max_concurrency is not None, the or self._limit part is redundant and can be confusing. Simplifying this expression will improve readability by relying on the class invariant.

Suggested change

	self._config.max_concurrency or self._limit,
	self._config.max_concurrency,

[gemini-code-assist]

Since __init__ ensures self._config.max_concurrency is not None, the or self._limit part is redundant and can be confusing. Simplifying this expression will improve readability by relying on the class invariant.

Suggested change

	and self._limit < (self._config.max_concurrency or self._limit)
	and self._limit < self._config.max_concurrency

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 439b2e67ff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

[chatgpt-codex-connector]

Catch asyncio timeout class in cooldown wait

Use asyncio.TimeoutError (or both exception classes) here instead of built-in TimeoutError. On Python 3.10, asyncio.wait_for(...) raises asyncio.TimeoutError, so this except does not match; when cooldown expires, acquire() raises out of the limiter and fails queued inference requests instead of resuming them. Since pyproject.toml declares requires-python = ">=3.10", this is a real runtime regression for supported environments.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0e3998aca0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

[chatgpt-codex-connector]

Retry rate-limited calls instead of re-raising immediately

When rate_limit is enabled, this branch updates the adaptive limiter and then immediately re-raises the GenerationRateLimitError. In the current execution path, uncaught async step exceptions propagate out of map_async and fail the shard/worker, so a single HTTP 429 still aborts the job before the reduced concurrency can stabilize subsequent requests. This makes the new adaptive limiter ineffective for normal generate(...) usage unless every caller adds custom exception handling.

Useful? React with 👍 / 👎.