Authenticate inference endpoints in-process; add secure_remote/modal_remote configs#451
Authenticate inference endpoints in-process; add secure_remote/modal_remote configs#451vertix wants to merge 8 commits into
secure_remote/modal_remote configs#451Conversation
Both wrap `remote` with an auth header read from the environment, so callers pass only the host: `.secure_remote` sends `Authorization: Bearer` from `POSITRONIC_INFERENCE_TOKEN`; `.modal_remote` sends `Modal-Key`/`Modal-Secret` from `MODAL_PROXY_TOKEN_ID`/`MODAL_PROXY_TOKEN_SECRET` on TLS :443. `serve.sh` gains opt-in Nebius `--auth token` via `NEBIUS_AUTH_TOKEN_SECRET`; `workflows/nebius/README.md` documents the flow.
`serve.sh` defaults `NEBIUS_AUTH_TOKEN_SECRET` to `positronic-serverless-inference-token`, so endpoints require `Authorization: Bearer <token>`; `NEBIUS_AUTH_TOKEN_SECRET=` opts out. `e2e.sh` opts out for its smoke check; docs and the remote-training skill switch to `.secure_remote`.
…s `--auth token`
vertix
changed the title
secure_remote and modal_remote authenticated inference configssecure_remote/modal_remote configs
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6512580b30
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3b7a946d67
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c7d93149d8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| # Our own Nebius endpoints answer plain HTTP on :8000 and validate the bearer token in-process. | ||
| # Front them with TLS instead by adding --policy.secure=True --policy.port=443. | ||
| secure_remote = remote.override(headers=bearer_headers) |
There was a problem hiding this comment.
When callers follow the new .secure_remote guidance for public Nebius endpoints, this preset inherits remote's secure=False / port 8000 defaults while adding Authorization: Bearer .... That sends the long-lived POSITRONIC_INFERENCE_TOKEN over plain HTTP/WebSocket, so anyone on the client-to-endpoint path can capture and replay it against the public ingress; require TLS for the authenticated preset or avoid recommending bearer-token auth on http:// endpoints.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid concern, but the literal remedy is not applicable to these endpoints, and I want to leave this open for a human call rather than resolve it.
Nebius serverless AI endpoints are bare public IPs over plain HTTP with no TLS (confirmed in the docs: the endpoint exposes a public_ip, every example is http://<ip>, and there is no certificate/hostname; a live test also showed the --auth token proxy drops the inference WebSocket upgrade). Consequences:
- Requiring
secure=Trueon.secure_remotewould make it speakwss:///https://to an endpoint that only serves plain HTTP, so it could no longer reach the endpoints it exists for. - "Don't use bearer-on-http" isn't an option either: bearer-on-http is the only mechanism that authenticates a Nebius endpoint at all, which is the entire purpose of this change.
So the real hardening is fronting the endpoint with a TLS terminator (hostname + cert), which is separate infrastructure work, out of scope for this PR. What I did land (139af3c) is to make the exposure explicit at the config site rather than silently recommend cleartext bearer. The token is also rotatable (see the README rotation recipe). Leaving this open as a documented known limitation pending a decision on whether to add TLS fronting.
1 participant
Summary
Authenticated access to served inference endpoints, on by default. Auth is enforced in the server,
not at the Nebius ingress: a live test showed Nebius
--auth tokengates plain HTTP but drops theinference WebSocket upgrade (
/api/v1/session→ 404 even with a valid token), so it can't protectthe actual inference path. Instead
VendorServervalidates the bearer token itself, and endpoints areserved open at the Nebius layer.
VendorServerreadsAUTH_TOKENfrom the environment and validatesAuthorization: Bearer <token>on the HTTP route (/api/v1/models) and the WebSocket routes(
/api/v1/session[/{model_id}]) — constant-time compare, rejecting a missing/invalid token on the WSbefore
accept(). Absent token ⇒ open server. Read once in the base class, so no per-vendor churn.serve.sh: injects the token as the container'sAUTH_TOKENvia--env-secret(
NEBIUS_AUTH_TOKEN_SECRET, defaultpositronic-serverless-inference-token), replacing--auth token.NEBIUS_AUTH_TOKEN_SECRET=serves an open endpoint;e2e.shopts out for its smoke check.remotethat attach the auth header from the environment, socallers pass only
--policy.host:.secure_remote→Authorization: BearerfromPOSITRONIC_INFERENCE_TOKEN(our Nebius endpoints, HTTP:8000).modal_remote→Modal-Key/Modal-SecretfromMODAL_PROXY_TOKEN_ID/MODAL_PROXY_TOKEN_SECRET, TLS:443workflows/nebius/README.mdand theremote-trainingskill describe the in-process-auth flow.CLAUDE.mdtweak (allow autonomous commits; prefer fewer test files).Test plan
Verified:
TestClientmatrix — authed server rejects missing/wrong token (HTTP 401,WS rejected) and accepts a valid token (HTTP 200, WS → ready); open server passes through. Regression
cases added to
test_vendor_server.pyexercise the same over real uvicorn + websockets.pytest --no-cov→ 598 passed, 7 skipped.ruff check/formatclean;bash -nclean.InferenceClient.new_session()→ ready. A prior authed (--auth token) endpoint showed tokenlessHTTP → 401, authed HTTP → 200, but authed WS → 404 — confirming the ingress drops the WS upgrade.
Not verified (needs an image rebuild + deploy): the
--env-secret AUTH_TOKEN=injection inside a liveNebius container — it mirrors the existing, working
--env-secret AWS_ACCESS_KEY_ID=line.