Move public nightlies to PyPI and keep local installs stable

[charlesbluca]

Description

Move public nightly publishing off TestPyPI and onto PyPI for nemo-retriever and the HF-derived nemotron-* packages, while leaving NVIDIA internal Artifactory publishing paths unchanged.

This PR updates the public nightly workflows and builder defaults to publish to PyPI with PYPI_API_TOKEN, removes public TestPyPI routing, and bumps HF nightly base versions so prerelease wheels sort after current stable releases.

For local inference installs, the published wheel now exposes only the standard local extra. That extra enumerates the full local stack and uses stable Nemotron release ranges up to the next major version. The former local-base and local-nightly extras are removed. Repository uv developer installs still get the latest Nemotron prerelease packages through the uv-only dev dependency group, which includes nemo_retriever[all], nemo_retriever[dev], and the explicit nemotron-* nightly constraints.

For users who install from published wheels, nemo-retriever[local] resolves stable Nemotron packages by default; --pre remains the opt-in escape hatch for prerelease/nightly Nemotron packages from PyPI. The README now documents that path near the local install instructions.

Validation performed:

• uv run --project nemo_retriever --no-dev --with pytest --with packaging python -m pytest ci/tests/test_huggingface_release_workflow.py nemo_retriever/tests/test_nemotron_ocr_v2_nightly.py nemo_retriever/tests/test_ci_workflows.py -q (33 passed)
• uv lock --check
• uv tree --only-group dev --locked confirmed uv dev installs include nemo-retriever[all, dev] plus the nightly nemotron-* packages.
• Temporary wheel metadata inspection confirmed no local-base or local-nightly extras and stable-range Requires-Dist entries for local.
• Docs search confirmed no stale local-nightly, uv-local, or TestPyPI install instructions outside negative assertions.
• HF nightly publish: https://github.com/NVIDIA/NeMo-Retriever/actions/runs/27162876322
• nemo-retriever nightly publish from this branch: https://github.com/NVIDIA/NeMo-Retriever/actions/runs/27164263927

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[greptile-apps]

Greptile Summary

This PR migrates public nightly publishing for nemo-retriever and the HF-derived nemotron-* packages from TestPyPI to PyPI, hardens CI token handling, and refactors the [local] extra from pinned Nemotron versions to stable semver ranges with a separate uv-only dev dependency group for nightly prerelease overlays.

• Workflow hardening: pypi-nightly-publish.yml gains a permissions: contents: read block and moves from -p "${token}" (token visible in process listings) to a TWINE_PASSWORD env var; huggingface-nightly.yml removes the TestPyPI branch entirely and adds per-package nightly_base_version matrix fields so nightly versions sort after current stable releases.
• Dependency model: [local] now uses >=x.y.z,<next-major ranges for Nemotron packages; the new [dependency-groups].dev section provides uv-only nightly constraints (!=stable-pin exclusions) without leaking prerelease specifiers into published wheel metadata.
• Test coverage: New tests in test_ci_workflows.py, test_huggingface_release_workflow.py, and test_nemotron_ocr_v2_nightly.py enforce the PyPI-only routing, permissions scoping, TWINE_PASSWORD usage, and correct version range structure.

Confidence Score: 5/5

Safe to merge — both previously flagged issues (token in process listing and missing permissions block) are resolved, and the dependency-range and workflow changes are well-tested.

The two blocking findings from the prior review are directly addressed: TWINE_PASSWORD is now set as an env var instead of a -p CLI argument, and pypi-nightly-publish.yml now carries permissions: contents: read. The remaining changes (TestPyPI removal, version range widening, dev dependency group) are straightforward and backed by 33 passing tests including a uv lock --check.

No files require special attention.

Important Files Changed

Filename	Overview
.github/workflows/pypi-nightly-publish.yml	Drops TestPyPI routing, adds workflow-level permissions: contents: read, and migrates token handling to TWINE_PASSWORD env var — both previously flagged security findings are resolved here.
.github/workflows/huggingface-nightly.yml	Removes TestPyPI branch (hard-codes PyPI URLs), adds nightly_base_version matrix field for three non-OCR packages, and simplifies upload flag logic; already carried permissions: contents: read at workflow level.
ci/scripts/nightly_build_publish.py	Updates CLI defaults (--repository-url, --token-env) to PyPI; _twine_upload already passes the token via TWINE_PASSWORD env var rather than as a CLI arg — no substantive logic change.
nemo_retriever/pyproject.toml	Converts pinned Nemotron versions in [local] extra to stable semver ranges (>=x,<next-major), adds PEP 735 [dependency-groups] dev section with nightly exclusion constraints, and removes the test-pypi uv index.
nemo_retriever/tests/test_ci_workflows.py	Adds three new tests enforcing that public nightly workflows target only PyPI, carry least-privilege permissions, and use TWINE_PASSWORD env var — directly codifying the security properties introduced by this PR.
nemo_retriever/tests/test_nemotron_ocr_v2_nightly.py	Expands pyproject assertions to cover stable ranges in [local], nightly exclusion constraints in [dependency-groups].dev, and absence of TestPyPI index — good coverage of the dependency-range change.
ci/tests/test_huggingface_release_workflow.py	Adds two tests verifying nightly base versions per matrix entry and that nightly_build_publish.py defaults to PyPI — straightforward regression guards for the workflow changes.
nemo_retriever/README.md	Adds a short --pre opt-in snippet for users who want nightly Nemotron packages after installing nemo-retriever[local].

Sequence Diagram

sequenceDiagram
    participant GH as GitHub Actions (schedule / dispatch)
    participant WF as nightly workflow
    participant Script as nightly_build_publish.py
    participant PyPI as PyPI (upload.pypi.org)

    GH->>WF: trigger (cron or workflow_dispatch)
    WF->>WF: resolve upload_to (default: pypi)
    WF->>Script: python ci/scripts/nightly_build_publish.py --repository-url https://upload.pypi.org/legacy/ --token-env PYPI_API_TOKEN
    Script->>Script: clone HF repo, patch version
    Script->>Script: build wheel
    Script->>Script: _twine_upload sets TWINE_PASSWORD in env
    Script->>PyPI: twine upload (token via TWINE_PASSWORD)
    PyPI-->>Script: 200 OK
    Script-->>WF: exit 0

Loading

_{Reviews (2): Last reviewed commit: "Address PyPI nightly workflow feedback" | Re-trigger Greptile}