Migrate development and CI environments to Pixi - #723
Draft
MridulS wants to merge 7 commits into
Draft
Conversation
pace_neutrons needs a MATLAB runtime, but it imports fine without one, so pytest.importorskip in sqw_horace_test does not trigger. Having it in 'test' therefore forced every test environment to provide MATLAB, rather than letting those tests skip. Splitting it out lets the regular test run skip them and confines the MATLAB setup to a single dedicated job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mantid is only distributed as a conda package while the rest of the stack is on PyPI, so development requires resolving across both ecosystems at once. Pixi does that in a single solve and writes one lock file, replacing the combination of pip-compile-multi, tox and a hand-maintained conda environment file that had to be kept in agreement by hand. Seven environments, all in one lock: default tests, resolved from PyPI, i.e. what a pip user gets mantid Mantid tests, resolved from conda so the stack is ABI-consistent docs documentation; extends mantid, several notebooks use it sqw SQW/Horace tests, needs pace_neutrons and MATLAB mypy type checking lint formatting and static analysis build sdist and wheel Notable details: - The mantid environment is restricted to linux-64 and osx-arm64. Mantid and scipp cannot coexist on win-64; see the comment in pixi.toml for the full derivation. Mantid has shipped osx-arm64 builds since 6.13, so the Mantid tests now run natively on Apple Silicon. - exclude-newer gives a two week baseline so a freshly published, occasionally broken build cannot break the lock the moment it lands. The Scipp stack is exempted from it, since that is what we want to test against immediately. Both the conda and the PyPI table are needed, because the mantid and docs environments resolve scipp from conda and the others from PyPI. - The docs task depends on a task running 'dot -c'. conda-forge's graphviz registers its layout plugins from a post-link script, and pixi does not run those, so rendering otherwise fails with "no layout engine support for dot". - codespell skips pixi.lock, which is full of package names it misreads, and the .pixi directory. The .tox entry it replaces was already redundant, .tox being gitignored and therefore never seen by pre-commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine workflows become four. ci.yml now runs the pixi environments directly. The Mantid job loses setup-micromamba, .buildconfig/ci-linux.yml and 'conda develop src'; the test job loses tox. The reusable test.yml existed only to parameterise the tox environment and is gone. weekly_windows_macos.yml is folded into ci.yml as matrix entries. Running macOS and Windows on every PR is cheap now that the lock is frozen and there is no resolution step. nightly.yml replaces nightly_at_main, nightly_at_main_lower_bound, nightly_at_release and unpinned. Following scipp/ess, there is no job resolving Scipp nightly wheels: the locked environments already test against the newest Scipp release via the exclude-newer exemptions, so the remaining question is the ends of the declared ranges, which uv answers with --resolution=lowest-direct and --resolution=highest. The SQW/Horace tests get their own job, which is the only one that needs MATLAB. docs.yml drops the apt install of pandoc and graphviz, since both now come from the docs environment, along with its 'version' input, which required a second micromamba setup; the 'branch' input covers building older docs. fetch-depth: 0 is set wherever setuptools_scm needs to determine a version. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All superseded by the pixi workspace and no longer referenced by CI. tox.ini 6 environments, now pixi tasks requirements/ 22 pip-compile-multi files requirements/make_base.py generated the .in files from pyproject.toml .buildconfig/ci-linux.yml hand-pinned conda env for Mantid and docs docs/environments/ orphaned, referenced from nowhere .buildconfig/ci-linux.yml in particular duplicated the dependency set already declared in pyproject.toml, pinned separately and by hand, which is the main thing this migration removes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rewrite dependency-management.md around pixi.toml and pixi.lock, and describe why both the conda and PyPI exclude-newer tables exist and how the nightly uv jobs cover the ends of the dependency ranges. getting-started.md loses the note about installing pandoc through the OS package manager, which the docs environment now provides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pyproject.toml>2and SciPy>1.14mainbranches of the Scipp stack, including Tof, SQW, and MATLAB/Horaceprefix-dev/setup-pixiv0.10.1 and Pixi v0.77.0How to test
With Pixi 0.77 or newer installed, reviewers can exercise the migration directly from the repository root with
pixi lock --check,pixi run -e default test,pixi run -e mantid test-mantid,pixi run -e docs docs doctest,pixi run -e docs docs html,pixi run -e docs docs linkcheck,pixi run -e lint lint, andpixi run -e build check-build. The Mantid and docs environments support Linux and Apple Silicon;pixi run -e sqw test-sqwadditionally requires MATLAB with the Compiler SDK and now fails during an explicit preflight when MATLAB is missing instead of silently skipping Horace coverage.Validation performed
site-packagesThe former standalone mypy task is not carried forward because it currently reports approximately 1,420 pre-existing errors and was not part of CI. Restoring useful type-checking coverage is better handled as a focused follow-up than exposing a Pixi task that cannot pass.