feat: meson-python support

[lucascolley]

scipy/scipy#25922
mesonbuild/meson-python#721
https://github.com/dnicolodi/rust-pyo3-meson

Currently need to patch in these files in order to build a Rust extension in SciPy with PyO3 v0.29.2.

cc @dnicolodi @bonzini @ngoldbaum @davidhewitt

[dnicolodi]

@lucascolley What's the reason you decided to submit the work of others as your own PR?

[dnicolodi]

update them for compatibility with v0.29.2 / dev version

The latest commits in https://github.com/dnicolodi/rust-pyo3-meson should work for v0.29.2

[codspeed-hq]

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 147 untouched benchmarks

---

_{Comparing lucascolley:meson-python (eb1dcd5) with main (dfdbc46)}

[lucascolley]

@lucascolley What's the reason you decided to submit the work of others as your own PR?

Just wanted to get a PR open for discussion! I included proper attribution in the commit. Obviously feel free to open a PR from your account instead if you want.

[dnicolodi]

Just wanted to get a PR open for discussion! I included proper attribution in the commit. Obviously feel free to open a PR from your account instead if you want.

For discussion an issue would have been enough. Unless you want to do the work to polish this enough for inclusion in PyO3, this PR only makes it more cumbersome to iterate on the commits.

[bonzini]

A draft PR does have the advantage of running CI. I don't really care - I am not a PyO3 developer and did this work just as a proof of concept that I wouldn't have time to complete.

[alex]

I can't speak for any of the other Pyo3 maintainers, but I'm not super familiar with meson. I personally need a lot more context on:

• What the benefits of us merging this are?
• How do we verify it works?
• When does it need to be updated as we make changes to these packages in the future?

[lucascolley]

What the benefits of us merging this are?

Projects which use the meson-python build system (e.g. NumPy, SciPy, scikit-learn) would be able to add PyO3 extensions without needing to each carry around copies of this code — they could just depend on PyO3 via cargo.toml in the usual way.

So, the benefits for projects which use meson-python and would like to add PyO3 extensions are quite large. No such projects exist yet since the build system support has thus far been inadequate, but, thanks to a lot of work on the meson side, we are getting there :). Whether that is a use-case the PyO3 maintainers are interested in supporting, well, I'll leave that up to you 😉 (but David did ask me about this already at EuroPython this year).

How do we verify it works?

We could add some small example projects like https://github.com/dnicolodi/rust-pyo3-meson to CI and check that they build and are usable. When there are users, I suppose it should be possible to also set up downstream CI against nightly versions of PyO3.

When does it need to be updated as we make changes to these packages in the future?

From my understanding (the meson folks in the thread know more), when changes are made to https://github.com/PyO3/pyo3/blob/main/pyo3-ffi/build.rs, equivalent changes should also be made to https://github.com/lucascolley/pyo3/blob/meson-python/pyo3-ffi/meson.build.

Of course, you could also decide to (perhaps initially) only support a subset of the build configuration space via meson, if it is anticipated that only that subset will be used downstream.

[ngoldbaum]

With a PyO3 maintainer hat on: I'm very interested in enabling Rust and PyO3 use in scientific Python projects. That means I'd be happy to commit to help review and maintain this functionality.

I agree with Alex that integration tests along the lines of our existing Maturin and setuptools-rust integration tests is needed before we could merge this.

[bonzini]

but David did ask me about this already at EuroPython this year

I may or may not have bragged about Meson 1.12's Cargo improvements to him at RustWeek. :)

Note that the equivalent of pyo3-build-config lives directly in Meson (it's what SciPy uses to configure and build its own extension modules), so you don't have to maintain that part, only pyo3-ffi's build script is converted to Meson's DSL here.

[alex]

I think for us to have any chance maintaining this, we need some sort of CI integration (ideally that will catch when the behavior of build.rs changes so we know we're forced to update this) and documentation.

I have a strong background degree of skepticism of maintaining multiple build systems. But if @ngoldbaum says this would enable more memory safety in the scientific ecosystem and there's not a better way to accomplish that, then probably its worth it.

[ngoldbaum]

and there's not a better way to accomplish that

Here's my understanding of why we've landed here. If I'm wrong about this I'd appreciate it if one of the people more familiar with Meson internals could chime in.

Meson is a meta-build system and works at the abstraction level of a C configure script, but for all languages. As such, it doesn't integrate with cargo and instead calls rustc directly. That means build.rs scripts are incompatible with how Meson has decided to support Rust.

Most scientific Python projects migrated from numpy.distutils or setuptools to meson-python for the Python 3.12 release. As such, meson-python/PyO3 compatibility is a must for any future use of Rust in projects like NumPy, SciPy, scikit-learn, and scikit-image.

I would personally really like to be able to write small utilities in Rust for NumPy instead of C, purely for developer ergonomics and the ability to use the Rust package ecosystem and standard library. Memory safety is an excellent side benefit.

Of course it'll be up to each project to decide to add Rust code and add a build dependency on Rust but I would like to get us from a world where the blocker is "Meson needs to add support, Rust is a no-go. Let's write more C" to a place where we can have a discussion about the technical merits of adopting Rust alongside C, C++, and Cython.

[alex]

That all makes sense to me. I guess one question I have is whether "meson builds builds.rs and runs it before running rustc over the crate" is a better direction to go? That approach isn't compatible with bazel, for example, but I wasn't sure if it made sense for meson.

[lucascolley]

That all makes sense to me. I guess one question I have is whether "meson builds builds.rs and runs it before running rustc over the crate" is a better direction to go? That approach isn't compatible with bazel, for example, but I wasn't sure if it made sense for meson.

There is a lengthy discussion in mesonbuild/meson#13779.

[alex]

My read of that thread (and please correct me if I misread any of this) is that:

Having meson build and run build.rs works.

However, it's disfavored for a few reasons:

1. Because of code quality/abstraction issues in the existing rust code
2. Because it may encourage the rust ecosystem to further depend on build.rs, instead of encouraging the ecosystem to move towards more declarative build configurations
3. Because it's not yet required for most crates that meson wants to support.

(1) seems like it shouldn't drive a decision, (3) seems like it's contradicted by this PR :-), and (2) is very interesting.

(2) is basically aligned with rust-lang/cargo#14948 -- and I'm personally very enthusiastic about this. (I've been trying to get conditional compilation for version numbers into rust for a while, since it reduces most of the build.rs for stuff I depend on.) It's not clear to me that what meson chooses to support is really a primary driver of that issue making progress though. (Certainly the impact here isn't going to be reducing our build.rs usage, it's going to be doing mesonbuild integration.)

I'm not quite sure how this all cashes out, but it does seem like at a minimum, both approaches are viable.

[eli-schwartz]

That all makes sense to me. I guess one question I have is whether "meson builds builds.rs and runs it before running rustc over the crate" is a better direction to go? That approach isn't compatible with bazel, for example, but I wasn't sure if it made sense for meson.

The reasons for meson are not entirely dissimilar to bazel.

While it is technically possible to compile build.rs and run it, doing so would have to occur during the "meson setup" stage, before a ninja file is emitted, since build.rs changes the command line arguments passed to rustc (and therefore its output is part of what is used to emit the ninja file). As such, it is not technically possible to support any build.rs with imports of another crate.

For the same reason, meson supports compiling C or C++ code at configure time:

cxx = meson.get_compiler('cpp')

thing = cxx.run('''
// some C++ code goes here
    ''')

if thing.returncode() == 0
    do_something_with_me = thing.stdout()
fi

... but only if this code does not depend on other build targets in the current project, including system libraries that have been configured to build locally/statically as a "subproject wrap" and are thus build artifacts rather than mere -lfoo linker flags.

Rust is not on an unequal(ly advantageous) footing here. In a sense it is slightly worse off since you can't compile and install a system crate to /usr/lib64 in advance.

Typically in C/C++ land, the motivation for compiling code snippets at configure time is to do checks similar to the builtin checks which already exist for cxx.sizeof(x) or cxx.compute_int(x) or cxx.has_function(x), which use fixed code snippets that rely only on the standard library and basically involve introspecting the platform. Sometimes one still needs to introspect the platform but for less common data, and a custom code snippet is needed. Usually cxx.links() is good enough and stdout of the program isn't needed.

My understanding is that in rust, build.rs usually does need external crates, and indeed this is the case for pyo3-ffi, so that ship sank before it set sail. But of course one is fully empowered to experiment with

rustc = meson.get_compiler('rust')

rustc.run(files('build.rs'))

If that works, then the mentioned meson PR proposing to automatically generate the exact same code without requiring an explicit meson.build file to be provided by the crate itself, would also work. At the cost of:

• not making it obvious which crates will work correctly,
• possibly generating additional source files meson doesn't know about / how to track, which is a very common thing in the rust ecosystem; in the worst case scenario this is an entire static C library
• if build.rs tries to detect system dependencies via pkgconfig instead of using the system-deps declarative Cargo.toml approach, build.rs won't see the existing detected version from meson (imagine a project with mixed C and rust code, where both the C code and the rust code depend on the same C library), and may pull in a different version causing a diamond dependency problem
• requiring to fork+exec a heavyweight compiler, potentially many times

(2) is basically aligned with rust-lang/cargo#14948 -- and I'm personally very enthusiastic about this. (I've been trying to get conditional compilation for version numbers into rust for a while, since it reduces most of the build.rs for stuff I depend on.)

Yes, stuff like this will be a huge help. GCC has supported this since the dinosaurs ruled the earth and it has proved its use many many times over.