numpy is imported unconditionally

[crusaderky]

We have many unit tests that generate Nannies with spawn method. In theory, those tests should become a lot faster if numpy was imported only if needed.
This is not the case today, because of

1. two places that import pandas; see blockers:
   • Defer pandas import on worker in P2P shuffle #5695
   • Don't import packages just for version compatibility check #5723
   • get_versions without importing packages #5724
2. two import blosc statements in distributed.protocol.compression

xfailed unit test is already in #5695.

Note that this means that fixing this issue means that, across all unit tests that do use numpy data on nannies, import numpy will be moved from @gen_cluster start to the first time where a numpy array or function is deserialised - which in turn may cause flakiness if the test time margins are too tight.

CC @gjoseph92

[gjoseph92]

Yes, I also just looked into the same thing. Maybe we should provide a helper in #5724 for checking if a package exists (via importlib.metadata/pkg_resources) without actually importing it, so we can use that on blosc?
Also xref #5269.

[crusaderky]

Maybe we should provide a helper in #5724 for checking if a package exists (via importlib.metadata/pkg_resources) without actually importing it, so we can use that on blosc?

Not that simple - distributed.protocol.compression fills up a dict of compress/decompress callables at module import time, so we'd need to be a bit more creative.

[gjoseph92]

Blosc/python-blosc#159 seems stalled, but we could probably suggest just not importing the tests in top-level __init__.py. I'm not sure this is worth worrying about too much right now beyond trying to get an upstream change?

[crusaderky]

WON'T FIX

#5750 introduces a new unconditional import of numpy, whose benefits definitely exceed those of this ticket.

[jakirkham]

If you can think of better ways to handle that import, am open to other options 🙂