Arch Linux

python-distributed 2022.12.1 is not compatible with python-dask 2023.4.0.

As an example, I have adapted the simple computation from https://distributed.dask.org/en/latest/quickstart.html

$ cat distributed_simple.py
import dask
import dask.distributed

def square(x):
return x ** 2

def neg(x):
return -x

if __name__ == "__main__":
print(f"Dask: {dask.__version__}")
print(f"Distributed: {dask.distributed.__version__}")
client = dask.distributed.Client()
print(client)
A = client.map(square, range(10))
B = client.map(neg, A)
total = client.submit(sum, B)
print(f"Result: {total.result()}")

$ python distributed_simple.py
Dask: 2023.4.0
Distributed: 2022.12.1
<Client: 'tcp://127.0.0.1:32903' processes=4 threads=16, memory=62.74 GiB>
Traceback (most recent call last):
File "/tmp/distributed_simple.py", line 15, in <module>
A = client.map(square, range(10))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/distributed/client.py", line 2081, in map
futures = self._graph_to_futures(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/distributed/client.py", line 3013, in _graph_to_futures
dsk = dsk.__dask_distributed_pack__(self, keyset, annotations)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'HighLevelGraph' object has no attribute '__dask_distributed_pack__'


Note for clarity I have removed unrelated warnings from this output. If we create a venv and install the 2023.4.0 version of distributed into it:

$ python -m venv --system-site-packages distributed_test
$ ./distributed_test/bin/pip install distributed==2023.4.0
$ ./distributed_test/bin/python distributed_simple.py
Dask: 2023.4.0
Distributed: 2023.4.0
<Client: 'tcp://127.0.0.1:44523' processes=4 threads=16, memory=62.74 GiB>
Result: -285

Note that https://github.com/dask/distributed/blob/main/pyproject.toml specifies an exact version of Dask that distributed relies on.

Could distributed please be updated to a compatible version, and in the future would it be possible to ensure that the two packages get updated simultaneously? Or at least make sure sufficient tests are run during the check() phase of dask to see it hasn't broken distributed?