Parallel

This module provides a common interface to parallelization backends. The list of supported backends is here. Backends can be selected with the backend argument of an instance of ParallelConfig, as seen in the examples below.

We use executors to submit tasks in parallel. The basic high-level pattern is

from pydvl.utils.parallel import init_executo
from pydvl.utils.config import ParallelConfig

config = ParallelConfig(backend="ray")
with init_executor(max_workers=1, config=config) as executor:
    future = executor.submit(lambda x: x + 1, 1)
    result = future.result()
assert result == 2

Running a map-reduce job is also easy:

from pydvl.utils.parallel import init_executor
from pydvl.utils.config import ParallelConfig

config = ParallelConfig(backend="joblib")
with init_executor(config=config) as executor:
    results = list(executor.map(lambda x: x + 1, range(5)))
assert results == [1, 2, 3, 4, 5]

There is an alternative map-reduce implementation MapReduceJob which internally uses joblib's higher level API with Parallel()

---

Last update: 2023-09-02
Created: 2023-09-02