Skip to content

pydvl.parallel.backend

CancellationPolicy

Bases: Flag

Policy to use when cancelling futures after exiting an Executor.

Note

Not all backends support all policies.

ATTRIBUTE DESCRIPTION
NONE

Do not cancel any futures.
PENDING

Cancel all pending futures, but not running ones.
RUNNING

Cancel all running futures, but not pending ones.
ALL

Cancel all pending and running futures.

ParallelBackend

Abstract base class for all parallel backends.

executor abstractmethod classmethod 

executor(
    max_workers: int | None = None,
    *,
    config: ParallelConfig | None = None,
    cancel_futures: CancellationPolicy | bool = CancellationPolicy.PENDING
) -> Executor

Returns a futures executor for the parallel backend.

Source code in src/pydvl/parallel/backend.py 
56
57
58
59
60
61
62
63
64
65
66
@classmethod
@abstractmethod
def executor(
    cls,
    max_workers: int | None = None,
    *,
    config: ParallelConfig | None = None,
    cancel_futures: CancellationPolicy | bool = CancellationPolicy.PENDING,
) -> Executor:
    """Returns a futures executor for the parallel backend."""
    ...

init_parallel_backend 

init_parallel_backend(
    config: ParallelConfig | None = None, backend_name: str | None = None
) -> ParallelBackend

Initializes the parallel backend and returns an instance of it.

The following example creates a parallel backend instance with the default configuration, which is a local joblib backend.

If you don't pass any arguments, then by default it will instantiate the JoblibParallelBackend:

Example 
parallel_backend = init_parallel_backend()

To create a parallel backend instance with for example ray as a backend, you can pass the backend name as a string:.

Example 
parallel_backend = init_parallel_backend(backend_name="ray")

The following is an example of the deprecated way for instantiating a parallel backend:

Example 
config = ParallelConfig()
parallel_backend = init_parallel_backend(config)
PARAMETER DESCRIPTION
backend_name

Name of the backend to instantiate.

TYPE: str | None DEFAULT: None

config

(DEPRECATED) Object configuring parallel computation, with cluster address, number of cpus, etc.

TYPE: ParallelConfig | None DEFAULT: None

Source code in src/pydvl/parallel/backend.py 
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
@deprecated(
    target=True,
    args_mapping={"config": "config"},
    deprecated_in="0.9.0",
    remove_in="0.10.0",
)
def init_parallel_backend(
    config: ParallelConfig | None = None, backend_name: str | None = None
) -> ParallelBackend:
    """Initializes the parallel backend and returns an instance of it.

    The following example creates a parallel backend instance with the default
    configuration, which is a local joblib backend.

    If you don't pass any arguments, then by default it will instantiate
    the JoblibParallelBackend:

    ??? Example
        ```python
        parallel_backend = init_parallel_backend()
        ```

    To create a parallel backend instance with for example `ray` as a backend,
    you can pass the backend name as a string:.

    ??? Example
        ```python
        parallel_backend = init_parallel_backend(backend_name="ray")
        ```


    The following is an example of the deprecated
    way for instantiating a parallel backend:

    ??? Example
        ``` python
        config = ParallelConfig()
        parallel_backend = init_parallel_backend(config)
        ```

    Args:
        backend_name: Name of the backend to instantiate.
        config: (**DEPRECATED**) Object configuring parallel computation,
            with cluster address, number of cpus, etc.


    """
    if backend_name is None:
        if config is None:
            backend_name = "joblib"
        else:
            backend_name = config.backend

    try:
        parallel_backend_cls = ParallelBackend.BACKENDS[backend_name]
    except KeyError:
        raise NotImplementedError(f"Unexpected parallel backend {backend_name}")
    return parallel_backend_cls(config)  # type: ignore

available_cpus 

available_cpus() -> int

Platform-independent count of available cores.

FIXME: do we really need this or is os.cpu_count enough? Is this portable?

RETURNS DESCRIPTION
int

Number of cores, or 1 if it is not possible to determine.
Source code in src/pydvl/parallel/backend.py 
184
185
186
187
188
189
190
191
192
193
194
195
196
def available_cpus() -> int:
    """Platform-independent count of available cores.

    FIXME: do we really need this or is `os.cpu_count` enough? Is this portable?

    Returns:
        Number of cores, or 1 if it is not possible to determine.
    """
    from platform import system

    if system() != "Linux":
        return os.cpu_count() or 1
    return len(os.sched_getaffinity(0))  # type: ignore