Backend

CancellationPolicy

Bases: Flag

Policy to use when cancelling futures after exiting an Executor.

Note

Not all backends support all policies.

ATTRIBUTE	DESCRIPTION
NONE	Do not cancel any futures.
PENDING	Cancel all pending futures, but not running ones.
RUNNING	Cancel all running futures, but not pending ones.
ALL	Cancel all pending and running futures.

BaseParallelBackend

Abstract base class for all parallel backends.

executor(max_workers=None, config=ParallelConfig(), cancel_futures=CancellationPolicy.PENDING) abstractmethod classmethod

Returns an executor for the parallel backend.

Source code in src/pydvl/parallel/backend.py

@classmethod
@abstractmethod
def executor(
    cls,
    max_workers: int | None = None,
    config: ParallelConfig = ParallelConfig(),
    cancel_futures: CancellationPolicy = CancellationPolicy.PENDING,
) -> Executor:
    """Returns an executor for the parallel backend."""
    ...

init_parallel_backend(config)

Initializes the parallel backend and returns an instance of it.

The following example creates a parallel backend instance with the default configuration, which is a local joblib backend.

Example

config = ParallelConfig()
parallel_backend = init_parallel_backend(config)

To create a parallel backend instance with a different backend, e.g. ray, you can pass the backend name as a string to the constructor of ParallelConfig.

Example

config = ParallelConfig(backend="ray")
parallel_backend = init_parallel_backend(config)

PARAMETER	DESCRIPTION
config	instance of ParallelConfig with cluster address, number of cpus, etc. TYPE: ParallelConfig

Source code in src/pydvl/parallel/backend.py

def init_parallel_backend(config: ParallelConfig) -> BaseParallelBackend:
    """Initializes the parallel backend and returns an instance of it.

    The following example creates a parallel backend instance with the default
    configuration, which is a local joblib backend.

    ??? Example
        ``` python
        config = ParallelConfig()
        parallel_backend = init_parallel_backend(config)
        ```

    To create a parallel backend instance with a different backend, e.g. ray,
    you can pass the backend name as a string to the constructor of
    [ParallelConfig][pydvl.utils.config.ParallelConfig].

    ??? Example
        ```python
        config = ParallelConfig(backend="ray")
        parallel_backend = init_parallel_backend(config)
        ```

    Args:
        config: instance of [ParallelConfig][pydvl.utils.config.ParallelConfig]
            with cluster address, number of cpus, etc.


    """
    try:
        parallel_backend_cls = BaseParallelBackend.BACKENDS[config.backend]
    except KeyError:
        raise NotImplementedError(f"Unexpected parallel backend {config.backend}")
    return parallel_backend_cls.create(config)  # type: ignore

available_cpus()

Platform-independent count of available cores.

FIXME: do we really need this or is os.cpu_count enough? Is this portable?

RETURNS	DESCRIPTION
int	Number of cores, or 1 if it is not possible to determine.

Source code in src/pydvl/parallel/backend.py

def available_cpus() -> int:
    """Platform-independent count of available cores.

    FIXME: do we really need this or is `os.cpu_count` enough? Is this portable?

    Returns:
        Number of cores, or 1 if it is not possible to determine.
    """
    from platform import system

    if system() != "Linux":
        return os.cpu_count() or 1
    return len(os.sched_getaffinity(0))  # type: ignore

effective_n_jobs(n_jobs, config=ParallelConfig())

Returns the effective number of jobs.

This number may vary depending on the parallel backend and the resources available.

PARAMETER	DESCRIPTION
n_jobs	the number of jobs requested. If -1, the number of available CPUs is returned. TYPE: int
config	instance of ParallelConfig with cluster address, number of cpus, etc. TYPE: ParallelConfig DEFAULT: ParallelConfig()

RETURNS	DESCRIPTION
int	The effective number of jobs, guaranteed to be >= 1.

RAISES	DESCRIPTION
RuntimeError	if the effective number of jobs returned by the backend is < 1.

Source code in src/pydvl/parallel/backend.py

def effective_n_jobs(n_jobs: int, config: ParallelConfig = ParallelConfig()) -> int:
    """Returns the effective number of jobs.

    This number may vary depending on the parallel backend and the resources
    available.

    Args:
        n_jobs: the number of jobs requested. If -1, the number of available
            CPUs is returned.
        config: instance of [ParallelConfig][pydvl.utils.config.ParallelConfig] with
            cluster address, number of cpus, etc.

    Returns:
        The effective number of jobs, guaranteed to be >= 1.

    Raises:
        RuntimeError: if the effective number of jobs returned by the backend
            is < 1.
    """
    parallel_backend = init_parallel_backend(config)
    if (eff_n_jobs := parallel_backend.effective_n_jobs(n_jobs)) < 1:
        raise RuntimeError(
            f"Invalid number of jobs {eff_n_jobs} obtained from parallel backend {config.backend}"
        )
    return eff_n_jobs

---

Last update: 2023-12-21
Created: 2023-12-21