Skip to content

pydvl.valuation.parallel

This module defines some utilities used in the parallel processing of valuation methods.

In particular, it defines a flag that can be used to signal across parallel processes to stop computation. This is useful when utility computations are expensive or batched together.

The flag is created by the fit method of valuations within a make_parallel_flag context manager, and passed to implementations of EvaluationStrategy.process. The latter calls the flag to detect if the computation should stop.

Flag

Bases: ABC

Abstract class for flags

To check a flag, call it as a function or check it in a boolean context. This will return True if the flag is set, and False otherwise.

MultiprocessingFlag 

MultiprocessingFlag(name: str)

Bases: Flag

A flag for signalling across processes using shared memory.

Source code in src/pydvl/valuation/parallel.py 
94
95
def __init__(self, name: str):
    self._flag = shared_memory.SharedMemory(name, create=False, size=1)

ThreadingFlag 

ThreadingFlag()

Bases: Flag

A trivial flag for signalling across threads.

Source code in src/pydvl/valuation/parallel.py 
75
76
def __init__(self):
    self._flag = False

make_parallel_flag 

make_parallel_flag()

A context manager that creates a flag for signalling across parallel processes. The type of flag created is based on the active parallel backend.

Source code in src/pydvl/valuation/parallel.py 
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
@contextmanager
def make_parallel_flag():
    """A context manager that creates a flag for signalling across parallel processes.
    The type of flag created is based on the active parallel backend."""
    backend = _get_active_backend()[0]

    if isinstance(backend, MultiprocessingBackend) or isinstance(backend, LokyBackend):
        flag = MultiprocessingFlag.create()
    elif isinstance(backend, ThreadingBackend):
        flag = ThreadingFlag()
    else:
        raise NotImplementedError()

    try:
        yield flag
    finally:
        flag.unlink()