Skip to content

pydvl.valuation.methods.loo

This module implements Leave-One-Out (LOO) valuation.

This is the simplest example of marginal-contribution-based valuation method. It is defined as:

\[ v_\text{LOO}(i) = U(N) - U(N_{-i}), \]

where \(U\) is the utility function, \(N\) is the set of all indices, and \(i\) is the index of interest.

Strictly speaking, LOO can be seen as a semivalue where the coefficients are zero except for \(k=|D|-1.\)

Changing LOO

LOOValuation is preconfigured to stop once all indices have been visited once. In particular, it uses a default LOOSampler with a FiniteSequentialIndexIteration. If you want to change this behaviour, the easiest way is to subclass and replace the constructor.

LOOValuation 

LOOValuation(utility: UtilityBase, progress: bool = False)

Bases: SemivalueValuation

Computes LOO values for a dataset.

Source code in src/pydvl/valuation/methods/loo.py 
48
49
50
51
52
53
54
55
56
def __init__(self, utility: UtilityBase, progress: bool = False):
    self.result: ValuationResult | None = None
    super().__init__(
        utility,
        LOOSampler(batch_size=1, index_iteration=FiniteSequentialIndexIteration),
        # LOO is done when every index has been updated once
        MinUpdates(n_updates=1),
        progress=progress,
    )

values 

values(sort: bool = False) -> ValuationResult

Returns a copy of the valuation result.

The valuation must have been run with fit() before calling this method.

PARAMETER DESCRIPTION
sort

Whether to sort the valuation result by value before returning it.

TYPE: bool DEFAULT: False

Returns: The result of the valuation.

Source code in src/pydvl/valuation/base.py 
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
def values(self, sort: bool = False) -> ValuationResult:
    """Returns a copy of the valuation result.

    The valuation must have been run with `fit()` before calling this method.

    Args:
        sort: Whether to sort the valuation result by value before returning it.
    Returns:
        The result of the valuation.
    """
    if not self.is_fitted:
        raise NotFittedException(type(self))
    assert self.result is not None

    from copy import copy

    r = copy(self.result)
    if sort:
        r.sort()
    return r

log_coefficient 

log_coefficient(n: int, k: int) -> float

The LOOSampler returns only complements of {idx}, so the weight is either 1/n (the probability of a set of size n-1) or 0 if k != n-1. We cancel this out here so that the final coefficient is either 1 if k == n-1 or 0 otherwise.

Source code in src/pydvl/valuation/methods/loo.py 
58
59
60
61
62
63
def log_coefficient(self, n: int, k: int) -> float:
    """The LOOSampler returns only complements of {idx}, so the weight is either
    1/n (the probability of a set of size n-1) or 0 if k != n-1. We cancel
    this out here so that the final coefficient is either 1 if k == n-1 or 0
    otherwise."""
    return float(-np.log(max(1, n))) if k == n - 1 else -np.inf