Skip to content

pydvl.valuation.scorers.supervised

This module provides a SupervisedScorer class that wraps scoring functions for supervised problems with additional information.

Supervised scorers can be constructed in the same way as in scikit-learn: either from known strings or from a callable. Greater values must be better. If they are not, a negated version can be used, see scikit-learn's make_scorer().

SupervisedScorer holds the test data used to evaluate the model.

Named scorer

It is possible to use all named scorers from scikit-learn.

from pydvl.valuation import Dataset, SupervisedScorer

train, test = Dataset.from_arrays(X, y, train_size=0.7)
model = SomeSKLearnModel()
scorer = SupervisedScorer("accuracy", test, default=0, range=(0, 1))

Model scorer

It is also possible to use the score() function from the model if it defines one:

from pydvl.valuation import Dataset, SupervisedScorer

train, test = Dataset.from_arrays(X, y, train_size=0.7)
model = SomeSKLearnModel()
scorer = SupervisedScorer(model, test, default=0, range=(-np.inf, 1))

SupervisedScorer 

SupervisedScorer(
    scoring: str
    | SupervisedScorerCallable[SupervisedModelT, ArrayT]
    | SupervisedModelT,
    test_data: Dataset,
    default: float,
    range: tuple[float, float] = (-float("inf"), float("inf")),
    name: str | None = None,
)

Bases: Generic[SupervisedModelT, ArrayT], Scorer

A scoring callable that takes a model, data, and labels and returns a scalar.

PARAMETER DESCRIPTION
scoring

Either a string or callable that can be passed to get_scorer.

TYPE: str | SupervisedScorerCallable[SupervisedModelT, ArrayT] | SupervisedModelT

test_data

Dataset where the score will be evaluated.

TYPE: Dataset

default

score to be used when a model cannot be fit, e.g. when too little data is passed, or errors arise.

TYPE: float

range

numerical range of the score function. Some Monte Carlo methods can use this to estimate the number of samples required for a certain quality of approximation. If not provided, it can be read from the scoring object if it provides it, for instance if it was constructed with compose_score().

TYPE: tuple[float, float] DEFAULT: (-float('inf'), float('inf'))

name

The name of the scorer. If not provided, the name of the function passed will be used.

TYPE: str | None DEFAULT: None

Tensor Support

SupervisedScorer supports both NumPy arrays and PyTorch tensors. Subclasses specialising in either type must specify the array type in the generic type.

New in version 0.5.0

Changed in version 0.10.0

This is now SupervisedScorer and holds the test data used to evaluate the model.

New in version 0.11.0

Added generic type support for arrays and tensor compatibility.

Source code in src/pydvl/valuation/scorers/supervised.py 
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
def __init__(
    self,
    scoring: str
    | SupervisedScorerCallable[SupervisedModelT, ArrayT]
    | SupervisedModelT,
    test_data: Dataset,
    default: float,
    range: tuple[float, float] = (-float("inf"), float("inf")),
    name: str | None = None,
):
    super().__init__()
    if isinstance(scoring, SupervisedModel):
        from sklearn.metrics import check_scoring

        self._scorer = check_scoring(scoring)
        if name is None:
            name = f"Default scorer for {scoring.__class__.__name__}"
    elif isinstance(scoring, str):
        self._scorer = get_scorer(scoring)
        if name is None:
            name = scoring
    else:
        self._scorer = scoring
        if name is None:
            name = getattr(scoring, "__name__", "scorer")
    self.test_data = test_data
    self.default = default
    # TODO: auto-fill from known scorers ?
    self.range = range
    self.name = name

SupervisedScorerCallable

Bases: Protocol[SupervisedModelT, ArrayT]

Signature for a supervised scorer