Skip to content

pydvl.valuation.scorers.utils

compose_score 

compose_score(
    scorer: SupervisedScorer,
    transformation: Callable[[float], float],
    name: str,
) -> SupervisedScorer

Composes a scoring function with an arbitrary scalar transformation.

Useful to squash unbounded scores into ranges manageable by data valuation methods.

Example:

sigmoid = lambda x: 1/(1+np.exp(-x))
compose_score(Scorer("r2"), sigmoid, range=(0,1), name="squashed r2")
PARAMETER DESCRIPTION
scorer

The object to be composed.

TYPE: SupervisedScorer

transformation

A scalar transformation

TYPE: Callable[[float], float]

range

The range of the transformation. This will be used e.g. by [Utility][pydvl.valuation.utility.Utility] for the range of the composite scorer.
name

A string representation for the composition, for str().

TYPE: str

RETURNS DESCRIPTION
SupervisedScorer

The composite SupervisedScorer.
Source code in src/pydvl/valuation/scorers/utils.py 
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
def compose_score(
    scorer: SupervisedScorer,
    transformation: Callable[[float], float],
    name: str,
) -> SupervisedScorer:
    """Composes a scoring function with an arbitrary scalar transformation.

    Useful to squash unbounded scores into ranges manageable by data valuation
    methods.

    Example:

    ```python
    sigmoid = lambda x: 1/(1+np.exp(-x))
    compose_score(Scorer("r2"), sigmoid, range=(0,1), name="squashed r2")
    ```

    Args:
        scorer: The object to be composed.
        transformation: A scalar transformation
        range: The range of the transformation. This will be used e.g. by
            [Utility][pydvl.valuation.utility.Utility] for the range of the
            composite scorer.
        name: A string representation for the composition, for `str()`.

    Returns:
        The composite [SupervisedScorer][pydvl.valuation.scorers.SupervisedScorer].
    """

    class CompositeSupervisedScorer(SupervisedScorer):
        def __call__(self, model: SupervisedModel) -> float:
            raw = super().__call__(model)
            return transformation(raw)

    new_scorer = CompositeSupervisedScorer(
        scoring=scorer._scorer,
        test_data=scorer.test_data,
        default=transformation(scorer.default),
        range=(transformation(scorer.range[0]), transformation(scorer.range[1])),
        name=name,
    )
    return new_scorer