pydvl.value.shapley.classwise
¶
Classwise Shapley (Schoch et al., 2022)^{1} offers a Shapley framework tailored for classification problems. Let \(D\) be a dataset, \(D_{y_i}\) be the subset of \(D\) with labels \(y_i\), and \(D_{y_i}\) be the complement of \(D_{y_i}\) in \(D\). The key idea is that a sample \((x_i, y_i)\), might enhance the overall performance on \(D\), while being detrimental for the performance on \(D_{y_i}\). The Classwise value is defined as:
where \(S_{y_i} \subseteq D_{y_i} \setminus \{i\}\) and \(S_{y_i} \subseteq D_{y_i}\).
Analysis of Classwise Shapley
For a detailed analysis of the method, with comparison to other valuation techniques, please refer to the main documentation.
In practice, the quantity above is estimated using Monte Carlo sampling of the powerset and the set of index permutations. This results in the estimator
with \(S^{(1)}, \dots, S^{(K)} \subseteq T_{y_i},\) \(\sigma^{(1)}, \dots, \sigma^{(L)} \in \Pi(T_{y_i}\setminus\{i\}),\) and \(\sigma^{(l)}_{:i}\) denoting the set of indices in permutation \(\sigma^{(l)}\) before the position where \(i\) appears. The sets \(T_{y_i}\) and \(T_{y_i}\) are the training sets for the labels \(y_i\) and \(y_i\), respectively.
Notes for derivation of test cases
The unit tests include the following manually constructed data: Let \(D=\{(1,0),(2,0),(3,0),(4,1)\}\) be the test set and \(T=\{(1,0),(2,0),(3,1),(4,1)\}\) the train set. This specific dataset is chosen as it allows to solve the model
in closed form \(\beta = \frac{\text{dot}(x, y)}{\text{dot}(x, x)}\). From the closedform solution, the tables for inclass accuracy \(a_S(D_{y_i})\) and outofclass accuracy \(a_S(D_{y_i})\) can be calculated. By using these tables and setting \(\{S^{(1)}, \dots, S^{(K)}\} = 2^{T_{y_i}}\) and \(\{\sigma^{(1)}, \dots, \sigma^{(L)}\} = \Pi(T_{y_i}\setminus\{i\})\), the Monte Carlo estimator can be evaluated (\(2^M\) is the powerset of \(M\)). The details of the derivation are left to the eager reader.
References¶

Schoch, Stephanie, Haifeng Xu, and Yangfeng Ji. CSShapley: Classwise Shapley Values for Data Valuation in Classification. In Proc. of the ThirtySixth Conference on Neural Information Processing Systems (NeurIPS). New Orleans, Louisiana, USA, 2022. ↩
ClasswiseScorer
¶
ClasswiseScorer(
scoring: Union[str, ScorerCallable] = "accuracy",
default: float = 0.0,
range: Tuple[float, float] = (0, 1),
in_class_discount_fn: Callable[[float], float] = lambda x: x,
out_of_class_discount_fn: Callable[[float], float] = np.exp,
initial_label: Optional[int] = None,
name: Optional[str] = None,
)
Bases: Scorer
A Scorer designed for evaluation in classification problems. Its value is computed from an inclass and an outofclass "inner score" (Schoch et al., 2022) ^{1}. Let \(S\) be the training set and \(D\) be the valuation set. For each label \(c\), \(D\) is factorized into two disjoint sets: \(D_c\) for inclass instances and \(D_{c}\) for outofclass instances. The score combines an inclass metric of performance, adjusted by a discounted outofclass metric. These inner scores must be provided upon construction or default to accuracy. They are combined into:
where \(f\) and \(g\) are continuous, monotonic functions. For a detailed explanation, refer to section four of (Schoch et al., 2022)^{ 1}.
Warning
Metrics must support multiple class labels if you intend to apply them
to a multiclass problem. For instance, the metric 'accuracy' supports
multiple classes, but the metric f1
does not. For a twoclass
classification problem, using f1_weighted
is essentially equivalent to
using accuracy
.
PARAMETER  DESCRIPTION 

scoring 
Name of the scoring function or a callable that can be passed to Scorer.
TYPE:

default 
Score to use when a model fails to provide a number, e.g. when too little was used to train it, or errors arise.
TYPE:

range 
Numerical range of the score function. Some Monte Carlo methods
can use this to estimate the number of samples required for a
certain quality of approximation. If not provided, it can be read
from the 
in_class_discount_fn 
Continuous, monotonic increasing function used to discount the inclass score. 
out_of_class_discount_fn 
Continuous, monotonic increasing function used to discount the outofclass score. 
initial_label 
Set initial label (for the first iteration) 
name 
Name of the scorer. If not provided, the name of the inner scoring
function will be prefixed by 
New in version 0.7.1
Source code in src/pydvl/value/shapley/classwise.py
estimate_in_class_and_out_of_class_score
¶
estimate_in_class_and_out_of_class_score(
model: SupervisedModel,
x_test: NDArray[float_],
y_test: NDArray[int_],
rescale_scores: bool = True,
) > Tuple[float, float]
Computes inclass and outofclass scores using the provided inner scoring function. The result is
In this context, for label \(c\) calculations are executed twice: once for \(D_c\) and once for \(D_{c}\) to determine the inclass and outofclass scores, respectively. By default, the raw scores are multiplied by \(\frac{D_c}{D}\) and \(\frac{D_{c}}{D}\), respectively. This is done to ensure that both scores are of the same order of magnitude. This normalization is particularly useful when the inner score function \(a_S\) is calculated by an estimator of the form \(\frac{1}{N} \sum_i x_i\), e.g. the accuracy.
PARAMETER  DESCRIPTION 

model 
Model used for computing the score on the validation set.
TYPE:

x_test 
Array containing the features of the classification problem. 
y_test 
Array containing the labels of the classification problem. 
rescale_scores 
If set to True, the scores will be denormalized. This is particularly useful when the inner score function \(a_S\) is calculated by an estimator of the form \(\frac{1}{N} \sum_i x_i\).
TYPE:

RETURNS  DESCRIPTION 

Tuple[float, float]

Tuple containing the inclass and outofclass scores. 
Source code in src/pydvl/value/shapley/classwise.py
compute_classwise_shapley_values
¶
compute_classwise_shapley_values(
u: Utility,
*,
done: StoppingCriterion,
truncation: TruncationPolicy,
done_sample_complements: Optional[StoppingCriterion] = None,
normalize_values: bool = True,
use_default_scorer_value: bool = True,
min_elements_per_label: int = 1,
n_jobs: int = 1,
parallel_backend: Optional[ParallelBackend] = None,
config: Optional[ParallelConfig] = None,
progress: bool = False,
seed: Optional[Seed] = None
) > ValuationResult
Computes an approximate Classwise Shapley value by sampling independent permutations of the index set for each label and index sets sampled from the powerset of the complement (with respect to the currently evaluated label), approximating the sum:
where \(\sigma_{:i}\) denotes the set of indices in permutation sigma before the position where \(i\) appears and \(S\) is a subset of the index set of all other labels (see the main documentation for details).
PARAMETER  DESCRIPTION 

u 
Utility object containing model, data, and scoring function. The scorer must be of type ClasswiseScorer.
TYPE:

done 
Function that checks whether the computation needs to stop.
TYPE:

truncation 
Callable function that decides whether to interrupt processing a permutation and set subsequent marginals to zero.
TYPE:

done_sample_complements 
Function checking whether computation needs to stop. Otherwise, it will resample conditional sets until the stopping criterion is met.
TYPE:

normalize_values 
Indicates whether to normalize the values by the variation in each class times their inclass accuracy.
TYPE:

done_sample_complements 
Number of times to resample the complement set for each permutation.
TYPE:

use_default_scorer_value 
The first set of indices is the sampled complement set. Unless not otherwise specified, the default scorer value is used for this. If it is set to false, the base score is calculated from the utility.
TYPE:

min_elements_per_label 
The minimum number of elements for each opposite label.
TYPE:

n_jobs 
Number of parallel jobs to run.
TYPE:

parallel_backend 
Parallel backend instance to use
for parallelizing computations. If
TYPE:

config 
(DEPRECATED) Object configuring parallel computation, with cluster address, number of cpus, etc.
TYPE:

progress 
Whether to display a progress bar.
TYPE:

seed 
Either an instance of a numpy random number generator or a seed for it.
TYPE:

RETURNS  DESCRIPTION 

ValuationResult

ValuationResult object containing computed data values. 
New in version 0.7.1
Source code in src/pydvl/value/shapley/classwise.py
237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 
