API Reference

Task & Metric Classes

Core classes for defining selective inference tasks and metrics

SelectiveInferenceTask: Wrap a trained LightningModule to attach selection results during inference.
SelectiveMetric: Evaluate a metric on full, selected, and rejected subsets.
RiskCoverageMetric: Build a risk-coverage curve from scores and per-sample errors.
RiskCoverage: Container for risk-coverage results.

Uncertainty scoring methods for embedding-based approaches

scores.CosineScore: Returns the KNN-distance based on the cosine distance to the nearest samples.
scores.CosineClassWiseScore: Class-wise version of scores.knn.CosineScore.
scores.EuclideanScore: Returns the KNN-distance based on the Euclidean distance to the nearest samples.
scores.EuclideanClassWiseScore: Class-wise version of scores.knn.EuclideanScore.
scores.MahalanobisScore: Returns the Mahalanobis distance to the training samples distribution.
scores.MahalanobisClassWiseScore: Class-wise version of .scores.knn.MahalanobisScore.
scores.PCAScore: Returns uncertainty scores based on PCA reconstruction errors.
scores.PyODScore: Uncertainty scores based on detectors supplied by PyOD.
scores.utils.TensorPCA: Tensor-based PCA.

Uncertainty scoring methods based on model logits

scores.EnergyScore: Free energy of the logit distribution.
scores.EnergyClassWiseScore: Class-wise version of ~seapig.scores.logits.EnergyScore.
scores.EntropyScore: Predictive entropy (worst label for multilabel).
scores.EntropyClassWiseScore: Class-wise version of ~seapig.scores.logits.EntropyScore.
scores.MarginScore: Negative top-two logit margin.
scores.MarginClassWiseScore: Class-wise version of ~seapig.scores.logits.MarginScore.
scores.SoftmaxScore: Negative maximum predicted probability.
scores.SoftmaxClassWiseScore: Class-wise version of ~seapig.scores.logits.SoftmaxScore.
scores.MutualInformationScore: Mutual information (BALD) uncertainty score for ensembles / MC-dropout.
scores.MutualInformationClassWiseScore: Class-wise version of ~seapig.scores.logits.MutualInformationScore.
scores.PredictiveVarianceScore: Variance of predicted probabilities across members.
scores.PredictiveVarianceClassWiseScore: Class-wise version of ~seapig.scores.logits.PredictiveVarianceScore.

Uncertainty scoring methods based on other principles

Abstract base classes

scores.UncertaintyScore: Abstract Base Class for Uncertainty Scores.
scores.EmbeddingScore: Base class for embedding-based uncertainty scores.
scores.KNNScore: Abstract base class for KNN distance-based uncertainty scores.
scores.LogitScore: Base class for logit-based uncertainty scores.
scores.ClassWiseScore: Base class-wise wrapper.

Helpers for progress and logging