Reference

Task & Metric Classes

Core classes for defining selective inference tasks and metrics

SelectiveInferenceTask: Wrap a trained LightningModule to attach selection results during inference.
SelectiveMetric: Evaluate a metric on full, selected, and rejected subsets.
RiskCoverageMetric: Build a risk-coverage curve from scores and per-sample errors.
RiskCoverage: Container for risk-coverage results.

Confidence scoring methods for embedding-based approaches

scores.CosineScore: Returns the KNN-distance based on the cosine distance to the nearest samples.
scores.EuclideanScore: Returns the KNN-distance based on the Euclidean distance to the nearest samples.
scores.MahalanobisScore: Returns the Mahalanobis distance to the training samples distribution.
scores.PCAScore: Returns confidence scores based on PCA reconstruction errors.
scores.PyODScore: Confidence scores based on detectors supplied by PyOD.

Confidence scoring methods based on model logits

Confidence scoring methods based on other principles

Abstract base classes

scores.ConfidenceScore: Abstract Base Class for Confidence Scores.
scores.EmbeddingScore: Base class for embedding-based confidence scores.
scores.KNNScore: Abstract base class for KNN distance-based confidence scores.
scores.LogitScore: Base class for logit-based confidence scores.

Helpers for progress and logging