scores.ConfidenceScore
Abstract Base Class for Confidence Scores.
Usage
scores.ConfidenceScore()Confidence scores quantify the deviation of query samples from the training distribution. Low scores indicate likely inliers (samples similar to training), while high scores indicate likely outliers (samples deviating from training). Samples with scores exceeding the threshold are excluded from prediction.
Attributes
trained: bool-
Whether the score has been trained. Defaults to
False. train_required: bool-
Whether training is required before scoring. Defaults to
False. cal_required: bool-
Whether calibration is required before selecting. Defaults to
False. calibrated: bool-
Whether the score has been calibrated. Defaults to
False. scores: torch.Tensor or None-
Confidence scores of the calibration samples. Low scores indicate likely inliers, high scores indicate likely outliers.
threshold: torch.Tensor or None-
Rejection threshold. Samples with scores higher than this value are excluded from prediction.
device: str-
Device to which internal tensors are put. Defaults to
"cpu". ident: str- String identifying the confidence score implementation.
Attributes
| Name | Description |
|---|---|
| cal_required | bool(x) -> bool |
| calibrated | bool(x) -> bool |
| train_required | bool(x) -> bool |
| trained | bool(x) -> bool |
cal_required
bool(x) -> bool
cal_required: bool = False
Returns True when the argument x is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
calibrated
bool(x) -> bool
calibrated: bool = False
Returns True when the argument x is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
train_required
bool(x) -> bool
train_required: bool = False
Returns True when the argument x is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
trained
bool(x) -> bool
trained: bool = False
Returns True when the argument x is true, False otherwise. The builtins True and False are the only two instances of the class bool. The class bool is a subclass of the class int, and cannot be subclassed.
Methods
| Name | Description |
|---|---|
| fit() | Fit a confidence score on training data. |
| get_threshold() | Get the current threshold value. |
| is_calibrated() | Return boolean indicating if the score is already calibrated. |
| is_trained() | Return boolean indicating if the score is already trained. |
| plot() | Plot densities for confidence scores. |
| requires_calibration() | Return boolean indicating if the score requires calibration. |
| requires_training() | Return boolean indicating if the score requires training. |
| score() | Calculate the confidence score for a tensor of samples. |
| select() | Select samples for prediction based on their confidence score. |
| set_calibrated() | Set a boolean that the score is already calibrated. |
| set_threshold() | Set a threshold based on a specific quantile on the available scores. |
| set_trained() | Set a boolean that the score is already trained. |
fit()
Fit a confidence score on training data.
Usage
fit(*args, **kwargs)X is used as training samples to fit the underlying method, while Y is an optional parameter that can be used to compute reference scores for the decision threshold (calibration set).
Subclasses define the exact parameter signatures and accepted input modes (precomputed tensors or model + DataLoader).
get_threshold()
Get the current threshold value.
Usage
get_threshold()is_calibrated()
Return boolean indicating if the score is already calibrated.
Usage
is_calibrated()is_trained()
Return boolean indicating if the score is already trained.
Usage
is_trained()plot()
Plot densities for confidence scores.
Usage
plot(query_scores=None, bins=100)By default, this method plots densities for the confidence scores. Optionally, it can also plot densities for query_scores.
Parameters
query_scores: torch.Tensor | None = None-
A
torch.Tensorrepresenting query scores to include in the plot. Defaults toNone. bins: int = 100-
An
intindicating the number of bins to use for density estimation. Defaults to100.
requires_calibration()
Return boolean indicating if the score requires calibration.
Usage
requires_calibration()requires_training()
Return boolean indicating if the score requires training.
Usage
requires_training()score()
Calculate the confidence score for a tensor of samples.
Usage
score(*args, **kwargs)Returns scores where low values indicate likely inliers and high values indicate likely outliers.
select()
Select samples for prediction based on their confidence score.
Usage
select(*args, **kwargs)Samples with scores lower than the threshold are selected for prediction, while samples with scores higher than the threshold are excluded.
Returns
dict[str, torch.Tensor]-
A dict with keys
'score'(raw confidence scores) and'selected'(boolean selection mask).
set_calibrated()
Set a boolean that the score is already calibrated.
Usage
set_calibrated()set_threshold()
Set a threshold based on a specific quantile on the available scores.
Usage
set_threshold(q=0.99)Samples with scores higher than this threshold are excluded from prediction.
Parameters
q: float = 0.99-
Quantile in the interval
(0, 1)used to compute the threshold from the stored calibration scores. Defaults to0.99.
Raises
ValueError- If no calibration scores are available yet.
set_trained()
Set a boolean that the score is already trained.
Usage
set_trained()See Also
- seapig.scores.knn.EuclideanScore: KNN-based score using Euclidean distance.
- seapig.scores.knn.CosineScore: KNN-based score using cosine distance.
- seapig.scores.pca.PCAScore: PCA reconstruction error score.
- seapig.scores.logits.SoftmaxScore: Softmax probability score.