scores.EntropyScore

Predictive entropy (worst label for multilabel).

Usage

Source

scores.EntropyScore(
    temperature=None,
    task="multiclass",
    per_member=False,
)

Methods

Name Description
fit() Fit the score on reference logits.
score() Compute uncertainty scores for query logits.
select() Select samples for prediction based on their uncertainty score.
set_threshold() Set a threshold based on a specific quantile on the available scores.
get_threshold() Get the current threshold value.
plot() Plot densities for uncertainty scores.

fit()

Fit the score on reference logits.

Usage

Source

fit(
    X=None,
    Y=None,
    temp_scale=False,
    model=None,
    loader=None,
    outdir=None,
    prefix=None,
    *args,
    **kwargs
)

This method supports two usage modes:

  1. Precomputed logits: Supply logits directly via X, with optional labels via Y for temperature fitting.
  2. On-the-fly extraction: Supply a model with a .logits() method and a DataLoader to extract logits automatically.

You must use either logits OR model+loader, but not both.

Parameters
X: torch.Tensor or None = None

Reference logits. Shape depends on task (see class docstring). Required when not using model and loader.

Y: torch.Tensor or None = None

Optional labels for temperature fitting. Shape/type depends on task.

temp_scale: bool = False

Boolean indicating if temperature scaling is to be applied. Defaults to False. If set to True labels are required.

model: torch.nn.Module or None = None

Model with a .logits(x) method. Required when not using precomputed logits.

loader: DataLoader or None = None

DataLoader yielding batches for inference. Required when using model.

outdir: Path or str or None = None

Optional directory to save/load logits. Only used with model and loader.

prefix: str or None = None
Optional prefix for saved files. Only used with model and loader.
Notes

Labels are required for temperature fitting to minimize NLL for the task.

score()

Compute uncertainty scores for query logits.

Usage

Source

score(query_logits=None, model=None, loader=None, outdir=None, prefix=None)

This method supports two usage modes:

  1. Precomputed logits: Supply query logits via query_logits.
  2. On-the-fly extraction: Supply a model with an .logits() method and a DataLoader to extract logits automatically.

You must use either logits (query_logits) OR model+loader, but not both.

# Mode 1: Precomputed logits
from seapig.scores import EntropyScore
my_score = EntropyScore()
scores = my_score.score(query_logits=test_logits)

# Mode 2: On-the-fly extraction
my_score = EntropyScore()
scores = my_score.score(model=model, loader=test_dl)
Parameters
query_logits: torch.Tensor = None

Logits for samples to score. Shape depends on task.

model: torch.nn.Module | None = None

A torch.nn.Module with an .embed() method. Required when not using X.

loader: DataLoader[Batch] | None = None

A torch.utils.data.DataLoader returning torch.Tensors or dicts with the "image" key. Required when using model.

outdir: Path | None = None

A pathlib.Path pointing to a directory for saving/loading embeddings. Only used with model and loader.

prefix: str | None = None
A str used as filename prefix for saved embeddings. Only used with model and loader.
Returns
torch.Tensor
1-D tensor of shape (N,). Lower values indicate lower uncertainty.

select()

Select samples for prediction based on their uncertainty score.

Usage

Source

select(query_logits=None, model=None, loader=None, outdir=None, prefix=None)

Samples with scores lower than the threshold are selected for prediction, while samples with scores higher than the threshold are excluded.

Parameters
query_logits: torch.Tensor = None

Logits for samples to select. Shape depends on task.

model: torch.nn.Module | None = None

A torch.nn.Module with an .embed() method. Required when not using X.

loader: DataLoader[Batch] | None = None

A torch.utils.data.DataLoader returning torch.Tensors or dicts with the "image" key. Required when using model.

outdir: Path | None = None

A pathlib.Path pointing to a directory for saving/loading embeddings. Only used with model and loader.

prefix: str | None = None
A str used as filename prefix for saved embeddings. Only used with model and loader.
Returns
dict[str, torch.Tensor]
A dict with keys 'score' (uncertainty scores) and 'selected' (boolean mask where True means the sample is selected).

set_threshold()

Set a threshold based on a specific quantile on the available scores.

Usage

Source

set_threshold(q=0.99)

Samples with scores higher than this threshold are excluded from prediction.

Parameters
q: float = 0.99
Quantile in the interval (0, 1) used to compute the threshold from the stored calibration scores. Defaults to 0.99.
Raises
ValueError
If no calibration scores are available yet.

get_threshold()

Get the current threshold value.

Usage

Source

get_threshold()

plot()

Plot densities for uncertainty scores.

Usage

Source

plot(query_scores=None, bins=100)

By default, this method plots densities for the uncertainty scores. Optionally, it can also plot densities for query_scores.

Parameters
query_scores: torch.Tensor | None = None

A torch.Tensor representing query scores to include in the plot. Defaults to None.

bins: int = 100
An int indicating the number of bins to use for density estimation. Defaults to 100.