1 rri_metrics.torchmetrics_single

rri_metrics.torchmetrics_single

Stateful one-step VIN and candidate-ranking evaluation metrics.

1.1 Classes

Name Description
CandidateTopKOracleHitMetric Accumulate candidate top-k oracle-hit rate.
LabelHistogram Accumulate label counts for ordinal classes.
RriErrorStats Accumulate finite-pair bias/variance statistics for RRI errors.
SelectedActionOracleComparisonMetric Accumulate selected-action oracle rank and regret diagnostics.
VinMetrics Container for VIN metrics computed from candidate rankings.
VinMetricsConfig Configuration for VIN torchmetrics bundles.

1.1.1 CandidateTopKOracleHitMetric

rri_metrics.torchmetrics_single.CandidateTopKOracleHitMetric(top_k=1)

Accumulate candidate top-k oracle-hit rate.

This metric evaluates one-step candidate ranking directly: a table is a hit when the model’s top-k predicted rows include at least one finite oracle-best candidate under the hard action mask. It keeps non-finite model scores separate from oracle labels, matching candidate_topk_oracle_hit.

1.1.1.1 Attributes

Name Description
hit_total Tensor["", float32] sum of finite table-level oracle hits.
hit_count Tensor["", float32] number of comparable candidate tables.

1.1.1.2 Methods

Name Description
update Accumulate one batch of candidate prediction/oracle tables.
compute Return finite mean top-k hit rate or NaN when empty.
1.1.1.2.1 update
rri_metrics.torchmetrics_single.CandidateTopKOracleHitMetric.update(
    predicted_scores,
    oracle_values,
    valid_mask=None,
    *,
    dim=-1,
)

Accumulate one batch of candidate prediction/oracle tables.

1.1.1.2.2 compute
rri_metrics.torchmetrics_single.CandidateTopKOracleHitMetric.compute()

Return finite mean top-k hit rate or NaN when empty.

1.1.2 LabelHistogram

rri_metrics.torchmetrics_single.LabelHistogram(num_classes)

Accumulate label counts for ordinal classes.

1.1.2.1 Attributes

Name Description
counts Tensor["K", int64] per-class counts, reduced by distributed sum.

1.1.3 RriErrorStats

rri_metrics.torchmetrics_single.RriErrorStats()

Accumulate finite-pair bias/variance statistics for RRI errors.

1.1.3.1 Attributes

Name Description
sum_error Tensor["", float32] sum of finite prediction-minus-label errors.
sum_error_sq Tensor["", float32] sum of squared finite errors.
count Tensor["", float32] count of finite error pairs.

1.1.4 SelectedActionOracleComparisonMetric

rri_metrics.torchmetrics_single.SelectedActionOracleComparisonMetric()

Accumulate selected-action oracle rank and regret diagnostics.

This metric evaluates a policy’s selected candidate against the finite oracle-labelled candidate table. It is intentionally separate from loss and training metrics: invalid selections are counted only through the valid table rate, and uncomparable rows do not contribute zero-valued regret or rank samples.

1.1.4.1 Attributes

Name Description
regret_total Tensor["", float32] sum of finite selected-action regrets.
regret_count Tensor["", float32] number of finite regret samples.
rank_total Tensor["", float32] sum of finite selected-action oracle ranks.
rank_count Tensor["", float32] number of finite rank samples.
percentile_total Tensor["", float32] sum of finite selected-action percentiles.
percentile_count Tensor["", float32] number of finite percentile samples.
valid_table_count Tensor["", float32] number of comparable candidate tables.
table_count Tensor["", float32] number of candidate tables presented.

1.1.4.2 Methods

Name Description
update Accumulate one batch of selected candidate indices and oracle values.
compute Return finite means plus the comparable-table rate.
1.1.4.2.1 update
rri_metrics.torchmetrics_single.SelectedActionOracleComparisonMetric.update(
    oracle_values,
    selected_indices,
    valid_mask,
    *,
    dim=-1,
)

Accumulate one batch of selected candidate indices and oracle values.

1.1.4.2.2 compute
rri_metrics.torchmetrics_single.SelectedActionOracleComparisonMetric.compute()

Return finite means plus the comparable-table rate.

1.1.5 VinMetrics

rri_metrics.torchmetrics_single.VinMetrics(num_classes, enable_spearman=True)

Container for VIN metrics computed from candidate rankings.

1.1.5.1 Attributes

Name Description
spearman Optional buffered Spearman correlation metric for this stage.
confusion Stateful ordinal-class confusion matrix for this stage.
label_hist Stateful ordinal-label histogram for this stage.
has_updates Tensor["", bool] indicating whether this stage received data.

1.1.6 VinMetricsConfig

rri_metrics.torchmetrics_single.VinMetricsConfig()

Configuration for VIN torchmetrics bundles.

1.1.6.1 Attributes

Name Description
target_type Factory target for aria_nbv.configs.base.BaseConfig.setup_target.
num_classes Number of ordinal classes used for confusion/histogram metrics.
enable_spearman Enable rank-correlation metrics that buffer all predictions/targets.

1.2 Functions

Name Description
topk_accuracy_from_probs Compute top-k accuracy from class probabilities.

1.2.1 topk_accuracy_from_probs

rri_metrics.torchmetrics_single.topk_accuracy_from_probs(
    probs,
    labels,
    *,
    top_k,
)

Compute top-k accuracy from class probabilities.

1.2.1.1 Parameters

Name Type Description Default
probs Tensor Tensor["N K"] class probabilities. required
labels Tensor Tensor["N"] integer class labels. required
top_k int Number of highest-probability classes to consider. required

1.2.1.2 Returns

Name Type Description
Tensor Tensor[""] scalar accuracy in [0, 1].
Back to top