1 rri_metrics.ranking

rri_metrics.ranking

Non-differentiable candidate-ranking evaluation.

This module provides comparisons of actor-selected or predicted candidates against finite hard-valid Oracle label tables without changing selection behavior.

1.1 Classes

Name Description
SelectedActionOracleComparison Per-table oracle diagnostics for a selected candidate action.

1.1.1 SelectedActionOracleComparison

rri_metrics.ranking.SelectedActionOracleComparison(
    selected_oracle_regret,
    selected_oracle_rank,
    selected_oracle_percentile,
    valid_table,
)

Per-table oracle diagnostics for a selected candidate action.

1.1.1.1 Attributes

Name Type Description
selected_oracle_regret Tensor Tensor["B"] oracle-best value minus the selected candidate’s oracle value. Lower is better; oracle-best ties have zero regret.
selected_oracle_rank Tensor Tensor["B"] one-based selected-candidate rank among finite hard-valid oracle labels. Tied oracle-best candidates have rank 1.
selected_oracle_percentile Tensor Tensor["B"] rank percentile in [0, 1] where 1 means oracle-best and 0 means worst among comparable finite labels.
valid_table Tensor Tensor["B"] mask for tables with a finite valid selected label and at least one finite valid oracle candidate.

1.2 Functions

Name Description
candidate_topk_oracle_hit Report whether predicted top-k rows include an oracle-best candidate.
selected_action_oracle_comparison Compare selected candidate actions against oracle-best candidate labels.

1.2.1 candidate_topk_oracle_hit

rri_metrics.ranking.candidate_topk_oracle_hit(
    predicted_scores,
    oracle_values,
    valid_mask=None,
    *,
    top_k=1,
    dim=-1,
)

Report whether predicted top-k rows include an oracle-best candidate.

The oracle-best tied set is computed from finite oracle values under the hard mask. Non-finite predicted scores are excluded from the predicted top-k set without removing their oracle labels, so a model that emits NaN on the true best row records a miss when other finite predictions exist. Predicted ties at the kth boundary are included to avoid arbitrary tie-breaking.

1.2.1.1 Parameters

Name Type Description Default
predicted_scores Tensor Candidate scores produced by a model or policy. required
oracle_values Tensor Candidate oracle values with the same shape as predicted_scores. required
valid_mask Tensor | None Optional hard-validity mask broadcastable to predicted_scores. None
top_k int Number of predicted candidates to consider before kth-boundary ties are expanded. 1
dim int Candidate dimension. -1

1.2.1.2 Returns

Name Type Description
Tensor Float hit indicator per candidate table. Empty oracle-labeled tables or
Tensor tables with no finite predictions return NaN.

1.2.2 selected_action_oracle_comparison

rri_metrics.ranking.selected_action_oracle_comparison(
    oracle_values,
    selected_indices,
    valid_mask,
    *,
    dim=-1,
)

Compare selected candidate actions against oracle-best candidate labels.

This reducer is for policy evaluation, not supervision. It reads a selected candidate index per table and reports how far that action is from the finite oracle-best row under the hard action mask. Invalid selected indices, masked selected rows, non-finite selected labels, and empty oracle tables return NaN diagnostics with valid_table=False.

1.2.2.1 Parameters

Name Type Description Default
oracle_values Tensor Candidate oracle values, such as target reward or finite-horizon oracle return. required
selected_indices Tensor Selected candidate indices with shape equal to oracle_values after removing dim; values may be integer or floating tensors, but non-finite or non-integral entries are treated as invalid selections. required
valid_mask Tensor Boolean hard-validity mask broadcastable to oracle_values. required
dim int Candidate dimension. -1

1.2.2.2 Returns

Name Type Description
SelectedActionOracleComparison SelectedActionOracleComparison with per-table regret, one-based rank,
SelectedActionOracleComparison rank percentile, and comparability mask.
Back to top