1 lightning.finite_horizon.objective

lightning.finite_horizon.objective

Compute the complete finite-horizon fitted-Q learning transaction.

This module owns the scientific computation between an admitted QhBatch and the scalar optimized by Lightning. It performs the joint scalar-horizon scorer query, constructs dense one-step and selected recursive targets, applies the distinct support masks, and reduces candidate losses to state means and state means to equally weighted horizon means. The lifecycle module owns optimizer stepping, target-network mutation, logging, and checkpoint state.

For non-empty realized horizon set :math:\mathcal H, the utility scalar is

.. math::

L = {hH} {sS_h} L_{s,h}.

At :math:h=1, :math:L_{s,1} is a mean over every finite labelled hard-valid candidate. At :math:h>1, it is the selected-action Huber or CORAL loss against the Double-Q target. This order prevents wide candidate tables or more common horizons from dominating the objective.

The opt-in continuing profile treats saved rollout length as a collection cutoff. Its explicit lookahead queries are independent of that countdown; only an actual successor with finite supported backup actions supplies a recursive target. The h=1 query remains directly supervised at the cutoff.

The transaction adds configured feasibility BCE and within-state ranking auxiliaries to this utility scalar for training. Checkpoint selection retains the utility scalar alone; detached auxiliary statistics expose the existing profile-specific support denominators without changing evaluation targets.

1.1 Classes

Name Description
QhFittedQObjective Horizon-balanced dense-Q1 and recursive fitted-Q objective.

1.1.1 QhFittedQObjective

lightning.finite_horizon.objective.QhFittedQObjective(
    loss,
    dense_q1_targets,
    dense_q1_support,
    recursive_targets,
    recursive_support,
    loss_sum_by_horizon,
    state_count_by_horizon,
    candidate_count_by_horizon,
    absolute_error_sum_by_horizon,
    ranking_accuracy_sum_by_horizon,
    ranking_state_count_by_horizon,
    ranking_pair_count_by_horizon,
    selected_regret_sum_by_horizon,
    selected_regret_state_count_by_horizon,
    gradient_components=None,
    q1_coral_loss_sum=None,
    q1_huber_loss_sum=None,
    recursive_requested_horizon=None,
)

Horizon-balanced dense-Q1 and recursive fitted-Q objective.

dense_q1_targets supervises every labelled hard-valid candidate with immediate reward at an explicit h=1 query. recursive_targets supervises only the factual selected action at h>1 and may include a masked h-1 bootstrap. Collapsing these into one candidate table would erase the requested-horizon index and can assign two different targets to the same state-candidate pair. Dense candidate losses are averaged within each factual state first; state means are then averaged within horizon, and non-empty horizons receive equal weight.

The returned sufficient statistics are local to the current process. The utility-only scalar loss applies the distributed denominator correction requested by :func:_compute_qh_objective, while all reporting tensors remain detached.

1.1.1.1 Attributes

Name Description
loss Tensor["", float32] horizon-balanced utility/checkpoint-selection scalar.
dense_q1_targets Tensor["B S N", float32] immediate rewards for explicit h=1 queries.
dense_q1_support Tensor["B S N", bool] finite hard-valid candidate support for dense Q1.
recursive_targets Tensor["B S", float32] selected-action fitted targets at queried h>1.
recursive_support Tensor["B S", bool] factual selected-transition support at h>1.
loss_sum_by_horizon Tensor["H_max", float32\|float64] local state-normalized loss sums.
state_count_by_horizon Tensor["H_max", int64] local state contributions; chain states remain dependent.
candidate_count_by_horizon Tensor["H_max", int64] local raw candidate-label contributions.
absolute_error_sum_by_horizon Tensor["H_max", float32\|float64] state-normalized absolute-error sums.
ranking_accuracy_sum_by_horizon Tensor["H_max", float32\|float64] within-state pairwise-accuracy sums.
ranking_state_count_by_horizon Tensor["H_max", int64] states with at least one unequal-target pair.
ranking_pair_count_by_horizon Tensor["H_max", int64] raw unequal-target candidate pairs.
selected_regret_sum_by_horizon Tensor["H_max", float32\|float64] state-level greedy regret sums.
selected_regret_state_count_by_horizon Tensor["H_max", int64] states with complete counterfactual regret support.
gradient_components Named Tensor["", float32] differentiable normalized utility components.
q1_coral_loss_sum Tensor["", float32\|float64] detached threshold-BCE state-mean sum.
q1_huber_loss_sum Tensor["", float32\|float64] detached native-Q Huber state-mean sum.
recursive_requested_horizon Explicit continuing queries of shape [B,S], or None for bounded profiles.
Back to top