#import "../../template/utils/diagram.typ": flex-caption #import "../../../shared/macros.typ": * #import "../../../shared/symbols.typ": symb #import "../../../shared/equations.typ": eqs #import "../../draft_markers.typ": thesis_status, review_note, research_todo, decision_todo #import "../../../shared/tables.typ": publication-table == Replay Eligibility and Finite-Horizon Learning Gate The factual rollout tables determine which records may supervise each learning problem. Five predicates answer different questions. *Proposal validity* says that a materialized row passes the declared proposal and hard-geometry checks. *Action support* says that the admitted actor protocol permits selecting it. *Label support* adds an admitted target-label state and a finite root-gain label. *Selected support* marks the one factual action taken on this branch. *Bootstrap support* admits its continuation: a reproducible successor and the supported lower-horizon candidates required by the objective. Padding and modality presence remain separate structural predicates. A feasible side view whose render fails can remain an action while lacking a label; an unselected labelled view can supervise immediate gain while lacking a factual successor. Selection uses action support, immediate loss uses action and label support, and recursive loss uses the selected transition and its continuation evidence. The diagnostic RRI audit has its own availability predicate, so a missing auxiliary metric does not remove a finite root-gain label. Exclusion reports retain the failed predicate for each operation. Equal counts in two masks do not establish that their definitions, information contracts, or label populations are identical. Dense one-step supervision requires finite immediate labels for every action in the declared comparison support. Legacy subset supervision instead learns only from its labelled eligible rows; completing a rollout does not make that subset dense. Exact two-step supervision is narrower: the factual first action must have a stored reward and either an explicit terminal outcome, whose continuation is exactly zero, or a valid successor step with finite one-step root-gain labels for every hard-valid successor candidate. #review_note(kind: "keep", id: "E09", position: right, note: [Keep this completeness requirement. Missing feasible successor labels prevent an exact maximum; a small supported subset cannot certify the same target or population.], )[This completeness condition makes the masked successor maximum exact over the stored action set; one finite label is insufficient because an unlabeled hard-valid action could have the true maximum.] #review_note(kind: "clarify", id: "R260920-E01", position: auto, note: [Separate exact-Q2 completeness from recursive one-step label admission. Remove the implied nesting and align the table and Double-Q wording; learned continuation is not certified multi-step evidence.], source: [Method, Fitted-Q recursion; shared/equations/rl.typ: qh_supported_successor_set], gate: [blocking for understanding recursive evidence; reconcile the three descriptions now], )[Recursive supervision has a different continuation requirement]: it uses a factual selected transition and immediate reward. When continuation is required, it evaluates a learned lower-horizon value on a recorded successor. The current bootstrap set contains hard-valid successor rows with finite one-step labels; it does not certify a factual multi-step return for each of those rows. Exact two-step supervision instead needs complete immediate labels over the stored hard-valid successor set, so that no unlabelled feasible action could change its maximum. The derived finite-horizon arrays align these fields on a padded state--candidate view; they do not create labels for unobserved transitions, make selected GT depth actor-visible, or turn sparse long-horizon action support into dense support. #figure( publication-table( columns: (0.78fr, 1.10fr, 1.42fr), header: ([*Learning surface*], [*Purpose*], [*Minimum factual evidence*]), rows: ( [dense $h=1$], [immediate candidate value], [finite one-step root-gain labels on every action in the declared support], [exact $h=2$], [base-case finite-support value], [selected reward plus either terminal outcome or complete finite one-step labels over the hard-valid successor action set], [recursive $h>1$], [variable-horizon fitted value], [selected reward and boundary convention; continuation uses a stored successor, hard-valid rows with finite one-step labels, and a learned lower-horizon estimate], [behavior return], [policy-conditioned Monte-Carlo control], [complete rewards for the declared query, or a verified task boundary, plus behavior-policy identity], ), ), caption: flex-caption([Required replay evidence by learning target. Dense immediate labels, exact two-step targets, recursive optimal-continuation targets, and behavior-policy returns are distinct supervision surfaces.], [Required replay evidence by learning target.]), ) The measured root/replay compatibility and batch probes are recorded in @ssec:recorded-replay-compatibility. Readable storage and successful joins do not establish admission to a particular learning objective. #figure( publication-table( columns: (0.72fr, 1.18fr, 1.25fr), header: ([*Activity or claim*], [*Required evidence*], [*Limit*]), rows: ( [Executable scoring], [compatible checkpoint, declared input/state protocol, hard support and supported query], [A runnable scorer does not establish predictive accuracy or policy benefit.], [One-step prediction], [finite held-out immediate labels; source-matched inputs, scene split and prediction/ranking measures], [Does not require a two-step or policy-endpoint result.], [Multi-step prediction], [supported targets for each declared horizon and budget; independent reference matching the stated continuation], [An exact-Q2 backup check certifies the target construction, not the learned prediction.], [Observation-updated state], [typed causal reader, deterministic fusion, source masks and no-future-observation tests], [Construction tests do not establish state sufficiency or policy benefit.], [Policy improvement], [held-out paired endpoints under equal budget and a matched evaluation contract], [Policy outcomes do not establish unbiased numerical values. Optional myopic controls require their own frozen scorer.], ), ), caption: flex-caption([Claim-specific evidence requirements. Executable scoring, held-out prediction and acquisition outcomes answer different questions.], [Claim-specific evidence requirements.]), ) #thesis_status( implementation: "partial", evidence: "pending", citation: [@FittedQIteration-ernst2005 @FixedHorizonTD-deAsis2020 @DoubleDQN-vanHasselt2015 @CQL-kumar2020 @BCQ-fujimoto2019], source: "aria_nbv/aria_nbv/data_handling/qh_data/batching.py; aria_nbv/aria_nbv/lightning/finite_horizon/datamodule.py; aria_nbv/aria_nbv/lightning/finite_horizon/module.py; aria_nbv/aria_nbv/lightning/finite_horizon/objective.py; aria_nbv/aria_nbv/lightning/finite_horizon/certification.py; aria_nbv/aria_nbv/rollouts/storage/qh_reader.py", gate: [populated held-out exact-Q2 receipt, supported H>2 targets, compatible checkpoint, frozen state protocol, and independent held-out oracle re-evaluation], )[The scalar requested-horizon A0/A1 controls, dense-Q1 plus selected-recursion Double-Q learner, feasibility auxiliary, manifest-bound trained-horizon support, and bounded stratified exact-Q2 certification surface are implemented. A1 remains the default; a qualifying population receipt, task-sufficient dynamic state, and comparative policy evidence remain unavailable.] Method defines the scorer and externally masked selection rule. The masked argmax is already the discrete decision rule. A separate actor network and online data collection are not required to train or execute this finite-candidate policy. Batch fitted Q iteration explicitly learns a greedy Q function from a fixed collection of transitions by repeatedly solving supervised regression problems @FittedQIteration-ernst2005. === Prediction support and independent backup checks The comparisons use Method's immediate and recursive targets (@ssec:thesis-horizon-recursive-offline-learning). Immediate supervision includes finite hard-valid labels at each realized state; recursive supervision needs its selected transition and the successor support required by the chosen profile. Padding and parallel batching add no observations. The pure one-step control preserves factual budgets and acquired prefixes while omitting recursive targets and feasibility supervision, so it does not also test joint validity learning. Two-step verification separates arithmetic from empirical agreement. Exact-table injection tests the backup calculation. A frozen-bundle comparison measures its error against the factual exact two-step reference on supported rows, using @ssec:thesis-exact-two-step-target. Agreement checks the backup target, not the learned two-step prediction or policy outcome. Physical scenes define the statistical clusters even when several stores or chains reuse them. Cross-store deduplication remains unverified; the minimum of five provenance groups in the existing check is neither proof of five independent scenes nor a power calculation. Missing support is unavailable evidence, not zero error. The sampling strata, profile-specific boundaries, support gates and batching procedure are retained in @ssec:repro-replay-validation. Longer-horizon claims require independent outcomes; the terminal-step proxy alone cannot supply them. === DDQN-style and behavior-return controls The Double-Q protocol uses Method's online selector and delayed evaluator on the same supported successor rows. Horizon one has no continuation. Terminal and empty-action masks can also yield zero continuation in the legacy countdown profile; record absence can therefore leave an immediate-only target. The continuing profile instead excludes recursive rows without a recorded, finite-label-supported successor. Comparisons must name the profile and verify its admission, rather than equating a recording cutoff with a task endpoint. A simpler single-estimator frozen lower-horizon maximum remains a planned matched control until it has an executable configuration and receipt. Its comparison must hold state, action support and scalar horizon targets fixed. As distinguished from the greedy value objective in @ssec:thesis-horizon-recursive-offline-learning, a retained chain also yields a complete observed return for a declared query length, using the reward sum defined in Foundations and specialized in Chapter 3. A shorter retained segment can describe the gain actually observed along that collector, but it is incomplete evidence for a longer query. Its unobserved suffix is not filled with zeros. A complete behavior-return comparison therefore requires continuity and the same measurement convention throughout the query, or an independently verified task endpoint. #review_note(kind: "keep", id: "E15", position: right, note: [Preserve the named behavior-policy estimand. Pooling returns from different collectors does not create optimal-continuation labels; clarify whether this control is essential or conditional.], )[These observed returns describe continuation under the named collector $mu$, rather than greedy finite-support continuation.] A predictor fitted to them is a behavior-return control. Interpreting it as the expected $Q^mu$ additionally requires appropriate coverage and information assumptions, and the mean-value check for Huber fitting described in Method. Behavior returns from random-valid, greedy, softmax, and oracle-lookahead chains must therefore remain identified rather than pooled as if they represented one optimal value function. For a fixed implemented oracle policy #symb.rl.oracle_policy, each complete chain supplies #symb.rl.sampled_oracle_return. Estimating its expected return requires repeated outcomes from the declared conditional population, with policy, initial information and decision protocol held fixed. Branches sharing a prefix and trajectories sharing a scene can carry common sources of dependence; their count alone does not establish independent support. Sampling weights and uncertainty estimates must respect the sampling design, such dependence, and any branch-selection probabilities. A deterministic replay of the same scene and seed can reproduce a return without adding a new outcome. Selecting the best explored return answers a search question. Beam pruning retains the highest accumulated selection scores among explored partial paths; when those scores are reward sums, this selects strong realized returns within the explored tree. Its retained maximum is neither an unbiased estimate of a fixed policy's expected return nor a certificate of #symb.rl.ideal_oracle_q. Reward-scored search and expected-return estimation must therefore report their sampling and retention rules separately. // evidence: // - @ReinforcementLearning-sutton2018 -> docs/literature/pdf/RLbook2020.pdf#page=80-89 (fixed-policy expectation and sample averages) // - rollouts/replay/engine.py:1270-1290,1378-1383; state.py:245-270 (shared root and retained paths) #decision_todo( [ORACLE-BRANCH01 — Choose branch probabilities, branching depths, action-sampling policy and computation budget for a matched collection comparison. Compare a schedule favouring early branches with a flat depth schedule under the same rendering budget. Early branching explores different prefixes; late branching explores alternatives after a shared prefix. Which allocation supplies more useful continuation evidence depends on the task and measured acquisition cost.], source: [Implemented branching and beam collector; ideal-oracle and sampled-return distinction.], gate: [Freeze the estimand, depth schedule, probabilities, selection and retention rules, and equal-cost budget before comparing collection policies; preserve shared-prefix dependence in analysis.], ) DDQN-style estimation addresses one source of maximization bias. It does not create missing successor transitions, make unsupported actions reliable, or repair state aliasing. CQL and BCQ motivate explicit offline-support diagnostics because a greedy learned policy can select actions whose multi-step consequences are weakly represented in the behavior data @CQL-kumar2020 @BCQ-fujimoto2019. Conservative regularization is introduced only if those diagnostics reveal systematic unsupported-action overestimation. #research_todo( [RRC10 — Decide whether to run or defer a bounded same-model one-step warm start, single-estimator continuation control, or Huber versus squared-loss comparison. Each asks a different optimization or mean-prediction question.], source: [Selected fitted-Q/Huber objective; reward and continuation research, 2 October 2026], gate: [Record a chosen comparison and its result, or explicit deferral; do not make these optional studies an architecture-wide obligation.], ) === Horizon-balanced replay and evaluation In a densely labelled corpus, one-step rows can vastly outnumber selected-action targets at longer horizons. An unweighted row mean can therefore favour the myopic task while hiding longer-horizon failure. The learning contract instead averages candidates within each state, states within each horizon, and then non-empty horizons uniformly. It binds this weighting to the maximum horizon, replay and candidate configurations, behavior-policy vocabulary, reward and return semantics, and discount. The evidence bundle records the realized state and candidate counts for every trained horizon, and online inference rejects horizons without positive training support. Alternative sampling or weighting schemes require a new versioned learning identity and remain in the development register. The primary diagnostic panel reports, separately for each horizon, admitted states and candidate labels, physical-scene and target coverage, state-averaged value loss and calibration error, and ranking and selected-action regret only where the required counterfactual comparison table exists. Counts accompany each diagnostic: no eligible comparison is unavailable evidence, not zero error. Dense one-step tables currently support ranking and regret; selected-transition recursion at longer horizons reports zero counterfactual comparison support. Candidate errors and pairwise accuracies are first reduced within factual state, and selected-action regret contributes one value per supported state. The evidence bundle retains behavior and family composition, bootstrap and boundary fractions, online/target disagreement, and selected-chain outcome fields for diagnosis. Marginal RRI, cumulative RRI and cumulative root gain remain distinct quantities. Held-out endpoint outcomes evaluate the resulting policy, rather than its fit to learning targets. A single validation loss can select a model only under a prespecified horizon aggregation. Cross-stage admission and checkpoint compatibility retain the contracts in @tab:thesis-learning-readiness; diagnostic fields and full identity inventories remain in their reproducibility owners. The optimality claim remains bounded under either interface. A learned finite-horizon value can approximate the best continuation only within the sampled finite candidate generator, hard-validity regime, represented actor state, and offline transition support. The same checkpoint cannot silently mix a pose-only state with privileged selected-depth, sensor-like, or actor-visible dynamic states. Longer horizons can expose information lost by a pose-only approximation: two histories ending at the same pose may leave different surfaces observed and therefore different useful continuations. Whether that distinction matters is an empirical property of the task and represented state. Selected-observation geometry can preserve it, but geometry updates alone do not establish Markov sufficiency, and longer horizons do not universally require that representation. The pose-only control remains a declared approximation whose limits must be checked against the same causal information boundary. #research_todo( [Specify the selected-observation fusion comparison in RQ2: which evidence is updated, which representation receives it, and which factor changes between matched controls. Do not infer this protocol from the longer-horizon information-loss risk or from an implemented privileged state control.], source: [RQ2 representation study; Method actor-state contract], gate: [Declare the compared information states and align their manifests and chapter descriptions.], ) The readiness gates in @tab:thesis-learning-readiness separate executable learning infrastructure from supported inference and policy evidence. A dynamic state claim additionally requires deterministic selected-observation fusion and no-future-observation tests. Those tests establish causality; held-out matched comparison is still needed to establish a useful state representation.