1 rollouts.storage
rollouts.storage
Persisted rollout contracts, Zarr I/O, validation, and typed readers.
The package keeps encoding and validation details private while exposing the small store interfaces used by generation, training, and inspection callers.
1.1 Package guide
aria_nbv.rollouts.storage owns the persisted replay interface: schema tables, manifest bindings, Zarr writes, read-only access, validation, finite-horizon chain decoding, and presentation-free row projections. Replay policy and state transitions remain in rollouts.replay; inspection, plotting, and training objectives consume this package but do not define storage semantics.
1.2 Supported use
from aria_nbv.rollouts.storage import RolloutZarrStoreReader
reader = RolloutZarrStoreReader("rollouts.zarr")
validation = reader.validate()
if not validation.ok:
raise ValueError(validation.errors)
qh_arrays = reader.q_h_view()The package interface hides table encoding and validation order. reader.py owns strict read-only access, writer.py owns normalized materialization, validation.py owns corruption detection, and _tables.py owns the shared schema and encoding implementation. Manifests, trace lineage, QH chain reads, and the typed read model remain distinct contracts because they change and are verified independently.
generation ──> writer ──> schema/tables <── validation <── reader
│ │
manifests QH/read model1.3 Scientific and persistence contracts
Factual sources/, targets/, rollouts/, steps/, and candidates/ tables are authoritative. The padded q_h/ arrays are a validated training projection. Candidate order, row ids, action and label masks, selected-depth alignment, reason-code bits, lineage hashes, and manifest digests are versioned persisted facts. Invalid actions remain masked rows; they are never encoded as low-RRI or zero-Q labels. Camera poses use the recorded world/camera conventions, depth is stored in metres, and readers never repair or mutate a store.
Directed-orbit transition evidence is an additive version-1 step bundle: direction uses -1 for unavailable and 0/1 for clockwise/counterclockwise; progress, selected increment, and remaining angle use NaN when unavailable. All four fields share availability, and rollout termination preserves the specific replay-owned reason rather than collapsing every early stop.
Optional target-selection audit measurements preserve availability separately from value: unavailable floating-point fields are stored as NaN, unavailable integer counts as -1, and measured zero remains 0. Typed projections decode the unavailable sentinels as None. Legacy stores may omit these optional arrays, which has the same unavailable meaning.
The established aria_nbv.rollouts.zarr_store, trace, read_model, qh_reader, and qh_geometry leaves remain targeted transition imports. New code should use aria_nbv.rollouts.storage and its named leaves. Shard manifest imports use aria_nbv.rollouts.storage.shard_manifest directly.
1.4 Verification
From aria_nbv/ run:
uv run ruff check aria_nbv/rollouts/storage
uv run pytest tests/rollouts/test_zarr_store.py \
tests/rollouts/test_dataset_writer.py \
tests/rollouts/test_read_model.py \
tests/rollouts/test_qh_reader.pyAdd the Oracle, Lightning, app, or Rerun consumer tests when their import seam changes. The active thesis discussion of replay lineage and decision-state relations is in docs/typst/thesis/sections/03-oracle-and-data-generation/03-03-replay-stores-and-diagnostics.typ.
1.5 Attributes
| Name | Description |
|---|---|
| DEFAULT_RETURN_SEMANTICS | Default return target family for root-normalized Q_H replay views. |
| Q_H_ARRAY_NAMES | Arrays persisted in the derived finite-candidate q_h/ training view. |
| Q_H_REWARD_METRIC | Candidate field used as the default Q_H training reward. |
| Q_H_TD_SEMANTICS | Q_H TD contract persisted on the q_h/ group. |
| ROLLOUT_ZARR_SCHEMA_ID | Schema id stored as a root attribute on rollout replay stores. |
| ROLLOUT_ZARR_SCHEMA_VERSION | Target-first rollout schema with hot candidate provenance and core parent-depth history. |
1.6 Classes
| Name | Description |
|---|---|
| RolloutZarrStoreConfig | Filesystem, replay, and oracle-retention settings for one rollout store. |
| RolloutZarrStoreReader | Open a completed standalone rollout replay store in read-only mode. |
| RolloutZarrValidationResult | Validation summary for a rollout Zarr store. |
| RolloutZarrWriteResult | Summary of one rollout Zarr write. |
1.6.1 RolloutZarrStoreConfig
rollouts.storage.RolloutZarrStoreConfig()Filesystem, replay, and oracle-retention settings for one rollout store.
The destination is independent of the immutable VIN offline cache. Direct writes own and replace store_dir; validated completed stores should be treated as immutable inputs. Cluster jobs obtain crash-safe lifecycle semantics through temporary shard writes and atomic final promotion.
1.6.1.1 Attributes
| Name | Description |
|---|---|
| paths | Repository path defaults used to resolve cache artifact destinations. |
| store_dir | Standalone destination replaced by a direct write; never the VIN source store. |
| return_semantics | Versioned return family recorded for the derived finite-candidate view. |
| discount_gamma | Discount applied only to non-terminal selected TD transitions. |
| target_protocol_version | Contract version separating target task inputs from oracle labels. |
| reason_code_version | Version of persisted candidate-invalidity bit positions. |
| field_retention_policy | Named policy describing which heavy oracle/audit payloads are retained. |
| source_offline_store_version | Immutable VIN source-cache version copied into rollout provenance. |
| split_manifest_hash | Hash binding the source split and ordered row ownership records. |
| q_h_chunk_states | Number of state rows per chunk in the persisted derived q_h/ view. |
| target_eval_crop_max_points | Fixed row width for oracle/eval target crop point payloads. |
| target_eval_crops_enabled | Persist oracle/eval target crop point payloads for sampled audit shards. |
| oracle_query_mode | Persisted hard-oracle candidate-query profile for Q_H supervision. |
| label_support_semantics | Persisted relation between actor-valid rows and finite Q labels. |
1.6.2 RolloutZarrStoreReader
rollouts.storage.RolloutZarrStoreReader(store_dir)Open a completed standalone rollout replay store in read-only mode.
The reader never repairs, migrates, or backfills arrays. Canonical factual tables remain the source of truth; q_h/ is a validated derived cache.
1.6.2.1 Methods
| Name | Description |
|---|---|
| candidate_shell_index | Return the immutable shell-ordered candidate-row lookup for this reader. |
| array | Read an array, returning -1 for supported unavailable provenance. |
| array_rows | Read only selected first-axis rows from one canonical Zarr array. |
| validate | Validate persisted contracts, optionally skipping unconsumed depth payloads. |
| manifest | Read root attrs and the top-level sidecar manifest without payload arrays. |
| acquisition_row_context | Resolve a chain or step to its authenticated, original generation context. |
| q_h_view | Return the padded finite-candidate Q_H training view. |
1.6.2.1.1 candidate_shell_index
rollouts.storage.RolloutZarrStoreReader.candidate_shell_index()Return the immutable shell-ordered candidate-row lookup for this reader.
The index is reader-local and derived only from canonical candidate columns. Reusing it prevents repeated whole-table scans in consumers that project several rollout steps from the same completed store.
1.6.2.1.2 array
rollouts.storage.RolloutZarrStoreReader.array(path)Read an array, returning -1 for supported unavailable provenance.
1.6.2.1.3 array_rows
rollouts.storage.RolloutZarrStoreReader.array_rows(path, positions)Read only selected first-axis rows from one canonical Zarr array.
1.6.2.1.4 validate
rollouts.storage.RolloutZarrStoreReader.validate(
validate_selected_depth_payload=True,
)Validate persisted contracts, optionally skipping unconsumed depth payloads.
1.6.2.1.5 manifest
rollouts.storage.RolloutZarrStoreReader.manifest()Read root attrs and the top-level sidecar manifest without payload arrays.
1.6.2.1.6 acquisition_row_context
rollouts.storage.RolloutZarrStoreReader.acquisition_row_context(table, row_id)Resolve a chain or step to its authenticated, original generation context.
Physical packing renumbers rows but archives the source manifest and a hashed row mapping. Follow those bindings through repeated packing; return the acquisition manifest digest, original row ID and generation context. Missing, ambiguous or altered bindings fail rather than borrowing another chain’s configuration. Metadata is cached for this immutable reader.
1.6.2.1.7 q_h_view
rollouts.storage.RolloutZarrStoreReader.q_h_view(discount_gamma=None)Return the padded finite-candidate Q_H training view.
Let S be the number of materialized step/state rows and A the maximum step-local full-shell width. Candidate ids, masks, positions, one-step labels, and reason bitsets are ndarray["S A", ...]; source/target ids, selected-action indices, and TD linkage are ndarray["S", ...]. Shorter shells are padded with id -1, false masks, and NaN labels.
valid_action_mask is the hard actor-selectable mask. q_train_mask is stricter: it also requires a label-valid target/GT state and a finite explicit target-root-gain label. Finite target RRI remains an independent diagnostic-availability predicate. Invalidity is represented by masks and versioned reason bitsets, never by a low score. TD reward/linkage describes the factual selected action only; unselected action labels remain one-step oracle supervision.
1.6.2.1.7.1 Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| discount_gamma | float | None | When None, read the persisted q_h/ cache. Otherwise rebuild the view from canonical factual tables using this discount for non-terminal selected transitions. |
None |
1.6.2.1.7.2 Returns
| Name | Type | Description |
|---|---|---|
dict[str, np.ndarray] |
Mapping containing all Q_H_ARRAY_NAMES, preserving the |
|
dict[str, np.ndarray] |
S state axis and padded A full-shell action axis. |
1.6.3 RolloutZarrValidationResult
rollouts.storage.RolloutZarrValidationResult(
store_dir,
num_rollouts,
num_steps,
num_candidates,
errors=list(),
)Validation summary for a rollout Zarr store.
errors contains schema, linkage, mask, and lineage violations. Validation fails if candidate strategy ids, mixture ids, target protocol metadata, source hashes, or explicit target-root-gain rewards are missing.
1.6.3.1 Attributes
| Name | Description |
|---|---|
| store_dir | Resolved rollout-store directory checked in read-only mode. |
| num_rollouts | Validated number of normalized rollout rows. |
| num_steps | Validated number of normalized step/state rows. |
| num_candidates | Validated number of full-shell candidate rows. |
| errors | All discovered schema, linkage, mask, label, and provenance violations. |
| ok | Return True when no validation errors were found. |
1.6.4 RolloutZarrWriteResult
rollouts.storage.RolloutZarrWriteResult(
store_dir,
num_rollouts,
num_steps,
num_candidates,
manifest_path,
manifest_sha256,
)Summary of one rollout Zarr write.
Counts refer to materialized row tables, not source VIN samples. One source sample can contribute multiple targets, rollout recipes, beam chains, steps, and full-shell candidate rows.
1.6.4.1 Attributes
| Name | Description |
|---|---|
| store_dir | Resolved standalone rollout-store directory written by the session. |
| num_rollouts | Number of retained trajectory-chain rows in rollouts/. |
| num_steps | Number of materialized state/transition rows in steps/. |
| num_candidates | Number of full-shell action rows in candidates/. |
| manifest_path | Human-readable provenance sidecar beside the Zarr payload. |
| manifest_sha256 | SHA-256 binding canonical manifest JSON to the store root attrs. |
1.7 Functions
| Name | Description |
|---|---|
| validate_rollout_zarr_store | Validate a rollout store, optionally omitting unconsumed selected-depth rows. |
| write_rollout_zarr_store | Replace a standalone destination with normalized rollout replay tables. |
1.7.1 validate_rollout_zarr_store
rollouts.storage.validate_rollout_zarr_store(
store_dir,
*,
validate_selected_depth_payload=True,
)Validate a rollout store, optionally omitting unconsumed selected-depth rows.
1.7.2 write_rollout_zarr_store
rollouts.storage.write_rollout_zarr_store(
store_dir,
records,
*,
return_semantics=DEFAULT_RETURN_SEMANTICS,
discount_gamma=1.0,
target_protocol_version=TargetInputProtocol.V0_GT_INPUT,
reason_code_version=INVALID_REASON_VERSION,
field_retention_policy='compact',
source_offline_store_version='unknown-source-version',
split_manifest_hash='unknown-split-manifest',
manifest_context=None,
selected_depth_enabled=True,
selected_depth_width_px=240,
selected_depth_height_px=240,
selected_depth_chunk_steps=16,
selected_depth_renderer='Pytorch3DDepthRenderer',
selected_depth_znear_m=0.001,
selected_depth_zfar_m=20.0,
selected_depth_source_resolution='exact_output_size',
q_h_chunk_states=64,
target_eval_crop_max_points=DEFAULT_TARGET_EVAL_CROP_MAX_POINTS,
target_eval_crops_enabled=False,
oracle_query_mode='legacy_unspecified',
label_support_semantics='subset_of_action_v1',
)Replace a standalone destination with normalized rollout replay tables.
Candidate shells remain complete in the factual tables; valid-only oracle vectors are expanded through masks, and q_h/ is derived from the normalized step/candidate rows. Selected mesh depth and optional target evaluation crops remain rollout-owned oracle/audit artifacts.
1.7.2.1 Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| store_dir | Path | str | Destination directory opened in Zarr write mode. A direct call replaces existing content at this path. | required |
| records | Sequence[_RolloutWriteRecord] |
Root-target-recipe rollout results with source/target lineage. | required |
| return_semantics | str | Versioned return family recorded on the derived view. | DEFAULT_RETURN_SEMANTICS |
| discount_gamma | float | Discount for non-terminal selected TD transitions. | 1.0 |
| target_protocol_version | TargetInputProtocol | str | Target input/label boundary contract. | TargetInputProtocol.V0_GT_INPUT |
| reason_code_version | str | Candidate-invalidity code-table version. | INVALID_REASON_VERSION |
| field_retention_policy | str | Named heavy-field retention policy. | 'compact' |
| source_offline_store_version | str | Immutable VIN source-cache version. | 'unknown-source-version' |
| split_manifest_hash | str | Hash of split and ordered source-row ownership. | 'unknown-split-manifest' |
| manifest_context | RolloutStoreManifestContext | None | Optional resolved config, invocation, runtime, and shard provenance for the sidecar. | None |
| selected_depth_enabled | bool | Whether selected-only oracle depth is retained. | True |
| selected_depth_width_px | int | Selected-depth raster width in pixels. | 240 |
| selected_depth_height_px | int | Selected-depth raster height in pixels. | 240 |
| selected_depth_chunk_steps | int | Selected-depth rows per Zarr chunk. | 16 |
| selected_depth_renderer | str | Renderer identity stored as provenance. | 'Pytorch3DDepthRenderer' |
| selected_depth_znear_m | float | None | Near clipping distance in metres. | 0.001 |
| selected_depth_zfar_m | float | None | Far clipping distance in metres. | 20.0 |
| selected_depth_source_resolution | str | Resolution-policy provenance label. | 'exact_output_size' |
| q_h_chunk_states | int | State rows per derived-view chunk. | 64 |
| target_eval_crop_max_points | int | Fixed point width P_max for audit crops. |
DEFAULT_TARGET_EVAL_CROP_MAX_POINTS |
| target_eval_crops_enabled | bool | Whether optional oracle target crops persist. | False |
| oracle_query_mode | str | Closed oracle-query support identity for Q_H labels. | 'legacy_unspecified' |
| label_support_semantics | str | Closed relationship between labels and actor-valid rows. | 'subset_of_action_v1' |
1.7.2.2 Returns
| Name | Type | Description |
|---|---|---|
| RolloutZarrWriteResult | Row counts, resolved destination, and sidecar manifest digest. |
1.7.2.3 Notes
This low-level direct writer is not a temp-to-final transaction. Use the rollout shard runner for validated atomic promotion. In either path the VIN source cache is opened only as an input and is never modified.