1 data_handling

data_handling

Expose typed ASE/EFM views and immutable VIN offline-store entrypoints.

The package root exports the stable cross-module contracts for raw ASE/EFM access: :class:AseEfmDataset, its config, and the zero-copy typed :class:EfmSnippetView and :class:VinSnippetView views. It also exports the VinOracleBatch payload and read-side configuration for immutable VinOfflineDataset stores.

Specialized codecs, diagnostics, writers, adapters, and storage internals stay owned by their leaf modules. aria_nbv.data_handling.qh_data owns the validated rollout-to-actor join and keeps scorer-visible tensors separate from supervision, but remains a leaf import rather than widening this one-step root. Oracle sample generation and target selection are pipeline concerns owned outside aria_nbv.data_handling; this package neither constructs oracle labels nor chooses target entities.

1.1 Package guide

aria_nbv.data_handling is the typed boundary between upstream ASE/ATEK-EFM assets and ARIA-NBV training or evaluation objects. It owns raw snippet views, immutable VIN stores, and the finite-horizon join between rollout chains and their exact actor-visible source rows.

The central invariant is actor/oracle separation:

Actor-visible state includes observed snippets, calibration and trajectory, semidense/EVL evidence, detected targets, finite candidate poses, causal history, budget, and masks.
Privileged supervision includes GT scene/target geometry, oracle candidate renders, target-RRI labels, and evaluation crops.
Invalid candidates are masks plus reason codes, never low-RRI examples.

1.2 Public Entry Points

The package root exports the stable raw-view and immutable-store contracts: AseEfmDataset, AseEfmDatasetConfig, EfmSnippetView, VinSnippetView, VinOracleBatch, VinOfflineDataset, VinOfflineDatasetConfig, and VinOfflineStoreConfig.

Specialized writer, reader, diagnostic, and QH contracts stay in their leaf modules:

Module	Responsibility
`ase_efm`	Stream typed EFM snippets and normalize their semantic-name metadata.
`vin_store`	Immutable indexed one-step evidence stores.
`qh_data.views`	Actor tensors, supervision tensors, chain and batch DTOs.
`qh_data.dataset`	Validate rollout-to-actor lineage and materialize chains.
`qh_data.batching`	Pad complete chains without conflating mask meanings.

1.3 Two-Store Architecture

The physical stores remain separate:

vin_offline stores expensive immutable one-step substrate such as actor blocks, candidate evidence, and optional oracle labels.
rollouts.zarr stores factual multi-step replay, selected observations, derived QH rows, and stable references back to VIN source rows.

The current rollout compatibility identifier is 2.0-target-rollout-provenance; ROLLOUT_ZARR_SCHEMA_VERSION in aria_nbv.rollouts.storage remains the executable owner and the focused QH test keeps this operator-facing value synchronized.

Raw streams and full meshes remain in their upstream-managed locations. Joined readers avoid copying those assets or cached backbone blocks into every rollout row.

1.4 Build and Inspect Data

Run commands from aria_nbv/.

Download a bounded ASE/ATEK-EFM subset:

uv run nbv-downloader -m list -c efm -n 10
uv run nbv-downloader -m download -c efm --ns 1 --max-shards 1

Validate a VIN build before writing, then build and inspect it:

uv run nbv-build-offline \
  --config-path ../.configs/build_vin_offline_81286.toml \
  --dry-run
uv run nbv-build-offline \
  --config-path ../.configs/build_vin_offline_81286.toml
uv run nbv-offline-info summary --store vin_offline
uv run nbv-offline-info tree --store vin_offline

Generate and validate a local rollout smoke store:

uv run nbv-build-rollouts \
  --config-path ../.configs/smoke/build_rollouts_v1_smoke.toml \
  --dry-run
uv run nbv-build-rollouts \
  --config-path ../.configs/smoke/build_rollouts_v1_smoke.toml
uv run nbv-rollouts-info \
  --store ../.data/offline_cache/rollouts_v1_smoke.zarr \
  --validate

Completed stores are immutable inputs. Rebuild incompatible stores with the current writer; never patch manifests, indices, split arrays, or Zarr payloads to satisfy validation.

1.5 Materialize a QH Dataset

QhDatasetConfig joins one or more ordered rollout stores to one immutable VIN actor store through actor, or multiple compatible immutable roots through actors. Both inputs normalize to the same qualified source path, which preflights every source reference before exposing a chain:

from pathlib import Path

from aria_nbv.data_handling.qh_data import QhDatasetConfig
from aria_nbv.data_handling.vin_store.store import VinOfflineStoreConfig
from aria_nbv.utils import Stage

dataset = QhDatasetConfig(
    rollout_store_dirs=(Path("rollouts-train.zarr"),),
    actor=VinOfflineStoreConfig(store_dir=Path("vin_offline")),
    split=Stage.TRAIN,
    root_evl_profile="evl_v1",
    selected_observation_protocol="none",
    experiment_profile="qh_cf0_v1",
).setup_target()

chain = dataset[0]
actor = chain.actor
supervision = chain.supervision

For a composed population, replace actor=... with actors=(VinOfflineStoreConfig(store_dir=Path("root-a")), VinOfflineStoreConfig(store_dir=Path("root-b"))); the two options are exclusive. Each source sample is qualified by its persisted root manifest hash and local index. The exact shard, row, scene and split must also match. Roots must share the supported storage capability, producer configuration, tensor shapes, voxel extents and named experiment profile; differing voxel geometry is rejected.

The existing DataLoader samples uniformly by rollout chain and uses the existing collation and worker-local storage handles. Storage shards are physical files, not permanent worker assignments. Stage population provenance remains exact, even when actor representations are compatible across stages.

QhActorTensors contains only scorer-legal evidence. QhSupervision contains labels, selected-transition facts, and masks used by the objective. Audit metadata remains CPU-only and outside device transfer.

1.6 Inspection

Use the lightweight CLIs for manifest and table checks, Streamlit for operational browsing, and Rerun for spatial evidence:

uv run nbv-st
uv run nbv-rerun-inspect \
  --config-path ../.configs/rerun_offline.toml \
  --rollout-store ../.data/offline_cache/rollouts_v1_smoke.zarr \
  --rollout-index 0 \
  --rollout-context required \
  --spawn

1.7 Troubleshooting

Store version mismatch: rebuild with the current writer; readers fail closed and intentionally provide no migration facade.
Missing manifest or lineage: treat the artifact as partial or stale and regenerate it from the bound source store.
Existing destination: choose a new immutable output path. Use overwrite only for a deliberately disposable smoke artifact.
QH source mismatch: rebuild the actor and rollout corpus from the same source manifest and split identity.
Selected-depth mismatch: use a compatible experiment profile; CF0 and the privileged CF+ selected-depth profile are separate contracts.

1.8 Detailed Contracts and Verification

Field shapes, frames, masks, and lifecycle rules live in public docstrings and the generated API reference. Rollout persistence is documented by the rollout guide.

cd aria_nbv
uv run ruff check aria_nbv/data_handling
uv run pytest tests/data_handling/test_vin_offline_store.py
uv run pytest tests/data_handling/test_public_api_contract.py
uv run pytest tests/data_handling/test_qh.py

1.9 Store identity

store_identity captures selected paths, canonical targets, alias/stat metadata and bounded authoritative version/seal fields. It supplies the same identity and pre/post-acquisition guard to VIN diagnostics and higher dataset composition without importing readers, array payloads, Oracle or Lightning. Its established DatasetBundle* identity records retain their versioned digest and normalized selection behavior. Import capture_dataset_bundle_generation and assert_dataset_bundle_generation_current from this leaf; detailed path and error contracts stay in its docstrings. tests/test_dataset_bundle.py covers replacement, symlinks, invalid/oversized authority files and bounded reads.

# data_handling { #aria_nbv.data_handling } `data_handling` Expose typed ASE/EFM views and immutable VIN offline-store entrypoints. The package root exports the stable cross-module contracts for raw ASE/EFM access: :class:`AseEfmDataset`, its config, and the zero-copy typed :class:`EfmSnippetView` and :class:`VinSnippetView` views. It also exports the :class:`VinOracleBatch` payload and read-side configuration for immutable :class:`VinOfflineDataset` stores. Specialized codecs, diagnostics, writers, adapters, and storage internals stay owned by their leaf modules. :mod:`aria_nbv.data_handling.qh_data` owns the validated rollout-to-actor join and keeps scorer-visible tensors separate from supervision, but remains a leaf import rather than widening this one-step root. Oracle sample generation and target selection are pipeline concerns owned outside :mod:`aria_nbv.data_handling`; this package neither constructs oracle labels nor chooses target entities.  ## Package guide `aria_nbv.data_handling` is the typed boundary between upstream ASE/ATEK-EFM assets and ARIA-NBV training or evaluation objects. It owns raw snippet views, immutable VIN stores, and the finite-horizon join between rollout chains and their exact actor-visible source rows. The central invariant is actor/oracle separation: - Actor-visible state includes observed snippets, calibration and trajectory, semidense/EVL evidence, detected targets, finite candidate poses, causal history, budget, and masks. - Privileged supervision includes GT scene/target geometry, oracle candidate renders, target-RRI labels, and evaluation crops. - Invalid candidates are masks plus reason codes, never low-RRI examples. ## Public Entry Points The package root exports the stable raw-view and immutable-store contracts: `AseEfmDataset`, `AseEfmDatasetConfig`, `EfmSnippetView`, `VinSnippetView`, `VinOracleBatch`, `VinOfflineDataset`, `VinOfflineDatasetConfig`, and `VinOfflineStoreConfig`. Specialized writer, reader, diagnostic, and QH contracts stay in their leaf modules: | Module | Responsibility | | --- | --- | | `ase_efm` | Stream typed EFM snippets and normalize their semantic-name metadata. | | `vin_store` | Immutable indexed one-step evidence stores. | | `qh_data.views` | Actor tensors, supervision tensors, chain and batch DTOs. | | `qh_data.dataset` | Validate rollout-to-actor lineage and materialize chains. | | `qh_data.batching` | Pad complete chains without conflating mask meanings. | ## Two-Store Architecture ![Data-store architecture](../figures/diagrams/data_handling/mermaid/data_store_architecture.svg) The physical stores remain separate: - `vin_offline` stores expensive immutable one-step substrate such as actor blocks, candidate evidence, and optional oracle labels. - `rollouts.zarr` stores factual multi-step replay, selected observations, derived QH rows, and stable references back to VIN source rows. The current rollout compatibility identifier is `2.0-target-rollout-provenance`; `ROLLOUT_ZARR_SCHEMA_VERSION` in `aria_nbv.rollouts.storage` remains the executable owner and the focused QH test keeps this operator-facing value synchronized. Raw streams and full meshes remain in their upstream-managed locations. Joined readers avoid copying those assets or cached backbone blocks into every rollout row. ## Build and Inspect Data Run commands from `aria_nbv/`. Download a bounded ASE/ATEK-EFM subset: ```sh uv run nbv-downloader -m list -c efm -n 10 uv run nbv-downloader -m download -c efm --ns 1 --max-shards 1 ``` Validate a VIN build before writing, then build and inspect it: ```sh uv run nbv-build-offline \ --config-path ../.configs/build_vin_offline_81286.toml \ --dry-run uv run nbv-build-offline \ --config-path ../.configs/build_vin_offline_81286.toml uv run nbv-offline-info summary --store vin_offline uv run nbv-offline-info tree --store vin_offline ``` Generate and validate a local rollout smoke store: ```sh uv run nbv-build-rollouts \ --config-path ../.configs/smoke/build_rollouts_v1_smoke.toml \ --dry-run uv run nbv-build-rollouts \ --config-path ../.configs/smoke/build_rollouts_v1_smoke.toml uv run nbv-rollouts-info \ --store ../.data/offline_cache/rollouts_v1_smoke.zarr \ --validate ``` Completed stores are immutable inputs. Rebuild incompatible stores with the current writer; never patch manifests, indices, split arrays, or Zarr payloads to satisfy validation. ## Materialize a QH Dataset `QhDatasetConfig` joins one or more ordered rollout stores to one immutable VIN actor store through `actor`, or multiple compatible immutable roots through `actors`. Both inputs normalize to the same qualified source path, which preflights every source reference before exposing a chain: ```python from pathlib import Path from aria_nbv.data_handling.qh_data import QhDatasetConfig from aria_nbv.data_handling.vin_store.store import VinOfflineStoreConfig from aria_nbv.utils import Stage dataset = QhDatasetConfig( rollout_store_dirs=(Path("rollouts-train.zarr"),), actor=VinOfflineStoreConfig(store_dir=Path("vin_offline")), split=Stage.TRAIN, root_evl_profile="evl_v1", selected_observation_protocol="none", experiment_profile="qh_cf0_v1", ).setup_target() chain = dataset[0] actor = chain.actor supervision = chain.supervision ``` For a composed population, replace `actor=...` with `actors=(VinOfflineStoreConfig(store_dir=Path("root-a")), VinOfflineStoreConfig(store_dir=Path("root-b")))`; the two options are exclusive. Each source sample is qualified by its persisted root manifest hash and local index. The exact shard, row, scene and split must also match. Roots must share the supported storage capability, producer configuration, tensor shapes, voxel extents and named experiment profile; differing voxel geometry is rejected. The existing DataLoader samples uniformly by rollout chain and uses the existing collation and worker-local storage handles. Storage shards are physical files, not permanent worker assignments. Stage population provenance remains exact, even when actor representations are compatible across stages. `QhActorTensors` contains only scorer-legal evidence. `QhSupervision` contains labels, selected-transition facts, and masks used by the objective. Audit metadata remains CPU-only and outside device transfer. ## Inspection Use the lightweight CLIs for manifest and table checks, Streamlit for operational browsing, and Rerun for spatial evidence: ```sh uv run nbv-st uv run nbv-rerun-inspect \ --config-path ../.configs/rerun_offline.toml \ --rollout-store ../.data/offline_cache/rollouts_v1_smoke.zarr \ --rollout-index 0 \ --rollout-context required \ --spawn ``` ## Troubleshooting - **Store version mismatch:** rebuild with the current writer; readers fail closed and intentionally provide no migration facade. - **Missing manifest or lineage:** treat the artifact as partial or stale and regenerate it from the bound source store. - **Existing destination:** choose a new immutable output path. Use overwrite only for a deliberately disposable smoke artifact. - **QH source mismatch:** rebuild the actor and rollout corpus from the same source manifest and split identity. - **Selected-depth mismatch:** use a compatible experiment profile; CF0 and the privileged CF+ selected-depth profile are separate contracts. ## Detailed Contracts and Verification Field shapes, frames, masks, and lifecycle rules live in public docstrings and the [generated API reference](index.qmd). Rollout persistence is documented by the [rollout guide](rollouts.qmd). ```sh cd aria_nbv uv run ruff check aria_nbv/data_handling uv run pytest tests/data_handling/test_vin_offline_store.py uv run pytest tests/data_handling/test_public_api_contract.py uv run pytest tests/data_handling/test_qh.py ``` ## Store identity `store_identity` captures selected paths, canonical targets, alias/stat metadata and bounded authoritative version/seal fields. It supplies the same identity and pre/post-acquisition guard to VIN diagnostics and higher dataset composition without importing readers, array payloads, Oracle or Lightning. Its established `DatasetBundle*` identity records retain their versioned digest and normalized selection behavior. Import `capture_dataset_bundle_generation` and `assert_dataset_bundle_generation_current` from this leaf; detailed path and error contracts stay in its docstrings. `tests/test_dataset_bundle.py` covers replacement, symlinks, invalid/oversized authority files and bounded reads.