1 datasets

datasets

Cross-store dataset composition; import each workflow from its owning leaf.

1.1 Package guide

aria_nbv.datasets composes immutable VIN roots, rollout supervision and artifact evidence. It sits above source/store readers and may invoke Lightning for explicit Q_H admission. It does not own frame math, rollout schemas, labels or training objectives. There are no package-root re-exports.

For available root/rollout stores, data locations, discovery, Streamlit inspection and immutable increments, start at the store overview.

1.2 Workflows and hidden coordination

bundles.inspection acquires a complete lightweight evidence product from metadata and index files, preserving source identity and readiness reasons.
bundles.readiness owns explicit corpus admission and batch preview. It constructs a Q_H DataModule only for that requested operation.
bundles.deep_scan owns opt-in array statistics and target inventory. It checks generation before and after acquisition and freezes the result.
bundles.export serializes already acquired products without reopening stores.
store_inventory refreshes cross-store metadata for the CLI and explicit app inventory action; generated snapshots remain derived and unverified.
scene_split freezes scene-wise development train/validation membership and routes existing immutable shards to the production per-stage dataset interface. Its assignment is external; physical source splits and campaign seals remain unchanged.
topology builds the cohesive graph of source, shard, modality and artifact relations. It reads metadata and file existence, not candidate arrays.

The dependency direction is readiness/deep_scan → inspection → store readers, with export consuming the completed products. Both dataset composition and VIN diagnostics use data_handling.store_identity for replacement-sensitive path and metadata identity. That lower owner imports only the standard library; it never imports this package, Oracle or Lightning. Identity DTO names and the versioned digest preimage are preserved to retain cache/evidence semantics. A forwarding bundles.generation module would add no behavior, so callers use the identity owner directly. Filename-only VIN discovery belongs to data_handling.vin_store.store.discover_vin_store_dirs, shared with topology.

1.3 Usage

from pathlib import Path
from aria_nbv.data_handling.store_identity import capture_dataset_bundle_generation
from aria_nbv.datasets.bundles.inspection import DatasetBundleSummaryRequest, inspect_dataset_bundle

selection, generation = capture_dataset_bundle_generation(Path("root-store"), ())
evidence = inspect_dataset_bundle(DatasetBundleSummaryRequest(selection, generation))

For a deep scan, pass the same selection and generation to bundles.deep_scan.DatasetBundleDeepRequest and inspect_dataset_bundle_deep. Light inspection does not imply Q_H readiness. Validation and complete array scans remain explicit, potentially expensive operations. No path repairs, migrates or writes the selected stores. Alias replacement, authoritative seals, row populations, hard invalidity masks and source hashes retain their meanings.

Python callers migrate from aria_nbv.dataset_bundle to the owning workflow leaf and from aria_nbv.dataset_topology to aria_nbv.datasets.topology. There are no old-root compatibility facades. Public docstrings own exact fields, shapes, units, failure behavior and immutable-result contracts. The active replay-store thesis section owns scientific interpretation; this relocation does not change its claims.

1.4 Verification

From the repository root:

cd aria_nbv
uv run pytest tests/test_dataset_bundle.py tests/test_dataset_topology.py \
  tests/app/panels/test_rollout_dataset_readiness.py \
  tests/app/panels/test_offline_dataset_cache_lifecycle.py \
  tests/app/panels/test_store_selection.py \
  tests/data_handling/test_vin_offline_store.py tests/data_handling/test_public_api_contract.py

These exercise alias/seal replacement, bounded metadata acquisition, opt-in arrays, generation guards, population/readiness parity, detached export and Streamlit cache lifetime. Test patches target the owner that executes the read.

1.5 Incremental development train/validation split

The development corpus uses an approximately 80/20 split by distinct scene, not sample, shard or candidate. Validation gets ceil(N / 5) scenes; all snippets, targets and rollout replicas of each scene stay together. This is a development partition, not an untouched confirmatory test set.

After publishing an increment, explicitly extend the existing plan:

uv run python -m aria_nbv.datasets.scene_split \
  --collection /absolute/path/to/new-collection \
  --output /absolute/offline-cache/rollout_supervision/development_scene_split.json

The command reads completed shard manifests, retains all earlier assignments, adds only new scenes and collection locators, and atomically replaces the plan outside immutable stores. Then refresh the store inventory. Repeated calls with the same collection preserve the same membership. Packed shards crossing the two stages are rejected rather than silently assigned to one stage.

Rollout Dataset loads this plan for its catalog and Q_H readiness requests. Training callers use the same SceneSplitPlan.store_paths_by_stage(stores) result to build QhDatasetConfig for each stage with split=None, and pass the datasets to QhDataModule(train=..., val=...). The DataModule independently checks scene disjointness and learning contracts. One dataset still binds one immutable actor root; different root fragments require separate compatible dataset compositions. A scene assignment does not upgrade legacy oracle label support to a dense-valid guarantee.

1.5.1 Shards and workers

Workers execute work units; they do not define permanent storage partitions. The recipe experiment collector uses one source sample per immutable shard so each attempt has its own timeout, rejection, retry and atomic publication. Increments add new shards and receipts; they never append into a sealed Zarr store. Generic production planning already supports --rows-per-shard on nbv-plan-rollout-shards; use that path when larger checkpoint units and their failure cost are appropriate, rather than assigning one shared store to each worker or repacking historical evidence.

# datasets { #aria_nbv.datasets } `datasets` Cross-store dataset composition; import each workflow from its owning leaf.  ## Package guide `aria_nbv.datasets` composes immutable VIN roots, rollout supervision and artifact evidence. It sits above source/store readers and may invoke Lightning for explicit Q_H admission. It does not own frame math, rollout schemas, labels or training objectives. There are no package-root re-exports. For available root/rollout stores, data locations, discovery, Streamlit inspection and immutable increments, start at the [store overview](https://github.com/JanDuchscherer104/ARIA-NBV/blob/main/aria_nbv/aria_nbv/datasets/STORE_OVERVIEW.md). ## Workflows and hidden coordination - `bundles.inspection` acquires a complete lightweight evidence product from metadata and index files, preserving source identity and readiness reasons. - `bundles.readiness` owns explicit corpus admission and batch preview. It constructs a Q_H DataModule only for that requested operation. - `bundles.deep_scan` owns opt-in array statistics and target inventory. It checks generation before and after acquisition and freezes the result. - `bundles.export` serializes already acquired products without reopening stores. - `store_inventory` refreshes cross-store metadata for the CLI and explicit app inventory action; generated snapshots remain derived and unverified. - `scene_split` freezes scene-wise development train/validation membership and routes existing immutable shards to the production per-stage dataset interface. Its assignment is external; physical source splits and campaign seals remain unchanged. - `topology` builds the cohesive graph of source, shard, modality and artifact relations. It reads metadata and file existence, not candidate arrays. The dependency direction is `readiness/deep_scan → inspection → store readers`, with export consuming the completed products. Both dataset composition and VIN diagnostics use `data_handling.store_identity` for replacement-sensitive path and metadata identity. That lower owner imports only the standard library; it never imports this package, Oracle or Lightning. Identity DTO names and the versioned digest preimage are preserved to retain cache/evidence semantics. A forwarding `bundles.generation` module would add no behavior, so callers use the identity owner directly. Filename-only VIN discovery belongs to `data_handling.vin_store.store.discover_vin_store_dirs`, shared with topology. ## Usage ```python from pathlib import Path from aria_nbv.data_handling.store_identity import capture_dataset_bundle_generation from aria_nbv.datasets.bundles.inspection import DatasetBundleSummaryRequest, inspect_dataset_bundle selection, generation = capture_dataset_bundle_generation(Path("root-store"), ()) evidence = inspect_dataset_bundle(DatasetBundleSummaryRequest(selection, generation)) ``` For a deep scan, pass the same selection and generation to `bundles.deep_scan.DatasetBundleDeepRequest` and `inspect_dataset_bundle_deep`. Light inspection does not imply Q_H readiness. Validation and complete array scans remain explicit, potentially expensive operations. No path repairs, migrates or writes the selected stores. Alias replacement, authoritative seals, row populations, hard invalidity masks and source hashes retain their meanings. Python callers migrate from `aria_nbv.dataset_bundle` to the owning workflow leaf and from `aria_nbv.dataset_topology` to `aria_nbv.datasets.topology`. There are no old-root compatibility facades. Public docstrings own exact fields, shapes, units, failure behavior and immutable-result contracts. The active [replay-store thesis section](../typst/thesis/sections/03-oracle-and-data-generation/03-03-replay-stores-and-diagnostics.typ) owns scientific interpretation; this relocation does not change its claims. ## Verification From the repository root: ```sh cd aria_nbv uv run pytest tests/test_dataset_bundle.py tests/test_dataset_topology.py \ tests/app/panels/test_rollout_dataset_readiness.py \ tests/app/panels/test_offline_dataset_cache_lifecycle.py \ tests/app/panels/test_store_selection.py \ tests/data_handling/test_vin_offline_store.py tests/data_handling/test_public_api_contract.py ``` These exercise alias/seal replacement, bounded metadata acquisition, opt-in arrays, generation guards, population/readiness parity, detached export and Streamlit cache lifetime. Test patches target the owner that executes the read. ## Incremental development train/validation split The development corpus uses an approximately 80/20 split by **distinct scene**, not sample, shard or candidate. Validation gets `ceil(N / 5)` scenes; all snippets, targets and rollout replicas of each scene stay together. This is a development partition, not an untouched confirmatory test set. After publishing an increment, explicitly extend the existing plan: ```sh uv run python -m aria_nbv.datasets.scene_split \ --collection /absolute/path/to/new-collection \ --output /absolute/offline-cache/rollout_supervision/development_scene_split.json ``` The command reads completed shard manifests, retains all earlier assignments, adds only new scenes and collection locators, and atomically replaces the plan outside immutable stores. Then refresh the store inventory. Repeated calls with the same collection preserve the same membership. Packed shards crossing the two stages are rejected rather than silently assigned to one stage. Rollout Dataset loads this plan for its catalog and Q_H readiness requests. Training callers use the same `SceneSplitPlan.store_paths_by_stage(stores)` result to build `QhDatasetConfig` for each stage with `split=None`, and pass the datasets to `QhDataModule(train=..., val=...)`. The DataModule independently checks scene disjointness and learning contracts. One dataset still binds one immutable actor root; different root fragments require separate compatible dataset compositions. A scene assignment does not upgrade legacy oracle label support to a dense-valid guarantee. ### Shards and workers Workers execute work units; they do not define permanent storage partitions. The recipe experiment collector uses one source sample per immutable shard so each attempt has its own timeout, rejection, retry and atomic publication. Increments add new shards and receipts; they never append into a sealed Zarr store. Generic production planning already supports `--rows-per-shard` on `nbv-plan-rollout-shards`; use that path when larger checkpoint units and their failure cost are appropriate, rather than assigning one shared store to each worker or repacking historical evidence.