1 data_handling.vin_store.store

data_handling.vin_store.store

Low-level storage primitives for the VIN offline dataset format.

This module owns the immutable on-disk layout of the VIN offline dataset:

  • path and split configuration,
  • per-shard block materialization helpers,
  • manifest and sample-index loading, and
  • Zarr-backed random-access reads for fixed-size tensor blocks.
  • indexed per-row MessagePack reads for optional diagnostic payloads.

1.1 Attributes

Name Description
ACTOR_ONLY_DATASET_VERSION Explicit actor-root capability: actor inputs and label assets, no scene supervision.
OFFLINE_DATASET_VERSION Version of the unchanged scene-labelled VIN offline dataset format.

1.2 Classes

Name Description
VinOfflineStoreConfig Filesystem configuration for one immutable VIN offline dataset.
VinOfflineStoreReader Read immutable VIN offline datasets with Zarr-backed random access.

1.2.1 VinOfflineStoreConfig

data_handling.vin_store.store.VinOfflineStoreConfig()

Filesystem configuration for one immutable VIN offline dataset.

1.2.1.1 Attributes

Name Description
paths Project path resolver.
store_dir Root directory containing the immutable VIN offline dataset.
manifest_filename Filename of the top-level manifest.
sample_index_filename Filename of the global sample index.
shards_dirname Directory containing immutable shard subdirectories.
splits_dirname Directory containing split membership arrays.
manifest_path Return the resolved path to the store-owned manifest.json.
sample_index_path Return the resolved path to the global sample_index.jsonl.
shards_dir Return the absolute shard root directory.
splits_dir Return the absolute split-array directory.

1.2.1.2 Methods

Name Description
split_path Return the split-array path for one split.
write_split_indices Persist split membership arrays.
read_split_indices Load the global sample indices for one split.
1.2.1.2.1 split_path
data_handling.vin_store.store.VinOfflineStoreConfig.split_path(split)

Return the split-array path for one split.

1.2.1.2.1.1 Parameters
Name Type Description Default
split str Split name such as "all", "train", or "val". required
1.2.1.2.1.2 Returns
Name Type Description
Path Absolute split-array path.
1.2.1.2.2 write_split_indices
data_handling.vin_store.store.VinOfflineStoreConfig.write_split_indices(
    split_to_indices,
)

Persist split membership arrays.

1.2.1.2.2.1 Parameters
Name Type Description Default
split_to_indices dict[str, np.ndarray] Split membership arrays keyed by split name. required
1.2.1.2.3 read_split_indices
data_handling.vin_store.store.VinOfflineStoreConfig.read_split_indices(split)

Load the global sample indices for one split.

1.2.1.2.3.1 Parameters
Name Type Description Default
split str Split name such as "all", "train", or "val". required
1.2.1.2.3.2 Returns
Name Type Description
np.ndarray Global sample indices for the requested split.

1.2.2 VinOfflineStoreReader

data_handling.vin_store.store.VinOfflineStoreReader(config)

Read immutable VIN offline datasets with Zarr-backed random access.

1.2.2.1 Methods

Name Description
get_split_records Return index records for the requested split.
read_actor_snippet Decode the actor-visible VIN blocks for one immutable source row.
read_backbone_evidence Decode persisted actor-visible root EVL evidence for one source row.
read_numeric_block Read one numeric block row for a sample.
read_optional_record Read one optional per-row diagnostic record.
1.2.2.1.1 get_split_records
data_handling.vin_store.store.VinOfflineStoreReader.get_split_records(split)

Return index records for the requested split.

1.2.2.1.1.1 Parameters
Name Type Description Default
split Stage | None Lifecycle stage to select, or None for every stored row. required
1.2.2.1.1.2 Returns
Name Type Description
list[VinOfflineIndexRecord] Ordered index records for the split.
1.2.2.1.2 read_actor_snippet
data_handling.vin_store.store.VinOfflineStoreReader.read_actor_snippet(
    record,
    *,
    device='cpu',
)

Decode the actor-visible VIN blocks for one immutable source row.

Tensors never alias Zarr buffers; shard handles reopen after worker forks.

1.2.2.1.2.1 Parameters
Name Type Description Default
record VinOfflineIndexRecord Global sample-index record selecting the source row. required
device str | torch.device Device receiving the decoded tensors. 'cpu'
1.2.2.1.2.2 Returns
Name Type Description
VinSnippetView class:VinSnippetView with world points Tensor["P C", float32],
VinSnippetView valid point length Tensor["1", int64],
VinSnippetView and world-from-rig :class:PoseTW history Tensor["T 12", float32].
VinSnippetView The persisted world-from-snippet gauge is PoseTW Tensor["1 12", float32].
1.2.2.1.3 read_backbone_evidence
data_handling.vin_store.store.VinOfflineStoreReader.read_backbone_evidence(
    record,
    *,
    device='cpu',
)

Decode persisted actor-visible root EVL evidence for one source row.

Returns None when the immutable source store did not materialize a backbone. Required voxel-frame blocks fail with rebuild guidance rather than being replaced by zero tensors; optional head/evidence blocks are represented as None by :class:EvlBackboneOutput.

1.2.2.1.4 read_numeric_block
data_handling.vin_store.store.VinOfflineStoreReader.read_numeric_block(
    record,
    block_name,
)

Read one numeric block row for a sample.

1.2.2.1.4.1 Parameters
Name Type Description Default
record VinOfflineIndexRecord Global sample-index record. required
block_name str Logical block name. required
1.2.2.1.4.2 Returns
Name Type Description
np.ndarray NumPy array view for the requested sample row.
1.2.2.1.5 read_optional_record
data_handling.vin_store.store.VinOfflineStoreReader.read_optional_record(
    record,
    block_name,
)

Read one optional per-row diagnostic record.

1.2.2.1.5.1 Parameters
Name Type Description Default
record VinOfflineIndexRecord Global sample-index record. required
block_name str Logical block name. required
1.2.2.1.5.2 Returns
Name Type Description
Any | None Stored per-row Python object or None.

1.3 Functions

Name Description
discover_vin_store_dirs Discover immutable VIN stores below root from their two index files.
validate_vin_offline_manifest Validate root schema and capability without opening shard payloads.

1.3.1 discover_vin_store_dirs

data_handling.vin_store.store.discover_vin_store_dirs(root)

Discover immutable VIN stores below root from their two index files.

Discovery checks filenames only and does not validate or open shard arrays. Ordinary collection directories are traversed without following directory symlinks. Recognized VIN stores and Zarr payloads are traversal boundaries: their payloads are not collections of nested VIN stores. Returned absolute paths are sorted for deterministic manifest-hash matching.

1.3.2 validate_vin_offline_manifest

data_handling.vin_store.store.validate_vin_offline_manifest(manifest)

Validate root schema and capability without opening shard payloads.

Readers and metadata-only bundle inspection share this admission contract. V11 roots carry actor inputs and must exclude scene-supervision blocks; V10 roots retain the scene-labelled contract. This check does not establish payload integrity or objective-specific training readiness.

1.3.2.1 Parameters

Name Type Description Default
manifest VinOfflineManifest Decoded manifest whose schema, block declarations, and free-input provenance are checked without mutation. required

1.3.2.2 Raises

Name Type Description
ValueError The schema, capability, block declarations, or declared backbone free-input provenance violate the reader contract.
Back to top