1 data_handling.vin_store.format

data_handling.vin_store.format

Typed manifest and index records for the VIN offline dataset format.

The new offline dataset format is an immutable indexed-shard layout optimized for multi-worker random access. This module owns the normalized metadata records shared by the writer and runtime dataset reader:

  • the top-level dataset manifest,
  • per-shard block descriptors, and
  • sample-index records used for global random access and split membership.

1.1 Classes

Name Description
VinOfflineBlockSpec Descriptor for one stored block inside a shard.
VinOfflineIndexRecord Global sample-index entry for VIN offline random access.
VinOfflineManifest Top-level manifest for one immutable VIN offline dataset.
VinOfflineMaterializedBlocks Declare optional payload families present in a VIN offline store.
VinOfflineShardSpec Descriptor for one immutable dataset shard.

1.1.1 VinOfflineBlockSpec

data_handling.vin_store.format.VinOfflineBlockSpec(
    name,
    kind,
    paths,
    dtype=None,
    shape=None,
    optional=False,
)

Descriptor for one stored block inside a shard.

1.1.1.1 Attributes

Name Type Description
name str Logical block name, for example "vin.points_world".
kind str Storage kind such as "zarr_array" or "msgpack_indexed_records".
paths list[str] Relative array names or file paths that materialize the block.
dtype str | None NumPy dtype name for numeric blocks.
shape list[int] | None Full stored array shape for numeric blocks.
optional bool Whether the block may be absent for some datasets.

1.1.1.2 Methods

Name Description
zarr_array_path Return the hierarchical Zarr path for a logical block name.
msgpack_records_path Return the msgpack filename used for one logical record block.
msgpack_records_offsets_path Return the offsets filename used for indexed record blocks.
for_zarr_array Build a block descriptor for one Zarr-backed numeric array.
for_indexed_msgpack_records Build a block descriptor for indexed per-row MessagePack records.
1.1.1.2.1 zarr_array_path
data_handling.vin_store.format.VinOfflineBlockSpec.zarr_array_path(name)

Return the hierarchical Zarr path for a logical block name.

1.1.1.2.1.1 Parameters
Name Type Description Default
name str Logical block name, for example "oracle.p3d.R". required
1.1.1.2.1.2 Returns
Name Type Description
str Zarr array path relative to the shard root.
1.1.1.2.2 msgpack_records_path
data_handling.vin_store.format.VinOfflineBlockSpec.msgpack_records_path(name)

Return the msgpack filename used for one logical record block.

1.1.1.2.2.1 Parameters
Name Type Description Default
name str Logical block name. required
1.1.1.2.2.2 Returns
Name Type Description
str Shard-local msgpack filename.
1.1.1.2.3 msgpack_records_offsets_path
data_handling.vin_store.format.VinOfflineBlockSpec.msgpack_records_offsets_path(
    name,
)

Return the offsets filename used for indexed record blocks.

1.1.1.2.3.1 Parameters
Name Type Description Default
name str Logical block name. required
1.1.1.2.3.2 Returns
Name Type Description
str Shard-local NumPy offsets filename.
1.1.1.2.4 for_zarr_array
data_handling.vin_store.format.VinOfflineBlockSpec.for_zarr_array(
    name,
    array_path,
    dtype,
    shape,
    optional=False,
)

Build a block descriptor for one Zarr-backed numeric array.

1.1.1.2.4.1 Parameters
Name Type Description Default
name str Logical block name. required
array_path str Relative Zarr array path inside the shard. required
dtype str Stored NumPy dtype name. required
shape list[int] Full stored array shape. required
optional bool Whether the block is optional. False
1.1.1.2.4.2 Returns
Name Type Description
Self Block descriptor for the stored array.
1.1.1.2.5 for_indexed_msgpack_records
data_handling.vin_store.format.VinOfflineBlockSpec.for_indexed_msgpack_records(
    name,
    relative_payload_path,
    relative_offsets_path,
    num_records,
    optional=True,
)

Build a block descriptor for indexed per-row MessagePack records.

1.1.1.2.5.1 Parameters
Name Type Description Default
name str Logical block name. required
relative_payload_path str Shard-local concatenated payload blob path. required
relative_offsets_path str Shard-local NumPy offsets path. required
num_records int Number of stored per-row records. required
optional bool Whether the block is optional. True
1.1.1.2.5.2 Returns
Name Type Description
Self Block descriptor for the indexed record block.

1.1.2 VinOfflineIndexRecord

data_handling.vin_store.format.VinOfflineIndexRecord(
    sample_index,
    sample_key,
    scene_id,
    snippet_id,
    split,
    shard_id,
    row,
)

Global sample-index entry for VIN offline random access.

1.1.2.1 Attributes

Name Type Description
sample_index int Global zero-based sample index.
sample_key str Stable dataset sample key.
scene_id str ASE scene identifier.
snippet_id str ASE snippet identifier.
split str Canonical split membership: all, train, or val.
shard_id str Shard that stores the sample.
row int Zero-based row offset inside the shard.

1.1.2.2 Methods

Name Description
read_many Read the global sample index.
write_many Write the global sample index.
1.1.2.2.1 read_many
data_handling.vin_store.format.VinOfflineIndexRecord.read_many(path)

Read the global sample index.

1.1.2.2.1.1 Parameters
Name Type Description Default
path Path sample_index.jsonl path. required
1.1.2.2.1.2 Returns
Name Type Description
list[Self] Parsed sample-index records.
1.1.2.2.2 write_many
data_handling.vin_store.format.VinOfflineIndexRecord.write_many(path, records)

Write the global sample index.

1.1.2.2.2.1 Parameters
Name Type Description Default
path Path Destination sample_index.jsonl path. required
records list[Self] Global sample-index records to persist. required

1.1.3 VinOfflineManifest

data_handling.vin_store.format.VinOfflineManifest(
    version,
    created_at,
    source,
    oracle,
    vin,
    materialized_blocks,
    stats=dict(),
    provenance=dict(),
    shards=list(),
)

Top-level manifest for one immutable VIN offline dataset.

1.1.3.1 Attributes

Name Type Description
version int Dataset-format version.
created_at str UTC creation timestamp.
source dict[str, Any] Raw dataset provenance and configuration snapshot.
oracle dict[str, Any] Oracle-label pipeline provenance and storage policy.
vin dict[str, Any] VIN-specific materialization settings.
materialized_blocks VinOfflineMaterializedBlocks Flags for optional stored blocks.
stats dict[str, Any] Aggregate dataset statistics.
provenance dict[str, Any] Writer provenance and build hints.
shards list[VinOfflineShardSpec] Immutable shard descriptors.

1.1.3.2 Methods

Name Description
write Persist the manifest to disk.
read Load a manifest from disk.
1.1.3.2.1 write
data_handling.vin_store.format.VinOfflineManifest.write(path)

Persist the manifest to disk.

1.1.3.2.1.1 Parameters
Name Type Description Default
path Path Destination manifest path. required
1.1.3.2.2 read
data_handling.vin_store.format.VinOfflineManifest.read(path)

Load a manifest from disk.

1.1.3.2.2.1 Parameters
Name Type Description Default
path Path Manifest JSON path. required
1.1.3.2.2.2 Returns
Name Type Description
'VinOfflineManifest' Deserialized manifest.

1.1.4 VinOfflineMaterializedBlocks

data_handling.vin_store.format.VinOfflineMaterializedBlocks(
    backbone,
    depths,
    candidate_pcs,
    gt_obbs=False,
    detected_obbs=False,
    trajectory=False,
)

Declare optional payload families present in a VIN offline store.

The flags describe store-wide availability, while each shard’s VinOfflineBlockSpec entries remain the authoritative physical inventory. Backbone and detected-OBB blocks are actor-visible EVL evidence; GT OBBs and candidate depths remain oracle supervision or diagnostics.

1.1.4.1 Attributes

Name Description
backbone Whether checkpoint-dependent EVL voxel/evidence outputs are materialized.
depths Whether oracle-rendered candidate depth maps and validity masks are materialized.
candidate_pcs Whether oracle candidate point-cloud diagnostic records are materialized.
gt_obbs Whether ASE GT OBB label/evaluation blocks are materialized.
detected_obbs Whether actor-visible EVL detected-OBB blocks are materialized.
trajectory Whether MPS/EFM pose timestamps and world-frame gravity are materialized.

1.1.5 VinOfflineShardSpec

data_handling.vin_store.format.VinOfflineShardSpec(
    shard_id,
    relative_dir,
    row_start,
    num_rows,
    blocks=dict(),
)

Descriptor for one immutable dataset shard.

1.1.5.1 Attributes

Name Type Description
shard_id str Stable shard identifier, for example "shard-000003".
relative_dir str Relative directory containing the shard files.
row_start int Global row offset covered by the shard.
num_rows int Number of samples stored in the shard.
blocks dict[str, VinOfflineBlockSpec] Stored block descriptors keyed by logical block name.
Back to top