1 data_handling.vin_store.writer

data_handling.vin_store.writer

Prepared-row and shard codecs for immutable VIN offline datasets.

This module owns the normalized rows and immutable shard writes used by the VIN offline dataset. It provides:

  • PreparedVinOfflineSample as the normalized in-memory row representation,
  • helpers for turning oracle-label outputs into fixed numeric blocks plus optional lazy diagnostic record blocks, and
  • shard flushing helpers reused by tests and alternate builders.

The writer stores training-critical tensors as fixed-size NumPy arrays for Zarr-backed random access. Rich per-row msgspec records are opt-in diagnostics because the numeric blocks are the canonical offline training contract.

1.1 Classes

Name Description
PreparedVinOfflineSample Normalized offline row before shard materialization.

1.1.1 PreparedVinOfflineSample

data_handling.vin_store.writer.PreparedVinOfflineSample(
    sample_key,
    scene_id,
    snippet_id,
    free_input_provenance=None,
    numeric_blocks=dict(),
    record_blocks=dict(),
)

Normalized offline row before shard materialization.

1.1.1.1 Attributes

Name Type Description
sample_key str Stable sample key for the row.
scene_id str ASE scene identifier.
snippet_id str ASE snippet identifier.
numeric_blocks dict[str, NDArray[Any]] Fixed-size numeric blocks stored as Zarr arrays.
record_blocks dict[str, Any] Lazy diagnostic payloads stored as msgspec records.

Additional attribute identifiers: free_input_provenance

1.2 Functions

Name Description
assign_offline_splits Assign deterministic split membership to global sample indices.
flush_prepared_samples_to_shard Materialize a list of prepared rows into one immutable shard.
prepare_vin_offline_sample Normalize actor inputs and optional scene supervision into immutable row blocks.

1.2.1 assign_offline_splits

data_handling.vin_store.writer.assign_offline_splits(records, val_fraction)

Assign deterministic split membership to global sample indices.

1.2.1.1 Parameters

Name Type Description Default
records list[VinOfflineIndexRecord] Global sample-index records. required
val_fraction float Requested validation fraction. required

1.2.1.2 Returns

Name Type Description
dict[str, NDArray[Any]] Mapping from split name to global sample-index arrays.

1.2.2 flush_prepared_samples_to_shard

data_handling.vin_store.writer.flush_prepared_samples_to_shard(
    shard_index,
    shard_dir,
    rows,
)

Materialize a list of prepared rows into one immutable shard.

1.2.2.1 Parameters

Name Type Description Default
shard_index int Zero-based shard index. required
shard_dir Path Destination shard directory. required
rows list[PreparedVinOfflineSample] Prepared sample rows. required

1.2.2.2 Returns

Name Type Description
tuple[VinOfflineShardSpec, list[VinOfflineIndexRecord]] Shard descriptor plus local sample-index records.

1.2.3 prepare_vin_offline_sample

data_handling.vin_store.writer.prepare_vin_offline_sample(
    scene_id,
    snippet_id,
    vin_snippet,
    candidates,
    depths,
    rri,
    candidate_pcs,
    backbone_out,
    max_candidates,
    source_sample=None,
    include_depths=True,
    include_candidate_pcs=True,
    include_backbone=True,
    include_diagnostic_payloads=False,
    include_gt_obbs=True,
    include_detected_obbs=True,
    include_trajectory_metadata=True,
    backbone_numeric_keep_fields=None,
    backbone_payload_keep_fields=None,
    sample_key=None,
    actor_only=False,
    reference_pose_world_rig=None,
)

Normalize actor inputs and optional scene supervision into immutable row blocks.

1.2.3.1 Parameters

Name Type Description Default
scene_id str ASE scene identifier. required
snippet_id str ASE snippet identifier. required
vin_snippet VinSnippetView Canonical VIN snippet for the row. required
candidates CandidateSamplingResult | None Optional candidate-sampling payload for diagnostics. required
depths CandidateDepths | None Candidate-depth payload aligned with labels; absent for actor-only roots. required
rri RriResult | None Oracle metrics aligned with rendered candidates; absent for actor-only roots. required
candidate_pcs CandidatePointClouds | None Optional candidate point clouds for diagnostics. required
backbone_out EvlBackboneOutput | None Optional backbone outputs for training or diagnostics. required
source_sample EfmSnippetView | None Optional raw EFM snippet used for compact GT modalities. None
max_candidates int Maximum number of candidates stored in fixed blocks. required
include_depths bool Whether to materialize numeric depth blocks. True
include_candidate_pcs bool Whether candidate point clouds may be written when rich diagnostic payloads are enabled. True
include_backbone bool Whether to materialize backbone outputs. True
include_diagnostic_payloads bool Whether to write rich msgpack records such as full depth DTOs, candidate DTOs, candidate point clouds, and full backbone payloads. Defaults off because numeric blocks are the canonical training contract. False
include_gt_obbs bool Whether to persist compact GT OBB tensors from the raw snippet. True
include_detected_obbs bool Whether to persist compact detected OBB tensors from the backbone. True
include_trajectory_metadata bool Whether to persist trajectory timestamps and gravity. True
backbone_numeric_keep_fields set[str] | None Optional EVL backbone field keep-list for fixed numeric blocks. None preserves legacy behavior by writing all supported numeric fields. None
backbone_payload_keep_fields set[str] | None Optional EVL backbone field keep-list for rich diagnostic payloads. None preserves legacy behavior by serializing all available fields. None
sample_key str | None Optional explicit sample key. None
actor_only bool Omit scene supervision completely for V11 actor roots. False
reference_pose_world_rig PoseTW | None Canonical yaw-adjusted root chart, required for actor-only rows; PoseTW["12"], translation in metres. None

1.2.3.2 Returns

Name Type Description
PreparedVinOfflineSample Prepared row ready for shard materialization.
Back to top