1 data_handling.vin_store.repack

data_handling.vin_store.repack

Lossless physical resharding of one immutable VIN root store.

This transformation preserves global indices and exact values, but changes manifest and physical shard identity. Existing dependent rollouts retain their original root; binding them to a repacked root requires a coordinated rewrite. No generation, relabeling, acquisition or scientific qualification occurs here.

1.1 Classes

Name Description
RootRepackResult New immutable layout and its origin-qualified row mapping receipt.

1.1.1 RootRepackResult

data_handling.vin_store.repack.RootRepackResult(
    store_dir,
    receipt_path,
    num_samples,
    num_shards,
)

New immutable layout and its origin-qualified row mapping receipt.

1.1.1.1 Attributes

Name Description
store_dir Freshly published root-store directory; existing destinations reject.
receipt_path Receipt binding the output rows to their source manifest and inventory.
num_samples Number of retained observations in the published root store.
num_shards Number of physical Zarr shards in the published root store.

1.2 Functions

Name Description
repack_root_store Reshard all root observations into a fresh, atomically published directory.
repack_root_stores Merge/split compatible roots, retaining origin-qualified observations.

1.2.1 repack_root_store

data_handling.vin_store.repack.repack_root_store(
    source,
    destination,
    *,
    rows_per_shard=64,
    read_chunk_rows=1,
)

Reshard all root observations into a fresh, atomically published directory.

Rows keep their sample index/key, scene/snippet and learning split. Every numeric and indexed-record block is passed through the canonical reader and shard writer with unchanged dtype/value/optional status. Heterogeneous block inventories or row shapes reject explicitly instead of filling or dropping fields. The receipt contains a total bijection qualified by original manifest digest and full source inventory. Original acquisition sidecars remain exact. Source bytes are rechecked before promotion; failure leaves no final output.

This single-store operation neither merges replicas nor rewrites dependent rollout references. Consumers of existing rollouts must keep the old root.

1.2.2 repack_root_stores

data_handling.vin_store.repack.repack_root_stores(
    sources,
    destination,
    *,
    samples_per_store=None,
    rows_per_shard=64,
    read_chunk_rows=1,
)

Merge/split compatible roots, retaining origin-qualified observations.

Destination indices are contiguous and local to each new root. Sample keys, scene/snippet, physical split, all block values/dtypes and optional records remain exact. The receipt links each new row to its original manifest/version, index/key and physical shard/row. Distinct original manifests never deduplicate by sample key. Identical manifests deduplicate only with equal full inventories; conflicting replicas reject. Acquisition sidecars and manifests are archived verbatim per origin. Unknown schemas, incompatible retained metadata, and backend compositions that erase QH’s empirical-pooling block reject.

Each output bounds materialization by rows_per_shard. The containing destination publishes atomically after exact block comparison and source currentness verification; it never overwrites an existing output.

Back to top