1 rollouts.storage.shard_manifest
rollouts.storage.shard_manifest
Deterministic JSONL shard manifests for rollout generation.
Rollout shard manifests split VIN offline source rows into Slurm-friendly work units. Each JSONL row owns one rollout shard and lists the ordered VIN sample rows that the shard must process. The manifest is intentionally small and human-inspectable; heavy rollout data remains in standalone Zarr stores.
1.1 Attributes
| Name | Description |
|---|---|
| ROLLOUT_SHARD_MANIFEST_VERSION | Version label for rollout generation JSONL shard manifests. |
| ROLLOUT_SHARD_OWNER_FILENAME | Shard owner/provenance sidecar written before final promotion. |
| ROLLOUT_SHARD_SUCCESS_FILENAME | Completion marker written last for a validated rollout shard. |
| ROLLOUT_SOURCE_MANIFEST_VERSION | Version label for profile-independent ordered VIN source manifests. |
1.2 Classes
| Name | Description |
|---|---|
| RolloutShardCampaignBinding | Optional campaign identity carried by a shard entry. |
| RolloutShardEntry | One deterministic rollout generation shard entry. |
| RolloutShardRow | One VIN offline source row owned by a rollout shard. |
| RolloutSourceManifest | Profile-independent ordered VIN rows shared by rollout campaigns. |
1.2.1 RolloutShardCampaignBinding
rollouts.storage.shard_manifest.RolloutShardCampaignBinding(
campaign_id,
plan_hash,
work_unit_hash,
target_id,
profile_hash,
explicit_target_hash,
generation_revision_hash='',
)Optional campaign identity carried by a shard entry.
1.2.1.1 Methods
| Name | Description |
|---|---|
| to_jsonable | Return stable JSON evidence. |
| from_jsonable | Decode a campaign binding. |
1.2.1.1.1 to_jsonable
rollouts.storage.shard_manifest.RolloutShardCampaignBinding.to_jsonable()Return stable JSON evidence.
1.2.1.1.2 from_jsonable
rollouts.storage.shard_manifest.RolloutShardCampaignBinding.from_jsonable(
payload,
)Decode a campaign binding.
1.2.2 RolloutShardEntry
rollouts.storage.shard_manifest.RolloutShardEntry(
shard_id,
split,
rows,
writer_config_hash,
source_manifest_hash,
source_cache_version,
split_manifest_hash,
source_store_dir,
manifest_version=ROLLOUT_SHARD_MANIFEST_VERSION,
campaign_binding=None,
campaign_split=None,
generation_revision_hash='',
)One deterministic rollout generation shard entry.
1.2.2.1 Attributes
| Name | Description |
|---|---|
| shard_id | Canonical shard-000000 style rollout-generation identifier. |
| split | VIN dataset split shared by all owned source rows. |
| rows | Ordered immutable VIN source rows assigned to this output shard. |
| writer_config_hash | Hash of the complete writer configuration used for drift rejection. |
| source_manifest_hash | Hash of the immutable VIN offline-store manifest. |
| source_cache_version | VIN source cache schema/version recorded in generated lineage. |
| split_manifest_hash | Hash binding split name and ordered source-row records. |
| source_store_dir | Portable VIN source-store basename/cache identity used for local reopening. |
| manifest_version | JSONL ownership-contract version. |
| campaign_binding | Optional campaign identity; None preserves legacy manifests. |
| campaign_split | Authoritative campaign split, distinct from the VIN source split. |
1.2.2.2 Methods
| Name | Description |
|---|---|
| from_jsonable | Decode and canonicalize one JSONL shard ownership entry. |
| to_jsonable | Return a stable JSON-compatible shard payload. |
| validate | Raise when the entry violates the shard-manifest contract. |
1.2.2.2.1 from_jsonable
rollouts.storage.shard_manifest.RolloutShardEntry.from_jsonable(payload)Decode and canonicalize one JSONL shard ownership entry.
1.2.2.2.1.1 Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| payload | dict[str, Any] | Parsed JSON object containing source rows and provenance hashes. | required |
1.2.2.2.1.2 Returns
| Name | Type | Description |
|---|---|---|
| 'RolloutShardEntry' | Typed entry. Call :meth:validate before using it to build a |
|
| 'RolloutShardEntry' | rollout store. |
1.2.2.2.2 to_jsonable
rollouts.storage.shard_manifest.RolloutShardEntry.to_jsonable()Return a stable JSON-compatible shard payload.
1.2.2.2.3 validate
rollouts.storage.shard_manifest.RolloutShardEntry.validate()Raise when the entry violates the shard-manifest contract.
1.2.3 RolloutShardRow
rollouts.storage.shard_manifest.RolloutShardRow(
order,
sample_index,
sample_key,
scene_id,
snippet_id,
split,
source_shard_id,
source_shard_row,
campaign_split=None,
)One VIN offline source row owned by a rollout shard.
1.2.3.1 Attributes
| Name | Description |
|---|---|
| order | Zero-based processing order within the containing source/shard manifest. |
| sample_index | Stable VIN dataset sample index used to reopen the source row. |
| sample_key | Canonical VIN sample key used to reject index drift. |
| scene_id | Source scene identifier persisted into rollout lineage. |
| snippet_id | Source snippet identifier persisted into rollout lineage. |
| split | VIN dataset split; all rows in one entry must share it. |
| source_shard_id | Identifier of the immutable VIN storage shard containing the row. |
| source_shard_row | Zero-based row within source_shard_id. |
| campaign_split | Authoritative campaign split; split remains VIN source split. |
1.2.3.2 Methods
| Name | Description |
|---|---|
| from_index_record | Build a manifest row from a VIN offline index record. |
| from_jsonable | Decode one JSON row while restoring numeric lineage fields. |
| to_jsonable | Return a stable JSON-compatible row payload. |
| hash_record | Return the row fields used for deterministic lineage hashing. |
| matches_record | Return whether a VIN offline index record matches this manifest row. |
1.2.3.2.1 from_index_record
rollouts.storage.shard_manifest.RolloutShardRow.from_index_record(
record,
*,
order,
)Build a manifest row from a VIN offline index record.
1.2.3.2.2 from_jsonable
rollouts.storage.shard_manifest.RolloutShardRow.from_jsonable(payload)Decode one JSON row while restoring numeric lineage fields.
1.2.3.2.2.1 Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| payload | dict[str, Any] | Mapping emitted by :meth:to_jsonable. |
required |
1.2.3.2.2.2 Returns
| Name | Type | Description |
|---|---|---|
| 'RolloutShardRow' | Typed source-row ownership record. Entry-level validation checks | |
| 'RolloutShardRow' | split consistency and contiguous ordering after decoding. |
1.2.3.2.3 to_jsonable
rollouts.storage.shard_manifest.RolloutShardRow.to_jsonable()Return a stable JSON-compatible row payload.
1.2.3.2.4 hash_record
rollouts.storage.shard_manifest.RolloutShardRow.hash_record()Return the row fields used for deterministic lineage hashing.
1.2.3.2.5 matches_record
rollouts.storage.shard_manifest.RolloutShardRow.matches_record(record)Return whether a VIN offline index record matches this manifest row.
1.2.4 RolloutSourceManifest
rollouts.storage.shard_manifest.RolloutSourceManifest(
split,
rows,
source_manifest_hash,
source_cache_version,
split_manifest_hash,
source_store_dir,
manifest_version=ROLLOUT_SOURCE_MANIFEST_VERSION,
)Profile-independent ordered VIN rows shared by rollout campaigns.
Candidate families, policies, retention settings, and output paths do not belong here. Profile-specific shard manifests bind this frozen source set to a writer configuration later.
1.2.4.1 Attributes
| Name | Description |
|---|---|
| split | VIN split shared by every ordered source row. |
| rows | Ordered immutable VIN source rows reused across compared profiles. |
| source_manifest_hash | Hash of the strict immutable VIN offline-store manifest. |
| source_cache_version | Strict VIN source cache schema/version. |
| split_manifest_hash | Hash binding the source manifest, split, and ordered source rows. |
| source_store_dir | Portable source-store basename/cache identity, never a checkout path. |
| manifest_version | Ordered source-manifest contract version. |
1.2.4.2 Methods
| Name | Description |
|---|---|
| from_index_records | Freeze ordered VIN index records into a profile-independent manifest. |
| from_jsonable | Decode one ordered source-manifest payload. |
| to_jsonable | Return a stable JSON-compatible source-manifest payload. |
| validate | Raise when source provenance, ordering, or row identity drifts. |
1.2.4.2.1 from_index_records
rollouts.storage.shard_manifest.RolloutSourceManifest.from_index_records(
records,
*,
source_manifest_hash,
source_cache_version,
source_store_dir,
)Freeze ordered VIN index records into a profile-independent manifest.
1.2.4.2.2 from_jsonable
rollouts.storage.shard_manifest.RolloutSourceManifest.from_jsonable(payload)Decode one ordered source-manifest payload.
1.2.4.2.3 to_jsonable
rollouts.storage.shard_manifest.RolloutSourceManifest.to_jsonable()Return a stable JSON-compatible source-manifest payload.
1.2.4.2.4 validate
rollouts.storage.shard_manifest.RolloutSourceManifest.validate()Raise when source provenance, ordering, or row identity drifts.
1.3 Functions
| Name | Description |
|---|---|
| build_rollout_split_manifest_hash | Hash one ordered source-row set using rollout-lineage semantics. |
| canonical_rollout_shard_id | Return the canonical shard-000000 style rollout shard id. |
| load_rollout_shard_entry | Load one validated manifest entry by canonical or numeric shard id. |
| read_rollout_shard_manifest | Read and validate all entries in a rollout shard JSONL manifest. |
| read_rollout_source_manifest | Read and validate one profile-independent ordered source manifest. |
| write_rollout_shard_manifest | Write rollout shard entries as stable JSONL. |
| write_rollout_source_manifest | Write one validated profile-independent source manifest as stable JSON. |
1.3.1 build_rollout_split_manifest_hash
rollouts.storage.shard_manifest.build_rollout_split_manifest_hash(
source_manifest_hash,
split,
records,
)Hash one ordered source-row set using rollout-lineage semantics.
1.3.2 canonical_rollout_shard_id
rollouts.storage.shard_manifest.canonical_rollout_shard_id(value)Return the canonical shard-000000 style rollout shard id.
1.3.3 load_rollout_shard_entry
rollouts.storage.shard_manifest.load_rollout_shard_entry(path, shard_id)Load one validated manifest entry by canonical or numeric shard id.
1.3.3.1 Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| path | Path | str | Rollout shard JSONL manifest. | required |
| shard_id | str | int | Numeric id or canonical shard-000000 token. |
required |
1.3.3.2 Returns
| Name | Type | Description |
|---|---|---|
| RolloutShardEntry | Matching validated ownership entry. |
1.3.3.3 Raises
| Name | Type | Description |
|---|---|---|
| KeyError | If the requested shard is absent from the manifest. |
1.3.4 read_rollout_shard_manifest
rollouts.storage.shard_manifest.read_rollout_shard_manifest(path)Read and validate all entries in a rollout shard JSONL manifest.
Blank lines are ignored. Malformed rows, duplicate shard ids, mixed splits, or non-contiguous row ownership fail before any rollout generation begins.
1.3.5 read_rollout_source_manifest
rollouts.storage.shard_manifest.read_rollout_source_manifest(path)Read and validate one profile-independent ordered source manifest.
1.3.6 write_rollout_shard_manifest
rollouts.storage.shard_manifest.write_rollout_shard_manifest(path, entries)Write rollout shard entries as stable JSONL.
1.3.7 write_rollout_source_manifest
rollouts.storage.shard_manifest.write_rollout_source_manifest(path, manifest)Write one validated profile-independent source manifest as stable JSON.