1 oracle.pipelines.shards
oracle.pipelines.shards
Plan deterministic shards and promote validated rollout stores atomically.
Each manifest entry owns an ordered subset of immutable VIN source rows. A build writes a fresh temporary rollout Zarr store, validates it, records its owner sidecar, then renames the directory into place and writes _SUCCESS last. Existing partial or mismatched paths are preserved for operator review rather than overwritten.
1.1 Classes
| Name | Description |
|---|---|
| RolloutShardCampaignStatus | Status summary for all shards planned by one rollout manifest. |
| RolloutShardOwnershipConflictError | A stale temporary or final path prevents safe shard ownership. |
| RolloutShardRunResult | Result of one strict rollout shard build or resume check. |
| RolloutShardStatus | Status for one planned rollout shard in a generation campaign. |
1.1.1 RolloutShardCampaignStatus
oracle.pipelines.shards.RolloutShardCampaignStatus(
manifest_path,
final_root,
shards,
)Status summary for all shards planned by one rollout manifest.
1.1.1.1 Attributes
| Name | Description |
|---|---|
| manifest_path | Resolved JSONL manifest that defines the campaign. |
| final_root | Directory under which manifest shard ids map to final stores. |
| shards | Per-manifest-entry statuses in deterministic manifest order. |
| counts | Count succeeded, failed, incomplete, and missing planned shards. |
1.1.1.2 Methods
| Name | Description |
|---|---|
| to_jsonable | Return a stable JSON-compatible campaign status payload. |
1.1.1.2.1 to_jsonable
oracle.pipelines.shards.RolloutShardCampaignStatus.to_jsonable()Return a stable JSON-compatible campaign status payload.
1.1.2 RolloutShardOwnershipConflictError
oracle.pipelines.shards.RolloutShardOwnershipConflictError(
message,
*,
shard_entry,
output_tmp,
output_final,
)A stale temporary or final path prevents safe shard ownership.
1.1.3 RolloutShardRunResult
oracle.pipelines.shards.RolloutShardRunResult(
final_dir,
skipped,
success_path,
owner_path,
store_result=None,
outcome='succeeded',
reason=None,
reason_code=None,
recipe_name=None,
target_task_sampling=None,
)Result of one strict rollout shard build or resume check.
1.1.3.1 Attributes
| Name | Description |
|---|---|
| final_dir | Final promoted standalone rollout-store directory. |
| skipped | Whether an already validated, provenance-matching shard was reused. |
| success_path | Completion marker path written only after final promotion. |
| owner_path | Owner/provenance sidecar path carried through the promoted directory. |
| store_result | Fresh write summary, or None when a completed shard was skipped. |
| reason_code | Machine-readable Oracle invalidity code for a measurement rejection. |
| recipe_name | Recipe whose Oracle measurement was rejected, if applicable. |
| target_task_sampling | Complete opt-in target-pool inspection, including rejected attempts. |
| manifest_path | Manifest path for fresh or skipped shards. |
1.1.4 RolloutShardStatus
oracle.pipelines.shards.RolloutShardStatus(
shard_id,
split,
final_dir,
status,
num_source_rows,
success_path,
owner_path,
failed_markers=(),
errors=(),
)Status for one planned rollout shard in a generation campaign.
1.1.4.1 Attributes
| Name | Description |
|---|---|
| shard_id | Canonical generation-shard identifier from the campaign manifest. |
| split | VIN dataset split owned by the shard. |
| final_dir | Deterministic final rollout-store directory for the shard. |
| status | Observed lifecycle state derived from final and failure markers. |
| num_source_rows | Number of immutable VIN source rows assigned by the manifest. |
| success_path | Expected path of the write-last completion marker. |
| owner_path | Expected path of the promoted owner/provenance sidecar. |
| failed_markers | Failure sidecars from attempts for this shard, ordered by path. |
| errors | Validation or lifecycle problems explaining an incomplete state. |
1.1.4.2 Methods
| Name | Description |
|---|---|
| to_jsonable | Return a stable JSON-compatible status payload. |
1.1.4.2.1 to_jsonable
oracle.pipelines.shards.RolloutShardStatus.to_jsonable()Return a stable JSON-compatible status payload.
1.2 Functions
| Name | Description |
|---|---|
| plan_rollout_shards | Plan deterministic source-row rollout shard entries from a writer config. |
| plan_rollout_source_manifest | Freeze the source reader’s ordered rows independently of rollout profile. |
| run_rollout_shard | Build one rollout shard with strict temp-to-final promotion semantics. |
| summarize_rollout_shard_campaign | Summarize completion state for a rollout shard campaign. |
| write_rollout_shard_manifest_from_config | Plan and write a rollout shard JSONL manifest. |
| write_rollout_source_manifest_from_config | Plan and write one profile-independent ordered source manifest. |
1.2.1 plan_rollout_shards
oracle.pipelines.shards.plan_rollout_shards(config, *, rows_per_shard)Plan deterministic source-row rollout shard entries from a writer config.
Planning opens the configured VIN offline dataset, preserves its selected sample-index order within each split, and records enough row lineage for a later build job to reject config/source drift before expensive rendering.
1.2.2 plan_rollout_source_manifest
oracle.pipelines.shards.plan_rollout_source_manifest(source)Freeze the source reader’s ordered rows independently of rollout profile.
The returned contract deliberately excludes candidate families, policies, retention, and output paths so paired campaigns can derive their own shard manifests from one reviewed source population.
1.2.3 run_rollout_shard
oracle.pipelines.shards.run_rollout_shard(
config,
*,
shard_entry,
output_tmp,
output_final,
invocation=None,
)Build one rollout shard with strict temp-to-final promotion semantics.
A completed final directory is skipped only when the Zarr store validates, the embedded shard manifest matches the requested entry, and _owner.json is still the payload named by _SUCCESS.json. Any stale temp directory or incomplete/tampered final directory is left for operator review. Diagnostic captures receive a fresh operational directory per build attempt; their original frozen configuration identity and previous attempt bytes are retained.
1.2.4 summarize_rollout_shard_campaign
oracle.pipelines.shards.summarize_rollout_shard_campaign(
manifest_path,
*,
final_root,
)Summarize completion state for a rollout shard campaign.
The summary is manifest-driven: every planned shard is checked at its deterministic final directory, and failed attempts are discovered from _FAILED.<shard_id>.*.json sidecars written beside final shards. This gives operators a retry list without scanning large Zarr payload trees.
1.2.5 write_rollout_shard_manifest_from_config
oracle.pipelines.shards.write_rollout_shard_manifest_from_config(
config,
*,
manifest_path,
rows_per_shard,
)Plan and write a rollout shard JSONL manifest.
1.2.6 write_rollout_source_manifest_from_config
oracle.pipelines.shards.write_rollout_source_manifest_from_config(
source,
*,
manifest_path,
)Plan and write one profile-independent ordered source manifest.