1 datasets.scene_split

datasets.scene_split

Frozen scene assignments for incremental development-corpus composition.

This plan selects existing immutable shards for train and validation. It never changes their physical source split, campaign provenance, labels or seals. All replicas of a scene share one assignment. Extensions retain earlier membership rather than reranking the enlarged population.

1.1 Classes

Name Description
SceneSplitPlan An explicit development population and its stable scene membership.

1.1.1 SceneSplitPlan

datasets.scene_split.SceneSplitPlan(
    seed,
    train_scene_ids,
    validation_scene_ids,
    collection_dirs,
)

An explicit development population and its stable scene membership.

1.1.1.1 Attributes

Name Description
seed Seed for SHA-256 scene ordering, independent of filesystem order.
train_scene_ids Distinct training scenes; every sample and replica stays in this stage.
validation_scene_ids Distinct validation scenes, disjoint from training scenes.
collection_dirs Exact collection directories eligible for this external assignment.

1.1.1.2 Methods

Name Description
digest Bind retained evidence to the exact assignments and collection scope.
includes_store Accept only exact shards below declared collection directories.
stage_for_scenes Select one stage; reject missing assignments and mixed-stage shards.
store_paths_by_stage Route exact selected stores through canonical manifest inspection.
1.1.1.2.1 digest
datasets.scene_split.SceneSplitPlan.digest()

Bind retained evidence to the exact assignments and collection scope.

1.1.1.2.2 includes_store
datasets.scene_split.SceneSplitPlan.includes_store(path)

Accept only exact shards below declared collection directories.

1.1.1.2.3 stage_for_scenes
datasets.scene_split.SceneSplitPlan.stage_for_scenes(scene_ids)

Select one stage; reject missing assignments and mixed-stage shards.

Store-level routing works only when every scene in a shard belongs to the same stage. A packed shard crossing train/validation must be split by its generator or consumed through a scene-filtering dataset.

1.1.1.2.4 store_paths_by_stage
datasets.scene_split.SceneSplitPlan.store_paths_by_stage(paths)

Route exact selected stores through canonical manifest inspection.

1.2 Functions

Name Description
annotate_scene_split_inventory Add external learning assignment without replacing persisted provenance.
extend_scene_split_plan Assign approximately 80/20 scenes while preserving every prior decision.
main Explicitly create or extend a scene split from completed collection shards.
read_scene_split_plan Read a frozen plan with collection paths relative to its own directory.
write_scene_split_plan Persist assignments outside immutable collections with portable locators.

1.2.1 annotate_scene_split_inventory

datasets.scene_split.annotate_scene_split_inventory(rows, plan)

Add external learning assignment without replacing persisted provenance.

1.2.2 extend_scene_split_plan

datasets.scene_split.extend_scene_split_plan(
    scene_ids,
    collection_dirs,
    *,
    existing=None,
    seed=20261006,
)

Assign approximately 80/20 scenes while preserving every prior decision.

Validation gets ceil(N / 5) scenes (rounding error is under one scene). Only new scenes can move the ratio toward that target during increments. An existing assignment is never changed, even when a later population cannot attain the target without moving earlier scenes.

1.2.3 main

datasets.scene_split.main()

Explicitly create or extend a scene split from completed collection shards.

1.2.4 read_scene_split_plan

datasets.scene_split.read_scene_split_plan(path)

Read a frozen plan with collection paths relative to its own directory.

1.2.5 write_scene_split_plan

datasets.scene_split.write_scene_split_plan(path, plan)

Persist assignments outside immutable collections with portable locators.

Back to top