1 rollouts.inspection.profile_comparison

rollouts.inspection.profile_comparison

Paired endpoint evaluation for frozen rollout-profile experiments.

The reducer consumes an independently declared set of intended experiment cells and exactly one terminal outcome for each profile and cell. A cell is identified by scene, target, generation seed, selection seed, and temperature. Completed outcomes carry raw root and final reconstruction errors plus hashes of the measurement receipts that produced them. A separate root-cloud digest identifies the measured content because receipt digests may also cover metadata. The producer’s assertion that those measurements were independently recomputed is admitted as provenance; this reducer cannot establish independence from JSON alone.

For every shared-complete cell, the endpoint gain is computed by the canonical rri_metrics.returns kernel,

\[ J_e^{(H)} = \frac{e_0 - e_H}{e_0 + \epsilon}. \]

The shared-complete mask is frozen from terminal statuses before any gains are read. Paired differences are averaged across seeds within each temperature, then equally across supported temperatures within a scene, and finally equally across scenes. Missing or failed cells are never converted to zero gain. The result is conditional on shared completion and therefore does not estimate unconditional rollout quality.

1.1 Classes

Name	Description
ProfileComparisonContractError	Raised when receipts cannot support the declared paired comparison.

1.1.1 ProfileComparisonContractError

rollouts.inspection.profile_comparison.ProfileComparisonContractError()

Raised when receipts cannot support the declared paired comparison.

1.2 Functions

Name	Description
canonical_json_bytes	Serialize evaluator output deterministically and reject non-finite JSON.
evaluate_profile_comparison	Reduce paired measurement receipts into a schema-v2 checkpoint.
evaluator_fingerprint	Return the digest of this evaluator module for frozen assignments.

1.2.1 canonical_json_bytes

rollouts.inspection.profile_comparison.canonical_json_bytes(value)

Serialize evaluator output deterministically and reject non-finite JSON.

1.2.2 evaluate_profile_comparison

rollouts.inspection.profile_comparison.evaluate_profile_comparison(
    payload,
    *,
    reference_manifest_bytes,
    expected_reference_manifest_sha256,
)

Reduce paired measurement receipts into a schema-v2 checkpoint.

1.2.2.1 Parameters

Name	Type	Description	Default
payload	Mapping[str, Any]	Frozen comparison envelope. The required input schema is documented by this module and enforced fail-closed at admission.	required
reference_manifest_bytes	bytes	Exact preregistered intended-cell manifest.	required
expected_reference_manifest_sha256	str	Digest frozen before outcomes were acquired. It must match the envelope and exact manifest bytes.	required

1.2.2.2 Returns

Name	Type	Description
	dict[str, Any]	A JSON-compatible schema-v2 performance checkpoint. The additional
	dict[str, Any]	profile_comparison member contains the conditional estimand,
	dict[str, Any]	outcome counts, cell-level gains, per-scene effects, per-temperature
	dict[str, Any]	effects, and bootstrap receipt needed for scientific review. This
	dict[str, Any]	function performs no W&B or OMX side effects.

1.2.2.3 Raises

Name	Type	Description
	ProfileComparisonContractError	If identities, intended cells, terminal outcomes, budgets, contracts, or numerical receipts are incomplete or incompatible.

1.2.2.4 Notes

The reducer requires independent_endpoint_evaluation=true and raw measurement hashes for completed rows, but cannot prove that claim from JSON. The experiment adapter remains responsible for producing and retaining the independently recomputed measurement artifacts.

1.2.3 evaluator_fingerprint

rollouts.inspection.profile_comparison.evaluator_fingerprint()

Return the digest of this evaluator module for frozen assignments.