1 data_handling.atek_downloads.download_stats

data_handling.atek_downloads.download_stats

Helpers for reporting local download coverage and snippet counts for ATEK shards.

The module provides read-only coverage summaries over downloaded WebDataset archives. It compares local shard/sample-key counts with ASE/ATEK URL manifests without opening EFM tensors or mutating the acquisition directories.

1.1 Classes

Name Description
DownloadedAtekStats Summary of available vs downloaded ATEK data for a given config.

1.1.1 DownloadedAtekStats

data_handling.atek_downloads.download_stats.DownloadedAtekStats(
    config_name,
    expected_scenes,
    expected_shards,
    downloaded_scenes,
    downloaded_shards,
    downloaded_snippets,
    snippets_per_shard,
    snippet_count_is_estimate,
)

Summary of available vs downloaded ATEK data for a given config.

1.1.1.1 Attributes

Name Description
config_name ATEK config name (e.g., "efm").
expected_scenes Number of GT-mesh scenes expected for this config.
expected_shards Total number of shards expected across GT-mesh scenes for this config.
downloaded_scenes Number of GT-mesh scenes present on disk (at least one shard downloaded).
downloaded_shards Number of shard tar files found on disk across GT-mesh scenes.
downloaded_snippets Estimated or exact number of snippet samples in downloaded shard tar files.
snippets_per_shard If constant, the inferred number of snippets per shard.
snippet_count_is_estimate Whether downloaded_snippets was inferred from a sampled constant.

1.2 Functions

Name Description
compute_downloaded_atek_stats Compute local download coverage for an ATEK config.
count_snippets_in_tar Count snippet samples in a shard tar.

1.2.1 compute_downloaded_atek_stats

data_handling.atek_downloads.download_stats.compute_downloaded_atek_stats(
    metadata,
    data_root,
    config_name,
    snippet_sample_size=10,
)

Compute local download coverage for an ATEK config.

1.2.1.1 Parameters

Name Type Description Default
metadata ASEMetadata Parsed ASE metadata (meshes + ATEK shard manifests). required
data_root Path Project data root directory (e.g., .data). required
config_name str ATEK config name (e.g., "efm"). required
snippet_sample_size int Number of shard tars to sample to infer a constant snippet-per-shard count. If the sample is not constant, the function falls back to an exact scan of all shard tars. 10

1.2.1.2 Returns

Name Type Description
DownloadedAtekStats Download stats for GT-mesh scenes under config_name.

1.2.2 count_snippets_in_tar

data_handling.atek_downloads.download_stats.count_snippets_in_tar(path)

Count snippet samples in a shard tar.

We define one snippet as one WebDataset “sample key”. In ASE ATEK shards, each snippet includes a *.sequence_name.txt file, so we count these entries.

1.2.2.1 Parameters

Name Type Description Default
path Path Path to a .tar shard. required

1.2.2.2 Returns

Name Type Description
int Number of snippets in this shard.
Back to top