spacr.example_archives

The published example datasets: what they are, and how to fetch them.

This module is separate from spacr.qt.hf_download, which owns the same downloads with a Qt progress dialog wrapped round them, and it imports PySide6 at module scope. spacr-download has to work on a cluster login node with no display and no Qt installed at all, so everything that is only about the data – which repositories publish it, what each archive is called, how a stream is verified, how an archive is safely unpacked, and what has to be repaired on arrival – lives here instead. hf_download imports these names back out, so every existing GUI caller (and every test that patches one of them on that module) keeps working unchanged.

This module is distinct from spacr.example_data, which fetches the Regression example screen’s count and score CSVs from a GitHub release. That is a different set of files, a different host and a different transport. This module is about the .tar archives of imaging data: the Mask demo plate, the Measure example, the Annotate/Classify example, the Replication and Recruitment assay examples, and – through spacr.screen_data – the pieces of the published TSG101 screen.

EXAMPLE_SETS carries one size per set, as spacr.screen_data.ScreenAsset does, so the picker and the CLI can show sizes before anything is fetched.

Classes

ExampleSet

One published example dataset, described before it is downloaded.

Functions

check_archive_sha256(→ None)	Refuse an archive whose SHA-256 differs from its pin.
download_archive(→ pathlib.Path)	Stream one file from the HF repo to dest_dir/basename.
example_plate_folder(→ pathlib.Path)	Return the shared plate folder the plate example sets unpack into.
example_set(→ ExampleSet)	The set called key.
example_set_folder(→ pathlib.Path)	Where the set called key unpacks: its folder beside the plate.
expand_measure_arrays(→ None)	Write each .npz back out as the .npy Measure reads.
explain_download_failure(→ str)	Convert a download exception into a message with cause and remedy.
extract_example_archive(→ int)	Unpack a downloaded example archive under dest.
make_the_example_paths_absolute(→ int)	Rewrite a downloaded example's stored paths to its unpack location.

Module Contents

class spacr.example_archives.ExampleSet

One published example dataset, described before it is downloaded.

Parameters:

• key – what a user types to ask for this one.

• repo – the Hugging Face dataset repository it is published in.

• summary – one line saying what the set is and what it is for.

• bytes – the archive’s approximate size: the figure the GUI buttons state (“about 280 MB”), taken from the archive at publication. It is not checked against the download. The exact length is verified during the transfer, against what the server declares.

• markers – what the unpacked set leaves behind, as globs relative to the folder it unpacks into. Presence is tested on these rather than on the folder, because all three sets share one plate directory: the directory existing says nothing about which of them is in it.

• expands_npz – whether .npz arrays have to be written back out as the .npy Measure reads. See expand_measure_arrays().

• in_default – whether spacr-download with no arguments fetches it. False for the OPS and Align & Stitch samples: together they are 0.6 GB of one screen’s tiles, and adding them would push the default run past CONFIRM_ABOVE_BYTES, so the default run would require confirmation. They are fetched when named, and by all.

• folder – the folder under the example-data root the set unpacks into. plate1 – the shared plate – for the three sets that are stages of one plate; a folder of its own for a set that would collide with them.

is_present(folder) → bool

Whether this set is already unpacked under folder.

Every marker has to match. A half-unpacked set – a transfer that died between the images and the database – is therefore reported as absent and is downloaded again.

Parameters:

folder – the plate directory the set unpacks into.

property archive: str

The .tar this set ships as.

spacr.example_archives.check_archive_sha256(file_name: str, digest: str) → None

Refuse an archive whose SHA-256 differs from its pin.

Parameters:

• file_name – the archive’s name in its repository.

• digest – the hex SHA-256 of the bytes that arrived.

Raises:

IOError – when EXAMPLE_ARCHIVE_SHA256 pins file_name to another digest.

spacr.example_archives.download_archive(repo_id: str, file_name: str, dest_dir: pathlib.Path, *, progress: Callable[[int, int | None], None] | None = None, chunk_size: int = 1 << 15) → pathlib.Path

Stream one file from the HF repo to dest_dir/basename.

Uses plain streamed HTTP rather than the hf_hub download machinery, so a one-off demo download creates no Hugging Face cache directory.

The body lands in a sibling .part file and is only moved onto the final path once every advertised byte has arrived. A dropped connection therefore never leaves a truncated file at the destination.

Parameters:

• repo_id – the Hugging Face dataset repository.

• file_name – the path within it; only the basename is kept on disk.

• dest_dir – the folder the file lands in.

• progress – called as progress(written, expected) after each chunk, where expected is None when the server declared no length. A CLI draws a percentage from it; the GUI does not use it, because its workers emit Qt signals from inside their own loops.

• chunk_size – 32 KB by default. An archive of gigabytes passes a bigger one.

spacr.example_archives.example_plate_folder() → pathlib.Path

Return the shared plate folder the plate example sets unpack into.

~/.cache/spacr/example_data/plate1 – a real spaCR plate directory, holding whichever of merged/, data/, measurements/ and settings/ have been downloaded.

The sets share one folder because they are used together: data/ holds the crops and measurements/measurements.db indexes them. Each archive’s members are relative to this folder, so the three unpack into it side by side and compose into a plate that Measure, Annotate and Classify can all be pointed at.

The assay examples are not stages of this plate and each has a folder of its own beside it; see example_set_folder().

spacr.example_archives.example_set(key: str) → ExampleSet

The set called key.

Parameters:

key – an EXAMPLE_SETS key, such as import, mask, measure, annotate, dose, control_chart or hit.

Raises:

KeyError – naming the keys that do exist. A typo therefore raises rather than returning None.

spacr.example_archives.example_set_folder(key: str) → pathlib.Path

Where the set called key unpacks: its folder beside the plate.

Parameters:

key – an EXAMPLE_SETS key.

Returns:

~/.cache/spacr/example_data/<folder>. Not created here.

Raises:

KeyError – as example_set() does.

spacr.example_archives.expand_measure_arrays(merged: pathlib.Path) → None

Write each .npz back out as the .npy Measure reads.

The compression is a transport detail – it halves a 700 MB download – and Measure loads npy. Converting on arrival keeps that entirely inside the downloader rather than teaching every reader about a second format.

The .npz is removed afterwards: keeping both would double the disk cost of an example dataset, and nothing reads the .npz afterwards.

This is a module function rather than a method: after_extract runs on the download thread, and calling a worker method such as _MeasureExampleWorker(dest)._expand_arrays(...) would construct a QObject on that thread, which the GUI thread may then not touch. The function needs no self.

Parameters:

merged – the folder the .npz arrays were unpacked into.

spacr.example_archives.explain_download_failure(exc: BaseException) → str

Convert a download exception into a message with cause and remedy.

Downloading test data requires a network connection. For the ordinary offline case str(exc) is a nested urllib3 dump:

(MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443):
Max retries exceeded with url: /api/datasets/... (Caused by
NewConnectionError('<urllib3.connection.HTTPSConnection object at
0x7e8d...>: Failed to establish a new connection: [Errno 101] Network
is unreachable'))"), '(Request ID: 73ac20ed-...)')

It states neither that the machine is offline nor what to do. The three conditions recognised are: no network, the huggingface_hub extra not installed, and a truncated transfer. Each gets a sentence naming the cause and the way out; anything else keeps its own message with the same closing advice attached.

Parameters:

exc – the exception raised inside the download worker.

Returns:

a multi-line message for the failure dialog.

spacr.example_archives.extract_example_archive(archive, dest) → int

Unpack a downloaded example archive under dest.

Members are extracted with filter="data". A tar can name ../../etc/something or an absolute path, and a plain extractall writes there, so an unfiltered unpack would let anyone who can publish to the repository write anywhere the user can write. The filter rejects those members, along with device nodes, setuid bits and symlinks pointing outside the tree.

Python 3.12 and later have it built in. Older interpreters get an explicit check instead of a silent unfiltered unpack.

Parameters:

• archive – the .tar on disk.

• dest – the folder to unpack into.

Returns:

how many members were written.

spacr.example_archives.make_the_example_paths_absolute(root) → int

Rewrite a downloaded example’s stored paths to its unpack location.

A measurements database stores absolute paths to its crops, which name the machine that made it and resolve nowhere else. The published copy stores them relative to the dataset root instead, so it is portable and carries no account name – and this function turns them back into absolute paths.

The settings files are rewritten the same way: they carry DATASET_PLACEHOLDER where the unpack location goes, so a user can press Run without first editing a path.

Idempotent. A path that is already absolute is left alone, so running this twice – a re-download over an existing copy – does not produce /home/me/data//home/me/data/....

Parameters:

root – the folder the dataset was unpacked into.

Returns:

how many values were rewritten.