spacr.cli_download

spacr-download — fetch spaCR’s published example data, with no GUI.

Downloads the example datasets that the application’s “Load test data” and “Load example data” buttons and the screen-data picker fetch, without a display. This is for machines such as cluster nodes, where the data has to be on disk before a batch job starts.

With no arguments it fetches the example sets – Import, Mask, Measure, Annotate/Classify, Replication, Recruitment and the synthetic Invasion set – which come to about 1.8 GB. The OPS and Align & Stitch samples are fetched only when named (spacr-download ops stitch) or with all. It does not fetch the published TSG101 screen, which is 33 GB. The screen is opt-in, is asked for in pieces, and is confirmed before it starts. The pieces are spacr.screen_data’s, unchanged: four measurement databases of about 0.5 GB and four crop folders of about 8 GB, any subset of which can be named.

--list prints every piece with its size and whether it is already on disk, and the total size of the current selection, without opening a network connection. --dry-run prints the same listing.

Importing this module must stay light – no Qt, no torch, no matplotlib – so spacr-download --help answers instantly on a login node with a cold NFS cache. That is why the download primitives live in spacr.example_archives rather than in spacr.qt.hf_download, which imports PySide6 at module scope. tests/test_cli_download.py pins it.

Usage:

spacr-download                            # every example set (~1.8 GB)
spacr-download --list                     # what exists, what is here
spacr-download measure annotate           # two of the seven
spacr-download --screen measurements      # the four databases (~2.1 GB)
spacr-download --screen crops --plate 1   # one plate of crops (~8.9 GB)
spacr-download all --yes                  # everything, screen included

Exit codes:

0  everything asked for is on disk, or a confirmation was declined
1  a download failed; the pieces that succeeded are still on disk
2  bad arguments, not enough disk space, or a large download that could not
   be confirmed because nothing was there to confirm it

Exceptions

SelectionError

Raised for an unknown example name, screen kind or plate number.

Classes

Piece

One archive this command can fetch, priced and located.

Functions

build_parser(→ argparse.ArgumentParser)

Return the spacr-download argument parser.

build_plan(→ List[Piece])

One Piece per thing to fetch, with its size and its folder.

cmd_download(→ int)

spacr-download proper. See main() for the exit codes.

default_destination(→ pathlib.Path)

~/.cache/spacr/example_data -- where the GUI already looks.

download(→ Tuple[List[Piece], List[Tuple[Piece, ...)

Fetch every piece, and keep going after one fails.

example_folder(→ pathlib.Path)

The one plate folder the three plate example sets unpack into.

fetch_piece(→ int)

Download one archive, unpack it, and delete the archive.

main(→ int)

spacr-download entry point.

pieces_to_fetch(→ List[Piece])

The plan minus what is already on disk.

render_listing(→ str)

Every piece, its size, whether it is here, and what the total would be.

resolve_selection(, *, screen, plates, ...)

Turn what the user typed into the exact list of things to fetch.

room_for(→ Optional[str])

Return a low-disk-space message, or None when there is enough room.

screen_folder(→ pathlib.Path)

Return the folder for one screen plate: <dest>/screen/plate<N>.

screen_notice(→ str)

Return the notice explaining how to request the screen data.

Module Contents

exception spacr.cli_download.SelectionError[source]

Bases: Exception

Raised for an unknown example name, screen kind or plate number.

Always exit code 2: nothing was attempted, so it is an argument problem rather than a failed download.

Initialize self. See help(type(self)) for accurate signature.

class spacr.cli_download.Piece[source]

One archive this command can fetch, priced and located.

Examples and screen pieces are described by two different dataclasses – ExampleSet and ScreenAsset – because they answer to two different publishers. This is the one shape the listing, the size arithmetic and the download loop all work in, so none of them has to know which kind of thing it is holding.

Parameters:

  • key – how a selection names it, unique across both publishers.

  • name – what the listing’s first column shows.

  • detail – the rest of the row – a summary for an example set, the archive name for a screen piece.

  • repo – the dataset repository it is published in.

  • archive – the .tar to fetch from it.

  • folder – where it unpacks.

  • bytes – the archive’s size, so a selection’s total is known before downloading.

  • present – whether it is already unpacked where it would go. Computed when the plan is built rather than looked up later, so the listing and the download agree about what will happen.

  • expands_npz – whether the unpacked arrays have to be rewritten as .npy afterwards.

spacr.cli_download.build_parser() → argparse.ArgumentParser[source]

Return the spacr-download argument parser.

Building it imports nothing beyond the standard library and two dependency-light spaCR modules, so --help is instant.

spacr.cli_download.build_plan(examples: Sequence[spacr.example_archives.ExampleSet], assets: Sequence[spacr.screen_data.ScreenAsset], dest) → List[Piece][source]

One Piece per thing to fetch, with its size and its folder.

Parameters:

  • examples – the example sets to include.

  • assets – the screen pieces to include.

  • dest – the root everything unpacks under.

spacr.cli_download.cmd_download(args: argparse.Namespace, *, out=None, err=None) → int[source]

spacr-download proper. See main() for the exit codes.

Parameters:

  • args – the parsed command line.

  • out – stream for ordinary output; sys.stdout when None.

  • err – stream for errors; sys.stderr when None.

Returns:

the process exit code.

spacr.cli_download.default_destination() → pathlib.Path[source]

~/.cache/spacr/example_data – where the GUI already looks.

Chosen so that a plate fetched by this command is the plate the application’s own “Load example data” buttons point at: spacr.example_archives.example_plate_folder() is <this>/plate1. Someone who ran spacr-download before opening the GUI should find the data already there, not download it twice.

spacr.cli_download.download(pieces: Sequence[Piece], *, out=None, err=None, quiet: bool = False) → Tuple[List[Piece], List[Tuple[Piece, BaseException]]][source]

Fetch every piece, and keep going after one fails.

A failed piece does not stop the remaining pieces; a crop plate can take an hour to download. Each failure is reported as it happens and repeated in the summary, and the caller turns a non-empty result into exit code 1.

Parameters:

  • pieces – what to fetch, in order.

  • out – stream for progress; sys.stdout when None.

  • err – stream for failures; sys.stderr when None.

  • quiet – print neither the per-piece line nor the percentage.

Returns:

(finished, failed). Both are needed and neither can be derived from the other: an interrupt stops the loop, so the pieces that were never attempted are in neither list, and a summary that subtracted the failures from the selection would report them as downloaded.

spacr.cli_download.example_folder(dest) → pathlib.Path[source]

The one plate folder the three plate example sets unpack into.

The sets share one folder because they compose: the Measure example’s merged/, the Annotate example’s data/ and measurements/, and the Mask demo’s raw images are three stages of the same plate, and spaCR expects to be pointed at a plate.

The assay sets are not stages of that plate and each unpacks into a folder of its own; see spacr.example_archives.example_set_folder().

Parameters:

dest – the root everything unpacks under.

spacr.cli_download.fetch_piece(piece: Piece, *, out=None, progress: bool = True) → int[source]

Download one archive, unpack it, and delete the archive.

The archive is removed as soon as it is unpacked: it is a second copy of everything just written, and for a crop plate that is another 8 GB.

Parameters:

  • piece – what to fetch.

  • out – where progress is drawn; sys.stdout when None.

  • progress – draw a percentage while it arrives.

Returns:

how many members were unpacked.

spacr.cli_download.main(argv: Sequence[str] | None = None) → int[source]

spacr-download entry point.

Parameters:

argv – argument list; sys.argv[1:] when None.

Returns:

0 done, 1 a download failed, 2 bad arguments or no room.

spacr.cli_download.pieces_to_fetch(plan: Sequence[Piece], *, force: bool = False) → List[Piece][source]

The plan minus what is already on disk.

--force keeps everything, which is how a truncated or edited copy gets repaired: the archive is re-fetched and unpacked over what is there.

Parameters:

  • plan – the pieces a selection resolved to.

  • force – fetch even what is already on disk.

spacr.cli_download.render_listing(plan: Sequence[Piece], chosen: Sequence[Piece], dest, *, force: bool = False) → str[source]

Every piece, its size, whether it is here, and what the total would be.

plan is the whole inventory and chosen is the selection, so the listing answers both “what is there?” and “what would this command do?” at once. A * marks the selected rows.

Parameters:

  • plan – every piece there is.

  • chosen – the pieces this run would fetch.

  • dest – the root everything unpacks under.

  • force – whether pieces already on disk count towards the total.

spacr.cli_download.resolve_selection(what: Sequence[str] = (), *, screen: str | None = None, plates: Sequence[int] = ()) → Tuple[List[spacr.example_archives.ExampleSet], List[spacr.screen_data.ScreenAsset]][source]

Turn what the user typed into the exact list of things to fetch.

Kept apart from argparse so the rules can be read – and tested – as rules rather than as a parser’s side effects.

Parameters:

  • what – names and groups: an example key, examples, screen or all. Empty means the default.

  • screen – measurements, crops, all or None. Naming a kind is itself a request for the screen, so --screen crops needs no positional argument to go with it.

  • plates – which screen plates; empty means all of them.

Returns:

(example sets, screen assets) in listing order.

Raises:

SelectionError – on a name, kind or plate that does not exist. A typo must not quietly select nothing and then report success.

spacr.cli_download.room_for(pieces: Sequence[Piece], dest) → str | None[source]

Return a low-disk-space message, or None when there is enough room.

The peak is not the total: each archive is unpacked and then deleted, so what has to fit at once is everything that will be kept plus the largest single archive still on disk beside its own unpacked copy.

Returns None when the free space cannot be read at all. A filesystem that will not answer is not evidence of a full one, and refusing a download over a failed statvfs would break the command on exactly the network filesystems a cluster user has.

Parameters:

  • pieces – what would be downloaded.

  • dest – where it would go; the nearest existing parent is measured, because the destination itself may not have been made yet.

spacr.cli_download.screen_folder(dest, plate: int) → pathlib.Path[source]

Return the folder for one screen plate: <dest>/screen/plate<N>.

Every plate’s measurements archive unpacks to measurements/measurements.db and every plate’s crop archive unpacks to data/ – the same two paths, four times over. Unpacked into one folder the fourth plate would silently overwrite the third, and spacr.screen_data.ScreenAsset.is_present() would report a plate as downloaded because a different plate’s file is sitting where its own would go. The plate number is therefore part of the path, and each plate folder can be opened on its own.

Parameters:

  • dest – the root everything unpacks under.

  • plate – which screen plate.

spacr.cli_download.screen_notice() → str[source]

Return the notice explaining how to request the screen data.

Printed on the default run, which does not fetch the screen, so the user learns that the screen is published and how to request it.

Nested helpers

_progress.report(written: int, expected: int | None) → None

Redraw the terminal line when the integer percentage changes.

An unknown or zero total cannot yield a useful percentage and is ignored; every emitted update is flushed so it is immediately visible.

spacr/cli_download.py:494