spacr.data_manager

Workflow inputs and outputs

Data Manager

Inspect project files and disk usage. Removing derived artifacts invalidates downstream uses of those files; original images must remain recoverable.

Open: the application’s Help/tools menus.

Inputs and outputs below include conditional alternatives. The guidance and handoff notes say which route applies.

Inputs

• Run history and artifacts — Project run records, settings, output paths, artifact provenance, status and logs.

Outputs

• Run history and artifacts — Project run records, settings, output paths, artifact provenance, status and logs.

API reference.

Module tutorial.

Measure project disk usage, and prune or archive regenerable data.

An imaging screen is mostly intermediates. stack/ is the raw files re-shaped, merged/ is stack/ with the masks concatenated on, data/ is one PNG per object, and a plate that arrives as 40 GB of TIFFs leaves 400 GB behind. This module reports where the space in a project is used, and clears derived data without touching the raw microscope images, which cannot be regenerated.

Deletion is conservative: only artifacts shown to be regenerable are deleted.

Disk usage

scan_project() walks the project once and reports bytes per artifact kind — raw images, channel stacks, merged arrays, masks, crops, databases, model weights — measured from the filesystem, not from what the registry happens to have recorded. Every file is reconciled against spacr.artifacts.Registry:

• a file under a registered artifact’s path is attributed to that artifact’s kind;

• a file under none of them is unregistered — its bytes are reported (and labelled by the project layout, as a guess, so the number is readable) but its provenance is unknown;

• a registered artifact whose path is no longer on disk is reported as missing, because a registry that lists missing files is not safe to prune from.

Pruning

An artifact is prunable only when it is regenerable, which requires all of the following:

1. the registry knows it — anything unregistered is never a candidate;

2. its kind is not an original (ORIGINAL_KINDS): nothing in spacr.ports declares that it produces raw images or sequencing reads, so nothing can make them again;

3. a declared producing module exists for its kind, and the module the registry recorded is one of them;

4. that module’s inputs are complete — every recorded input artifact is still registered and still on disk, and spacr.ports.check_ready() says the module could run in this project right now;

5. it was written by a run that finished (status == "complete");

6. what is on disk still fingerprints to what was registered, so a folder somebody has since dropped a file into is not prunable;

7. it lives inside the project, reached without following a symlink;

8. every artifact sharing its path passes all of the above — three kinds live in measurements.db, and deleting the file for one of them destroys the other two;

9. no other registered artifact sits inside it or around it, because the bytes belong to the innermost one and a plan must not under-report what it deletes.

Anything that fails any of those is kept, and PrunePlan.kept says which rule kept it. Unknown provenance always means keep.

Rule 3 also protects other kinds. ports.producers_of is empty for raw-images and sequencing-reads, which is why originals are safe — and it is empty today for channel-stacks too, so stack/ is reported as unregistered bytes and never offered, even though it is one of the largest intermediates. This changes only when a port declares that it produces them.

Nothing is deleted without a plan. plan_prune() returns exactly what would go and how much it frees; prune() refuses to run unless it is handed PrunePlan.token, a digest over that exact path set and byte total, so a confirmation cannot authorise a deletion other than the one that was shown.

Deleting is count first, delete second, verify. The tree is re-fingerprinted against the plan before anything is removed; the registry write is a count and a write on one predicate, checked for equality and rolled back on any difference. See _verified_write().

Archiving

plan_archive() and archive() move a project, or a subset of it, somewhere else and leave a record: a manifest at the destination, a ledger at the origin, and rows in the destination’s registry carrying the provenance the artifacts arrived with. The registry records where every artifact went.

Public API

scan_project, ProjectUsage, KindUsage, format_usage

Where the space went.

plan_prune, PrunePlan, PruneCandidate, PruneSkip, format_prune_plan, prune, PruneResult

What can safely go, and the deletion that is gated on it.

plan_archive, ArchivePlan, archive, ArchiveResult

Moving it, with a record.

is_prunable

The predicate, on its own, for anything that wants to ask.

Exceptions

ArchiveError	An archive could not be carried out, or could not be verified.
ConfirmationRequired	A destructive call arrived without the plan's confirmation token.
DataManagerError	Base class for the errors this module raises.
PruneAborted	A prune stopped before removing anything, and nothing was removed.
PruneIncomplete	A prune deleted some of the plan and could not finish it.

Classes

ArchiveItem	One top-level entry an archive would move.
ArchivePlan	What an archive would move, and where to.
ArchiveResult	Record of a completed archive.
ArtifactUsage	One registered artifact, measured on disk.
KindUsage	Disk usage of one artifact kind in a project.
ProjectUsage	Disk usage of a project, reconciled against the artifact registry.
PruneCandidate	One thing a prune would delete.
PrunePlan	Exactly what a prune would delete, and what it would leave.
PruneResult	Record of a completed prune.
PruneSkip	One thing that was considered and kept, and why.

Functions

archive(→ ArchiveResult)	Move a project somewhere else and leave a record of where it went.
format_prune_plan(→ str)	Render a PrunePlan as the text shown before a deletion.
format_usage(→ str)	Render a ProjectUsage as a block of text.
human_bytes(→ str)	Format a byte count with a decimal unit, such as "1.4 GB".
is_prunable() → str)	Return "" when artifact may be pruned, else the reason to keep.
plan_archive(→ ArchivePlan)	Plan which entries archiving a project, or part of one, would move.
plan_prune(→ PrunePlan)	Plan which registered artifacts can be deleted; nothing is deleted.
prune(→ PruneResult)	Carry out a plan. Irreversible, and gated on the plan being unchanged.
scan_project(→ ProjectUsage)	Measure a project and reconcile it against the artifact registry.

Module Contents

exception spacr.data_manager.ArchiveError

Bases: DataManagerError

An archive could not be carried out, or could not be verified.

Initialize self. See help(type(self)) for accurate signature.

exception spacr.data_manager.ConfirmationRequired

Bases: DataManagerError

A destructive call arrived without the plan’s confirmation token.

Initialize self. See help(type(self)) for accurate signature.

exception spacr.data_manager.DataManagerError

Bases: Exception

Base class for the errors this module raises.

Initialize self. See help(type(self)) for accurate signature.

exception spacr.data_manager.PruneAborted

Bases: DataManagerError

A prune stopped before removing anything, and nothing was removed.

Raised when the tree no longer matches the plan, or when a registry write changed a different number of rows than the count that gated it. When it is raised, nothing on disk was deleted.

Initialize self. See help(type(self)) for accurate signature.

exception spacr.data_manager.PruneIncomplete

Bases: DataManagerError

A prune deleted some of the plan and could not finish it.

Distinct from PruneAborted, which means nothing was removed; this type means some files were removed.

Initialize self. See help(type(self)) for accurate signature.

class spacr.data_manager.ArchiveItem

One top-level entry an archive would move.

Parameters:

• source – where it is now.

• destination – where it would go.

• size_bytes – its size.

• n_files – files it holds.

• kind – its kind, when an artifact claims it.

• artifact_ids – the registry rows it carries.

class spacr.data_manager.ArchivePlan

What an archive would move, and where to.

Parameters:

• root – the project being archived from.

• destination – the folder it would move into.

• items – the top-level entries that would move.

• total_bytes – how much moves.

• total_files – how many files.

• whole_project – True when every top-level entry is moving.

• token – the confirmation; see PrunePlan.token.

• created_utc – when the plan was made.

__bool__() → bool

True when there is something to move.

class spacr.data_manager.ArchiveResult

Record of a completed archive.

Parameters:

• root – the origin.

• destination – where it went.

• moved – (source, destination) per entry, in the order moved.

• total_bytes – bytes moved.

• manifest_path – the manifest written at the destination.

• ledger_path – the record left at the origin.

• registered – artifacts registered in the destination’s registry.

• finished_utc – when it finished.

class spacr.data_manager.ArtifactUsage

One registered artifact, measured on disk.

Parameters:

• path – the file or folder.

• kinds – every kind registered at this path, ranked.

• artifacts – the registry rows at this path, newest first.

• size_bytes – bytes actually there now.

• n_files – files actually there now.

• exists – whether anything is there at all.

property kind: str

The kind this path is reported under.

class spacr.data_manager.KindUsage

Disk usage of one artifact kind in a project.

Parameters:

• kind – a spacr.ports kind, or OTHER_KIND.

• label – the display name.

• size_bytes – bytes on disk, measured by walking the project.

• n_files – files counted.

• n_paths – distinct artifact paths (a folder counts once).

• n_artifacts – registry rows of this kind.

• registered_bytes – of size_bytes, how much sits under a registered artifact.

• unregistered_bytes – the rest — bytes under no registered artifact.

• recorded_bytes – what the registry says those artifacts weigh. A gap against size_bytes means the folder changed since it was registered, which is exactly when a prune must decline.

• shared_paths – artifacts of this kind sitting at a path whose bytes are counted under a different kind — the three kinds inside measurements.db. Reported so a kind showing zero bytes reads as “counted next door” rather than “gone”.

property drifted: bool

True when disk and registry disagree about this kind’s size.

class spacr.data_manager.ProjectUsage

Disk usage of a project, reconciled against the artifact registry.

Parameters:

• root – the project root.

• total_bytes – every byte under root, symlinks excluded.

• total_files – every file under root, symlinks excluded.

• kinds – per-kind breakdown, largest first.

• artifacts – one entry per registered artifact path.

• unregistered – (path, size_bytes) for the largest unregistered top-level entries, largest first — the bytes no artifact claims.

• unregistered_bytes – their total.

• unregistered_files – how many files that is.

• missing – artifacts the registry has whose path is gone.

• outside – registered artifacts whose path is not under root.

• symlinks – links found and not followed. Never counted, never deleted: a link can point outside the project, where a recursive delete must not reach.

• errors – paths that could not be read, as strings.

• scanned_utc – when the walk ran.

__str__() → str

The full report; see format_usage().

artifact_at(path: str) → ArtifactUsage | None

Return the entry for one registered path, or None.

Parameters:

path – registered artifact path to resolve absolutely and look up.

kind(kind: str) → KindUsage

Return the row for one kind, zeroed when it is absent.

Parameters:

kind – artifact-kind key to look up.

property registered_bytes: int

Bytes sitting under an artifact the registry knows about.

class spacr.data_manager.PruneCandidate

One thing a prune would delete.

Parameters:

• path – the file or folder that would go.

• kind – its kind.

• module – the module that would make it again.

• artifact_ids – every registry row at this path.

• size_bytes – what deleting it frees, measured now.

• n_files – files it holds.

• inventory_digest – the content fingerprint the plan was made against. prune() re-computes it and refuses on any difference, so a change to the tree since planning stops the delete.

• sample_files – up to twenty paths, for showing a user what this is.

• downstream – artifact ids derived from this one — results that would no longer be reproducible from what is left.

• regenerate_with – the sentence telling a user how to get it back.

property label: str

The kind’s display name.

class spacr.data_manager.PrunePlan

Exactly what a prune would delete, and what it would leave.

Produced by plan_prune() and consumed by prune(). Nothing else may be deleted: prune() takes a plan, not a folder.

Parameters:

• root – the project.

• candidates – what would go, largest first.

• kept – what was considered and kept, with the rule that kept it.

• total_bytes – what the prune frees.

• total_files – how many files that is.

• kinds – the kinds this plan was asked for.

• token – the confirmation. prune() refuses without it, and it is a digest over the exact path set and byte total, so a token taken from one plan cannot authorise a different one.

• unregistered_bytes – bytes in the project that no artifact claims — reported here because they are the reason a prune frees less than a user expected, and they are never candidates.

• created_utc – when the plan was made.

__bool__() → bool

True when there is something to delete.

__str__() → str

The full report; see format_prune_plan().

file_list() → Tuple[Tuple[str, ...], bool]

Enumerate every file this plan would delete, right now.

Walked fresh rather than stored: a plan for a project with millions of crops must not carry millions of strings, and the list a user is shown should be the list that is on disk when they look at it.

Returns:

(paths, truncated). truncated is True when the plan holds more than MAX_RECORDED_FILES files and the list was cut short.

property paths: Tuple[str, ...]

Every path this plan would delete, in plan order.

class spacr.data_manager.PruneResult

Record of a completed prune.

Parameters:

• root – the project.

• removed_paths – the artifact paths deleted, in plan order.

• removed_files – every file removed. Empty with files_truncated when the plan held more than MAX_RECORDED_FILES.

• files_truncated – the list was too long to keep.

• freed_bytes – bytes freed, as counted by the plan.

• n_files – files removed.

• registry_rows – registry rows marked or forgotten.

• forgotten – whether those rows were deleted rather than marked.

• finished_utc – when it finished.

class spacr.data_manager.PruneSkip

One thing that was considered and kept, and why.

Parameters:

• path – the file or folder.

• kind – its kind, as far as anything knows.

• size_bytes – what keeping it costs.

• reason – the rule that kept it, in a sentence a user can act on.

• artifact_id – the registry row, when there was one.

spacr.data_manager.archive(plan: ArchivePlan, *, confirm: str, registry: spacr.artifacts.Registry | None = None) → ArchiveResult

Move a project somewhere else and leave a record of where it went.

Three records are left, because one of them may move with the data:

• a manifest at the destination naming the origin, the time, the spaCR version and every artifact that arrived, with its provenance;

• a ledger at the origin — appended to, never overwritten — so a folder somebody finds nearly empty still says where its contents are;

• registry rows at the destination, one per artifact, carrying the module, settings hash and inputs the artifact arrived with plus extra['archived_from']. The destination is self-describing, and spacr.artifacts.by_project() on it answers.

The origin’s own rows are marked archived_to — through the same counted, verified write a prune uses — unless the registry file is itself moving, in which case the marks would travel with it and say the wrong thing.

Nothing is overwritten: a destination entry that already exists stops the call before anything moves.

Parameters:

• plan – from plan_archive().

• confirm – ArchivePlan.token.

• registry – the origin’s registry; opened from the root when omitted.

Returns:

an ArchiveResult.

Raises:

• ConfirmationRequired – without the right token.

• ArchiveError – when a destination entry exists, or when a move cannot be verified afterwards.

spacr.data_manager.format_prune_plan(plan: PrunePlan, *, limit: int = 20) → str

Render a PrunePlan as the text shown before a deletion.

Parameters:

• plan – the plan.

• limit – how many kept entries to explain.

spacr.data_manager.format_usage(usage: ProjectUsage, *, limit: int = 8) → str

Render a ProjectUsage as a block of text.

Parameters:

• usage – the result of scan_project().

• limit – how many unregistered paths to list.

spacr.data_manager.human_bytes(size: float) → str

Format a byte count with a decimal unit, such as "1.4 GB".

Parameters:

size – bytes.

Returns:

e.g. "1.4 GB". Powers of 1000, because that is what the disk vendor, df and the user’s storage quota all use.

spacr.data_manager.is_prunable(artifact: spacr.artifacts.Artifact, *, root: Any, registry: spacr.artifacts.Registry | None = None, group: Sequence[spacr.artifacts.Artifact] = ()) → str

Return "" when artifact may be pruned, else the reason to keep.

An empty string is the only value that means “delete this”. Every failure mode, including an unanticipated one, produces a sentence, so the artifact is kept.

Parameters:

• artifact – a registered artifact. An unregistered path cannot be passed here.

• root – the project root it must live inside.

• registry – the registry, for checking that its inputs survive.

• group – every artifact registered at the same path. A path is prunable only when all of them are: measurements.db carries three kinds, and deleting the file for one destroys the other two.

Returns:

"" or a reason.

spacr.data_manager.plan_archive(root: Any, destination: Any, *, registry: spacr.artifacts.Registry | None = None, paths: Sequence[str] | None = None, usage: ProjectUsage | None = None) → ArchivePlan

Plan which entries archiving a project, or part of one, would move.

Entries are top-level: a whole project archive moves every child of the root, and a subset archive moves the paths named. Moving a file out of a registered folder is not offered, so a registered folder is never split between two places.

Parameters:

• root – the project root.

• destination – the folder to move into. It must not already hold an entry with the same name; an archive never overwrites.

• registry – an open registry, for the provenance carried along.

• paths – entries to move. Defaults to everything under root.

• usage – a ProjectUsage, to save a second walk.

Returns:

an ArchivePlan.

Raises:

DataManagerError – when root is not a folder, when a named path is not inside it, or when the destination is inside the root.

spacr.data_manager.plan_prune(root: Any, *, registry: spacr.artifacts.Registry | None = None, kinds: Sequence[str] | None = None, usage: ProjectUsage | None = None, paths: Sequence[str] | None = None) → PrunePlan

Plan which registered artifacts can be deleted; nothing is deleted.

Every registered artifact in the project is put through is_prunable(). What passes becomes a candidate; what does not becomes a PruneSkip carrying the rule that kept it, so a user who expected to free 300 GB and was offered 12 GB can read why.

Unregistered bytes are never candidates. They are counted and reported (PrunePlan.unregistered_bytes), because “spaCR does not know what made this” is information, but the answer for them is always keep.

Parameters:

• root – the project root.

• registry – an open registry; opened from root when omitted. A project with no registry produces an empty plan, because nothing there has known provenance.

• kinds – kinds to consider. Defaults to DEFAULT_PRUNABLE_KINDS. Naming a PROTECTED_KINDS member opts it in — it still has to pass every safety rule. Naming an ORIGINAL_KINDS member does nothing: there is no path through this module that deletes an original.

• usage – a ProjectUsage from scan_project(), to save a second walk of a large project.

• paths – restrict to these artifact paths.

Returns:

a PrunePlan.

spacr.data_manager.prune(plan: PrunePlan, *, confirm: str, registry: spacr.artifacts.Registry | None = None, forget_rows: bool = False) → PruneResult

Carry out a plan. Irreversible, and gated on the plan being unchanged.

The steps run in this order:

1. the confirmation must equal PrunePlan.token, a digest over the plan’s exact paths and byte total, so a token cannot authorise a deletion other than the one it was shown for;

2. every candidate is re-fingerprinted and must still match the plan. Any difference — a file added, a file changed, the folder gone — aborts the whole call before a single delete;

3. the registry write happens next, inside one transaction, counted and verified (see _verified_write()). A mismatch rolls it back and raises with nothing on disk deleted;

4. only then are the files removed;

5. every path is checked to be gone afterwards.

Step 3 commits before step 4 deliberately. A crash between them leaves the registry saying an artifact was pruned while its files are still there — recoverable by running the prune again. The other order leaves files deleted that the registry still describes as present, which cannot be recovered from.

Parameters:

• plan – from plan_prune().

• confirm – PrunePlan.token.

• registry – the project registry; opened from the plan’s root when omitted.

• forget_rows – delete the registry rows instead of marking them. Off by default: the row is the recipe for making the data again.

Returns:

a PruneResult.

Raises:

• ConfirmationRequired – without the right token.

• PruneAborted – when the tree changed, or a count and its write disagreed. Nothing was deleted in either case.

• PruneIncomplete – when a path survived its own deletion — a permission, a busy file. Some of the plan did go, which is why this is a different type from PruneAborted.

spacr.data_manager.scan_project(root: Any, *, registry: spacr.artifacts.Registry | None = None) → ProjectUsage

Measure a project and reconcile it against the artifact registry.

One walk of the tree, then every file is attributed to the registered artifact whose path contains it — longest path wins — or to nobody. The per-kind numbers come from the filesystem; the registry’s own size_bytes is reported alongside so drift between the two is visible rather than assumed away.

Parameters:

• root – the project root.

• registry – an open spacr.artifacts.Registry. Omit and one is opened for root when a registry file exists — including the shared one spacr.artifacts.ARTIFACTS_DB_ENV points at. A project with no registry is scanned anyway, and reports every byte as unregistered, so nothing in it is prunable.

Returns:

a ProjectUsage.

Raises:

DataManagerError – when root is not a directory.

Nested helpers

_walk_project._note(exc: OSError) → None

Record an inaccessible walk entry without stopping the scan.

spacr/data_manager.py:535

scan_project._bucket(kind: str) → Dict[str, int]

Return the shared zero-initialized counters for kind.

spacr/data_manager.py:648