spacr.artifacts¶

The artifact registry: what produced every file in a spaCR project.

A spaCR project is a folder of derived data. merged/ came from raw images, measurements.db came from merged/, the model came from the crops the database indexes, and the hit list came from the model. The files themselves do not record these links, so without a registry a result cannot be checked against its inputs: after Mask is re-run with a different diameter, every downstream number is out of date and nothing marks it.

This module records it. Every output registers with:

the producing module ("mask", "measure", … — the keys spacr.ports and spacr.validate use);
a settings hash over the settings that could change the numbers (spacr.resume.COSMETIC_SETTINGS decides which cannot);
the spaCR version that produced it;
the input artifact ids it was derived from, making the project a DAG;
a timestamp, a path, and a content fingerprint.

Storage is SQLite — artifacts.db in the project root, one row per artifact plus one edge row per input — because two Measure workers, a GUI and a batch runner can all be touching a project at once, and a pickle cannot survive that. Writes go through spacr.database_concurrency.transaction() with an explicit lock budget, which is also why nothing here does its own retry arithmetic.

Public API¶

Registry: The registry for one project: register(), get(), by_kind(), by_project(), latest(), downstream_of(), upstream_of(), is_stale(), forget().
open_registry, and module-level register / by_kind / by_project / latest / downstream_of / is_stale: The same, resolved from a project path.
register_run_outputs: The one call a finished run makes; walks spacr.ports.declared_outputs() and registers what is there.
Artifact, Staleness, Fingerprint: The records that come back.
settings_hash, material_settings, content_fingerprint: The provenance primitives, usable on their own.

Classes¶

`Artifact`	One registered output and everything known about where it came from.
`Fingerprint`	A content fingerprint plus how it was arrived at.
`Registry`	The artifact registry for one project.
`Staleness`	Whether an artifact still matches what it was made from.

Functions¶

`by_kind`(→ List[Artifact])	Every artifact of `kind`; see `Registry.by_kind()`.
`by_project`(→ List[Artifact])	Every artifact in a project; see `Registry.by_project()`.
`content_fingerprint`(→ Fingerprint)	Fingerprint whatever is at `path`: a file, a folder, or nothing.
`downstream_of`(→ List[Artifact])	What an artifact invalidates; see `Registry.downstream_of()`.
`is_stale`(→ Staleness)	Whether an artifact is out of date; see `Registry.is_stale()`.
`latest`(→ Optional[Artifact])	The newest artifact of `kind`; see `Registry.latest()`.
`material_settings`(→ Dict[str, Any])	Return only the settings that can change the numbers.
`open_registry`(→ Registry)	Return the `Registry` for one project.
`register`(→ Artifact)	Register one artifact; see `Registry.register()`.
`register_run_outputs`(→ Tuple[Artifact, ...])	Register everything a finished run declared it would write.
`registry_path`(→ str)	Return the registry file for `project`.
`settings_hash`(→ str)	Return a digest over the material settings of a run.

Module Contents¶

class spacr.artifacts.Artifact[source]¶

One registered output and everything known about where it came from.

Parameters:

artifact_id – 16 hexadecimal characters, derived from the identity below. Registering byte-identical content from the same module, project, role and settings yields the same id, so a repeated registration updates one row rather than growing the table.
project – absolute project root.
kind – a spacr.ports kind, e.g. "measurements-db".
role – the producing module’s port role, e.g. "merged".
path – absolute path of the file or folder.
module – producing module key, e.g. "mask".
run_id – the run this came out of, when the caller knows it.
settings_hash – digest of the material settings; see settings_hash().
spacr_version – the version that produced it.
created_ns – registration time, time.time_ns().
created_utc – the same instant, ISO-8601.
fingerprint – content digest; see content_fingerprint().
fingerprint_method – how that digest was computed.
size_bytes – bytes on disk at registration.
n_files – files covered.
status – STATUS_COMPLETE, STATUS_PARTIAL or STATUS_FAILED.
settings – the material settings, kept so a later run can be diffed against this one rather than only compared by hash.
extra – anything else the producer wanted recorded.
inputs – artifact ids this was derived from.
schema_version – the registry layout that wrote the row.

__str__() → str[source]¶: One line: id, kind, module and path.

to_dict() → Dict[str, Any][source]¶: Return a JSON-serializable copy of the record.

property exists: bool[source]¶: True when the artifact is still on disk where it was registered.

class spacr.artifacts.Fingerprint[source]¶

A content fingerprint plus how it was arrived at.

Parameters:

digest – hexadecimal SHA-256, or "" when nothing was there.
method – "sha256" (a file read end to end), "sampled" (a large file: size plus its first and last megabyte), "tree" (a folder: every file’s relative path, size and mtime) or "missing".
size_bytes – total bytes covered.
n_files – number of files covered; 1 for a single file.

__bool__() → bool[source]¶: True when something was actually fingerprinted.

class spacr.artifacts.Registry(path: str | os.PathLike | None = None, *, project: str | os.PathLike | None = None, timeout: float = 30.0, create: bool = True)[source]¶

The artifact registry for one project.

One SQLite file, opened per operation and closed again: Measure workers, the GUI and a batch runner all register into the same project, and a connection held open across a whole run is a lock held across a whole run. Writes go through spacr.database_concurrency.transaction() with an explicit lock budget, so a second registration arriving mid-write waits rather than failing.

Parameters:

path – the registry file. Defaults to registry_path() for project.
project – the project root recorded on artifacts that do not name one of their own.
timeout – seconds a write may wait on the lock, in total.
create – create the file and its tables when missing. Pass False for a read-only consumer that must not conjure an empty registry.

Raises:

FileNotFoundError – when create is False and there is no registry.
ValueError – when neither path nor project is given.

Open this project’s registry, creating its schema when requested.

Constructor arguments and refusal cases are documented on Registry so the rendered class contract has one source.

all(*, limit: int | None = None) → List[Artifact][source]¶

Return every artifact in this registry, newest first.

Parameters:: limit – cap the number of rows returned.

by_kind(kind: str, *, project: str | None = None, module: str | None = None, limit: int | None = None) → List[Artifact][source]¶

Return every artifact of kind, newest first.

Parameters:

kind – a spacr.ports kind, e.g. "merged-arrays".
project – restrict to one project root.
module – restrict to one producing module.
limit – cap the number of rows returned.

by_project(project: str | None = None, *, kind: str | None = None, module: str | None = None, limit: int | None = None) → List[Artifact][source]¶

Return every artifact belonging to one project, newest first.

Parameters:

project – the project root; defaults to this registry’s own. Pass "" explicitly for “every project in this file”, which is what a shared registry (see ARTIFACTS_DB_ENV) holds.
kind – restrict to one kind.
module – restrict to one producing module.
limit – cap the number of rows returned.

downstream_of(artifact: str | Artifact, *, transitive: bool = True) → List[Artifact][source]¶

Return the artifacts derived from artifact.

Transitive by default: the question a user asks — “what does this invalidate?” — is about everything downstream, not only the immediate children.

Parameters:

artifact – id or Artifact.
transitive – follow the edges all the way down.

Returns:

registered descendants, newest first.

forget(artifact: str | Artifact) → int[source]¶

Delete one artifact row and the edges pointing out of it.

Edges pointing at it are left alone on purpose: a downstream artifact that names a vanished input must keep saying so, which is what makes it report as stale rather than as current.

Parameters:: artifact – id or Artifact.
Returns:: number of artifact rows deleted — 0 or 1.

get(artifact: str | Artifact) → Artifact | None[source]¶

Return one artifact by id, or None when it is not registered.

Parameters:: artifact – id or Artifact.

is_stale(artifact: str | Artifact, *, settings: Mapping[str, Any] | None = None) → Staleness[source]¶

Answer whether an upstream artifact or setting changed since this.

An artifact is stale when any of these hold:

one of its recorded inputs is no longer in the registry;
an input was registered again after this artifact was;
a newer artifact of the same kind now sits at an input’s path — the shape “Mask was re-run with a different diameter” takes;
an input is itself stale, transitively;
settings is supplied and its material hash differs from the one recorded here.

Parameters:

artifact – id or Artifact.
settings – the settings a caller is about to use, compared against the ones that produced the artifact.

Returns:

a Staleness; bool(result) is the answer and Staleness.reasons is what to show a user.

Return the most recent artifact matching the filters, or None.

The call auto-chaining makes: “what is the current measurements-db for this project?”.

Parameters:

kind – a spacr.ports kind.
project – restrict to one project root.
module – restrict to one producing module.
role – restrict to one port role.
path – restrict to one exact path.

register(*, module: str, kind: str, path: str | os.PathLike, role: str = '', project: str | os.PathLike | None = None, settings: Mapping[str, Any] | None = None, settings_digest: str = '', inputs: Sequence[str | Artifact] = (), run_id: str = '', status: str = STATUS_COMPLETE, extra: Mapping[str, Any] | None = None, fingerprint: Fingerprint | None = None) → Artifact[source]¶

Record one output and what it was made from.

The id is derived from the identity — project, module, kind, role, path, settings hash, content fingerprint — so registering the same content twice updates the existing row (refreshing its timestamp, run id and inputs) instead of adding a duplicate. Registering different content, or the same content under different settings, creates a new row, which is exactly what makes the older downstream artifacts stale.

Parameters:

module – producing module key, e.g. "mask".
kind – a spacr.ports kind.
path – the file or folder produced.
role – the module’s port role for this output.
project – project root; defaults to the registry’s own.
settings – the run’s settings. Only the material ones are hashed and stored.
settings_digest – use this digest instead of hashing settings — for a caller that already computed one.
inputs – artifact ids (or Artifact objects) this was derived from.
run_id – the run this came out of.
status – STATUS_COMPLETE, STATUS_PARTIAL or STATUS_FAILED.
extra – any additional JSON-safe provenance.
fingerprint – a precomputed Fingerprint; omit to compute one from path.

Returns:

the stored Artifact.

Raises:

ValueError – when module, kind or path is empty.

upstream_of(artifact: str | Artifact, *, transitive: bool = False) → List[Artifact][source]¶

Return the artifacts artifact was derived from.

Parameters:

artifact – id or Artifact.
transitive – follow inputs of inputs, to the roots of the DAG.

Returns:

registered ancestors, newest first. Input ids that are no longer registered are absent — is_stale() is what reports them.

class spacr.artifacts.Staleness[source]¶

Whether an artifact still matches what it was made from.

Stale means an upstream artifact or a material setting changed after this was produced. A deleted file is reported by missing instead — that is an availability problem, not a provenance one, and conflating the two hides both.

Parameters:

artifact_id – the artifact asked about.
stale – the answer. bool(staleness) is the same value.
reasons – human-readable sentences, one per finding.
causes – machine codes for the same findings, e.g. "upstream-newer".
missing – the artifact’s own path is no longer on disk.

__bool__() → bool[source]¶: True when the artifact is stale.

__str__() → str[source]¶: A one-line verdict with its reasons.

spacr.artifacts.by_kind(kind: str, *, project: str | os.PathLike | None = None, registry: Registry | None = None, **kwargs: Any) → List[Artifact][source]¶

Every artifact of kind; see Registry.by_kind().

Parameters:

kind – a spacr.ports kind.
project – the project root, used only to locate the registry file. It is not forwarded as a filter, so a shared registry (see ARTIFACTS_DB_ENV) also returns other projects’ artifacts.
registry – an open registry to use instead of opening one.
kwargs – passed through to Registry.by_kind().

spacr.artifacts.by_project(project: str | os.PathLike | None = None, *, registry: Registry | None = None, **kwargs: Any) → List[Artifact][source]¶

Every artifact in a project; see Registry.by_project().

Parameters:

project – the project root.
registry – an open registry to use instead of opening one.
kwargs – passed through to Registry.by_project().

spacr.artifacts.content_fingerprint(path: str | os.PathLike, *, full_hash_limit: int = FULL_HASH_LIMIT_BYTES) → Fingerprint[source]¶

Fingerprint whatever is at path: a file, a folder, or nothing.

A regular file is hashed end to end while it is small enough to be worth it, and above full_hash_limit from its size plus its first and last megabyte — enough to notice a rewritten database without spending minutes of I/O at the end of every run. A folder is fingerprinted from its file inventory: each file’s relative path, size and modification time, in sorted order. Symlinks are skipped — linked files are ignored and linked subfolders are not descended into — so a folder of links fingerprints as empty.

Parameters:

path – file or folder.
full_hash_limit – byte size above which a file is sampled instead of read end to end.

Returns:

a Fingerprint; method="missing" when nothing is there.

spacr.artifacts.downstream_of(artifact: str | Artifact, *, project: str | os.PathLike | None = None, registry: Registry | None = None, **kwargs: Any) → List[Artifact][source]¶

What an artifact invalidates; see Registry.downstream_of().

Parameters:

artifact – id or Artifact.
project – the project root.
registry – an open registry to use instead of opening one.
kwargs – passed through to Registry.downstream_of().

spacr.artifacts.is_stale(artifact: str | Artifact, *, project: str | os.PathLike | None = None, registry: Registry | None = None, **kwargs: Any) → Staleness[source]¶

Whether an artifact is out of date; see Registry.is_stale().

Parameters:

artifact – id or Artifact.
project – the project root.
registry – an open registry to use instead of opening one.
kwargs – passed through to Registry.is_stale().

spacr.artifacts.latest(kind: str, *, project: str | os.PathLike | None = None, registry: Registry | None = None, **kwargs: Any) → Artifact | None[source]¶

The newest artifact of kind; see Registry.latest().

Parameters:

kind – a spacr.ports kind.
project – the project root, used only to locate the registry file. It is not forwarded as a filter, so a shared registry (see ARTIFACTS_DB_ENV) may return another project’s artifact.
registry – an open registry to use instead of opening one.
kwargs – passed through to Registry.latest().

spacr.artifacts.material_settings(settings: Mapping[str, Any] | None) → Dict[str, Any][source]¶

Return only the settings that can change the numbers.

Verbosity, worker counts, plot cosmetics and the environment snapshot are dropped, using the same deny-list (spacr.resume.COSMETIC_SETTINGS, and spacr.resume.ENV_SETTINGS) that decides whether a resume is allowed. Any key nobody has classified counts as material, so a new knob is conservatively assumed to matter.

Parameters:: settings – a settings dict, or None.
Returns:: a new dict with the inconsequential keys removed.

spacr.artifacts.open_registry(project: str | os.PathLike | None = None, *, path: str | os.PathLike | None = None, create: bool = True) → Registry[source]¶

Return the Registry for one project.

Parameters:

project – the project root.
path – an explicit registry file, overriding project’s default.
create – create the file and tables when missing.

spacr.artifacts.register(*, registry: Registry | None = None, project: str | os.PathLike | None = None, **kwargs: Any) → Artifact[source]¶

Parameters:

registry – an open registry to use instead of opening one.
project – the project root, used to find the registry and recorded on the artifact.
kwargs – passed through to Registry.register().

spacr.artifacts.register_run_outputs(module: str, settings: Mapping[str, Any] | None = None, *, roots: Iterable[Any] | None = None, run_id: str = '', status: str = STATUS_COMPLETE, inputs: Sequence[str | Artifact] | None = None, registry: Registry | None = None, strict: bool = True) → Tuple[Artifact, ...][source]¶

The one call a pipeline entry point makes on completion. For each project root it walks spacr.ports.declared_outputs(), registers every produced port that is actually on disk, and links each one to the artifacts currently sitting at the module’s input ports — which is what turns a folder of files into a provenance DAG.

Parameters:

module – producing module key, e.g. "mask".
settings – the settings the run used; hashed into every artifact.
roots – project roots to record. Defaults to the one spacr.ports.project_root() derives from settings. A multi-plate run passes its whole src list.
run_id – the run this came out of.
status – STATUS_COMPLETE, STATUS_PARTIAL or STATUS_FAILED — a run that lost fields should say so.
inputs – explicit input artifact ids, overriding the lookup.
registry – an open registry to use for every root, instead of one per project.
strict – raise on failure. Pipelines pass False: a registry that cannot be written is worth one printed line, never a lost run.

Returns:

the artifacts registered, in declaration order.

spacr.artifacts.registry_path(project: str | os.PathLike | None = None) → str[source]¶

Return the registry file for project.

Parameters:: project – the project root. Ignored when ARTIFACTS_DB_ENV is set, which points every project at one shared registry.
Returns:: an absolute path. The file need not exist yet.
Raises:: ValueError – when no project is given and no override is set.

spacr.artifacts.settings_hash(settings: Mapping[str, Any] | None) → str[source]¶

Return a digest over the material settings of a run.

Two runs with the same digest do not differ in any material setting, so a downstream artifact whose recorded digest still matches the current settings is not stale on their account.

Parameters:: settings – a settings dict, or None.
Returns:: a lowercase SHA-256 hex digest (of {} when there are none).