spacr.scorecard

Score segmentations and classifiers against ground-truth labels.

Every published model carries a table comparing the finetuned model with the vanilla one on a held-out set, with all the common metrics for that model type. Two related modules measure something else:

• spacr.model_compare compares two models to EACH OTHER and says so in its own docstring – “neither model is ground truth”. It answers “do these two disagree”, which is a different and also useful question.

• spacr.seg_qc scores a field with NO labels at all: it flags splits, merges and implausible diameters from the mask alone.

Neither computes Dice or any other score against a hand-drawn truth mask. This module does, so a published number can state whether a model is more accurate, not only whether it differs from another model.

Matching predicted objects to labelled ones

Every number here rests on one choice: when is a predicted object THE SAME object as a labelled one? The literature’s answer is an IoU threshold, and the threshold is not a detail – a model that finds every cell and outlines them loosely scores well at 0.5 and badly at 0.9, and a model that finds half of them perfectly does the opposite. That is why average_precision() reports the whole sweep from 0.5 to 0.9 rather than one number, and why every precision/recall figure states the threshold it was matched at.

Matching is one-to-one and optimal, not greedy. A greedy pass down the IoU matrix is cheaper and gives a different answer when one prediction overlaps two truths: it takes the first pair it sees rather than the assignment that maximises total overlap, so the score depends on label ORDER, which is an implementation detail of whoever wrote the mask. match_objects() uses scipy.optimize.linear_sum_assignment, so relabelling either mask cannot change the result.

Dependencies and scope

The module does not import torch or anything that imports it. The Model Zoo imports without torch and a test asserts it, so reading a scorecard does not load a GPU stack. At module scope only numpy is imported; scipy, tifffile and matplotlib are imported inside the functions that need them.

Nothing here uploads or reads a model catalogue. The functions turn label arrays and scores into numbers and CSV text; scorecard_figure() is the only one that writes a file, and publishing the results is left to the caller.

Classes

HoldoutField	One labelled field: the image a model reads, and the truth it is
HoldoutScore	One model's result on one named, versioned held-out set.
HoldoutSet	A named, versioned set of labelled fields, as declared by a manifest.
Match	One IoU threshold's worth of matching, and what it implies.

Functions

average_precision(→ Dict[str, float])	AP at each IoU in the sweep, and the mean across it.
boundary_f1(→ Dict[str, float])	Boundary precision, recall and F1 at each pixel tolerance.
compare_against_baseline(→ Dict[str, Dict[str, float]])	Score two models on the same truth and report the difference.
compare_classifier_against_baseline(→ Dict[str, ...)	Two classifiers on the same labels, and the difference.
counts_and_areas(→ Dict[str, float])	Object-count error (signed) and the area error a measurement inherits.
dice(→ Dict[str, float])	Per-object Dice, averaged over matched pairs, plus the pixel-wise one.
headline(→ List[str])	Return the headline metric lines for a tooltip, then a count line.
iou_matrix(→ Tuple[numpy.ndarray, numpy.ndarray, ...)	(iou, truth ids, pred ids). IoU of every overlapping pair.
load_holdout(→ HoldoutSet)	Read a hold-out manifest.
match_objects(→ Match)	Match predicted objects to labelled ones, one to one and optimally.
read_scorecard_csv(→ Dict[str, Any])	Parse a published scorecard CSV back into an entry's metrics.
score_classifier(→ Dict[str, float])	Compute every classifier metric of the published table at a given threshold.
score_holdout(→ HoldoutScore)	Score a model over a whole held-out set.
score_model_on_holdout(→ HoldoutScore)	Run predict over a hold-out set and score it against the truth.
score_segmentation(→ Dict[str, float])	Compute every segmentation metric of the published table for one field.
scorecard_csv(→ str)	Return the rows as CSV text; the caller decides where to write it.
scorecard_figure(metrics, path, *[, title, dpi])	Draw finetuned against stock as paired bars, and write it to path.
scorecard_is_present(→ bool)	Whether metrics actually carries a scorecard.
scorecard_rows(→ List[Dict[str, object]])	Return the published comparison table, one row per metric.
splits_and_merges(→ Dict[str, int])	How many labelled objects were split, and how many were merged.
verify_holdout(→ List[str])	Check every truth mask is present and matches its digest.

Module Contents

class spacr.scorecard.HoldoutField

One labelled field: the image a model reads, and the truth it is scored against.

Parameters:

• image – path to the field, relative to the manifest.

• truth – path to the label mask, relative to the manifest.

• sha256 – digest of the TRUTH mask, or "".

class spacr.scorecard.HoldoutScore

One model’s result on one named, versioned held-out set.

Parameters:

• name – the set’s name, e.g. toxo_pv_holdout.

• version – the set’s version. Two numbers are comparable only when this matches, and it is carried into every published row for that reason rather than being assumed from context.

• metrics – the pooled scorecard.

• per_field – one scorecard per field, in the order they were given.

property n_fields: int

How many labelled fields this score was computed over.

Returns:

the field count.

class spacr.scorecard.HoldoutSet

A named, versioned set of labelled fields, as declared by a manifest.

Parameters:

• name – stable name, e.g. toxo_pv_holdout.

• version – the version two numbers must share to be comparable.

• fields – the labelled fields, in manifest order.

• root – directory the relative paths resolve against.

path_to(relative: str) → pathlib.Path

Resolve a manifest-relative path against this set’s root.

Manifest paths are relative so that a held-out set can be moved or shared without rewriting the manifest, and the versioned name refers to the same files on every machine.

Parameters:

relative – the path as the manifest spells it.

Returns:

the absolute path.

class spacr.scorecard.Match

One IoU threshold’s worth of matching, and what it implies.

Parameters:

• threshold – the IoU at which a pair counted as the same object.

• pairs – (truth index, pred index) for each matched pair, into the id arrays iou_matrix() returns.

• ious – the IoU of each matched pair, in the same order.

• n_truth – labelled objects present.

• n_pred – predicted objects present.

property average_precision: float

TP / (TP + FP + FN) – the segmentation-benchmark convention.

NOT the area under a precision-recall curve, despite the name. The segmentation literature uses this quantity and calls it AP, and reporting the other thing under the same label is how two papers’ numbers stop being comparable. Named explicitly here for that reason.

property f1: float

Harmonic mean of precision and recall, 0.0 when both are 0.

property false_negatives: int

Truth objects the model did not find.

Returns:

the count.

property false_positives: int

Predicted objects with no truth object to match.

Returns:

the count.

property precision: float

Fraction of predicted objects that matched a truth object.

Returns:

TP / n_pred, or 0 when the model predicted nothing.

property recall: float

Fraction of truth objects that a predicted object matched.

Returns:

TP / n_truth, or 0 when there was nothing to find.

property true_positives: int

Predicted objects that matched a truth object at this threshold.

Returns:

the count.

spacr.scorecard.average_precision(truth: numpy.ndarray, pred: numpy.ndarray, thresholds: Sequence[float] = DEFAULT_IOU_THRESHOLDS) → Dict[str, float]

AP at each IoU in the sweep, and the mean across it.

Parameters:

• truth – the ground-truth label image, 0 for background and one integer id per object.

• pred – the predicted label image, the same shape as truth.

spacr.scorecard.boundary_f1(truth: numpy.ndarray, pred: numpy.ndarray, tolerances: Sequence[int] = DEFAULT_BOUNDARY_TOLERANCES) → Dict[str, float]

Boundary precision, recall and F1 at each pixel tolerance.

A predicted boundary pixel counts as correct when a true boundary pixel lies within tolerance pixels of it, and vice versa. A tolerance is used because two people labelling the same cell disagree by a pixel or two, so a zero-tolerance boundary score measures the annotator as much as the model.

Parameters:

• truth – the ground-truth label image, 0 for background and one integer id per object.

• pred – the predicted label image, the same shape as truth.

spacr.scorecard.compare_against_baseline(truth: numpy.ndarray, finetuned: numpy.ndarray, vanilla: numpy.ndarray, **kwargs) → Dict[str, Dict[str, float]]

Score two models on the same truth and report the difference.

Parameters:

• truth – the ground-truth label image, 0 for background and one integer id per object.

• finetuned – the finetuned model’s label image for the same field.

• vanilla – the vanilla model’s label image for the same field.

Returns:

{"finetuned": ..., "vanilla": ..., "delta": ...}.

delta holds finetuned - vanilla for every numeric metric except match_iou, so the published comparison does not depend on a reader subtracting two columns by eye.

Both models are scored on the SAME array with the SAME settings, which is the only thing that makes the subtraction meaningful – and is why this takes three masks rather than two scorecards.

spacr.scorecard.compare_classifier_against_baseline(labels: Sequence[int], finetuned: Sequence[float], vanilla: Sequence[float], **kwargs) → Dict[str, Dict[str, float]]

Two classifiers on the same labels, and the difference.

The same shape as compare_against_baseline(), and for the same reason: the delta is what the published table is for. threshold and the counts are dropped from it – a difference in n is not a result, it is a sign the two were scored on different data, which this signature makes impossible.

Parameters:

• labels – ground truth per object, 0 or 1.

• finetuned – the finetuned classifier’s predicted probability of the positive class, one per label.

• vanilla – the vanilla classifier’s predicted probability of the positive class, one per label.

spacr.scorecard.counts_and_areas(truth: numpy.ndarray, pred: numpy.ndarray, threshold: float = 0.5) → Dict[str, float]

Object-count error (signed) and the area error a measurement inherits.

count_error is n_pred - n_truth and is signed because its direction distinguishes the failure: a model that finds too many objects is over-segmenting and one that finds too few is merging or missing, and an absolute count error does not show which.

Parameters:

• truth – the ground-truth label image, 0 for background and one integer id per object.

• pred – the predicted label image, the same shape as truth.

spacr.scorecard.dice(truth: numpy.ndarray, pred: numpy.ndarray, threshold: float = 0.5) → Dict[str, float]

Per-object Dice, averaged over matched pairs, plus the pixel-wise one.

The two numbers measure different things and are often confused. The per-object mean says how well a typical matched object is outlined; the pixel-wise figure ignores object identity and is dominated by the largest objects, so a model that misses ten small cells and outlines one large cell accurately scores well on the pixel-wise figure and badly on the per-object mean.

Parameters:

• truth – the ground-truth label image, 0 for background and one integer id per object.

• pred – the predicted label image, the same shape as truth.

spacr.scorecard.headline(metrics: Mapping[str, object], *, baseline: Mapping[str, object] | None = None, limit: int = 3) → List[str]

Return the headline metric lines for a tooltip, then a count line.

Parameters:

• metrics – a scorecard, or any mapping; unknown keys are ignored.

• baseline – the vanilla model’s scorecard, to show the difference.

• limit – how many metric lines to return before the count line (on N objects, N fields).

Returns:

lines, or [] when the mapping holds no scorecard at all – an entry with free-form metrics is left exactly as it was, because this must not turn somebody’s two-line note into a truncated table.

spacr.scorecard.iou_matrix(truth: numpy.ndarray, pred: numpy.ndarray) → Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]

(iou, truth ids, pred ids). IoU of every overlapping pair.

Parameters:

• truth – the ground-truth label image, 0 for background and one integer id per object.

• pred – the predicted label image, the same shape as truth.

spacr.scorecard.load_holdout(manifest_path) → HoldoutSet

Read a hold-out manifest.

Parameters:

manifest_path – path to the hold-out manifest JSON, which must declare name, version and fields.

Raises:

ValueError – when the manifest lacks a name, a version or any field. All three are refusals rather than defaults, and the version most of all: a set that does not say which version it is cannot be compared against anything, and a default would let one be published as though it could.

spacr.scorecard.match_objects(truth: numpy.ndarray, pred: numpy.ndarray, threshold: float = 0.5) → Match

Match predicted objects to labelled ones, one to one and optimally.

Parameters:

• truth – the ground-truth label image, 0 for background and one integer id per object.

• pred – the predicted label image, the same shape as truth.

• threshold – minimum IoU for a pair to count as the same object.

Uses linear_sum_assignment rather than a greedy pass so the result cannot depend on label order – see this module’s docstring.

spacr.scorecard.read_scorecard_csv(text: str) → Dict[str, Any]

Parse a published scorecard CSV back into an entry’s metrics.

The CSV is the source that the other surfaces render: the Hugging Face artifact, the tooltip, the API page and the Zoo screen all read numbers parsed by this function, so they cannot disagree.

It uses only the standard library csv module, not pandas. The Model Zoo imports without torch and a test asserts it, so browsing models and reading a scorecard work on a machine with neither torch nor cellpose; only re-running an evaluation needs them.

Parameters:

text – the CSV as written by scorecard_csv().

Returns:

{metric: {"finetuned": ..., "vanilla": ..., "delta": ...}} plus holdout and holdout_version when the rows carry them. Empty when the text has no rows.

spacr.scorecard.score_classifier(labels: Sequence[int], scores: Sequence[float], *, threshold: float = 0.5, calibration_bins: int = 10) → Dict[str, float]

Compute every classifier metric of the published table at a given threshold.

Parameters:

• labels – ground truth, 0 or 1.

• scores – predicted probability of the positive class.

• threshold – where a score becomes a positive call. Reported back in the result, because “precision 0.94” without it is not a claim anybody can check or reproduce.

AUROC, AUPRC, Brier and ECE are threshold-FREE and are the numbers that survive a reader disagreeing with the cutoff; everything else moves when the threshold does. Both kinds are here, and the result says which is which by carrying threshold beside them.

spacr.scorecard.score_holdout(pairs: Sequence[Tuple[numpy.ndarray, numpy.ndarray]], *, name: str, version: str, **kwargs) → HoldoutScore

Score a model over a whole held-out set.

Parameters:

• pairs – (truth, prediction) per field.

• name – the held-out set’s name, carried into the result.

• version – the held-out set’s version, carried into the result.

Pooled from the counts, not averaged from the ratios. A field with three objects and a field with three hundred are not equal evidence, and a mean of per-field precisions treats them as though they were – so a model that fails on one sparse field is punished as hard as one that fails on a confluent one. Precision, recall and F1 are recomputed from the summed true and false positives; only the per-object means (IoU, Dice, area error) are averaged, weighted by each field’s matched-object count.

spacr.scorecard.score_model_on_holdout(holdout: HoldoutSet, predict, *, read_mask=None, **kwargs) → HoldoutScore

Run predict over a hold-out set and score it against the truth.

Parameters:

• holdout – the hold-out set whose fields’ images are predicted and whose truth masks are read.

• predict – image path -> label array. Injected rather than imported: this module must keep importing without torch, and a segmentation model is the one thing that cannot.

• read_mask – path -> label array. Defaults to tifffile, which the package already depends on.

The returned HoldoutScore carries the set’s name and version, so a published number always states what it was measured on.

spacr.scorecard.score_segmentation(truth: numpy.ndarray, pred: numpy.ndarray, *, threshold: float = 0.5, iou_thresholds: Sequence[float] = DEFAULT_IOU_THRESHOLDS, boundary_tolerances: Sequence[int] = DEFAULT_BOUNDARY_TOLERANCES) → Dict[str, float]

Compute every segmentation metric of the published table for one field.

Parameters:

• truth – the ground-truth label image, 0 for background and one integer id per object.

• pred – the predicted label image, the same shape as truth.

• threshold – the IoU at which precision, recall, F1, Dice and the area error are matched. Reported in the result as match_iou so a published number can never be read without it.

The result always includes n_truth and n_pred, because a score such as a Dice of 0.91 cannot be interpreted without the number of objects it was computed on.

spacr.scorecard.scorecard_csv(rows: Sequence[Dict[str, object]]) → str

Return the rows as CSV text; the caller decides where to write it.

Parameters:

rows – the table rows, as from scorecard_rows(); the first row’s keys are the CSV columns, and no rows gives "".

spacr.scorecard.scorecard_figure(metrics: Mapping[str, object], path, *, title: str = '', dpi: int | None = None)

Draw finetuned against stock as paired bars, and write it to path.

The scorecard in graph form, for Hugging Face and for the API page, beside the CSV and the table. It reads the same parsed metrics the tooltip and the zoo screen read, so the picture cannot disagree with the numbers printed next to it.

Matplotlib is imported inside the function. The Model Zoo must be able to import this module with neither torch nor cellpose present – a test asserts it – and a drawing dependency at module scope would add matplotlib to every caller that only reads a scorecard.

Only a subset of metrics is drawn, because a scorecard holds about forty numbers and a chart of forty bars is hard to read. The ones in CHART_METRICS that the scorecard actually carries are drawn, in that order, and anything absent is skipped rather than drawn as zero – a missing metric and a metric that scored nothing look identical at a glance and mean opposite things.

Parameters:

• metrics – as read_scorecard_csv() returns.

• path – where to write the PNG.

• title – heading; the model’s name is the useful thing to pass.

Returns:

the path written.

Raises:

ValueError – when none of the chart metrics are present, since an empty chart published beside a model would imply it scored zero.

spacr.scorecard.scorecard_is_present(metrics: Mapping[str, Any]) → bool

Whether metrics actually carries a scorecard.

True when at least one value is a mapping with a non-None "finetuned" entry. Callers use it to show NO_SCORECARD instead of blank cells, as ModelEntry.provenance_known does for training provenance: a model with no numbers is not a model that scored zero.

Parameters:

metrics – an entry’s metrics mapping.

spacr.scorecard.scorecard_rows(finetuned: HoldoutScore, vanilla: HoldoutScore) → List[Dict[str, object]]

Return the published comparison table, one row per metric.

The table has four renderings – the CSV on Hugging Face, the tooltip, the API section and the zoo screen. This is the one place a difference is computed; the other renderings format these rows, so they cannot disagree.

Parameters:

• finetuned – the finetuned model’s score.

• vanilla – the vanilla model’s score, on the same set name and version.

Raises:

ValueError – when the two were scored on different sets (name or version differ). A table whose two columns came from different data looks the same as a valid one, so the mismatch is refused here.

spacr.scorecard.splits_and_merges(truth: numpy.ndarray, pred: numpy.ndarray, minimum_overlap: float = 0.1) → Dict[str, int]

How many labelled objects were split, and how many were merged.

Parameters:

• truth – the ground-truth label image, 0 for background and one integer id per object.

• pred – the predicted label image, the same shape as truth.

• minimum_overlap – fraction of the truth object a prediction must cover to count as overlapping it, so a one-pixel graze is not a split.

The two failures affect measurements differently. A split inflates the object count and halves the areas; a merge deletes an object and doubles one. Both are invisible to Dice at the field level, which is why they are counted separately rather than folded into it.

spacr.scorecard.verify_holdout(holdout: HoldoutSet) → List[str]

Check every truth mask is present and matches its digest.

Parameters:

holdout – the hold-out set whose truth masks are checked on disk.

Returns:

one line per problem; empty when the set is intact.

A digest is optional, but a missing one is reported as a problem. A set published without digests can still be scored, but nobody can then tell whether two people scored the same masks, which is what naming and versioning the set are for.

Nested helpers

read_scorecard_csv.number(value)

A cell as a number, or the raw string when it is not one.

A whole number is returned as an int and an empty cell as None. The CSV cannot distinguish a count from a measurement, and reading everything as float would render “on 12517.0 objects” in a tooltip.

spacr/scorecard.py:1012

score_holdout.weighted(key: str, weight_key: str = 'n_matched') → float

Average key across fields, weighted by how much each holds.

A plain mean would let a field with four objects count as much as one with four hundred, so a sparse corner of the plate could move the published number more than the plate does.

spacr/scorecard.py:674

score_model_on_holdout.read_mask(path)

Read a label image off disk.

The default, used when the caller passes none. Imported here rather than at module scope so scoring an already-loaded pair of arrays needs no tifffile.

spacr/scorecard.py:908