spacr.agreement¶

Workflow inputs and outputs¶

Annotator Agreement¶

Compare independent annotation columns and inspect discordant objects; agreement is separate from classification accuracy.

Open: Annotate → Annotator Agreement.

Inputs and outputs below include conditional alternatives. The guidance and handoff notes say which route applies.

Inputs

Training annotations — A chosen annotation column in measurements/measurements.db, table png_list; labels belong to object identities. Relevant tables, depending on the route: png_list. Relevant columns, depending on the route: prcfo.
Object crops — data/**/*_png when save_png is enabled; png_list indexes saved crops. Supported workflows can instead stream crops from merged arrays and masks. Relevant tables, depending on the route: png_list. Relevant columns, depending on the route: png_path, prcfo.

Outputs

Quality-control results — Stored project checks and QC reports; a missing check is not a passing result.
Figures and table exports — The output location chosen by the tool; exports describe the selected data and filters.

Before this module

Annotate: Choose independent annotation columns for the same objects.

API reference.

Module tutorial.

Multi-annotator agreement for spaCR annotation columns.

This module measures how far two or more annotators agree on the same crops and lists the crops on which they disagree. It reads only the png_list table of a measurements.db.

Annotation columns are the ones the Annotate app writes: an INTEGER column added to png_list with ALTER TABLE … ADD COLUMN (see spacr.qt.annotate_engine.ensure_annotation_column()), holding the class the annotator pressed — 1 for a left click, 2 for a right click — and NULL for every crop they have not looked at yet. Two annotators means two such columns over the same png_path rows.

Public API¶

cohens_kappa(a, b): κ for one pair of label vectors.
fleiss_kappa(matrix): κ for three or more annotators, from a subjects × categories count matrix.
agreement_report(db_path, columns): the whole picture for a database: per-pair κ, overall κ, per-class κ, confusion matrices, and the counts that make them interpretable.
disagreements(db_path, columns): exactly the rows the annotators labelled differently, for review.
format_agreement(report): the report as text.

Statistical conventions¶

An unlabelled cell is an abstention, not a disagreement. Every pair is scored on the rows both annotators labelled (pairwise complete cases). Rows where only one of them committed are counted and reported separately as n_abstained. Treating them as disagreements would silently deflate κ in proportion to how far behind the slower annotator is — which is a property of the calendar, not of the annotation.

κ has no value when there is no variance. κ = (pₒ − pₑ)/(1 − pₑ). If both annotators put every compared row in the same single class, pₑ = 1 and the denominator vanishes: the answer is undefined, not 1.0. If just one annotator used a single class, pₑ collapses onto the other’s marginal and κ is identically 0 no matter how well they agree. Both cases return nan with an explanation attached, because on a screen where 98 % of cells are negative they are the normal case, and a returned 0.0 or 1.0 there would misstate the agreement.

Raw agreement is reported next to κ, always. 95 % agreement with κ ≈ 0 is the prevalence paradox (Feinstein & Cicchetti, 1990), not a broken annotator: when one class dominates, chance agreement is already almost as high as the observed agreement, so κ has almost no room left.

The interpretation bands are a convention. Landis & Koch (1977) “slight/fair/moderate/substantial/almost perfect” is a rule of thumb with no distributional basis. It is reported as a label, and labelled as a convention.

Nothing here imports torch, cellpose or any GPU stack — pandas, numpy and the standard library only — so the Qt screen can compute agreement without waking a 4-second import chain.

Classes¶

`AgreementReport`	Result of `agreement_report()`: per-pair and overall κ with counts.
`PairAgreement`	Cohen's κ for one annotator pair, with its counts and caveats.

Functions¶

`agreement_report`(, labels)	Score how well two or more annotation columns agree.
`annotation_columns`(→ List[str])	Guess which columns of `table` hold human annotations.
`cohens_kappa`() → float)	Cohen's κ between two annotators' labels.
`confusion_matrix`() → pandas.DataFrame)	Return the `a` × `b` contingency table over rows both labelled.
`disagreements`(, complete_only, limit)	Return the rows the annotators labelled differently.
`fleiss_kappa`(→ float)	Fleiss' κ from a subjects × categories count matrix.
`format_agreement`(→ str)	Render an `AgreementReport` as a plain-text block.
`interpret_kappa`(→ str)	Return the Landis & Koch band for `kappa` (a convention).
`kappa_detail`(, name_a, name_b)	Compute Cohen's κ for `a` vs `b` with counts and confusion matrix.
`load_annotations`() → pandas.DataFrame)	Read `key` + `columns` from `table` into a normalised frame.
`table_columns`(→ List[str])	Return the column names of `table`, in declaration order.

Module Contents¶

class spacr.agreement.AgreementReport[source]¶

Result of agreement_report(): per-pair and overall κ with counts.

Parameters:

db_path – path of the source annotation database.
table – source table that holds the annotation columns.
key – column that identifies each annotated row.
columns – annotator columns, in report order.
pairs – one PairAgreement per unordered column pair.
overall_kappa – Cohen’s κ for two annotators, Fleiss’ κ for three or more (computed on rows every annotator labelled).
overall_method – name of the κ statistic used for the overall value.
overall_note – interpretive caveat or reason the overall value is undefined; empty when no caveat applies.
interpretation – Landis–Koch convention label for overall_kappa.
labels – ordered class universe used throughout the report.
per_class – one row per class — its one-vs-rest κ, how often the annotators were unanimous on it, and its prevalence. This is where “we agree on the negatives, we argue about the positives” shows up.
n_rows – total annotation-table rows examined.
n_complete – rows every annotator labelled.
n_partial – rows some but not all labelled — abstentions, not disagreements.
n_unlabelled – rows none of the annotators labelled.
n_disagreements – rows where two annotators who both committed chose differently. This is the review queue’s length.
percent_agreement – fraction of complete rows with unanimous labels.
convention – named interpretation scale applied to κ values.
warnings – report-level caveats that must be shown to the reader.

kappa_table() → pandas.DataFrame[source]¶: The per-pair numbers as a DataFrame, ready to render.

pair(a: str, b: str) → PairAgreement | None[source]¶

Return the PairAgreement for two columns, either order.

Parameters:

a – name of either annotator column in the pair.
b – name of the other annotator column in the pair.

property defined: bool[source]¶: False when the overall κ is nan.

property n_annotators: int[source]¶: Return the number of annotation columns included in the report.

class spacr.agreement.PairAgreement[source]¶

Cohen’s κ for one annotator pair, with its counts and caveats.

Parameters:

column_a – name of the first annotator column (the confusion-matrix row axis).
column_b – name of the second annotator column (the confusion-matrix column axis).
kappa – Cohen’s κ, or nan when it is undefined/degenerate — check defined before quoting it.
percent_agreement – raw pₒ, the fraction of compared rows where the two labels are identical. Always meaningful, even when κ is not.
expected_agreement – pₑ, agreement expected from the marginals alone.
n_compared – rows both annotators labelled — κ’s denominator.
n_agree – compared rows on which the two annotators agreed.
n_disagree – compared rows on which the two annotators disagreed.
n_abstained – rows exactly one of them labelled. Excluded from κ (an abstention is not a disagreement) and reported here instead.
n_neither – rows neither of them has reached yet.
labels – ordered class universe used for the confusion matrix.
confusion – a labels down the rows, b across the columns.
note – why κ is nan, or what to watch out for when it is not.
interpretation – convention label that explains the κ magnitude.

__str__() → str[source]¶

Return a readable one-line summary of this annotator pair.

The line includes both column names, signed three-decimal κ (or undefined), its interpretation, raw agreement, and compared-row count.

property defined: bool[source]¶: False when κ is nan — i.e. the data cannot support a κ.

spacr.agreement.agreement_report(db_path: str, columns: Sequence[str], table: str = PNG_TABLE, key: str = PNG_KEY, missing_values: Sequence[Any] = (), labels: Sequence[Any] | None = None) → AgreementReport[source]¶

Score how well two or more annotation columns agree.

Each pair is scored on the rows both of its annotators labelled. The overall κ for three or more annotators is Fleiss’ κ over the rows every annotator labelled, which is stricter — the report carries n_complete and n_partial so the difference is visible rather than silent.

Parameters:

db_path – path to measurements.db; opened read-only.
columns – two or more annotation columns of table.
table – table holding them (default png_list).
key – row key (default png_path).
missing_values – extra values to treat as abstentions, e.g. (0,) for legacy databases where 0 meant “not looked at”.
labels – optional fixed label universe.

Returns:

an AgreementReport.

Raises:

ValueError – for fewer than two distinct columns, or an unknown table/column.

spacr.agreement.annotation_columns(db_path: str, table: str = PNG_TABLE, key: str = PNG_KEY, max_classes: int = 20, min_labelled: int = 1, include_model_columns: bool = False) → List[str][source]¶

Guess which columns of table hold human annotations.

The Annotate app adds a plain INTEGER column per annotation pass, so an annotation column is one that is not part of the crop metadata, not written by a model, holds few distinct values, and has at least one non-NULL.

Model-written columns are excluded. A classifier writes into this same table — pred/cv_predictions from the CV stage, predictions/ml_pred from the ML one — and its class column is indistinguishable by shape from an annotation pass. Offering it made agreement_report score the classifier as a third annotator, which is a different question with the same units: on a real database, four “annotators” gave κ = -0.004 where the two humans agree at 0.471. See _MODEL_COLUMNS.

Parameters:

db_path – path to measurements.db.
table – table to inspect (default png_list).
key – row key, always excluded.
max_classes – reject columns with more distinct values than this — a continuous measurement is not an annotation.
min_labelled – reject columns with fewer labelled rows.
include_model_columns – offer the model’s own columns too. For the deliberate question “how well does the classifier agree with the annotators?”, which is model validation, not inter-annotator agreement. Off by default because it is never the question somebody means when they ask for agreement between annotators.

Returns:

candidate column names, in table order.

spacr.agreement.cohens_kappa(a: Sequence[Any], b: Sequence[Any], labels: Sequence[Any] | None = None, missing_values: Sequence[Any] = ()) → float[source]¶

Cohen’s κ between two annotators’ labels.

Rows either annotator left unlabelled are dropped (abstention, not disagreement). Returns nan — not 0.0 or 1.0 — when the compared rows carry no variance; kappa_detail() gives the same number plus the reason.

Parameters:

a – first annotator’s labels.
b – second annotator’s labels, row-aligned with a.
labels – optional label universe.
missing_values – extra values that count as abstentions.

Returns:

κ in [-1, 1], or nan when undefined.

Raises:

ValueError – when the sequences differ in length.

spacr.agreement.confusion_matrix(a: Sequence[Any], b: Sequence[Any], labels: Sequence[Any] | None = None, missing_values: Sequence[Any] = ()) → pandas.DataFrame[source]¶

Return the a × b contingency table over rows both labelled.

Parameters:

a – first annotator’s labels.
b – second annotator’s labels, row-aligned with a.
labels – label universe; inferred from the data when omitted.
missing_values – extra values that count as abstentions.

Returns:

integer DataFrame, a labels on the index.

spacr.agreement.disagreements(db_path: str, columns: Sequence[str], table: str = PNG_TABLE, key: str = PNG_KEY, missing_values: Sequence[Any] = (), complete_only: bool = False, limit: int | None = None) → pandas.DataFrame[source]¶

Return the rows the annotators labelled differently.

A row is a disagreement when at least two annotators committed to a label and those labels are not all the same. A row where one annotator abstained is not a disagreement — by default it is scored from the available labels, and dropped entirely if that leaves fewer than two labels.

Parameters:

db_path – path to measurements.db; opened read-only.
columns – two or more annotation columns.
table – table holding them (default png_list).
key – row key (default png_path) — the crop to look at.
missing_values – extra values to treat as abstentions.
complete_only – when True, only consider rows every annotator labelled.
limit – cap the number of returned rows (the review list can be long); None returns all of them.

Returns:

DataFrame with key, one column per annotator, plus n_labelled and n_classes, in table order.

Raises:

ValueError – for fewer than two distinct columns, or an unknown table/column.

spacr.agreement.fleiss_kappa(matrix: Any) → float[source]¶

Fleiss’ κ from a subjects × categories count matrix.

matrix[i][j] is how many annotators put subject i in category j. Every row must sum to the same number of annotators n ≥ 2.

Fleiss’ κ generalises Scott’s π rather than Cohen’s κ: it pools the annotators into one marginal distribution instead of giving each its own, so on two annotators it agrees with Scott’s π and will usually differ slightly from cohens_kappa().

Parameters:: matrix – 2-D array-like of non-negative counts.
Returns:: κ, or nan when every rating fell in one category (no variance — pₑ=1, so κ has no denominator).
Raises:: ValueError – for a ragged/empty matrix, negative counts, rows that disagree on the number of annotators, or fewer than 2 annotators per subject.

spacr.agreement.format_agreement(report: AgreementReport) → str[source]¶

Render an AgreementReport as a plain-text block.

Every κ is printed next to its raw percent agreement and the number of rows behind it, so an undefined or paradoxical κ reads as information rather than as a missing value.

Parameters:: report – the report to render.
Returns:: multi-line text, no trailing newline.

spacr.agreement.interpret_kappa(kappa: float) → str[source]¶

Return the Landis & Koch band for kappa (a convention).

Parameters:: kappa – a κ value, possibly nan.
Returns:: band name, or "undefined" for nan.

spacr.agreement.kappa_detail(a: Sequence[Any], b: Sequence[Any], labels: Sequence[Any] | None = None, missing_values: Sequence[Any] = (), name_a: str = 'a', name_b: str = 'b') → PairAgreement[source]¶

Compute Cohen’s κ for a vs b with counts and confusion matrix.

Only rows both annotators labelled enter the calculation. Rows where exactly one of them abstained are counted into PairAgreement.n_abstained; they are not disagreements.

Parameters:

a – first annotator’s labels (None/NaN/”” = abstention).
b – second annotator’s labels, row-aligned with a.
labels – label universe; inferred from the compared rows when omitted. Passing it keeps confusion matrices comparable across pairs.
missing_values – extra values that count as abstentions, e.g. (0,) for legacy databases where 0 meant “not looked at”.
name_a – name to record for the first column.
name_b – name to record for the second column.

Returns:

a PairAgreement.

Raises:

ValueError – when the two sequences differ in length.

spacr.agreement.load_annotations(db_path: str, columns: Sequence[str], table: str = PNG_TABLE, key: str = PNG_KEY, missing_values: Sequence[Any] = ()) → pandas.DataFrame[source]¶

Read key + columns from table into a normalised frame.

Labels come back normalised: integers where the value is integral, None for every abstention. Row order is the table’s own.

Parameters:

db_path – path to measurements.db (opened read-only).
columns – annotation columns to read.
table – table holding them (default png_list).
key – row-identifying column (default png_path).
missing_values – extra values to treat as abstentions.

Returns:

DataFrame with key first, then one column per annotator.

Raises:

ValueError – for an unknown table or column.

spacr.agreement.table_columns(db_path: str, table: str = PNG_TABLE) → List[str][source]¶

Return the column names of table, in declaration order.

Parameters:: db_path – path to the SQLite database, opened read-only.
Raises:: ValueError – when the database has no such table.

Nested helpers¶

kappa_detail._pack(kappa: float, p_o: float, p_e: float, note: str) → PairAgreement¶

Build one complete agreement result from the captured counts.

Parameters:

kappa – computed Cohen’s kappa, or NaN when it is undefined.
p_o – observed agreement fraction.
p_e – agreement fraction expected from the marginals.
note – explanation of the ordinary or degenerate result.

Returns:

a result containing the supplied statistics together with the captured annotator names, counts, labels, confusion matrix, and interpretation.

spacr/agreement.py:440