spacr.qt.widgets.control_chart

Control-chart statistics for a screening campaign: limits, rules and Z’.

A screening campaign is dozens of plates run over weeks. Hit calling, plate normalisation and Z’ all assume that the positive and negative controls behave the same way on every plate. When the controls drift — the cell line changes, a reagent lot changes, incubation conditions change — every downstream result shifts with them, and the analysis does not detect it, because it normalises to the controls it is given.

This module tracks one control value per plate over run order and puts limits round it, so drift is visible while the screen is running rather than after. It is pure numpy/pandas — no Qt anywhere in it, the same rule spacr.qt.widgets.graph_spec and spacr.qt.widgets.pca_model follow — so it is usable from a notebook, testable without a display, and free for a report or a QC gate to call.

Sigma estimation

Sigma comes from the average moving range divided by d2 = 1.128 (D2_MOVING_RANGE), never from the standard deviation of the series.

The standard deviation of the plate-to-plate series is inflated by exactly the drift the chart exists to detect. Take a control that slides linearly by half a unit per plate over thirty plates with no other noise at all:

• the SD of those thirty values is 4.40, so mean ± 3 SD is [4.0, 30.5] and every single point is comfortably inside. The chart says “in control” while the campaign slides from 10 to 24.5.

• the moving range only ever sees plate-to-plate change, which for a slow drift is 0.5 and stays 0.5. MR-bar / 1.128 = 0.443, so the limits are [13.42, 16.08] around a baseline centre of 14.75 — and 24 of the 30 plates fall outside them.

That asymmetry is the reason short-term (within-subgroup) variation is the only admissible sigma for a control chart, and it is pinned by a test. What the SD route would have said is computed anyway and reported — see sd_reference_limits() and ControlChartResult.sd_would_flag — so the two sets of limits can be compared directly.

Three estimators, and which one ran is always named

ESTIMATOR_MOVING_RANGE

Individuals / moving range (I-MR). One control value per plate is the normal case in a screen, so this is what ESTIMATOR_AUTO picks when every plate contributes a single control well. Sigma is mean(|x_i - x_{i-1}|) / 1.128; 1.128 is d2 for a subgroup of two, which is what a moving range of consecutive points is.

ESTIMATOR_SUBGROUP_S

X-bar / S, for a plate with several control wells (subgroup n > 1). The plotted point is the plate’s mean and sigma is S-bar / c4(n), where c4() is computed from the gamma function rather than looked up in a table. The published table stops at n = 25, does not interpolate, and is prone to transcription errors. c4(n) = sqrt(2/(n-1)) · Γ(n/2) / Γ((n-1)/2) is three lines, is exact for every n, and can be checked against the published constant — c4(5) = 0.9400 — which is what the test does.

Sigma of the plotted statistic is sigma_within / sqrt(n), and that is what the zones and the rules are stated in. When subgroup sizes differ between plates the limits differ per plate; they are computed per point rather than averaged, and the result says so.

ESTIMATOR_ROBUST / ESTIMATOR_MAD

Explicit, never implicit, and named in the result. spaCR measurements are skewed and one catastrophically bad plate inside the baseline drags the classical centre and MR-bar out far enough to swallow everything after it. The robust variant takes the median as the centre and either the median moving range (ESTIMATOR_ROBUST) or the MAD (ESTIMATOR_MAD) as the spread.

The two robust constants are not interchangeable, and the module keeps them separate. 1.4826 (MAD_SCALE) is 1/Φ⁻¹(0.75), the factor that makes the MAD of a normal sample an unbiased estimate of its SD. A moving range is |x_i - x_{i-1}|, the absolute value of a difference of two observations, so its scale is sqrt(2)·sigma and its median is sqrt(2)·Φ⁻¹(0.75)·sigma = 0.9539·sigma (MEDIAN_MR_CONSTANT). Multiplying a median moving range by 1.4826 overestimates sigma by exactly sqrt(2), which makes the limits 41% wider and can hide a drift. Both constants are computed from Φ⁻¹(0.75) here for the same reason c4 is computed from the gamma function.

Phase I and Phase II

Limits are estimated from a stated baseline set of plates and then applied forward. Limits computed from all the data, drift included, widen to contain the drift they are meant to detect.

The baseline is the first DEFAULT_BASELINE plates in run order by default (twenty, the textbook Phase I minimum for an individuals chart), or an explicit list of plate ids (ControlChartSpec.baseline_plates), or everything before a cut-off in the order column (ControlChartSpec.baseline_before). Whichever it was, the plates that formed it are in ControlChartResult.baseline_plates and named in ControlChartResult.report().

MIN_BASELINE is eight, because MR-bar over seven moving ranges is already dominated by one or two plate-to-plate jumps, and with fewer by one. A shorter baseline is refused.

Limits estimated from an out-of-control baseline are unreliable. When the baseline itself contains a violation, ControlChartResult.caveats() says so. Re-estimating without the flagged plates is offered (ControlChartSpec.reestimate) and is off by default, so plates are never removed from the baseline without the user asking. It runs one pass, not iterations to convergence: iterating shrinks the limits onto the tightest subset of the baseline and eventually flags everything.

The rules, and saying which one fired

The Nelson / Western Electric set, all eight, each a named constant with what it physically detects and its approximate false-alarm rate: RULE_BEYOND_3_SIGMA, RULE_NINE_ONE_SIDE, RULE_SIX_TRENDING, RULE_FOURTEEN_ALTERNATING, RULE_TWO_OF_THREE_BEYOND_2, RULE_FOUR_OF_FIVE_BEYOND_1, RULE_FIFTEEN_WITHIN_1, RULE_EIGHT_BEYOND_1.

Which points a rule flags is a convention, and an unstated one makes two implementations disagree about the same chart, so it is stated: every point that is part of the evidence is flagged. For rule 1 that is the point itself. For the run rules (2, 3, 4, 7, 8) it is every point of the run, reported as one maximal span rather than as a sliding window’s worth of overlapping alarms — which is what lets ControlChartResult.report() say “plates P22–P30 are nine in a row above the centre line” instead of listing three violations. For the “k of m” rules (5, 6) it is the qualifying points only: a window can trigger on two points beyond 2 sigma with a third sitting on the centre line, and that third point is not flagged because it is not part of the evidence.

The rule set is selectable, and it has to be, because running all eight is not eight times the sensitivity. Each rule has its own in-control false-alarm rate; RULE_ALARM_RATE carries them, ControlChartResult.caveats() does the arithmetic for the set that actually ran, and over a 200-plate campaign it reports the expected number of false alarms. RULES_DEFAULT is the Western Electric four (1, 2, 5, 6), which is the set the classical ARL figure was worked out for; RULES_ALL is all eight, and RULES_LIMITS_ONLY is rule 1 alone for a user who wants limits and nothing else.

Ordering

“Plate by plate over time” means the x axis is run order, and rules 2, 3, 4, 6, 7 and 8 are statements about a sequence, so they are meaningless when the order is wrong. An explicit order or date column (ControlChartSpec.order) is used when given. Without one the plate id is sorted with a natural, digit-aware key so plate 2 comes before plate 10, and ControlChartResult.order_inferred is set, the note is in ControlChartResult.notes, and ControlChartResult.report() says it in capitals.

The key is a deliberate extension of spacr.qt.widgets.graph_spec._sort_key(), not a copy of it: that one tries float(text) and falls back to a plain string compare, which puts P10 before P2 — fine for facet labels, wrong for a run order. Here the string is split into digit and non-digit runs and the digit runs compare as integers.

Z-prime

zprime_frame() computes the per-plate Z-factor when both a positive and a negative control are named, and zprime_chart() charts it with everything above — same estimator, same rules, same baseline. A Z’ that slides from 0.7 to 0.3 over a campaign is a drift in assay quality and is detected the same way.

Exceptions

ControlChartError

Raised when a chart cannot be computed; the message says how to fix it.

Classes

ControlChartResult	A chart, its limits, what fired, and everything needed to read it.
ControlChartSpec	What to chart, in what order, from which baseline, under which rules.
Violation	One rule firing, over one span of plates.

Functions

c4(→ float)	The unbiasing constant for the SD of a subgroup of n.
candidate_key_columns(→ Tuple[str, ...])	The columns worth offering as the plate id or the control label.
candidate_value_columns(→ Tuple[str, ...])	The columns worth offering as the charted measurement, sorted.
control_chart(→ ControlChartResult)	Chart spec's control value plate by plate over run order.
sd_reference_limits(→ Tuple[float, float, float, float])	(centre, sigma, lower, upper) from mean ± 3 SD, for comparison only.
zprime_chart(→ ControlChartResult)	zprime_frame(), charted with the same estimator, rules and baseline.
zprime_frame(→ pandas.DataFrame)	Per-plate Z-factor, one row per plate, in run order.

Module Contents

exception spacr.qt.widgets.control_chart.ControlChartError

Bases: ValueError

Raised when a chart cannot be computed; the message says how to fix it.

Raised rather than returned as an empty result. Every one of these is a sentence a user can act on — “18 plates carry a control value and the baseline needs 20” — and a caller that swallowed it would draw an empty axis with no explanation. The screen catches it and shows the text.

Initialize self. See help(type(self)) for accurate signature.

class spacr.qt.widgets.control_chart.ControlChartResult

A chart, its limits, what fired, and everything needed to read it.

Parameters:

• plates – one entry per point, in run order. The x axis.

• values – the plotted statistic per plate — the control value, or the plate’s mean control value when there is more than one control well.

• order_labels – what the order column said for each plate, as text. Equal to plates when the order was inferred.

• subgroup_sizes – control wells behind each point.

• subgroup_sd – within-plate SD per point; NaN where n < 2.

• centre – the centre line, estimated from Phase I only.

• sigma – sigma of the plotted statistic, which is what the zones and every rule are stated in. For X-bar/S that is sigma_within / sqrt(n); when subgroup sizes differ it is the median of sigma_at, and the per-point values are the ones applied.

• sigma_within – the within-plate sigma, before the sqrt(n). Equal to sigma for an individuals chart.

• sigma_at – sigma per point.

• lower – per-point lower limit (centre - 3 sigma).

• upper – per-point upper limit.

• z – (value - centre) / sigma_at per point — the units every rule is written in, so varying subgroup sizes do not affect the rules.

• estimator – which estimator actually ran. Never "auto".

• baseline – positional indices of the Phase I plates.

• baseline_excluded – plates dropped from Phase I by a re-estimation.

• violations – every rule firing, in run order then rule order.

• rules – the rule set that was run.

• value_column – the column the plotted value was read from.

• plate_column – the column that identifies each plate.

• order_inferred – the order was guessed from the plate id.

• degenerate – sigma came out zero — see caveats().

• sd_reference – (centre, sigma, lower, upper) from sd_reference_limits() over every point.

• sd_would_flag – how many points fall outside those SD limits.

__len__() → int

Plates on the chart.

caveats() → Tuple[str, ...]

The warnings a reader should see before relying on the chart.

false_alarm_rate() → float

Approximate probability that some selected rule fires on an in-control point.

1 - Π(1 - p_i) over the rules that ran. The independence it assumes is not true and the error has a known direction — see RULE_ALARM_RATE — so this overstates the alarm rate, which is the safe direction for a warning.

headline() → str

One sentence about the campaign, including any violations.

points_frame() → pandas.DataFrame

One row per plate — what the chart draws, and its CSV.

Everything a reader would want to re-plot elsewhere: the order label, the value, the subgroup, the limits that applied to that point, the z, whether it was Phase I and which rules fired on it.

report() → str

The full text report, as shown on screen and written to a file.

rules_at(index: int) → Tuple[int, ...]

Which rules fired on point index, ascending.

A point can trip several — a plate beyond 3 sigma is usually also the end of a two-of-three-beyond-2-sigma window — and reporting only the first would hide that the same plate is evidence for two different failures.

Parameters:

index – the point’s position in run order; outside 0 .. len(self) - 1 raises ControlChartError.

violations_frame() → pandas.DataFrame

One row per violation — the table the screen shows.

zone(index: int) → int

Which sigma band point index sits in: 0, 1, 2 or 3.

0 is inside 1 sigma, 3 is beyond 3 sigma. Signed nowhere — the side is the sign of z. What a renderer colours a marker by.

Parameters:

index – the point’s position in run order; outside 0 .. len(self) - 1 raises ControlChartError.

property baseline_plates: Tuple[str, ...]

The plate ids Phase I was estimated from, in run order.

property baseline_violations: Tuple[Violation, ...]

The violations that fell inside Phase I.

property flagged: numpy.ndarray

did any rule fire on it.

Type:

Boolean per point

class spacr.qt.widgets.control_chart.ControlChartSpec

What to chart, in what order, from which baseline, under which rules.

Frozen and JSON round-tripping, like GraphSpec and PCASpec, so a QC configuration is something a settings file or a report can carry and re-run against the next campaign.

Parameters:

• value – the measured column to chart. Required at chart time.

• plate – the column identifying the plate. Required at chart time. One point per distinct value of it.

• order – an explicit run-order or date column. Strongly preferred: without it the order is inferred from plate and every run-based rule rests on that inference.

• control_column – the column saying what each row is ("condition", "gene", "well_type"). None means the table is already only the control.

• control_levels – which level(s) of control_column are the control being charted. Required whenever control_column is set — a control column with no level named is a filter with no predicate.

• positive_levels – the positive control, for zprime_frame().

• negative_levels – the negative control, likewise.

• estimator – one of ESTIMATORS.

• rules – which of RULES_ALL to run.

• baseline_n – Phase I length when neither baseline_plates nor baseline_before is given.

• baseline_plates – an explicit Phase I set, by plate id.

• baseline_before – everything strictly before this value of the order column is Phase I. A date, for a campaign whose baseline is “the first week”.

• reestimate – when the baseline itself trips a rule, drop the flagged plates and estimate once more. Off by default; see the module docstring for why.

Raises:

ControlChartError – on an unknown estimator or rule id, a baseline shorter than MIN_BASELINE, or a control column with no level named — at the point the spec is built, not at render time.

__post_init__() → None

Normalise the columns and levels, and validate the chart settings.

Raises:

ControlChartError – if the estimator is unknown; if a rule is not a number, or not one of rules 1-8; if the baseline is shorter than the module’s minimum – MR-bar over fewer moving ranges is dominated by one or two plate-to-plate jumps; or if control_column and control_levels are given without each other.

describe() → str

One line, for a caption.

classmethod from_dict(payload: Mapping[str, Any]) → ControlChartSpec

Rebuild from to_dict().

Unknown keys ignored, missing keys defaulted, so a configuration written by another build still opens.

Parameters:

payload – a mapping as written by to_dict(); unknown keys are ignored and missing ones take their defaults.

classmethod from_json(text: str) → ControlChartSpec

Rebuild from to_json().

Parameters:

text – a JSON string as written by to_json().

to_dict() → Dict[str, Any]

Return a plain JSON-compatible dict with every field, always.

Every field is written even at its default, so the stored schema is stable across versions.

to_json() → str

to_dict() as sorted JSON.

with_baseline(*, n: int | None = None, plates: Sequence[str] | None = None, before: str | None = None) → ControlChartSpec

A copy with a different Phase I.

with_columns(*, value: str | None = None, plate: str | None = None, order: str | None = None) → ControlChartSpec

A copy pointed at different columns.

with_control(column: str | None, levels: Sequence[str]) → ControlChartSpec

A copy charting a different control.

Parameters:

• column – the column holding the control labels, or None.

• levels – the labels in that column that mark control wells.

with_estimator(estimator: str) → ControlChartSpec

A copy using a different sigma estimator.

Parameters:

estimator – the sigma estimator name, e.g. 'auto', 'moving_range', 'subgroup_s', 'robust' or 'mad'; stored without checking.

with_rules(rules: Sequence[int]) → ControlChartSpec

A copy running a different rule set.

Parameters:

rules – the rule numbers (1 to 8) to run.

class spacr.qt.widgets.control_chart.Violation

One rule firing, over one span of plates.

A maximal span, not a sliding window’s worth of overlapping alarms: twelve points in a row above the centre line is one violation of rule 2 covering twelve plates, which is what a reader needs, rather than four violations covering nine plates each, which is an artefact of the implementation.

Parameters:

• rule – the rule number, 1-8.

• start – first index of the span, into ControlChartResult.plates.

• end – last index of the span, inclusive.

• points – the indices actually flagged. Equal to the whole span for the run rules; only the qualifying points for rules 5 and 6, where a window can trigger with a non-qualifying point inside it.

• plates – the plate ids of points.

• side – ABOVE, BELOW or EITHER. For rule 3 it is the direction of travel (ABOVE for a rising trend) rather than a side of the centre line, because a trend can run entirely on one side of the centre or cross it and is a violation either way.

• in_baseline – whether any flagged point is inside Phase I. Limits estimated from an out-of-control baseline are unreliable, so this is carried per violation rather than recomputed by every reader.

describe() → str

The sentence ControlChartResult.report() prints.

Names the rule by number and in words: the number alone requires the reader to know Nelson’s numbering, and the words alone cannot be cross-referenced against another chart.

where() → str

The plates in words — "plate P17" or "plates P22-P30".

property span: int

Plates from start to end, inclusive.

spacr.qt.widgets.control_chart.c4(n: int) → float

The unbiasing constant for the SD of a subgroup of n.

c4(n) = sqrt(2/(n-1)) · Γ(n/2) / Γ((n-1)/2), which is E[s] / sigma for a normal sample of size n: the sample SD is a biased estimate of sigma, badly so for the small subgroups a plate’s control wells make, and S-bar / c4(n) is the correction.

Computed from the gamma function rather than looked up, because the published table stops at 25, does not interpolate, and is prone to transcription errors. The values agree with the published ones to every digit they print: c4(2) = 0.7979, c4(5) = 0.9400, c4(10) = 0.9727.

Parameters:

n – the subgroup size, converted to int; must be at least 2.

Raises:

ControlChartError – for n < 2. A subgroup of one has no spread to unbias, which is the whole reason the I-MR chart exists.

spacr.qt.widgets.control_chart.candidate_key_columns(frame: pandas.DataFrame) → Tuple[str, ...]

The columns worth offering as the plate id or the control label.

Categorical by the same classifier, plus anything named like a plate or a date, because plateID is high-cardinality enough in a big campaign that the classifier calls it a key and skips it — and a key is exactly what is wanted here. This is the one place in spaCR where “identifies rather than describes” is a recommendation rather than a disqualification.

Parameters:

frame – the per-well table whose columns are classified by spacr.qt.widgets.graph_spec.column_kinds(). Columns named like a plate, date, time, batch, run or order are added.

spacr.qt.widgets.control_chart.candidate_value_columns(frame: pandas.DataFrame) → Tuple[str, ...]

The columns worth offering as the charted measurement, sorted.

Continuous by spacr.qt.widgets.graph_spec.column_kinds(), which is the one column classifier in this codebase — the Local Data Filter’s rule, re-read. Reused rather than re-derived: the Graph Builder, the PCA screen and this one must agree about what cell_count is, so all three classify the same table the same way.

Parameters:

frame – the per-well table whose columns are classified by spacr.qt.widgets.graph_spec.column_kinds().

spacr.qt.widgets.control_chart.control_chart(frame: pandas.DataFrame, spec: ControlChartSpec | None = None) → ControlChartResult

Chart spec’s control value plate by plate over run order.

The whole policy is in the module docstring; the short version is that one point is drawn per plate in run order, sigma comes from short-term variation and never from the SD of the series, the limits are estimated from a stated baseline and applied forward, and every rule that fires is reported by number, in words, and with the plates it fired on.

Parameters:

frame – the per-well table holding the spec’s value, plate, order and control columns.

Raises:

ControlChartError – whenever the chart cannot be computed — with the reason and the fix in the message.

spacr.qt.widgets.control_chart.sd_reference_limits(values: Sequence[float]) → Tuple[float, float, float, float]

(centre, sigma, lower, upper) from mean ± 3 SD, for comparison only.

This is not an estimator option. It exists so a result can print the SD limits next to the moving-range limits: on a drifting series the SD limits are wide enough to contain the drift, and a chart drawn with them reports “in control” throughout.

Uses the sample SD (ddof=1) over every value given — limits computed from all the data, including the excursion they are supposed to detect, which is the error the comparison illustrates.

Parameters:

values – the plotted values; non-finite ones are ignored.

spacr.qt.widgets.control_chart.zprime_chart(frame: pandas.DataFrame, spec: ControlChartSpec) → ControlChartResult

zprime_frame(), charted with the same estimator, rules and baseline.

A thin composition rather than a second engine: the Z’ series is one value per plate, which is an individuals chart, which is what control_chart() already is. The order is carried as an explicit integer column, so the Z’ chart never reports an inferred order — the ordering decision was taken once, upstream.

Parameters:

• frame – the per-well table holding the spec’s value, plate, order and control columns.

• spec – the chart settings; must name control_column, positive_levels and negative_levels. Its estimator, rules and baseline are reused for the Z’ chart.

spacr.qt.widgets.control_chart.zprime_frame(frame: pandas.DataFrame, spec: ControlChartSpec) → pandas.DataFrame

Per-plate Z-factor, one row per plate, in run order.

Z' = 1 - 3(sd_pos + sd_neg) / |mean_pos - mean_neg| — the number a screener actually watches, and the one that says whether a plate could have detected anything at all. Charting it with the same limits and the same rules is the same question one level up: a Z’ that slides from 0.7 to 0.3 over a campaign is a screen that stopped working, plate by plate, with no single plate looking wrong.

Both controls need at least two wells on a plate for the SD to exist, so a plate with a single positive well produces no Z’ and is left out rather than given a zero.

Parameters:

• frame – the per-well table holding the spec’s value, plate, order and control columns.

• spec – the chart settings; must name control_column, positive_levels and negative_levels.

Returns:

columns plate, order, order_index, zprime, separation, and the per-control means, SDs and n.

Raises:

ControlChartError – when the spec does not name both controls.

Nested helpers

control_chart._finish(centre: float, sigma_within: float, baseline: np.ndarray, excluded: Tuple[str, ...]) → ControlChartResult

Assemble the result, flagging a degenerate spread.

A sigma at or below the tolerance means every point is a violation, so it is reported as DEGENERATE rather than charted – limits drawn from no spread say nothing about the process.

spacr/qt/widgets/control_chart.py:1708