spacr.gene_measurement_sweep

Test every gene against every measurement in one matrix operation.

sweep() asks which genes change which measurements across the whole screen at once rather than one gene at a time.

The associations are computed as one matrix product. A representative screen with 1,376 guides x 785 measurements over 1,366 wells is computed in 0.1 seconds; fitting the same pairs as about a million separate regressions takes hours.

The module applies three corrections. Each addresses an error found while sweeping one gene by hand:

• Identifier columns are excluded from the measurements. pathogen_object_label came out at p=2.5e-07 for EAF1, but it is a label: the selected cells sit lower in the segmentation’s numbering, which is a position in a list. pathogen_pathogen is the same column under another name (spearman 0.9979, identical in 141,626 of 226,467 rows).

• P values are corrected for the number of tests (Benjamini-Hochberg). With 785 measurements, pathogen_solidity at p=1.8e-03 does not survive the correction.

• Each measurement’s circularity is reported in its own column. The classification score is a function of the image, so a measurement it already tracks cannot corroborate a result derived from it. On this screen spearman(pred, pathogen_channel_1_mean_intensity) = -0.389, and the result that first looked strongest for GRA14 was that measurement, in the direction the correlation predicts.

Classes

HOUSE	Define spaCR's scientific-figure palette and sizing constants.
SweepResult	Result of sweep(): the effect grid and a tidy per-pair table.

Functions

gene_fractions(→ pandas.DataFrame)	Return each gene's fraction per well as the sum of its guides' fractions.
gene_of_guide(→ Optional[str])	Return the gene a guide name belongs to, for both spellings in use.
is_measurement(→ bool)	Whether name measures the object rather than naming it.
measurement_columns(→ List[str])	Every numeric column of frame that is a measurement.
measurement_family(→ str)	Which family name belongs to. See MEASUREMENT_FAMILIES.
plot_calibration(result[, path, title, level])	Plot observed against expected -log10(p) to check calibration.
plot_circularity(result[, path, alpha, title, level])	Plot whether each surviving measurement adds independent evidence.
plot_effect_against_representation(result[, path, ...])	Plot effect counts against each gene's effective representation.
plot_gene_profile(result, gene[, path, alpha, top, title])	Plot one gene's effects on its measurements, largest first.
plot_gene_similarity(result[, path, alpha, top, ...])	Plot the correlation between genes' effect profiles as a heatmap.
plot_grid_volcano(result[, path, alpha, title, level])	Plot effect against -log10(p) for every gene x measurement pair.
plot_guide_concordance(result[, path, alpha, top, title])	Plot how often a gene's own guides agree on the sign of each effect.
plot_measurement_families(result[, path, alpha, top, ...])	Plot each gene's significant measurements by family, as stacked bars.
plot_measurement_hits(result[, path, alpha, top, ...])	Plot measurements by how many genes move them past the correction.
plot_sweep(result[, path, alpha, max_circularity, ...])	Plot a clustered heatmap of the effects that survive the correction.
sweep(→ SweepResult)	Associate every guide with every measurement, blocked and corrected.

Module Contents

class spacr.gene_measurement_sweep.HOUSE

Define spaCR’s scientific-figure palette and sizing constants.

The palette follows recurring colors in published apicomplexan-genomics figures, including Waldman et al. (Cell, 2020; Figures 1 and 3) and Giuliano et al. (Nature Microbiology, 2024; Figure 1). Grey represents background data, while accent colors identify the smaller subset being interpreted. Family colors remain stable across panels so readers do not have to relearn category mappings.

Text, spines, and ticks use theme-aware colors selected by _readable() rather than this palette, which keeps figures legible in spaCR’s dark interface while retaining consistent scientific data colors.

class spacr.gene_measurement_sweep.SweepResult

Result of sweep(): the effect grid and a tidy per-pair table.

Parameters:

• table – tidy result frame with one row per fitted guide or gene-measurement pair and its effect, significance, support, and control evidence.

• effects – dense blocked association matrix indexed by fitted guide or gene and columned by measurement; these effects are repeated in table.

• n_wells – number of well identities shared by wells and fractions before guide-specific filtering; unlike table['n_wells'], this is screen-wide.

• n_blocks – number of distinct block labels removed from a fitted design, or zero when the sweep returned before fitting.

• dropped – numeric wells columns omitted from the effect grid, including automatically rejected identifier or duplicate columns and explicitly dropped measurements.

• circularity_known – whether at least three joined score values produced a finite measurement-to-score circularity; when false, circularity is NaN and survivors() refuses a circularity cutoff.

describe() → str

Summarize grid size, significance, circularity, and omitted inputs.

survivors(alpha: float = 0.05, max_circularity: float = 1.0) → pandas.DataFrame

Rows past the correction, optionally past a circularity bar too.

Raises:

ValueError – a circularity bar was asked for and the score never joined to the wells. Filtering on a column of NaN would return an empty table.

spacr.gene_measurement_sweep.gene_fractions(fractions: pandas.DataFrame, gene_of: Any | None = None) → pandas.DataFrame

Return each gene’s fraction per well as the sum of its guides’ fractions.

This is the rule the regression applies (wells_for_coefficient with guide_aggregation='sum'), so a gene-level sweep uses the same gene fractions as the fit that identified the gene.

Parameters:

• fractions – well-by-guide fraction matrix. Columns assigned to the same gene are summed row-wise.

• gene_of – guide name -> gene id. Defaults to gene_of_guide(). A guide for which it raises or returns no gene is left out.

Returns:

one column per gene. Guides that name no gene are left out rather than pooled into an “unknown” gene.

spacr.gene_measurement_sweep.gene_of_guide(guide: Any, prefix: str | None = None) → str | None

Return the gene a guide name belongs to, for both spellings in use.

spacr.hits.gene_of reads a regression design term – fraction:grna[225160_1] – and truncates at the first underscore. Applied to the bare TGGT1_225160_2 that a count table carries, that rule returns TGGT1, the organism, for every guide in the screen. Only a bracketed term is therefore passed to hits.gene_of.

A bare name is parsed by its number of underscore-separated parts. process_reads splits <organism>_ on exactly three components, so the gene of a name with three or more parts is the middle part(s), and the gene of a two-part name is the first part. This works for PF3D7_0100100_1 and for a human library without listing either organism.

When an organism prefix has been removed – the measured prefix, or one of GUIDE_PREFIXES – the remainder is read as <gene>_, so the gene is everything except the last part. TGGT1_ROP18_kinase_1 gives ROP18_kinase, not kinase.

Parameters:

• guide – bare guide name or bracketed regression design term to parse.

• prefix – an organism prefix measured from the library, as spacr.control_names.common_prefix() returns. When given, it is removed first and the remainder is read as <gene>_; this handles prefixes that are not in GUIDE_PREFIXES.

spacr.gene_measurement_sweep.is_measurement(name: str) → bool

Whether name measures the object rather than naming it.

Parameters:

name – candidate column name to classify.

spacr.gene_measurement_sweep.measurement_columns(frame: pandas.DataFrame) → List[str]

Every numeric column of frame that is a measurement.

Parameters:

frame – object-measurement table whose numeric columns are screened.

A column whose ranks correlate with an identifier column at above 0.99 is also excluded, whatever it is called. pathogen_pathogen passes the name test but matches the object label to four decimal places, so it is detected from its values rather than its name.

spacr.gene_measurement_sweep.measurement_family(name: Any) → str

Which family name belongs to. See MEASUREMENT_FAMILIES.

spacr.gene_measurement_sweep.plot_calibration(result: SweepResult, path: str | None = None, *, title: str = '', level: str | None = None)

Plot observed against expected -log10(p) to check calibration.

The observed P values against the uniform they would follow if nothing were real. A grid that hugs the diagonal has no signal; one that lifts off it at the left has some; one that lifts off everywhere has a systematic effect – a plate term that did not get blocked out, or an aggregation that correlated every well with itself.

The title reports the inflation factor lambda from the median P value; 0.9 to 1.15 is labelled calibrated. Check this plot before the others: with inflated P values their significant results are not reliable.

spacr.gene_measurement_sweep.plot_circularity(result: SweepResult, path: str | None = None, *, alpha: float = 0.05, title: str = '', level: str | None = None)

Plot whether each surviving measurement adds independent evidence.

A measurement the classifier already tracks cannot corroborate a result derived from that classifier. The plot exposes that dependence rather than presenting a correlated measurement as separate confirmation.

Every point is drawn because the correlation cutoff is a judgment. This lets users see where their hits fall and choose a threshold appropriate to their screen.

spacr.gene_measurement_sweep.plot_effect_against_representation(result: SweepResult, path: str | None = None, *, alpha: float = 0.05, title: str = '', level: str | None = None)

Plot effect counts against each gene’s effective representation.

The plot exposes representation as a possible confound without modifying the underlying statistic. It shows whether a gene’s number of significant measurements follows the overall representation trend or departs from it.

x is the gene’s effective well count (effective_wells, the participation ratio that sets the degrees of freedom of each p-value) and y is how many measurements it moved past the correction. With at least eight genes, a dotted line shows the linear trend. A gene above the line has more significant measurements than its effective well count predicts; a gene at the far right with a high count may owe its count to statistical weight.

The x axis uses effective wells rather than share. share is the median fraction a gene takes of the wells it is present in, so a rare gene can score high on it. On this module’s test fixture a gene in 18 of 120 wells has share 0.44 and one in all 120 has 0.20, the opposite of their statistical power. The participation ratio is the quantity the test used.

Controls are drawn with a distinct marker because their relationship to the trend provides a useful assay calibration.

Returns:

the matplotlib Figure, or None when nothing survived.

spacr.gene_measurement_sweep.plot_gene_profile(result: SweepResult, gene: Any, path: str | None = None, *, alpha: float = 0.05, top: int = 24, title: str = '')

Plot one gene’s effects on its measurements, largest first.

Shows the measurements past the correction for gene (or, when none pass, its largest effects), ordered by absolute effect, for example “GRA14 raises pathogen intensity and lowers pathogen area”.

Each bar is labelled with its measurement family, so a profile drawn from one family can be read as one finding rather than as many.

spacr.gene_measurement_sweep.plot_gene_similarity(result: SweepResult, path: str | None = None, *, alpha: float = 0.05, top: int = 30, title: str = '', level: str | None = None)

Plot the correlation between genes’ effect profiles as a heatmap.

Two genes in one pathway are expected to move the same measurements in the same direction. The other plots read each gene on its own; this one compares genes, which shows whether a hit list contains one shared phenotype or several.

Profiles are correlated across the whole effect row, not only the significant entries, so a shared sub-threshold pattern contributes to the similarity.

spacr.gene_measurement_sweep.plot_grid_volcano(result: SweepResult, path: str | None = None, *, alpha: float = 0.05, title: str = '', level: str | None = None)

Plot effect against -log10(p) for every gene x measurement pair.

This shows the whole grid, which plot_sweep() does not because it draws only survivors. A screen where most pairs are significant looks different here from one with a few strong effects.

Pairs past the correction are coloured by direction (green raises the measurement, rust lowers it) and the rest are grey. When circularity is known, significant pairs with circularity of at least 0.15 are ringed, because the classifier already tracks them; when it is not known, the legend says that circularity was not computed.

spacr.gene_measurement_sweep.plot_guide_concordance(result: SweepResult, path: str | None = None, *, alpha: float = 0.05, top: int = 20, title: str = '')

Plot how often a gene’s own guides agree on the sign of each effect.

Guide agreement is the internal control available in this design. Two guides against the same gene are independent perturbations of it, so agreement on the sign of an effect points to the gene, and disagreement points to guide-specific effects.

Requires a table with guide rows: sweep(..., level='guide') or 'both'. A gene-level table has nothing to compare and returns None.

Returns:

the matplotlib Figure, or None when there is nothing to compare.

spacr.gene_measurement_sweep.plot_measurement_families(result: SweepResult, path: str | None = None, *, alpha: float = 0.05, top: int = 14, title: str = '', level: str | None = None)

Plot each gene’s significant measurements by family, as stacked bars.

The significant measurements of each gene are counted per measurement family (six families plus “other”), which summarises hundreds of columns in a form that can be compared between genes.

The families are coarse on purpose – see MEASUREMENT_FAMILIES.

Parameters:

• result – completed guide/measurement sweep whose significant rows are grouped into measurement families.

• top – how many genes to draw, most-hits first.

Returns:

the matplotlib Figure, or None when nothing survived.

spacr.gene_measurement_sweep.plot_measurement_hits(result: SweepResult, path: str | None = None, *, alpha: float = 0.05, top: int = 26, title: str = '', level: str | None = None)

Plot measurements by how many genes move them past the correction.

This reads the grid along its measurement axis. A measurement moved by half the library does not discriminate between genes; it more likely reflects a plate effect, focus drift or confluence artefact, and it adds a hit to many genes. Measurements moved by at least half of the significant genes are ringed.

spacr.gene_measurement_sweep.plot_sweep(result: SweepResult, path: str | None = None, *, alpha: float = 0.05, max_circularity: float = 1.0, top: int = 40, title: str = '', level: str | None = None)

Plot a clustered heatmap of the effects that survive the correction.

The full grid is too large to read as an image: 1,240 guides x 767 measurements is 951,080 cells, all coloured whether or not they are significant. The heatmap therefore draws only survivors – the guides and measurements with at least one entry past the correction – and orders rows and columns by correlation clustering.

Parameters:

• result – what sweep() returned.

• top – the most guides and measurements to draw, lowest q first. Survivors beyond this limit are not drawn; read them from result.table.

Returns:

the matplotlib Figure, or None when nothing survived.

spacr.gene_measurement_sweep.sweep(wells: pandas.DataFrame, fractions: pandas.DataFrame, *, blocks: Sequence | None = None, scores: Sequence[float] | None = None, alpha: float = 0.05, controls: Sequence[str] | None = None, min_wells: int = 5, measurements: Iterable[str] | None = None, drop_measurements: Iterable[str] | None = None, drop_guides: Iterable[str] | None = None, max_share: float | None = None, max_wells_fraction: float | None = None, level: str = 'guide') → SweepResult

Associate every guide with every measurement, blocked and corrected.

Parameters:

• wells – one row per well, the measurements in its columns.

• fractions – one row per well, one column per guide, holding that guide’s fraction. Aligned to wells by index.

• blocks – the plate of each well. Each block’s mean is subtracted before fitting. Absent means one block, so a plate difference can then appear as a gene effect.

• scores – the per-well classification score, used only to compute each measurement’s circularity. Absent leaves the circularity column NaN and SweepResult.circularity_known False.

• controls – guide or gene names to mark as controls in the control column. They are not removed. The regression drops the control columns because they are not part of the contrast it fits; this sweep tests whether each guide or gene changes a measurement, and the controls’ results are reported for comparison.

• drop_measurements – columns to leave out, by name; the complement of measurements.

• drop_guides – guides or genes to leave out, by name. A guide is dropped when its own name or its gene (gene_of_guide()) is listed, so gene ids work at either level.

• max_share – drop a guide whose median well fraction, where it is present, is above this.

• max_wells_fraction – drop a guide present in more than this share of wells. This prevalence filter is separate from max_share, which measures median abundance where the guide is present. A guide can be common at low abundance or rare at high abundance, so the two filters identify different forms of over-representation.

• min_wells – guides present in fewer wells than this are dropped.

• level – 'guide', 'gene', or 'both'. A gene’s fraction in a well is the sum of its guides’ fractions, the same rule the regression applies (gene_fractions()).

Returns:

a SweepResult. With 'both' the table carries a level column and the guide rows stay reachable beside the gene ones.

Nested helpers

_order_like_neighbours.order(matrix)

Order one matrix axis by correlation-cluster neighbours.

Parameters:

matrix – numeric two-dimensional rows to arrange.

Returns:

identity indices below three rows; otherwise average- linkage leaf indices from correlation distances. Non-finite matrix values are normalized for distance calculation and undefined distances become 1.0. Any clustering failure is handled by the parent’s mean-order fallback.

spacr/gene_measurement_sweep.py:775