spacr.qt.synthetic

Synthetic datasets + saved settings for exercising every pipeline app.

The goal: give a developer (or a bug reporter) a one-line way to generate a demo folder that flows cleanly through every spacr pipeline — mask, measure, crop, classify, timelapse, map_barcodes — plus a matching settings CSV that plugs into the “Import settings…” button on each app screen.

Everything is reverse-engineered from what the pipelines actually consume:

• Filenames match the cellvoyager regex in spacr.utils._get_regex('.tif', 'cellvoyager'):

  <plateID>_<wellID>_T<timeID>F<fieldID>L<laserID>A<AID>Z<sliceID>C<chanID>.tif
  

• Channels are laid out in the order every mask default expects:

  C0 = nucleus, C1 = cell, C2 = pathogen, C3 = organelle
  

• Images are 16-bit uint16. Every channel of a field is drawn from one shared cell layout, so the nucleus really is inside the cell and the pathogen really is inside the same cell — the relationships measure_crop goes looking for when it links objects.

• Measure/crop demos ship the merged/*.npy stacks measure_crop actually reads (image planes first, then the label-mask planes), not a stand-in.

• Settings CSVs are written in the two-column Key,Value format, the first of the header pairs spacr.qt.screens.app_screen.AppScreen ._load_settings_csv tries, so the “Import settings…” button restores every value into the form. Reading one straight from Python means naming the columns — load_settings(path, setting_key="Key", setting_value="Value") — because spacr.utils.load_settings defaults to the other spelling (setting_key/setting_value, what spacr.io.save_settings_to_db writes) and raises rather than guessing.

Every generator is reproducible: identical inputs give byte-identical output on any machine, so two people comparing “the demo fails here” look at the same pixels. Seeds are not taken from hash((well, field, time, chan)), which PYTHONHASHSEED salts differently at every interpreter start.

Public API:

generate_mask_demo(dst, ...) -> DemoLayout
generate_measure_demo(dst, ...) -> DemoLayout
generate_crop_demo(dst, ...) -> DemoLayout
generate_classify_demo(dst, ...) -> DemoLayout
generate_timelapse_demo(dst, ...) -> DemoLayout
generate_map_barcodes_demo(dst, ...) -> DemoLayout
save_settings_csv(dst, settings) -> Path
demo_settings(app_key, src, channels=None) -> Dict[str, Any]

CLI:

python -m spacr.qt.synthetic mask /tmp/demo
python -m spacr.qt.synthetic all  /tmp/demo

Classes

DemoLayout

What a demo generator produced. Absolute paths only.

Functions

barcode_pool(→ List[str])	Return n distinct synthetic barcodes of length bases.
cellvoyager_filename(→ str)	Return a filename matching:
demo_settings(→ Dict[str, Any])	Return a spacr settings dict tailored for the demo dataset
generate_barcode_csv(→ pathlib.Path)	Write a name,sequence barcode CSV.
generate_classify_demo() → DemoLayout)	Write a synthetic Classify dataset of PNG crops and a png_list table.
generate_crop_demo(, fields, channels)	Same dataset as measure — Crop is measure with save_png on, and
generate_map_barcodes_demo(→ DemoLayout)	Populate dst with a self-contained map_barcodes demo:
generate_mask_demo(, fields, channels)	Populate dst with a folder that runs cleanly through the Mask
generate_measure_demo(, fields, channels)	Write a synthetic Measure dataset of merged image and label stacks.
generate_synthetic_fastq(→ List[pathlib.Path])	Write a gzip-compressed synthetic FASTQ pair carrying known barcodes.
generate_timelapse_demo(, fields, times, channels)	Write a synthetic Timelapse dataset with several frames per field.
main(→ int)	Generate one (or every) demo dataset via the python -m CLI.
save_settings_csv(→ pathlib.Path)	Write settings in the two-column Key,Value format that
synthetic_read(→ str)	Build one 150-base read carrying a (column, gRNA, row) triplet.

Module Contents

class spacr.qt.synthetic.DemoLayout

What a demo generator produced. Absolute paths only.

merged_files replaces the old mask_files: nothing in spaCR reads a folder of standalone label tiffs, and the measure/crop demos now ship the merged/*.npy stacks measure_crop actually opens — the label planes are the trailing planes of those arrays.

Parameters:

• src – absolute path of the demo folder, the value the demo’s settings use as src.

• image_dir – absolute path of the folder holding the generated raw images; the same as src except for demos that put them in a data subfolder.

spacr.qt.synthetic.barcode_pool(n: int, length: int, seed: int = 0) → List[str]

Return n distinct synthetic barcodes of length bases.

Parameters:

• n – how many to draw.

• length – barcode length in bases.

• seed – RNG seed — the same seed always returns the same pool.

Returns:

list of n unique uppercase DNA strings.

spacr.qt.synthetic.cellvoyager_filename(plate: str = 'plate1', well: str = 'A01', time: int = 1, field: int = 1, laser: int = 1, a: int = 1, slice_: int = 1, chan: int = 1, ext: str = 'tif') → str

Return a filename matching: <plateID>_<wellID>_T<timeID>F<fieldID>L<laserID>A<AID>Z<sliceID>C<chanID>.<ext>

spacr.qt.synthetic.demo_settings(app_key: str, src: str, channels: Sequence[int] | None = None) → Dict[str, Any]

Return a spacr settings dict tailored for the demo dataset generated by generate_<app>_demo.

Parameters:

• app_key – which app the settings are for.

• src – the demo folder (for map_barcodes, the demo root — its barcode CSVs are resolved relative to it).

• channels – the acquisition channels the dataset actually holds. Defaults to all four. Every *_channel and *_mask_dim key is derived from it, so a two-channel demo cannot advertise a pathogen channel it never acquired.

Values are the minimum needed to make the pipeline flow — real users will tweak thresholds + channel numbers to fit their data.

spacr.qt.synthetic.generate_barcode_csv(dst: pathlib.Path, names: Sequence[str], sequences: Sequence[str]) → pathlib.Path

Write a name,sequence barcode CSV.

This is the format spacr.sequencing.map_sequences_to_names reads — it requires both columns by name and rejects duplicate sequences. (The demo used to ship a FASTA, which that function cannot read at all.)

Parameters:

• dst – output .csv path.

• names – barcode names, aligned with sequences.

• sequences – barcode sequences.

Returns:

the resolved dst path.

Raises:

ValueError – when names and sequences do not hold the same number of entries. This compares row counts, not barcode lengths: the two CSVs written for a demo folder carry different barcode lengths (21 for gRNAs, 8 for wells). The check is needed because the zip below stops at the shorter list, so an unequal pair would drop barcodes from the table without a log message, and the reads of those barcodes would map to no name.

spacr.qt.synthetic.generate_classify_demo(dst: pathlib.Path, n_crops: int = 64, plate: str = 'plate1', wells: Sequence[str] = ('A01', 'A02', 'A03', 'A04')) → DemoLayout

Write a synthetic Classify dataset of PNG crops and a png_list table.

Classify reads PNG single-object crops and a measurements.db with a png_list table whose annotate column carries the class labels for training and testing.

This is a hand-built stand-in for a measured plate, not a replica of one. Two of the four properties below match measure_crop; the other two do not.

Crop names match. A real crop is <file_name>_<cell_id>.png where file_name is the merged stack’s <plate>_<well>_<field>_<time> (spacr.utils._generate_names()) — e.g. plate1_A01_1_1_1.png. That is exactly what this writes, and exactly what spacr.utils._map_wells_png parses plate/row/column/field back out of.

The cell_png leaf matches. measure_crop appends f"{crop_mode}_png/" to the folder, so a real cell crop does live under cell_png/ — which is what spacr.io.generate_training_dataset()’s png_path.str.contains(png_type) filter (png_type='cell_png') needs to see. Crops written flat as data/crop_000.png, the layout this replaced, were all filtered away and the run died on “got 0 classes”.

The folder above it does not match. measure_crop buckets every crop by what it contains first: data/<single|multiple|no>_nucleus/<single_pathogen|multiple_pathogens|uninfected>/<plate>_<well>/cell_png/. This demo writes data/<plate>_<well>/cell_png/ with no bucket folders — nothing downstream of png_type reads them, and inventing an infection status per synthetic crop would be a fiction the pixels do not support.

The png_list columns do not match either. spacr.utils.filepaths_to_database() writes png_path, file_name, plateID, rowID, columnID, fieldID, prcfo, cell_id, with the tokenised values rowID='r1'/columnID='c1'/fieldID='f1'. This table carries png_path, plateID, wellID, rowID, columnID, fieldID, timeID, label in plain form, plus the annotate column — which is the point: annotate is what dataset_mode='annotation' selects classes on, and a measure run never writes one. A human does, in the Annotate screen.

Parameters:

• dst – destination folder.

• n_crops – total number of crops, spread evenly over the wells.

• plate – plate ID baked into the crop names and png_list.

• wells – well IDs to spread the crops over; at least two, so the classifier can hold a whole well out.

Returns:

DemoLayout.

spacr.qt.synthetic.generate_crop_demo(dst: pathlib.Path, plate: str = 'plate1', wells: Iterable[str] = ('A01', 'A02'), fields: int = 2, channels: Iterable[int] = (0, 1, 2, 3)) → DemoLayout

Same dataset as measure — Crop is measure with save_png on, and writes PNG crops into per-object folders alongside the DB.

Builds the dataset directly rather than by calling generate_measure_demo(), so the folder holds exactly one settings_*.csv (settings_crop.csv, with PNG crops on) and “Import settings…” cannot pick the measure settings, which turn PNG crops off.

Parameters:

• dst – destination folder.

• plate – plate ID baked into every filename.

• wells – well IDs to emit.

• fields – fields per well.

• channels – acquisition channels to render.

Returns:

DemoLayout whose settings_csv is settings_crop.csv.

spacr.qt.synthetic.generate_map_barcodes_demo(dst: pathlib.Path, n_barcodes: int = 12, n_reads: int = 5000, seed: int = 0, n_rows: int = 4, n_columns: int = 6) → DemoLayout

Populate dst with a self-contained map_barcodes demo:

dst/
  barcodes/
    grna.csv            # ← N gRNA barcodes, name,sequence
    row.csv             # ← row (plate-row) barcodes
    column.csv          # ← column barcodes
  demo_R1_001.fastq.gz  # ← reads carrying those barcodes
  demo_R2_001.fastq.gz
  settings_map_barcodes.csv

The FASTQs sit in dst itself, not a fastq/ subfolder: the pipeline’s src is listed flat for *.fastq.gz (spacr.io.parse_gz_files), so a subfolder means zero samples found and a run that exits having written nothing.

Parameters:

• dst – destination folder.

• n_barcodes – number of unique gRNA barcodes to plant.

• n_reads – approximate total number of reads to emit.

• seed – RNG seed for reproducibility.

• n_rows – number of plate-row barcodes.

• n_columns – number of plate-column barcodes.

Returns:

DemoLayout describing the emitted files.

spacr.qt.synthetic.generate_mask_demo(dst: pathlib.Path, plate: str = 'plate1', wells: Iterable[str] = ('A01', 'A02'), fields: int = 2, channels: Iterable[int] = (0, 1, 2, 3)) → DemoLayout

Populate dst with a folder that runs cleanly through the Mask app. Layout:

dst/
  <plateID>_<wellID>_T01F<field>L01A01Z01C<chan>.tif
  settings_mask.csv

Parameters:

dst – destination folder; made absolute and created if absent.

spacr.qt.synthetic.generate_measure_demo(dst: pathlib.Path, plate: str = 'plate1', wells: Iterable[str] = ('A01', 'A02'), fields: int = 2, channels: Iterable[int] = (0, 1, 2, 3)) → DemoLayout

Write a synthetic Measure dataset of merged image and label stacks.

Measure reads what Mask produces: a merged/ folder of .npy stacks whose trailing planes are the label masks. The stacks are pre-built here, so Measure can run on the demo without a GPU or a Mask run.

Note

The organelle plane this writes is measured into nothing, and the defect is not in this module. Plane organelle_mask_dim of every merged stack carries 64 real labels, but a measure run over this folder writes cell/nucleus/pathogen/cytoplasm and no organelle table at all (verified: 4 fields → cell 64, nucleus 64, pathogen 71, cytoplasm 64, organelle absent).

All four organelle writes in spacr.measure._measure_crop_core are gated on settings.get('summarize_organelles_by') is not None, and spacr.settings.get_measure_crop_settings — the defaults every measure run is canonicalised through — never sets that key.

Defaulting the key for measure the way the Mask app does would not fix this, for two reasons that have to be fixed together.

1. set_default_settings_preprocess_generate_masks defaults it to the string 'cell', and measure.py tests it with "organelle" in settings['summarize_organelles_by'] — a substring test when the value is a str. Running this demo with summarize_organelles_by='cell' gives cell_organelle_summary (16 rows/field) and still no organelle table. Only a value containing 'organelle' writes the per-organelle table (['cell', 'organelle'] → organelle 64 rows/field, verified).

2. A list cannot be shipped today: spacr.settings.expected_types declares 'summarize_organelles_by': str, so spacr.validate.validate_settings rejects ['cell', 'organelle'] with “is a list, but str is expected” — a hard pre-flight error on a demo that must load clean. The tooltip and spacr.gui_utils both describe it as a list, and spacr.external_masks builds one; only the type table disagrees.

So the demo deliberately omits the key, and the wiring needed elsewhere is: widen expected_types['summarize_organelles_by'] to (str, list, type(None)); make get_measure_crop_settings default it to ['cell', 'organelle'] (safe — every write is separately gated on organelle_mask_dim is not None, which defaults to None); and add it to the measure section of spacr/qt/screens/settings_model.py so the Measure form can hold it — without a widget, apply_settings_dict drops it and collect() never emits it, so a CSV key would change what a CLI run measures and nothing about a GUI run.

Parameters:

dst – destination folder; made absolute and created if absent.

spacr.qt.synthetic.generate_synthetic_fastq(dst_dir: pathlib.Path, grnas: Sequence[str], rows: Sequence[str], columns: Sequence[str], n_reads: int = 5000, seed: int = 0, sample: str = FASTQ_SAMPLE, paired: bool = True) → List[pathlib.Path]

Write a gzip-compressed synthetic FASTQ pair carrying known barcodes.

Every read is one (column, gRNA, row) triplet in the frame the shipped barcode-mapping defaults parse, so unique_combinations.csv comes back with the planted wells and guides in it. Reads are spread evenly over the rows x columns wells, and within a well over the gRNAs with a skew, because a real screen has a handful of abundant guides and a long tail.

Parameters:

• dst_dir – folder to write into.

• grnas – gRNA barcode sequences (21 bases each).

• rows – row barcode sequences (8 bases each).

• columns – column barcode sequences (8 bases each).

• n_reads – approximate total number of reads; the real total is rounded down to a whole number of reads per well.

• seed – RNG seed for reproducible read pools.

• sample – sample name; the files are <sample>_R1_001.fastq.gz (and _R2_).

• paired – also write the R2 mate. R2 is the exact reverse complement of R1 — a perfectly overlapping pair — which is what spacr.sequencing’s paired path reduces to after it reverse-complements R2 and takes the per-base consensus.

Returns:

the written paths, R1 first.

Raises:

ValueError – when any barcode list is empty.

spacr.qt.synthetic.generate_timelapse_demo(dst: pathlib.Path, plate: str = 'plate1', wells: Iterable[str] = ('A01',), fields: int = 1, times: int = 8, channels: Iterable[int] = (0, 1)) → DemoLayout

Write a synthetic Timelapse dataset with several frames per field.

Tracking needs multi-T frames per (well, field). The files use the cellvoyager naming with T01..T<N>, and every frame holds the same cells shifted by a few pixels rather than a new random field.

Only the nucleus and cell channels are acquired, and the settings list only those channels, with no pathogen_channel or organelle_channel, so pre-flight accepts them.

Note

The dataset and settings this writes clear pre-flight, but the Timelapse pipeline still cannot consume them, and neither defect is in this module. Both were reproduced on this demo and both fixes were proved by patching the two functions at run time; with the pair applied the demo completes and writes 8 merged stacks, per-channel movies, a track-overlay GIF and a 16-track tracks/*.csv.

1. spacr.io._rename_and_organize_image_files names its stack files <plate>_<well>_<field>.npy when timelapse=True — dropping the timeID and max-projecting every timepoint of a field into one array — while spacr.io._generate_time_lists groups on <plate>_<well>_<field>_<time>.npy and skips anything with fewer than four underscore-separated parts. An 8-frame field becomes one stack/plate1_A01_1.npy, _generate_time_lists returns [], no *_norm_timelapse.npz is written, no masks are generated, and preprocess_generate_masks dies in _pivot_counts_table on no such table: object_counts. Emitting the timeID in both branches (the non-timelapse spelling is already exactly what _generate_time_lists parses) is the fix.

2. Past that, spacr.object.generate_cellpose_masks_sam hands spacr.timelapse._trackpy_track_cells a list of 2-D frames, and the tracking chain indexes it as an array: _track_by_iou does masks.shape[0] and _relabel_masks_based_on_tracks does np.zeros(masks.shape, …), both AttributeError: 'list' object has no attribute 'shape'. In the timelapse_mode='iou' path this demo asks for, the first one is swallowed by the except Exception retry loop in _facilitate_trackin_with_adaptive_removal, which then shrinks the search range 100 times and reports Failed to track after 100 attempts — a message about displacement for a bug about a type. Coercing once at the top of _trackpy_track_cells (masks = np.asarray(masks)) clears both.

Parameters:

dst – destination folder; made absolute and created if absent.

spacr.qt.synthetic.main(argv: list[str] | None = None) → int

Generate one (or every) demo dataset via the python -m CLI.

Parameters:

argv – optional argv list; defaults to sys.argv[1:].

Returns:

process exit code (0 on success).

spacr.qt.synthetic.save_settings_csv(path: pathlib.Path, settings: Dict[str, Any]) → pathlib.Path

Write settings in the two-column Key,Value format that spacr.utils.load_settings reads.

Parameters:

• path – CSV file to write; made absolute, its parent folder is created, and an existing file is overwritten.

• settings – settings to write, one Key,Value row each; None is written as an empty value and everything else with str().

spacr.qt.synthetic.synthetic_read(column_barcode: str, grna: str, row_barcode: str, prefix: str = SEQ_READ_PREFIX) → str

Build one 150-base read carrying a (column, gRNA, row) triplet.

The layout is the one documented above SEQ_TARGET; a read built here is recovered exactly by the shipped regex / target_sequence / offset_start / window_length defaults.

Parameters:

• column_barcode – 8-base column barcode.

• grna – 21-base gRNA barcode.

• row_barcode – 8-base row barcode.

• prefix – stagger placed before the anchor window.

Returns:

a 150-base read.

Raises:

ValueError – when a barcode is the wrong length — a silently mis-sized barcode would shift every downstream field by that many bases and map to nothing, which is far harder to see than a stop.

Nested helpers

_synth_field._peak() → float

One object’s peak intensity, jittered around the nominal.

spacr/qt/synthetic.py:345