Remote and distributed execution

spaCR can submit any module exposed by spacr-run to another workstation, a Slurm cluster, or a cloud/HPC command-line client. Jobs remain on the remote system when the spaCR GUI closes. Their identifiers, status, settings hashes and latest logs are stored locally and appear in Help → Distributed Jobs.

Execution profiles

An execution profile describes how to reach compute, not an analysis. Profiles never store passwords or API keys:

SSH workstation

Uploads the resolved settings JSON over SSH, starts spacr-run as a detached background process (output redirected to a log, stdin closed) so it continues if the SSH connection drops, and records its exit code.

Slurm cluster

Uploads settings, submits an sbatch script, polls squeue and then sacct, and cancels with scancel. The SSH host may be blank when the Slurm commands are installed locally.

Cloud / custom command

Runs configured submit/status/cancel argument templates. This supports cloud CLIs and site-specific schedulers without embedding vendor credentials in spaCR. The command must print a job identifier containing only letters, digits and _.:+-, and the status command should print a conventional state such as PENDING, RUNNING, SUCCEEDED, FAILED or CANCELLED.

Configure SSH keys, cloud authentication and VPN access outside spaCR. Test the same connection in a terminal before submitting a long run.

Shared data and path mapping

spaCR transfers the settings document, not an image dataset. Images must already be visible to the execution target through shared storage, a mirrored mount or a cloud-aware wrapper command.

When the mount paths differ, set a Local dataset root and Remote dataset root. Every absolute path nested in the settings below the local root is rewritten beneath the remote root. Paths outside it are preserved. For example:

local root:   /mnt/microscopy
remote root:  /cluster/projects/microscopy
local src:    /mnt/microscopy/experiment-7/plate-A
remote src:   /cluster/projects/microscopy/experiment-7/plate-A

Settings can be dragged onto the Distributed Jobs screen, selected with Browse, or handed off directly with Submit remote… in any ordinary module.

Settings files from the GUI

Every module screen with Import settings… has Export settings… next to it. It saves the values currently in the form, exactly what Run would use, to a CSV you choose. Nothing runs: the source folder is not listed or opened, so a form set up for a slow network share is saved at once. The dialog suggests <module>_settings.csv in the source folder.

The file has the layout spaCR writes into <src>/settings/ at the start of every run: a Key,Value header and one row per setting, with lists in Python form ([0, 1, 2]), True/False and empty cells for unset values. Paths are saved exactly as the form shows them. A .json copy with the same settings is written beside it. Import settings… reads the file back to the same form values, and the command line runs it unchanged:

spacr-run measure --settings measure_settings.csv --dry-run
spacr-run measure --settings measure_settings.csv

From Python, spacr.cli.load_settings_file reads it and spacr.cli.write_settings_file writes one. When the cluster mounts the data under another path, change src with --set src=/cluster/projects/plate-A or use the dataset roots of an execution profile.

After Submit remote…, Save settings CSV… on the Distributed Jobs screen saves the snapshot it received as the same file. Submitting is unchanged.

Model checkpoints on the execution target

Exported settings keep the model path and any recorded zoo key, publisher version and SHA-256. Existing pinned files must match that checksum. Missing files resolve only to the exact recorded identity, or an exact zoo key/filename when unpinned; no fuzzy substitution is made. Models are cached below the spaCR home directory (normally ~/.spacr/models), unless SPACR_MODEL_CACHE selects another directory. Processes share a download lock and verify cached checkpoints before downloading. On an offline node set SPACR_MODEL_DOWNLOAD=off before spacr-run: model metadata and weights are not downloaded. Existing local files and verified cached models remain usable; an uncached known model reports the checksum and cache path to copy. Unknown requests fail clearly. Custom paths do not gain a trusted catalogue identity from their filenames.

Jobs and nodes that share one model cache download a missing model once. The first job holds a lock file in the cache and refreshes it while it downloads; the others wait, then reuse the verified file. A lock that a crashed node left behind is removed after SPACR_MODEL_LOCK_STALE_SECONDS without a refresh (600 by default). When SPACR_MODEL_DOWNLOAD=off stops a run, the error ends with a shell command that downloads the checkpoint to the cache path and checks its SHA-256. Run it on a machine with network access that sees the same folder, or run it elsewhere and copy the file to that path.

Command-line workflow

The GUI and CLI share the same profile and job stores. Create a workstation profile and submit a settings export:

spacr-remote profile add gpu-box \
    --backend ssh \
    --host scientist@gpu-box \
    --workdir /shared/spacr \
    --local-root /mnt/lab \
    --remote-root /shared/lab

spacr-remote submit mask \
    --settings mask-settings.csv \
    --profile gpu-box

spacr-remote list --refresh
spacr-remote status JOB_ID --logs
spacr-remote watch JOB_ID --logs
spacr-remote cancel JOB_ID

A Slurm profile can be local or reached through a login host:

spacr-remote profile add lab-slurm \
    --backend slurm \
    --host scientist@login.cluster \
    --workdir /project/lab \
    --runner /project/lab/env/bin/spacr-run \
    --scheduler-options "--partition=gpu --gres=gpu:1 --time=12:00:00"

Custom/cloud templates

Templates are parsed as argument vectors; spaCR does not execute them with shell=True. Supported placeholders are:

{job_id}

The permanent local spaCR identifier.

{module}

The canonical spacr-run module name.

{settings}

The absolute local resolved-settings JSON. A cloud wrapper is responsible for uploading it if needed.

{external_id}

The identifier printed by the submit command, used by status/cancel/log commands.

{profile}

The profile display name.

If a cloud CLI prints JSON, set a Job-ID regular expression with a named id group. For example "jobId":\s*"(?P<id>[A-Za-z0-9-]+)".

Reliability and limitations

• A temporary polling failure is retained as a visible error and does not falsely mark a remote job failed.

• SSH jobs write an exit-code file atomically. Slurm uses accounting after a job leaves the queue.

• Local job records and profiles use advisory locks plus atomic replacement, so a GUI and spacr-remote process do not partially write JSON.

• Cancellation is a request. A scheduler or remote process may take time to stop, and pipeline-level checkpoint semantics still determine how much work can be resumed.

• Cloud templates intentionally do not provide an embedded shell. Put pipelines, uploads, quoting and vendor-specific JSON handling in a reviewed wrapper executable.

Python API

Persistent remote and distributed execution for spaCR pipelines.

The local GUI and spacr.cli already agree on one headless contract:

spacr-run MODULE --settings SETTINGS.json

This module transports that contract to a workstation over SSH, submits it to Slurm, or hands it to a user-configured cloud/HPC command. Submitted jobs are recorded locally and can be polled or cancelled after spaCR itself has closed.

No command uses shell=True. SSH necessarily invokes a remote login shell; all user-controlled values interpolated into its small fixed scripts are quoted with shlex.quote(), and host names are validated separately. Custom command profiles are parsed into argument vectors and substitute each placeholder inside one argument, so shell operators have no special meaning.

class spacr.remote_execution.CommandResult(returncode: int, stdout: str = '', stderr: str = '')

Bases: object

Result returned by the injectable command runner.

Parameters:

• returncode – process exit status, where zero denotes success.

• stdout – captured standard-output text.

• stderr – captured standard-error text.

returncode: int

stderr: str = ''

stdout: str = ''

class spacr.remote_execution.ExecutionProfile(name: str, backend: str, host: str = '', workdir: str = '', local_root: str = '', remote_root: str = '', runner: str = 'spacr-run', scheduler_options: str = '', submit_command: str = '', status_command: str = '', cancel_command: str = '', log_command: str = '', job_id_pattern: str = '', poll_seconds: int = 10, ssh_options: ~typing.Dict[str, str] = <factory>, ssh_interactive: bool = False, ssh_keychain: bool = False)

Bases: object

Connection and scheduler settings for one execution target.

local_root and remote_root describe the same shared or mirrored dataset. Every absolute string nested in the settings is rewritten when it lies below local_root. Image datasets are deliberately not copied: accidental recursive transfer of a multi-terabyte plate is worse than a clear pre-flight error.

For command profiles, command strings are tokenized with shlex.split() and support {job_id}, {module}, {settings} and {external_id} placeholders. They are argument templates, not shell scripts.

Parameters:

• name – human-readable profile name used to select this target.

• backend – execution mechanism, one of BACKENDS.

• host – OpenSSH target for SSH and remote Slurm; blank permits a Slurm scheduler running on the local host.

• workdir – absolute execution directory where spacr-jobs artifacts are created.

• local_root – local dataset prefix paired with remote_root for nested absolute-path rewriting.

• remote_root – equivalent dataset prefix visible to the target host.

• runner – headless executable invoked as runner module --settings path by SSH and Slurm backends.

• scheduler_options – additional shell-free arguments passed to sbatch.

• submit_command – custom-backend argument template containing {settings} and printing a job identifier.

• status_command – custom status template containing {external_id} and returning a state spaCR can normalize.

• cancel_command – custom cancellation template containing {external_id}.

• log_command – optional custom template for fetching job output.

• job_id_pattern – optional regular expression extracting a named id, first capture, or full match from submission output.

• poll_seconds – recommended client polling interval, from 2 through 3600 seconds.

• ssh_options – non-secret SSH configuration overrides, such as user, port, identity-file path and jump host; passwords and OTPs are forbidden.

• ssh_interactive – use a verified persistent SSH session with GUI keyboard-interactive prompts instead of the external OpenSSH command.

• ssh_keychain – remember the login password, never one-time codes or push answers, in the operating-system keychain; off by default.

backend: str

cancel_command: str = ''

classmethod from_dict(value: Mapping[str, Any]) → ExecutionProfile

Construct and validate a profile from JSON-compatible data.

Parameters:

value – stored profile fields keyed by their dataclass names.

host: str = ''

job_id_pattern: str = ''

local_root: str = ''

log_command: str = ''

name: str

poll_seconds: int = 10

remote_root: str = ''

runner: str = 'spacr-run'

scheduler_options: str = ''

ssh_interactive: bool = False

ssh_keychain: bool = False

ssh_options: Dict[str, str]

status_command: str = ''

submit_command: str = ''

to_dict() → Dict[str, Any]

Return a JSON-safe representation.

validate() → ExecutionProfile

Validate the profile and return self for fluent callers.

workdir: str = ''

class spacr.remote_execution.JobStore(path: PathLike | None = None)

Bases: object

Atomic persistent store for remote job metadata.

Parameters:

path – optional jobs JSON path; by default state_directory()/jobs.json. The advisory lock uses a sibling path with .lock appended.

get(job_id: str) → RemoteJob

Return a job by full ID or unambiguous prefix.

Parameters:

job_id – complete job identifier or an unambiguous prefix.

list() → List[RemoteJob]

Return newest jobs first.

save(job: RemoteJob) → None

Insert or replace one job atomically.

Parameters:

job – remote-job record to persist.

class spacr.remote_execution.ProfileStore(path: PathLike | None = None)

Bases: object

Atomic persistent store for execution profiles.

Parameters:

path – optional profiles JSON path; by default state_directory()/profiles.json. The advisory lock uses a sibling path with .lock appended.

delete(name: str) → bool

Delete one profile; return whether it existed.

Parameters:

name – profile name to delete case-insensitively.

get(name: str) → ExecutionProfile

Return a named profile or raise a user-facing error.

Parameters:

name – profile name to match case-insensitively.

list() → List[ExecutionProfile]

Return profiles sorted by case-insensitive name.

save(profile: ExecutionProfile) → None

Insert or replace one profile atomically.

Parameters:

profile – validated execution profile to persist.

exception spacr.remote_execution.RemoteExecutionError

Bases: RuntimeError

A profile, submission, polling, or cancellation error.

These errors are safe to show directly in the GUI: passwords and environment variables are never included in command rendering.

class spacr.remote_execution.RemoteJob(job_id: str, module: str, profile_name: str, backend: str, status: str = 'submitting', external_id: str = '', created_utc: str = <factory>, updated_utc: str = <factory>, settings_path: str = '', settings_sha256: str = '', remote_settings_path: str = '', remote_job_dir: str = '', log_reference: str = '', log_tail: str = '', exit_code: int | None = None, error: str = '', profile: ~typing.Dict[str, ~typing.Any] = <factory>)

Bases: object

Persistent local record of one submitted job.

Parameters:

• job_id – locally generated stable identifier for the job.

• module – resolved spaCR command-line module key to execute.

• profile_name – name of the execution profile used for submission.

• backend – backend recorded at submission time.

• status – compact lifecycle state, initially "submitting".

• external_id – validated process, scheduler, or provider identifier.

• created_utc – ISO-8601 construction timestamp.

• updated_utc – timestamp replaced whenever JobStore persists the record.

• settings_path – retained local path of the mapped settings JSON.

• settings_sha256 – SHA-256 digest of the exact serialized settings.

• remote_settings_path – uploaded settings path for SSH and Slurm.

• remote_job_dir – target-side directory containing job artifacts.

• log_reference – backend log location or custom-backend marker.

• log_tail – most recently fetched output or availability message.

• exit_code – normalized process exit status, or None until known.

• error – latest submission, polling, or cancellation failure; cleared after a successful refresh or cancellation.

• profile – submission-time profile snapshot, retained so the job remains operable after profile edits or deletion.

backend: str

created_utc: str

error: str = ''

exit_code: int | None = None

external_id: str = ''

classmethod from_dict(value: Mapping[str, Any]) → RemoteJob

Construct a job from a stored mapping, tolerating future fields.

Parameters:

value – stored job fields keyed by their dataclass names.

job_id: str

log_reference: str = ''

log_tail: str = ''

module: str

profile: Dict[str, Any]

profile_name: str

remote_job_dir: str = ''

remote_settings_path: str = ''

settings_path: str = ''

settings_sha256: str = ''

status: str = 'submitting'

to_dict() → Dict[str, Any]

Return a JSON-safe representation.

updated_utc: str

class spacr.remote_execution.RemoteJobManager(profile_store: ~spacr.remote_execution.ProfileStore | None = None, job_store: ~spacr.remote_execution.JobStore | None = None, runner: ~typing.Callable[[...], ~spacr.remote_execution.CommandResult] = <function _run_command>)

Bases: object

Submit, monitor, cancel and inspect persistent remote jobs.

All methods are synchronous and may perform network I/O. GUI callers must invoke them on a worker thread; the shipped Distributed Jobs screen does.

Parameters:

• profile_store – injectable persistent profile store; omitted creates the default ProfileStore.

• job_store – injectable persistent job store; omitted creates the default JobStore.

• runner – shell-free callable accepting an argument vector, timeout, and optional input and returning CommandResult.

cancel(job_id: str) → RemoteJob

Request cancellation and persist the result.

Parameters:

job_id – complete local job identifier or unambiguous prefix.

logs(job_id: str, lines: int = 200) → str

Retrieve and persist the tail of one remote job’s log.

Parameters:

job_id – complete local job identifier or unambiguous prefix.

refresh(job_id: str, *, include_logs: bool = True) → RemoteJob

Poll one non-terminal job and optionally retain its latest log tail.

Parameters:

job_id – complete local job identifier or unambiguous prefix.

refresh_all(*, include_logs: bool = False) → List[RemoteJob]

Poll every active job and return the complete newest-first list.

submit(module: str, settings: Mapping[str, Any], profile_name: str) → RemoteJob

Submit resolved settings through a named execution profile.

Parameters:

• module – headless spaCR module name accepted by spacr-run.

• settings – resolved module settings to serialize for the job.

• profile_name – name of the execution profile to use.

spacr.remote_execution.map_settings_paths(value: Any, local_root: str, remote_root: str) → Any

Recursively map absolute paths below one local root to a remote root.

Non-path strings and paths outside the configured root are unchanged. Mapping keys are intentionally preserved: setting names are not paths.

Parameters:

• value – nested settings value whose absolute path strings are mapped.

• local_root – local dataset root that mapped paths must lie below.

• remote_root – remote dataset root that replaces local_root.

spacr.remote_execution.state_directory() → Path

Return the persistent directory for profiles, jobs, settings and logs.

SPACR_REMOTE_STATE_DIR is intentionally supported for tests, portable deployments, and managed lab installations. Otherwise XDG state storage is used on Linux and a conventional per-user directory elsewhere.