Dear Cryosparc Team,
We are running cryosparc on a HPC resource for a multi-user environment. Currently we have 5.0.7+260921 installed, but the error already occurred with a previous 4.7.0 installation – and was the reason for us doing a fresh install with the most recent version .
Independent of the dataset, user and compute node architecture we consistently get a SIGSEGV error during reference based motion correction jobs. This error apparently occurs after calculation of hyperparameters (as we do get a downloadable refmotion_hyperparameters output file) but before doseweights are calculated (the download link leads to a blank). Interestingly, other job types run without problem s
What did we already tr y:
- We already tried this on A100 and H100 compute nodes (both with and without MIG activated)
- We tried cryosparc 4.7.0 and 5.0.7+260921
- We also tried to set NO_PAGELOCK to true (worker; recommended here Reference based motion correction job is terminating abnormally during cross validation - #36 by wtempel ) and/or IO_URING to false (master; recommended here https://discuss.cryosparc.com/t/sigsegv-in-core-ioengine-core-so-across-multiple-job-types/ 252 45/4)
- We tried to use single GPU only
- We tried to deactivate slicing GPU also computes trajectories (using 2 GPUs)
Environment we are using:
Both the A100 and the H100 nodes are running on driver 615.71.10 with CUDA 13.4 (as reported by nvidia-smi and confirmed by logfile during worker installation).
On a worker node (A100 type):
ramses31907 15:46:05 /projects/strubitem/software/cryosparc-5.0.7/cryosparc_worker [0]$ eval $(./bin/cryosparcw env)
ramses31907 15:47:36 /projects/strubitem/software/cryosparc-5.0.7/cryosparc_worker [0]$ env | grep PATH
__MODULES_SHARE_MODULEPATH=/projects/strubitem/modules/3DEM:2
MANPATH=/usr/share/man:
__MODULES_SHARE_MANPATH=:1
CRYOSPARC_DB_PATH=/projects/strubitem/databases/v2027
PYTHONPATH=/projects/strubitem/software/cryosparc-5.0.7/cryosparc_worker
MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LD_PRELOAD
NUMBA_CUDA_INCLUDE_PATH=/projects/strubitem/software/cryosparc-5.0.7/cryosparc_worker/.pixi/envs/worker/include
DEBUGINFOD_IMA_CERT_PATH=/etc/keys/ima:
PATH=/projects/strubitem/software/cryosparc-5.0.7/cryosparc_worker/.pixi/envs/worker/bin:/projects/sw/eb/bin:/projects/sw/eb/bin:/projects/strubitem/software/cryosparc-5.0.7/cryosparc_master/deps/external/mongodb/bin:/projects/strubitem/software/cryosparc-5.0.7/cryosparc_master/bin:/projects/strubitem/software/cryosparc-5.0.7/cryosparc_master/.pixi/envs/master/bin:/projects/strubitem/software/cryosparc-current/cryosparc_master/bin:/home/ebehrman/.local/bin:/home/ebehrman/bin:/projects/sw/eb/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
MODULEPATH=/etc/scl/modulefiles:/etc/scl/modulefiles:/projects/sw/eb/arch/zen4/modules/ai:/projects/sw/eb/arch/zen4/modules/bio:/projects/sw/eb/arch/zen4/modules/cae:/projects/sw/eb/arch/zen4/modules/chem:/projects/sw/eb/arch/zen4/modules/compiler:/projects/sw/eb/arch/zen4/modules/data:/projects/sw/eb/arch/zen4/modules/devel:/projects/sw/eb/arch/zen4/modules/geo:/projects/sw/eb/arch/zen4/modules/lang:/projects/sw/eb/arch/zen4/modules/lib:/projects/sw/eb/arch/zen4/modules/math:/projects/sw/eb/arch/zen4/modules/mpi:/projects/sw/eb/arch/zen4/modules/numlib:/projects/sw/eb/arch/zen4/modules/perf:/projects/sw/eb/arch/zen4/modules/phys:/projects/sw/eb/arch/zen4/modules/system:/projects/sw/eb/arch/zen4/modules/tobedone:/projects/sw/eb/arch/zen4/modules/toolchain:/projects/sw/eb/arch/zen4/modules/tools:/projects/sw/eb/arch/zen4/modules/vis:/etc/scl/modulefiles:/projects/sw/noeb/arch/zen4/modules:/projects/ramses-software/modules:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles:/projects/strubitem/modules/3DEM
ramses31907 15:47:44 /projects/strubitem/software/cryosparc-5.0.7/cryosparc_worker [0]$ uname -a
Linux ramses31907 5.14.0-570.150.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Oct 19 09:39:38 EDT 2026 x86_64 x86_64 x86_64 GNU/Linux
ramses31907 15:47:50 /projects/strubitem/software/cryosparc-5.0.7/cryosparc_worker [0]$ free -g
total used free shared buff/cache available
Mem: 503 21 483 0 2 481
Swap: 3 0 3
ramses31907 15:47:55 /projects/strubitem/software/cryosparc-5.0.7/cryosparc_worker [0]$ nvidia-smi
Thu Nov 5 15:48:01 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 615.71.10 KMD Version: 615.71.10 CUDA UMD Version: 13.4 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100-PCIE-40GB On | 00000000:81:00.0 Off | On |
| N/A 28C P0 32W / 250W | 203MiB / 40960MiB | N/A Default |
| | | Enabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| MIG devices: |
+------------------+----------------------------------+-----------+-----------------------+
| GPU GI CI MIG | Shared Memory-Usage | Vol| Shared |
| ID ID Dev | Shared BAR1-Usage | SM Unc| CE ENC DEC OFA JPG |
| | | ECC| |
|==================+==================================+===========+=======================|
| 0 1 0 0 | 102MiB / 20096MiB | 42 0 | 3 0 2 0 0 |
| | 0MiB / 12196MiB | | |
+------------------+----------------------------------+-----------+-----------------------+
| 0 2 0 1 | 102MiB / 20096MiB | 42 0 | 3 0 2 0 0 |
| | 0MiB / 12196MiB | | |
+------------------+----------------------------------+-----------+-----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
Representative job log:
================= CRYOSPARC =================
Project P1 Job J38
Master ramses4.host Port 61000
===========================================================================
MAIN PROCESS PID 644996
2026-11-05 02:46:49,330 core monitor INFO | MONITOR PROCESS PID 645006
2026-11-05 02:46:49,330 core monitor INFO | ========= monitor process now waiting for main process
2026-11-05 02:46:49,330 core heartbeat INFO | ========= Updating heartbeat
================= CRYOSPARC =================
Project P1 Job J38
Master ramses4.host Port 61000
===========================================================================
MAIN PROCESS PID 644996
========= updating job startup information at 2026-11-05 02:46:49.953626
========= now starting main process at 2026-11-05 02:46:51.158363
2026-11-05 02:46:51,621 core run INFO | Running job J38 of type reference_motion_correction
2026-11-05 02:46:51,621 core run INFO | Running job on hostname ramses_gpus
2026-11-05 02:46:51,626 core run INFO | Allocated Resources: lane='ramses_gpus' lane_type='cluster' hostname='ramses_gpus' target=SchedulerTarget(cache_path='/local/', cache_reserve_mb=10000, cache_quota_mb=None, lane='ramses_gpus', name='ramses_gpus', title='ramses_gpus', desc=None, hostname='ramses_gpus', worker_bin_path='/projects/strubitem/software/cryosparc-current/cryosparc_worker/bin/cryosparcw', config=Cluster(send_cmd_tpl='{{ command }}', qsub_cmd_tpl='sbatch {{ script_path_abs }}', qstat_cmd_tpl='squeue -j {{ cluster_job_id }}', qdel_cmd_tpl='scancel {{ cluster_job_id }}', qinfo_cmd_tpl="sinfo --format='%.8N %.6D %.10P %.6T %.14C %.5c %.6z %.7m %.7G %.9d %20E'", type='cluster', script_tpl='#!/usr/bin/env bash\n#### cryoSPARC cluster submission script template for SLURM\n## Available variables:\n## {{ run_cmd }} - the complete command string to run the job\n## {{ num_cpu }} - the number of CPUs needed\n## {{ num_gpu }} - the number of GPUs needed.\n## Note: the code will use this many GPUs starting from dev id 0\n## the cluster scheduler or this script have the responsibility\n## of setting CUDA_VISIBLE_DEVICES so that the job code ends up\n## using the correct cluster-allocated GPUs.\n## {{ ram_gb }} - the amount of RAM needed in GB\n## {{ job_dir_abs }} - absolute path to the job directory\n## {{ project_dir_abs }} - absolute path to the project dir\n## {{ job_log_path_abs }} - absolute path to the log file for the job\n## {{ worker_bin_path }} - absolute path to the cryosparc worker command\n## {{ run_args }} - arguments to be passed to cryosparcw run\n## {{ project_uid }} - uid of the project\n## {{ job_uid }} - uid of the job\n## {{ job_creator }} - name of the user that created the job (may contain spaces)\n## {{ cryosparc_username }} - cryosparc username of the user that created the job (usually an email)\n##\n## What follows is a simple SLURM script:\n\n#SBATCH --job-name cryosparc_{{ project_uid }}_{{ job_uid }}\n#SBATCH --time={{ (wallhours|default(48))|int }}:00:00\n#SBATCH --partition=gpu\n\n#SBATCH --output={{ job_log_path_abs }}\n#SBATCH --error={{ job_log_path_abs }}\n\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --cpus-per-task={{ num_cpu }}\n#SBATCH --mem={{ (ram_gb|float * (ram_multiplier|default(2))|float)|int}}G\n\n#SBATCH --gres localtmp:{{ (ssd_space|default(100))|int}}g\n#SBATCH --gpus=h100_2g.24gb:{{ num_gpu }}\n#SBATCH --gres-flags=enforce-binding\n\nmodule load cryosparc-current\nexport CUSTOM_DYNAMIC_SSD_PATH="/local/${SLURM_JOB_USER}.${SLURM_JOB_ID}"\n\n{{ run_cmd }}\n', custom_vars={'ram_multiplier': '3', 'ssd_space': '150', 'wallhours': '48'}, tpl_vars=['run_cmd', 'job_dir_abs', 'ram_gb', 'num_gpu', 'worker_bin_path', 'ram_multiplier', 'run_args', 'command', 'wallhours', 'job_log_path_abs', 'job_uid', 'cluster_job_id', 'num_cpu', 'project_dir_abs', 'cryosparc_username', 'project_uid', 'job_creator', 'ssd_space'], custom_var_names=['ram_multiplier', 'wallhours', 'ssd_space'])) slots=ResourceSlots(CPU=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17], GPU=[0, 1, 2], RAM=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]) fixed=FixedResourceSlots(SSD=False) licenses_acquired=3
2026-11-05 02:46:59,980 core heartbeat INFO | ========= Updating heartbeat
Transparent hugepages setting: [always] madvise never
Running job on hostname %s ramses_gpus
Allocated Resources : lane='ramses_gpus' lane_type='cluster' hostname='ramses_gpus' target=SchedulerTarget(cache_path='/local/', cache_reserve_mb=10000, cache_quota_mb=None, lane='ramses_gpus', name='ramses_gpus', title='ramses_gpus', desc=None, hostname='ramses_gpus', worker_bin_path='/projects/strubitem/software/cryosparc-current/cryosparc_worker/bin/cryosparcw', config=Cluster(send_cmd_tpl='{{ command }}', qsub_cmd_tpl='sbatch {{ script_path_abs }}', qstat_cmd_tpl='squeue -j {{ cluster_job_id }}', qdel_cmd_tpl='scancel {{ cluster_job_id }}', qinfo_cmd_tpl="sinfo --format='%.8N %.6D %.10P %.6T %.14C %.5c %.6z %.7m %.7G %.9d %20E'", type='cluster', script_tpl='#!/usr/bin/env bash\n#### cryoSPARC cluster submission script template for SLURM\n## Available variables:\n## {{ run_cmd }} - the complete command string to run the job\n## {{ num_cpu }} - the number of CPUs needed\n## {{ num_gpu }} - the number of GPUs needed.\n## Note: the code will use this many GPUs starting from dev id 0\n## the cluster scheduler or this script have the responsibility\n## of setting CUDA_VISIBLE_DEVICES so that the job code ends up\n## using the correct cluster-allocated GPUs.\n## {{ ram_gb }} - the amount of RAM needed in GB\n## {{ job_dir_abs }} - absolute path to the job directory\n## {{ project_dir_abs }} - absolute path to the project dir\n## {{ job_log_path_abs }} - absolute path to the log file for the job\n## {{ worker_bin_path }} - absolute path to the cryosparc worker command\n## {{ run_args }} - arguments to be passed to cryosparcw run\n## {{ project_uid }} - uid of the project\n## {{ job_uid }} - uid of the job\n## {{ job_creator }} - name of the user that created the job (may contain spaces)\n## {{ cryosparc_username }} - cryosparc username of the user that created the job (usually an email)\n##\n## What follows is a simple SLURM script:\n\n#SBATCH --job-name cryosparc_{{ project_uid }}_{{ job_uid }}\n#SBATCH --time={{ (wallhours|default(48))|int }}:00:00\n#SBATCH --partition=gpu\n\n#SBATCH --output={{ job_log_path_abs }}\n#SBATCH --error={{ job_log_path_abs }}\n\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --cpus-per-task={{ num_cpu }}\n#SBATCH --mem={{ (ram_gb|float * (ram_multiplier|default(2))|float)|int}}G\n\n#SBATCH --gres localtmp:{{ (ssd_space|default(100))|int}}g\n#SBATCH --gpus=h100_2g.24gb:{{ num_gpu }}\n#SBATCH --gres-flags=enforce-binding\n\nmodule load cryosparc-current\nexport CUSTOM_DYNAMIC_SSD_PATH="/local/${SLURM_JOB_USER}.${SLURM_JOB_ID}"\n\n{{ run_cmd }}\n', custom_vars={'ram_multiplier': '3', 'ssd_space': '150', 'wallhours': '48'}, tpl_vars=['run_cmd', 'job_dir_abs', 'ram_gb', 'num_gpu', 'worker_bin_path', 'ram_multiplier', 'run_args', 'command', 'wallhours', 'job_log_path_abs', 'job_uid', 'cluster_job_id', 'num_cpu', 'project_dir_abs', 'cryosparc_username', 'project_uid', 'job_creator', 'ssd_space'], custom_var_names=['ram_multiplier', 'wallhours', 'ssd_space'])) slots=ResourceSlots(CPU=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17], GPU=[0, 1, 2], RAM=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]) fixed=FixedResourceSlots(SSD=False) licenses_acquired=3
FSC No-Mask... 0.143 at 92.929 radwn. 0.5 at 78.132 radwn. Took 0.344s.
FSC With Mask... 0.143 at 98.500 radwn. 0.5 at 97.680 radwn. Took 0.146s.
2026-11-05 02:47:10,040 core heartbeat INFO | ========= Updating heartbeat
gpufft: creating new cufft plan (plan id 0 pid 644996)
gpu_id 0
ndims 3
dims 200 200 200
inembed 200 200 202
istride 1
idist 8080000
onembed 200 200 101
ostride 1
odist 4040000
batch 1
type R2C
wkspc automatic
Python traceback:
/projects/strubitem/software/cryosparc-current/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numba/cuda/cudadrv/nvrtc.py:257: UserWarning: NVRTC log messages whilst compiling kernel:
kernel(963): warning #177-D: variable "Nb2p1" was declared but never referenced
int Nb2p1 = Nb2 + 1;
^
Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"
warnings.warn(msg)
WARNING: io_uring support disabled (not supported by kernel), I/O performance may degrade
refmotion worker 4 (NVIDIA H100 MIG 2g.24gb)
scale (alpha): 6.831359
noise model (sigma2): 140.951462
TIME (s) SECTION
0.000022440 sanity
0.389275663 read movie
0.000002340 get gain, defects
0.087467881 read bg
0.001379703 read rigid
0.359411989 prep_movie
0.477621511 extract from frames
0.000312858 extract from refs
0.000000270 adj
0.000000020 bfactor
0.003718332 rigid motion correct
0.000174479 get noise, scale
1.319387488 --- TOTAL ---
[….]
refmotion worker 5 (NVIDIA H100 MIG 2g.24gb)
Min BFGS iterations: 241
Max BFGS iterations: 675
AVG TIME (s) SECTION
0.000000000 Cache IO
0.128127276 Optimize trajectories
0.009243391 Compute cross-validation
0.001075238 Save trajectory
0.138445905 --- TOTAL (Cros-Val) ---
refmotion worker 5 (NVIDIA H100 MIG 2g.24gb)
Min BFGS iterations: 382
Max BFGS iterations: 549
AVG TIME (s) SECTION
0.000000000 Cache IO
0.283514511 Optimize trajectories
0.022660346 Compute cross-validation
0.001888914 Save trajectory
0.308063771 --- TOTAL (Cros-Val) ---
Received SIGSEGV (addr=0000000000000000)
/projects/strubitem/software/cryosparc-current/cryosparc_worker/core/ioengine/core.so(traceback_signal_handler+0x113)[0x7fb785fb4873]
/lib64/libc.so.6(+0x3ebf0)[0x7fb79f63ebf0]
/projects/strubitem/software/cryosparc-current/cryosparc_worker/compute/jobs/motion_correction/bin_motion.so(+0x11520)[0x7fb7500fe520]
/projects/strubitem/software/cryosparc-current/cryosparc_worker/core/ioengine/../../lib/libgomp.so.1(GOMP_parallel+0x43)[0x7fb78d0d0c42]
/projects/strubitem/software/cryosparc-current/cryosparc_worker/compute/jobs/motion_correction/bin_motion.so(fspace_herm_rot_sum_real_AconjB+0x1d4)[0x7fb75011e164]
/projects/strubitem/software/cryosparc-current/cryosparc_worker/compute/jobs/motion_correction/bin_motion.so(read_movie_reconstruct_particles+0x2611)[0x7fb750125fb1]
/projects/strubitem/software/cryosparc-current/cryosparc_worker/compute/jobs/motion_correction/bin_motion.so(refmotion_pipeline_worker+0x4b6)[0x7fb750128016]
/lib64/libc.so.6(+0x8a19a)[0x7fb79f68a19a]
/lib64/libc.so.6(+0x10f100)[0x7fb79f70f100]
rax 0000000000000000 rbx 0000000000000000 rcx 0000000000000000 rdx 00007e307fb5ae00
rsi 0000000000007da0 rdi 00007faed000ee84 rbp 00007fb6d5ff9da0 rsp 00007fb6d5ff9ce0
r8 0000000000000190 r9 0000000000000027 r10 00000000000000c9 r11 0000000000000212
r12 0000000000001f67 r13 0000000000000004 r14 0000000000001f68 r15 00007fb6d5ff9e60
8b 0c c1 49 83 fc 02 0f 86 53 03 00 00 48 8d 71 04 48 89 d0 48 29 f0 48 83 f8 18 0f 86
3f 03 00 00 49 83 fc 06 0f 86 b4 03 00 00 48 8b 74 24 78 31 c0 48 c1 e6 05 66 2e 0f 1f
84 00 00 00 00 00
--> c5 fc 10 04 01 c5 fc 58 04 02 c5 fc 11 04 02 48 83 c0 20 48 39 f0 75 e8 48 8b 44
24 70 48 89 c6 41 f6 c6 07 0f 84 6d 03 00 00 48 83 7c 24 58 02 48 8b 7c 24 60 0f 86 70
03 00 00 c5 f8 77 4c 8d