Error in patch motion child process terminated unexpectedly

lmthomas · July 27, 2026, 3:56pm

Revisiting as this topic as it has come up before but looking at the answers they seem more specific to earlier versions, I have just upgraded to 5.0.6.

Error is

Child process with PID terminated unexpectedly with exit code 1

The error occurs right after CUDA is called from what I gather from the log file;

“[CPU: 280.3 MB Avail: 236.07 GB]

Calling CUDA init from 3199518

CPU: 280.3 MB Avail: 236.06 GB

Calling CUDA init from 3199519

CPU: 371.5 MB Avail: 236.07 GB

Child process with PID 3199518 terminated unexpectedly with exit code 1.

CPU: 371.5 MB Avail: 236.07 GB

Child process with PID 3199519 terminated unexpectedly with exit code 1.

CPU: 371.5 MB Avail: 236.07 GB

Unregistering child process pids…

CPU: 371.5 MB Avail: 236.07 GB

Outputting final results…

CPU: 375.5 MB Avail: 236.07 GB

'uid', 'movie_blob/path', 'movie_blob/shape', 'movie_blob/psize_A', 'movie_blob/is_gain_corrected', 'movie_blob/format', 'movie_blob/has_defect_file', 'movie_blob/import_sig', 'micrograph_blob/path', 'micrograph_blob/idx', 'micrograph_blob/shape', 'micrograph_blob/psize_A', 'micrograph_blob/format', 'micrograph_blob/is_background_subtracted', 'micrograph_blob/vmin', 'micrograph_blob/vmax', 'micrograph_blob/import_sig', 'micrograph_blob_non_dw/path', 'micrograph_blob_non_dw/idx', 'micrograph_blob_non_dw/shape', 'micrograph_blob_non_dw/psize_A', 'micrograph_blob_non_dw/format', 'micrograph_blob_non_dw/is_background_subtracted', 'micrograph_blob_non_dw/vmin', 'micrograph_blob_non_dw/vmax', 'micrograph_blob_non_dw/import_sig', 'micrograph_blob_non_dw_AB/path', 'micrograph_blob_non_dw_AB/idx', 'micrograph_blob_non_dw_AB/shape', 'micrograph_blob_non_dw_AB/psize_A', 'micrograph_blob_non_dw_AB/format', 'micrograph_blob_non_dw_AB/is_background_subtracted', 'micrograph_blob_non_dw_AB/vmin', 'micrograph_blob_non_dw_AB/vmax', 'micrograph_blob_non_dw_AB/import_sig', 'micrograph_thumbnail_blob_1x/path', 'micrograph_thumbnail_blob_1x/idx', 'micrograph_thumbnail_blob_1x/shape', 'micrograph_thumbnail_blob_1x/format', 'micrograph_thumbnail_blob_1x/binfactor', 'micrograph_thumbnail_blob_1x/micrograph_path', 'micrograph_thumbnail_blob_1x/vmin', 'micrograph_thumbnail_blob_1x/vmax', 'micrograph_thumbnail_blob_2x/path', 'micrograph_thumbnail_blob_2x/idx', 'micrograph_thumbnail_blob_2x/shape', 'micrograph_thumbnail_blob_2x/format', 'micrograph_thumbnail_blob_2x/binfactor', 'micrograph_thumbnail_blob_2x/micrograph_path', 'micrograph_thumbnail_blob_2x/vmin', 'micrograph_thumbnail_blob_2x/vmax', 'background_blob/path', 'background_blob/idx', 'background_blob/binfactor', 'background_blob/shape', 'background_blob/psize_A', 'rigid_motion/type', 'rigid_motion/path', 'rigid_motion/idx', 'rigid_motion/frame_start', 'rigid_motion/frame_end', 'rigid_motion/zero_shift_frame', 'rigid_motion/psize_A', 'spline_motion/type', 'spline_motion/path', 'spline_motion/idx', 'spline_motion/frame_start', 'spline_motion/frame_end', 'spline_motion/zero_shift_frame', 'spline_motion/psize_A'

CPU: 383.3 MB Avail: 236.06 GB

Done outputting results”

I did try to run

cryosparcw newcuda /usr/local/cuda

but is has an error “no such command”

I did run some of the command in the do this before posting and it appears that when I run

/sbin/ldconfig -p | grep -i cuda

It shows cuda v 11.2 and 10.1. while in the metatdata file from the job cuda v12 is listed.

So I am at a loss as to what to try.

Thank You,

Len Thomas

wtempel · July 27, 2026, 9:01pm

This command does not apply to CryoSPARC versions ≥ 4.4, which include the relevant CUDA libraries and require certain minimum versions of the nvidia driver.
What are the outputs of these commands

on the worker where the job ran and failed:
```
uname -a
nvidia-smi
```
on the CryoSPARC master (with appropriate project and job IDs):
```
cryosparcm job log P999 J123 | tail -n 40
```

lmthomas · July 27, 2026, 9:27pm

Here are the outputs of the commands you asked for.

>uname -a
Linux spgpu 4.18.0-477.15.1.el8_8.x86_64 #1 SMP Wed Jun 28 15:04:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 11142 G /usr/libexec/Xorg 125MiB |
| 0 N/A N/A 11381 G /usr/bin/gnome-shell 103MiB |
| 1 N/A N/A 11142 G /usr/libexec/Xorg 4MiB |
±----------------------------------------------------------------------------+

>cryosparcm job log P45 J289 | tail -n 40
File “compute/gpu/gpucore.py”, line 59, in compute.gpu.gpucore.initialize
RuntimeError: Nvidia driver supports CUDA Toolkit version 12.0 and older, but CUDA 12.8 support is required. Please update Nvidia driver to version 570.26 or newer.
2026-07-27 13:01:24,713 core heartbeat INFO | ========= Updating heartbeat
2026-07-27 13:01:34,718 core heartbeat INFO | ========= Updating heartbeat
2026-07-27 13:01:44,724 core heartbeat INFO | ========= Updating heartbeat
/spshared/apps/cryosparc4/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/spshared/apps/cryosparc4/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide
ret = ret.dtype.type(ret / rcount)
2026-07-27 13:01:50,745 core.data_management _dump_job_gridfs INFO | Dumping all of job’s images in the database to /data/UserData/kolelong/CS-fo-krios/J289/gridfs_data…
2026-07-27 13:01:50,746 core.data_management _dump_job_gridfs INFO | Done. Exported 0 images in 0.00s
2026-07-27 13:01:50,749 core.data_management _dump_job_events INFO | Done. Dumped 42 files in 0.00s
2026-07-27 13:01:50,756 core.data_management dump_job INFO | Dumped Job P45-J289 (Patch Motion Correction) in 0.01s
========= main process now complete at 2026-07-27 13:01:50.778347
2026-07-27 13:01:54,799 core heartbeat INFO | ========= Updating heartbeat
2026-07-27 13:01:54,802 core heartbeat INFO | Attempted to heartbeat for inactive Job P45-J289 (Patch Motion Correction) with status completed
2026-07-27 13:01:54,802 core monitor ERROR | ========= heartbeat failed
Traceback (most recent call last):
File “/spshared/apps/cryosparc4/cryosparc_worker/cli/monitor.py”, line 34, in monitor
assert heartbeat(conf.project, conf.job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
2026-07-27 13:02:04,802 core heartbeat INFO | ========= Updating heartbeat
2026-07-27 13:02:04,805 core heartbeat INFO | Attempted to heartbeat for inactive Job P45-J289 (Patch Motion Correction) with status completed
2026-07-27 13:02:04,805 core monitor ERROR | ========= heartbeat failed
Traceback (most recent call last):
File “/spshared/apps/cryosparc4/cryosparc_worker/cli/monitor.py”, line 34, in monitor
assert heartbeat(conf.project, conf.job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
2026-07-27 13:02:14,806 core heartbeat INFO | ========= Updating heartbeat
2026-07-27 13:02:14,809 core heartbeat INFO | Attempted to heartbeat for inactive Job P45-J289 (Patch Motion Correction) with status completed
2026-07-27 13:02:14,809 core monitor ERROR | ========= heartbeat failed
Traceback (most recent call last):
File “/spshared/apps/cryosparc4/cryosparc_worker/cli/monitor.py”, line 34, in monitor
assert heartbeat(conf.project, conf.job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
2026-07-27 13:02:14,809 core monitor ERROR | Killing main process pid 129350
2026-07-27 13:02:14,809 core monitor ERROR | ************* Connection to cryosparc command lost. Heartbeat failed 3 consecutive times.

wtempel · July 28, 2026, 6:20pm

For use with CryoSPARC v5, the driver needs to be updated to version 570.26 or newer.

lmthomas · July 28, 2026, 10:11pm

Thank you for the confirmation. I noticed that when looking at the log file.