I am writing here to report and want to seek your assistance regarding an issue I encountered while testing the operation of CryoSPARC v5.0.3.
I followed the T20S introductory tutorial, downloaded the EMPIAR-10025 dataset, and ran the prepared workflow using the default parameters.
Job 1 (Import Movies) completed successfully using *.tif and norm-ambibox05-0.mrc as input files. However, when I started Job 2 (Patch Motion Correction) using the outputs from Job 1, the job failed. The log displayed the following message:Child process with PID ### terminated unexpectedly with exit code 1. After this, the motion correction process did not proceed.
To troubleshoot, I downgraded CryoSPARC to versions 5.0.2 and 4.7.2 and tried the exact same procedure. Although the specific error messages differed slightly depending on the version, the job consistently failed to proceed normally.
I checked the log files and noticed that an AssertionError appeared in the job_log across all tested versions. I suspect this might be the root cause of the problem.
I have attached the error message output from Job 2 for your reference, and the errors occur in a machine of the following specs:
- Ubuntu24.04 installed on Windows11 wsl2
- CUDA ver. 12.8
- nVidia Driver ver. 595.79 (only installed in Windows)
- GeForce RTX 4080 Super x 2
Best regards,
Attatched log files:
- Event_log of J2:
P1 J2: Patch Motion Correction
CryoSPARC Running Version: v5.0.3
| DETAIL | VALUE
±----------------±-------------------+
| Title
| tuto01 empiar10025
| Description |
±----------------±-------------------+
| ID | P1
±----------------±-------------------+
Project Details
±----------------±-------------------+
|
|
|
|
±----------------±-------------------+
| Created by user | Tus Seibukka |
±----------------±-------------------+
Job Details
±----------------±--------------------------------------------------------+
| DETAIL | VALUE |
±----------------±--------------------------------------------------------+
| Title | |
±----------------±--------------------------------------------------------+
| Description | |
±----------------±--------------------------------------------------------+
| ID | J2 |
±----------------±--------------------------------------------------------+
| Job type | Patch Motion Correction |
±----------------±--------------------------------------------------------+
| Status | completed |
±----------------±--------------------------------------------------------+
| Created by user | Tus Seibukka |
±----------------±--------------------------------------------------------+
| Created | Wed Mar 18 2026 19:00:27 GMT+0900 (Japan Standard Time) |
±----------------±--------------------------------------------------------+
| Queued | Wed Mar 18 2026 19:34:19 GMT+0900 (Japan Standard Time) |
±----------------±--------------------------------------------------------+
| Launched | Wed Mar 18 2026 19:34:23 GMT+0900 (Japan Standard Time) |
±----------------±--------------------------------------------------------+
| Started | Wed Mar 18 2026 19:34:25 GMT+0900 (Japan Standard Time) |
±----------------±--------------------------------------------------------+
| Waiting | - |
±----------------±--------------------------------------------------------+
| Killed | - |
±----------------±--------------------------------------------------------+
| Completed | Wed Mar 18 2026 19:35:03 GMT+0900 (Japan Standard Time) |
±----------------±--------------------------------------------------------+
| Failed | - |
±----------------±--------------------------------------------------------+
| Last accessed | Thu Mar 19 2026 11:54:19 GMT+0900 (Japan Standard Time) |
±----------------±--------------------------------------------------------+
| Size | 88.62 KB |
±----------------±--------------------------------------------------------+
Inputs
movies
J1.imported_movies
±--------------------------------+
| MOVIE_BLOB |
±--------------------------------+
| J1.imported_movies.movie_blob.F |
±--------------------------------+
±-----------------------------------+
| GAIN_REF_BLOB |
±-----------------------------------+
| J1.imported_movies.gain_ref_blob.F |
±-----------------------------------+
±-----------------------------------+
| MSCOPE_PARAMS |±-----------------------------------+
| J1.imported_movies.mscope_params.F |
±-----------------------------------+
doseweights
No group connections
Parameters
General settings
±---------------------------------------±------±--------±-----±---------+
| PARAMETER | VALUE | DEFAULT | SPEC | ADVANCED |
±---------------------------------------±------±--------±-----±---------+
| Make motion diagnostic plots | True | X | | |
±---------------------------------------±------±--------±-----±---------+
| Number of movies to plot | 10 | X | | |
±---------------------------------------±------±--------±-----±---------+
| Only process this many movies | None | X | | |
±---------------------------------------±------±--------±-----±---------+
| Save results in 16-bit floating point | False | X | | |
±---------------------------------------±------±--------±-----±---------+
| Output denoiser training data | True | X | | |
±---------------------------------------±------±--------±-----±---------+
| Num. movies for denoiser training data | 200 | X | | |
±---------------------------------------±------±--------±-----±---------+
| Low-memory mode | False | X | | |
±---------------------------------------±------±--------±-----±---------+
Motion correction
±---------------------------------±------±--------±-----±---------+
| PARAMETER | VALUE | DEFAULT | SPEC | ADVANCED |
±---------------------------------±------±--------±-----±---------+
| Maximum alignment resolution (A) | 5 | X | | |
±---------------------------------±------±--------±-----±---------+
| B-factor during alignment | 500 | X | | |
±---------------------------------±------±--------±-----±---------+
| Start frame (included, 0-based) | 0 | X | | |
±---------------------------------±------±--------±-----±---------+
| End frame (excluded, 0-based) | None | X | | |
±---------------------------------±------±--------±-----±---------+
| Output F-crop factor | 1 | X | | |
±---------------------------------±------±--------±-----±---------+
| Override e/A^2 | None | X | | |
±---------------------------------±------±--------±-----±---------+
| Allow Variable Dose | False | X | | X |
±---------------------------------±------±--------±-----±---------+
| Calibrated smoothing | 0.5 | X | | X |
±---------------------------------±------±--------±-----±---------+
| Override knots Z | None | X | | X |
±---------------------------------±------±--------±-----±---------+
| Override knots Y | None | X | | X |
±---------------------------------±------±--------±-----±---------+
| Override knots X | None | X | | X |
±---------------------------------±------±--------±-----±---------+
| Hot pixel threshold | 10 | X | | X |
±---------------------------------±------±--------±-----±---------+
Random Seeds
±------------±----------±--------±-----±---------+
| PARAMETER | VALUE | DEFAULT | SPEC | ADVANCED |
±------------±----------±--------±-----±---------+
| Random seed | 623802727 | X | | X |
±------------±----------±--------±-----±---------+
Compute settings
±------------------------------±------±--------±-----±---------+
| PARAMETER | VALUE | DEFAULT | SPEC | ADVANCED |
±------------------------------±------±--------±-----±---------+
| Number of GPUs to parallelize | 2 | | X | |
±------------------------------±------±--------±-----±---------+
Outputs
micrographs±----------------------------+
| J2.exposure.micrograph_blob |
±----------------------------+
±----------------------------+
| MICROGRAPH_BLOB_NON_DW_AB |
±----------------------------+
| J2.exposure.micrograph_blob |
±----------------------------+
±-----------------------------+
| MICROGRAPH_THUMBNAIL_BLOB_1X |
±-----------------------------+
| J2.exposure.thumbnail_blob |
±-----------------------------+
±-----------------------------+
| MICROGRAPH_THUMBNAIL_BLOB_2X |
±-----------------------------+
| J2.exposure.thumbnail_blob |
±-----------------------------+
±-----------------------+
| MOVIE_BLOB |
±-----------------------+
| J2.exposure.movie_blob |
±-----------------------+
±----------------------------+
| MICROGRAPH_BLOB |
±----------------------------+
| J2.exposure.micrograph_blob |
±----------------------------+
±----------------------+
| BACKGROUND_BLOB |
±----------------------+
| J2.exposure.stat_blob |
±----------------------+
±-------------------+
| RIGID_MOTION |
±-------------------+
| J2.exposure.motion |
±-------------------+
±-------------------+
| SPLINE_MOTION |
±-------------------+
| J2.exposure.motion |
±-------------------+
±----------------------------+
| GAIN_REF_BLOB (PASSTHROUGH) |
±----------------------------+
| J2.exposure.gain_ref_blob |
±----------------------------+
±----------------------------+
| MSCOPE_PARAMS (PASSTHROUGH) |
±----------------------------+
| J2.exposure.mscope_params |
±----------------------------+
micrographs_incomplete
±----------------------------+
| MICROGRAPH_BLOB |
±----------------------------+
| J2.exposure.micrograph_blob |
±----------------------------+
±-------------------------+
| MOVIE_BLOB (PASSTHROUGH) |
±-------------------------+
| J2.exposure.movie_blob |
±-------------------------+
±----------------------------+
| GAIN_REF_BLOB (PASSTHROUGH) |
±----------------------------+
| J2.exposure.gain_ref_blob |
±----------------------------+±----------------------------+
±----------------------------+
| MSCOPE_PARAMS (PASSTHROUGH) |
±----------------------------+
| J2.exposure.mscope_params |
±----------------------------+
Target
EM01-WIN11.localdomain (node)> [2026-03-18 19:34:23.30] [CPU: 146.5 MB] License is valid.
Avail: 122.94 GB $$
> [2026-03-18 19:34:25.85] [CPU: 147.0 MB] Master running v5.0.3, worker running v5.0.3
Avail: 122.72 GB $$
> [2026-03-18 19:34:25.89] [CPU: 148.0 MB] Resources Allocated
Avail: 122.72 GB $$
> [2026-03-18 19:34:33.24] [CPU: 320.7 MB] [Avail: 122.60 GB]
--------------------------------------------------------------
> [2026-03-18 19:34:33.28] [CPU: 321.7 MB] Random seed: 623802727
Avail: 122.60 GB $$
> [2026-03-18 19:34:33.28] [CPU: 321.7 MB] [Avail: 122.60 GB]
Job will output denoiser training data for this many movies: 20
> [2026-03-18 19:34:33.28] [CPU: 321.7 MB] parent process is 14931
Avail: 122.60 GB $$
> [2026-03-18 19:34:33.31] [CPU: 240.5 MB] Calling CUDA init from 14980
Avail: 122.59 GB $$
> [2026-03-18 19:35:03.39] [CPU: 321.7 MB] Outputting final results…
Avail: 122.60 GB $$
> [2026-03-18 19:35:03.43] [CPU: 322.7 MB] [Avail: 122.60 GB]
--------------------------------------------------------------
> [2026-03-18 19:35:03.43] [CPU: 322.7 MB] Compiling job outputs…
Avail: 122.60 GB $$
> [2026-03-18 19:35:03.44] [CPU: 322.7 MB] [Avail: 122.60 GB]
Output dataset contains: [‘movie_blob’, ‘background_blob’, ‘micrograph_thumbnail_blob_2x’, ‘micrograph_blob_non_dw_AB’,
‘spline_motion’, ‘micrograph_thumbnail_blob_1x’, ‘micrograph_blob’, ‘micrograph_blob_non_dw’, ‘rigid_motion’]
> [2026-03-18 19:35:03.44] [CPU: 322.7 MB] [Avail: 122.60 GB]
Intersection of input and output has 0 item(s)
> [2026-03-18 19:35:03.45] [CPU: 322.7 MB] [Avail: 122.60 GB]
Passing through from input movies to output micrographs_incomplete
> [2026-03-18 19:35:03.45] [CPU: 322.7 MB] Output results are [‘micrograph_blob’].
Avail: 122.60 GB $$
> [2026-03-18 19:35:03.45] [CPU: 322.7 MB] [Avail: 122.60 GB]
Intersection of input and output has 20 item(s)
> [2026-03-18 19:35:03.46] [CPU: 322.7 MB] Updating job size…
Avail: 122.60 GB $$
> [2026-03-18 19:35:03.51] [CPU: 322.7 MB] [Avail: 122.60 GB]
--------------------------------------------------------------
> [2026-03-18 19:35:03.51] [CPU: 322.7 MB] Job complete. Total time 30.27s
- Job_log of J2:
Project P1 Job J2
Master EM01-WIN11.localdomain Port 62000MAIN PROCESS PID 14931
2026-03-18 19:34:24,369 core monitor INFO | MONITOR PROCESS PID 14939
2026-03-18 19:34:24,369 core monitor INFO | ========= monitor process now waiting for main process
2026-03-18 19:34:24,369 core heartbeat INFO | ========= Updating heartbeat
================= CRYOSPARC =================
Project P1 Job J2
Master EM01-WIN11.localdomain Port 62000MAIN PROCESS PID 14931
========= updating job startup information at 2026-03-18 19:34:25.071599
========= now starting main process at 2026-03-18 19:34:25.660103
2026-03-18 19:34:25,845 core run INFO | Running job J2 of type patch_motion_correction_multi
2026-03-18 19:34:25,845 core run INFO | Running job on hostname EM01-WIN11.localdomain
2026-03-18 19:34:25,845 core run INFO | Allocated Resources: lane=‘default’ lane_type=‘node’ hostname=‘EM01-WIN11.localdomain’ target=SchedulerTarget(cache_path=‘/mnt/wsl/ssd01/cryosparc_cache’, cache_reserve_mb=10000, cache_quota_mb=None, lane=‘default’, name=‘EM01-WIN11.localdomain’, title=‘Worker EM01-WIN11.localdomain (node)’, desc=None, hostname=‘EM01-WIN11.localdomain’, worker_bin_path=‘/home/cryosparcuser/cryosparc/cryosparc_worker/bin/cryosparcw’, config=Node(type=‘node’, ssh_str=‘cryosparcuser@EM01-WIN11.localdomain’, resource_slots=ResourceSlots(CPU=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127], GPU=[0, 1], RAM=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]), resource_fixed=FixedResourceSlots(SSD=True), monitor_port=None, gpus=[Gpu(id=0, name=‘NVIDIA GeForce RTX 4080 SUPER’, mem=17171480576), Gpu(id=1, name=‘NVIDIA GeForce RTX 4080 SUPER’, mem=17171480576)])) slots=ResourceSlots(CPU=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], GPU=[0, 1], RAM=[0, 1, 2, 3]) fixed=FixedResourceSlots(SSD=False) licenses_acquired=2
Transparent hugepages setting: always [madvise] never/home/cryosparcuser/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/_csot.py:119: UserWarning: MongoClient opened before fork. May not be entirely fork-safe, proceed with caution. See PyMongo’s documentation for details: MongoDB PyMongo Documentation - PyMongo Driver - MongoDB Docs
return func(self, *args, **kwargs)
/home/cryosparcuser/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/_csot.py:119: UserWarning: MongoClient opened before fork. May not be entirely fork-safe, proceed with caution. See PyMongo’s documentation for details: MongoDB PyMongo Documentation - PyMongo Driver - MongoDB Docs
return func(self, *args, **kwargs)
2026-03-18 19:34:35,198 core heartbeat INFO | ========= Updating heartbeat
DIE: cuMemsetD32Async(dest=0x190f200000, value=0x0, num_u32_elements=56955920, stream=(nil)): CUDA ERROR: (CUDA_ERROR_NOT_INITIALIZED) initialization error
Total: 7.685s
iosys:2:iosys:1:
iocb processing [hits 1]: 0.000035s 0.00% (0.192767s 2.51% with children)
iosys_sort_jobs [hits 1]: 0.000014s 0.00%
iosys_open_cacheable_fds [hits 1]: 0.000076s 0.00%
iosys_process_cached_fd_reads [hits 1]: 0.001652s 0.02% (0.192642s 2.51% with children)
io_uring fill sq [hits 1]: 0.000021s 0.00%
io_uring block [hits 1]: 0.190968s 2.48%
io_uring drain cq [hits 1]: 0.000001s 0.00%
slow_path [hits 1]: 0.000000s 0.00%iosys:0:
iocb processing [hits 1]: 0.000044s 0.00% (5.339897s 69.49% with children)
iosys_sort_jobs [hits 1]: 0.000002s 0.00%
iosys_open_cacheable_fds [hits 1]: 0.000000s 0.00%
iosys_process_cached_fd_reads [hits 1]: 0.001998s 0.03% (0.001999s 0.03% with children)
io_uring fill sq [hits 1]: 0.000000s 0.00%
io_uring block [hits 1]: 0.000000s 0.00%
io_uring drain cq [hits 1]: 0.000000s 0.00%
slow_path [hits 1]: 5.337853s 69.46%DIE: cuMemsetD32Async(dest=0x190f200000, value=0x0, num_u32_elements=56955920, stream=(nil)): CUDA ERROR: (CUDA_ERROR_NOT_INITIALIZED) initialization error
Total: 7.956s
iosys:2:iosys:1:
iocb processing [hits 1]: 0.000035s 0.00% (0.187766s 2.36% with children)
iosys_sort_jobs [hits 1]: 0.000020s 0.00%
iosys_open_cacheable_fds [hits 1]: 0.000056s 0.00%
iosys_process_cached_fd_reads [hits 1]: 0.001696s 0.02% (0.187655s 2.36% with children)
io_uring fill sq [hits 1]: 0.000022s 0.00%
io_uring block [hits 1]: 0.185936s 2.34%
io_uring drain cq [hits 1]: 0.000001s 0.00%
slow_path [hits 1]: 0.000000s 0.00%iosys:0:
iocb processing [hits 1]: 0.000041s 0.00% (5.590907s 70.27% with children)
iosys_sort_jobs [hits 1]: 0.000001s 0.00%
iosys_open_cacheable_fds [hits 1]: 0.000000s 0.00%
iosys_process_cached_fd_reads [hits 1]: 0.001737s 0.02% (0.001738s 0.02% with children)
io_uring fill sq [hits 1]: 0.000000s 0.00%
io_uring block [hits 1]: 0.000000s 0.00%
io_uring drain cq [hits 1]: 0.000000s 0.00%
slow_path [hits 1]: 5.589128s 70.25%2026-03-18 19:34:45,206 core heartbeat INFO | ========= Updating heartbeat
2026-03-18 19:34:55,214 core heartbeat INFO | ========= Updating heartbeat
/home/cryosparcuser/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/home/cryosparcuser/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide
ret = ret.dtype.type(ret / rcount)
2026-03-18 19:35:03,477 core.data_management _dump_job_gridfs INFO | Dumping all of job’s images in the database to /home/cryosparcuser/em/tuto01/CS-tuto01-empiar10025/J2/gridfs_data…
2026-03-18 19:35:03,478 core.data_management _dump_job_gridfs INFO | Done. Exported 0 images in 0.00s
2026-03-18 19:35:03,482 core.data_management _dump_job_events INFO | Done. Dumped 43 files in 0.00s
2026-03-18 19:35:03,492 core.data_management dump_job INFO | Dumped Job P1-J2 (Patch Motion Correction) in 0.02s
========= main process now complete at 2026-03-18 19:35:03.514629
2026-03-18 19:35:05,285 core heartbeat INFO | ========= Updating heartbeat
2026-03-18 19:35:05,289 core heartbeat INFO | Attempted to heartbeat for inactive Job P1-J2 (Patch Motion Correction) with status completed
2026-03-18 19:35:05,290 core monitor ERROR | ========= heartbeat failed
Traceback (most recent call last):
File “/home/cryosparcuser/cryosparc/cryosparc_worker/cli/monitor.py”, line 34, in monitor
assert heartbeat(conf.project, conf.job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
2026-03-18 19:35:15,290 core heartbeat INFO | ========= Updating heartbeat
2026-03-18 19:35:15,294 core heartbeat INFO | Attempted to heartbeat for inactive Job P1-J2 (Patch Motion Correction) with status completed
2026-03-18 19:35:15,294 core monitor ERROR | ========= heartbeat failed
Traceback (most recent call last):
File “/home/cryosparcuser/cryosparc/cryosparc_worker/cli/monitor.py”, line 34, in monitor
assert heartbeat(conf.project, conf.job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
2026-03-18 19:35:25,294 core heartbeat INFO | ========= Updating heartbeat
2026-03-18 19:35:25,298 core heartbeat INFO | Attempted to heartbeat for inactive Job P1-J2 (Patch Motion Correction) with status completed
2026-03-18 19:35:25,298 core monitor ERROR | ========= heartbeat failed
Traceback (most recent call last):
File “/home/cryosparcuser/cryosparc/cryosparc_worker/cli/monitor.py”, line 34, in monitor
assert heartbeat(conf.project, conf.job)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
2026-03-18 19:35:25,298 core monitor ERROR | Killing main process pid 14931
2026-03-18 19:35:25,299 core monitor ERROR | ************* Connection to cryosparc command lost. Heartbeat failed 3 consecutive times.