Hi all,
I have a motion correction job which has failed which is now unresponsive. The input was 4809 movies, 39 of which failed due to running out of VRAM - leaving 4770 complete micrographs. While this is quite normal, there was a subsequent error message reporting that movie_blob had the wrong length, being 4776 rather than 4770. As shown in the wrong length log below, the job output including motion data has length 4776 indicating some partial results for these 6 extra? As a result when the job is being finalised there is some file ordering error and the outputs are not useable. I cannot mark the job as complete to finish motion correcting the 39 and it does not show up in the main job list, only when viewing failed jobs.
VRAM error:
[CPU: 340.8 MB Avail: 114.26 GB]
Error occurred while processing J541/imported/005152026933172802222_FoilHole_17196693_Data_17202639_17202641_20230719_010651_fractions.tiff
Traceback (most recent call last):
File “/usr/local/cryosparc2_worker/cryosparc_compute/jobs/pipeline.py”, line 60, in exec
return self.process(item)
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 177, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 180, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 182, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 255, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 665, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 216, in cryosparc_compute.engine.cuda_core.transfer_ndarray_to_cudaarray
pycuda._driver.MemoryError: cuArrayCreate failed: out of memoryMarking J541/imported/005152026933172802222_FoilHole_17196693_Data_17202639_17202641_20230719_010651_fractions.tiff as incomplete and continuing…
Wrong length issue:
[CPU: 353.7 MB Avail: 113.45 GB]
Compiling job outputs…
[CPU: 353.8 MB Avail: 113.45 GB]Passing through outputs for output group micrographs from input group movies
[CPU: 358.9 MB Avail: 113.45 GB]This job outputted results [‘micrograph_blob_non_dw’, ‘micrograph_thumbnail_blob_1x’, ‘micrograph_thumbnail_blob_2x’, ‘micrograph_blob’, ‘background_blob’, ‘rigid_motion’, ‘spline_motion’]
[CPU: 358.9 MB Avail: 113.45 GB]Loaded output dset with 4776 items
[CPU: 358.9 MB Avail: 113.45 GB]Passthrough results [‘movie_blob’, ‘gain_ref_blob’, ‘mscope_params’]
[CPU: 358.9 MB Avail: 113.45 GB]Loaded passthrough dset with 4809 items
[CPU: 359.1 MB Avail: 113.45 GB]Intersection of output and passthrough has 4770 items
[CPU: 359.1 MB Avail: 113.45 GB]Passing through outputs for output group micrographs_incomplete from input group movies
[CPU: 359.1 MB Avail: 113.45 GB]This job outputted results [‘micrograph_blob’]
[CPU: 359.1 MB Avail: 113.45 GB]Loaded output dset with 39 items
[CPU: 359.1 MB Avail: 113.45 GB]Passthrough results [‘movie_blob’, ‘gain_ref_blob’, ‘mscope_params’]
[CPU: 359.1 MB Avail: 113.45 GB]Loaded passthrough dset with 4809 items
[CPU: 359.1 MB Avail: 113.45 GB]Intersection of output and passthrough has 39 items
[CPU: 359.1 MB Avail: 113.45 GB]Checking outputs for output group micrographs
[CPU: 360.0 MB Avail: 113.45 GB]Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 101, in cryosparc_compute.run.main
File “/usr/local/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py”, line 1032, in check_outputs
assert output_len == target_len, “Result %s had the wrong length (%d, should be %d). Note that outputs are still usable.” % (resname, output_len, target_len)
AssertionError: Result movie_blob had the wrong length (4770, should be 4776). Note that outputs are still usable.
CryoSPARC version v4.3.1. Motion correction run with default parameters. CryoSPARC is running fine for other jobs so I don’t think it’s an install issue…
Any help would be greatly appreciated.
Cheers,
Will