Hi everyone,
I’m trying to run “Patch Motion correction” on large (8912x8912x40) eer movies, and noticed a weird thing in the log file: for some frames, I get following error:
Error occurred while processing J35/imported/013972726808599632256_FoilHole_8444582_Data_8443190_8443192_20230209_143427_EER.eer
Traceback (most recent call last):
File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 60, in exec
return self.process(item)
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 177, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 180, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 182, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 255, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 497, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 414, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction.get_framedata
File "cryosparc_master/cryosparc_compute/engine/newgfourier.py", line 107, in cryosparc_compute.engine.newgfourier.do_fft_plan_inplace
File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cufft.py", line 353, in cufftExecR2C
cufftCheckStatus(status)
File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cufft.py", line 124, in cufftCheckStatus
raise e
cryosparc_compute.skcuda_internal.cufft.cufftInvalidPlan
Marking J35/imported/013972726808599632256_FoilHole_8444582_Data_8443190_8443192_20230209_143427_EER.eer as incomplete and continuing...
and for others – a different one:
Error occurred while processing J35/imported/001602825989841354997_FoilHole_8448638_Data_8443229_8443231_20230209_204850_EER.eer
Traceback (most recent call last):
File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 60, in exec
return self.process(item)
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 177, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 180, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 182, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 255, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 598, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 357, in cryosparc_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
File "/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/gpuarray.py", line 210, in __init__
self.gpudata = self.allocator(self.size * self.dtype.itemsize)
pycuda._driver.MemoryError: cuMemAlloc failed: out of memory
Marking J35/imported/001602825989841354997_FoilHole_8448638_Data_8443229_8443231_20230209_204850_EER.eer as incomplete and continuing...
even though I’m running in low-memory mode on 3x2080Ti (12 Gb) cards (machine with 256 Gb RAM, CS v4.1.2).
“Import movies” when fine with all the frames, with the header check.
Any suggestions on what’s going on and how to fix that?
