Motion corretion issue

Motion Correction
1

Hello,
I am running motion corr on 4000 movies, whenever I run it, it only correct the first movie and for the rest of the movies, it shows this error:

Error occurred while processing J67/imported/011070566505842330880_FoilHole_25201378_Data_25203422_25203424_20240425_113401_fractions.tiff
Traceback (most recent call last):
File “/opt/cryosparc3/cryosparc_worker/cryosparc_compute/jobs/pipeline.py”, line 59, in exec
return self.process(item)
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 210, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 213, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 242, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 219, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 292, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 710, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 205, in cryosparc_master.cryosparc_compute.gpu.gpucore.transfer_ndarray_to_cudaarray
File “/opt/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/devices.py”, line 232, in _require_cuda_context
return fn(*args, **kws)
File “/opt/cryosparc3/cryosparc_worker/cryosparc_compute/gpu/driver.py”, line 169, in create_array
handle = allocator()
File “/opt/cryosparc3/cryosparc_worker/cryosparc_compute/gpu/driver.py”, line 155, in
allocator = lambda: cuda_check_error(cuda.cuArrayCreate(desc), “Could not allocate GPU array”)
File “/opt/cryosparc3/cryosparc_worker/cryosparc_compute/gpu/driver.py”, line 284, in cuda_check_error
raise RuntimeError(f"{msg}: {err.name}")
RuntimeError: Could not allocate GPU array: CUDA_ERROR_OUT_OF_MEMORY

Marking J67/imported/011070566505842330880_FoilHole_25201378_Data_25203422_25203424_20240425_113401_fractions.tiff as incomplete and continuing…

I have tried to do motion corr on the fewer subset of movies, still this error occurs. I also put it on the low memory option, again this happened. I don’t understand what the problem is and I need to motion corr on CryoSparc since I want to use denoiser after. Can anyone help me with this?

2

@Freza065 Please can you post additional information

  1. outputs of the commands
    free -h
nvidia-smi
    on the worker node
  2. output of the command
    cryosparcm cli "get_job('P99', 'J199', 'version', 'job_type', 'params_spec', 'instance_information')"
    on the CryoSPARC master host where you replace P99 , J199 with the failed job’s project and job IDs, respectively
  3. is the scheduler lane to which the job was submitted of type cluster or of type node?
  4. Where other compute tasks running on the worker when the job failed?