I'm getting this error while processing motion correction job

Install, Configure and Update
1

Error occurred while processing J1/imported/010839289081933794573_14sep05c_00024sq_00003hl_00005es.frames.tif
Traceback (most recent call last):
File “/home/prince/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py”, line 59, in exec
return self.process(item)
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 210, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 213, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 242, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 219, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 292, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 710, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 205, in cryosparc_master.cryosparc_compute.gpu.gpucore.transfer_ndarray_to_cudaarray
File “/home/prince/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/devices.py”, line 232, in _require_cuda_context
return fn(*args, **kws)
File “/home/prince/cryosparc/cryosparc_worker/cryosparc_compute/gpu/driver.py”, line 169, in create_array
handle = allocator()
File “/home/prince/cryosparc/cryosparc_worker/cryosparc_compute/gpu/driver.py”, line 155, in
allocator = lambda: cuda_check_error(cuda.cuArrayCreate(desc), “Could not allocate GPU array”)
File “/home/prince/cryosparc/cryosparc_worker/cryosparc_compute/gpu/driver.py”, line 284, in cuda_check_error
raise RuntimeError(f"{msg}: {err.name}")
RuntimeError: Could not allocate GPU array: CUDA_ERROR_OUT_OF_MEMORY

Marking J1/imported/010839289081933794573_14sep05c_00024sq_00003hl_00005es.frames.tif as incomplete and continuing…

2

Welcome to the forum @itsingh9 .

Please can you post the output of the command

/home/prince/cryosparc/cryosparc_worker/bin/cryosparcw gpulist

on the worker computer where the job failed.

3

Same problem here. Could anyone help? :smiling_face_with_tear:

4

Welcome to the forum @AllanChinNTU .
Please can you post the outputs of these commands:

csprojectid=P99 # replace with actual project ID
csjobid=J199 # replace with id of a job that should be running
cryosparcm cli "get_job('$csprojectid', '$csjobid', 'job_type', 'version', 'instance_information', 'status',  'params_spec', 'errors_run')"
cryosparcm joblog $csprojectid $csjobid | tail -n 40
cryosparcm eventlog $csprojectid $csjobid | tail -n 40
5

Hi wtempel, thanks a lot for the speedy reply!!
The outputs are as the following,

{‘_id’: ‘6751f99733fbba321a816079’, ‘errors_run’: , ‘instance_information’: {‘CUDA_version’: ‘11.8’, ‘available_memory’: ‘60.43GB’, ‘cpu_model’: ‘Intel(R) Core™ i9-14900K’, ‘driver_version’: ‘12.2’, ‘gpu_info’: [{‘id’: 0, ‘mem’: 25756696576, ‘name’: ‘NVIDIA GeForce RTX 4090’, ‘pcie’: ‘0000:01:00’}], ‘ofd_hard_limit’: 1048576, ‘ofd_soft_limit’: 1024, ‘physical_cores’: 16, ‘platform_architecture’: ‘x86_64’, ‘platform_node’: ‘LingChao’, ‘platform_release’: ‘5.15.167.4-microsoft-standard-WSL2’, ‘platform_version’: ‘#1 SMP Tue Nov 5 00:21:55 UTC 2024’, ‘total_memory’: ‘62.68GB’, ‘used_memory’: ‘1.56GB’}, ‘job_type’: ‘patch_motion_correction_multi’, ‘params_spec’: {}, ‘project_uid’: ‘P2’, ‘status’: ‘completed’, ‘uid’: ‘J21’, ‘version’: ‘v4.6.0’}

================= CRYOSPARCW ======= 2024-12-06 03:06:08.347558 =========
Project P2 Job J21
Master LingChao. Port 39002

MAIN PROCESS PID 28483
========= now starting main process at 2024-12-06 03:06:08.347759
motioncorrection.run_patch cryosparc_compute.jobs.jobregister
/home/lingchao/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for <class ‘numpy.float64’> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/lingchao/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class ‘numpy.float64’> type is zero.
return self._float_to_str(self.smallest_subnormal)
/home/lingchao/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for <class ‘numpy.float32’> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/lingchao/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class ‘numpy.float32’> type is zero.
return self._float_to_str(self.smallest_subnormal)
MONITOR PROCESS PID 28485
========= monitor process now waiting for main process
========= sending heartbeat at 2024-12-06 03:06:09.319662

Running job on hostname %s localhost
Allocated Resources : {‘fixed’: {‘SSD’: False}, ‘hostname’: ‘localhost’, ‘lane’: ‘default’, ‘lane_type’: ‘node’, ‘license’: True, ‘licenses_acquired’: 1, ‘slots’: {‘CPU’: [0, 1, 2, 3, 4, 5], ‘GPU’: [0], ‘RAM’: [0, 1]}, ‘target’: {‘cache_path’: None, ‘cache_quota_mb’: None, ‘cache_reserve_mb’: 10000, ‘desc’: None, ‘gpus’: [{‘id’: 0, ‘mem’: 25756696576, ‘name’: ‘NVIDIA GeForce RTX 4090’}], ‘hostname’: ‘localhost’, ‘lane’: ‘default’, ‘monitor_port’: None, ‘name’: ‘localhost’, ‘resource_fixed’: {‘SSD’: False}, ‘resource_slots’: {‘CPU’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31], ‘GPU’: [0], ‘RAM’: [0, 1, 2, 3, 4, 5, 6, 7]}, ‘ssh_str’: ‘lingchao@localhost’, ‘title’: ‘Worker node localhost’, ‘type’: ‘node’, ‘worker_bin_path’: ‘/home/lingchao/cryosparc/cryosparc_worker/bin/cryosparcw’}}
========= sending heartbeat at 2024-12-06 03:06:19.332503
========= sending heartbeat at 2024-12-06 03:06:29.345800
HOST ALLOCATION FUNCTION: using numba.cuda.pinned_array
/home/lingchao/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/home/lingchao/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)

========= main process now complete at 2024-12-06 03:06:35.008904
Total: 24.223s
MAIN THREAD:

========= main process now complete at 2024-12-06 03:06:39.358916.
========= monitor process now complete at 2024-12-06 03:06:39.365134.
self.cooperative = ‘cudaCGGetIntrinsicHandle’ in lib.get_asm_str()
File “/home/lingchao/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/codegen.py”, line 119, in get_asm_str
return self._join_ptxes(self._get_ptxes(cc=cc))
File “/home/lingchao/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/codegen.py”, line 131, in _get_ptxes
arch = nvvm.get_arch_option(*cc)
File “/home/lingchao/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/nvvm.py”, line 414, in get_arch_option
arch = find_closest_arch((major, minor))
File “/home/lingchao/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/nvvm.py”, line 387, in find_closest_arch
raise NvvmSupportError(msg)
numba.cuda.cudadrv.error.NvvmSupportError: No supported GPU compute capabilities found. Please check your cudatoolkit version matches your CUDA version.

Marking J7/imported/005154033534275794310_14sep05c_c_00003gr_00014sq_00011hl_00004es.frames.tif as incomplete and continuing…
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 252 MB] [‘uid’, ‘movie_blob/path’, ‘movie_blob/shape’, ‘movie_blob/psize_A’, ‘movie_blob/is_gain_corrected’, ‘movie_blob/format’, ‘movie_blob/has_defect_file’, ‘movie_blob/import_sig’, ‘micrograph_blob/path’, ‘micrograph_blob/idx’, ‘micrograph_blob/shape’, ‘micrograph_blob/psize_A’, ‘micrograph_blob/format’, ‘micrograph_blob/is_background_subtracted’, ‘micrograph_blob/vmin’, ‘micrograph_blob/vmax’, ‘micrograph_blob/import_sig’, ‘micrograph_blob_non_dw/path’, ‘micrograph_blob_non_dw/idx’, ‘micrograph_blob_non_dw/shape’, ‘micrograph_blob_non_dw/psize_A’, ‘micrograph_blob_non_dw/format’, ‘micrograph_blob_non_dw/is_background_subtracted’, ‘micrograph_blob_non_dw/vmin’, ‘micrograph_blob_non_dw/vmax’, ‘micrograph_blob_non_dw/import_sig’, ‘micrograph_blob_non_dw_AB/path’, ‘micrograph_blob_non_dw_AB/idx’, ‘micrograph_blob_non_dw_AB/shape’, ‘micrograph_blob_non_dw_AB/psize_A’, ‘micrograph_blob_non_dw_AB/format’, ‘micrograph_blob_non_dw_AB/is_background_subtracted’, ‘micrograph_blob_non_dw_AB/vmin’, ‘micrograph_blob_non_dw_AB/vmax’, ‘micrograph_blob_non_dw_AB/import_sig’, ‘micrograph_thumbnail_blob_1x/path’, ‘micrograph_thumbnail_blob_1x/idx’, ‘micrograph_thumbnail_blob_1x/shape’, ‘micrograph_thumbnail_blob_1x/format’, ‘micrograph_thumbnail_blob_1x/binfactor’, ‘micrograph_thumbnail_blob_1x/micrograph_path’, ‘micrograph_thumbnail_blob_1x/vmin’, ‘micrograph_thumbnail_blob_1x/vmax’, ‘micrograph_thumbnail_blob_2x/path’, ‘micrograph_thumbnail_blob_2x/idx’, ‘micrograph_thumbnail_blob_2x/shape’, ‘micrograph_thumbnail_blob_2x/format’, ‘micrograph_thumbnail_blob_2x/binfactor’, ‘micrograph_thumbnail_blob_2x/micrograph_path’, ‘micrograph_thumbnail_blob_2x/vmin’, ‘micrograph_thumbnail_blob_2x/vmax’, ‘background_blob/path’, ‘background_blob/idx’, ‘background_blob/binfactor’, ‘background_blob/shape’, ‘background_blob/psize_A’, ‘rigid_motion/type’, ‘rigid_motion/path’, ‘rigid_motion/idx’, ‘rigid_motion/frame_start’, ‘rigid_motion/frame_end’, ‘rigid_motion/zero_shift_frame’, ‘rigid_motion/psize_A’, ‘spline_motion/type’, ‘spline_motion/path’, ‘spline_motion/idx’, ‘spline_motion/frame_start’, ‘spline_motion/frame_end’, ‘spline_motion/zero_shift_frame’, ‘spline_motion/psize_A’]
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] --------------------------------------------------------------
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Compiling job outputs…
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Passing through outputs for output group micrographs from input group movies
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] This job outputted results [‘micrograph_blob_non_dw’, ‘micrograph_blob_non_dw_AB’, ‘micrograph_thumbnail_blob_1x’, ‘micrograph_thumbnail_blob_2x’, ‘movie_blob’, ‘micrograph_blob’, ‘background_blob’, ‘rigid_motion’, ‘spline_motion’]
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Loaded output dset with 0 items
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Passthrough results [‘gain_ref_blob’, ‘mscope_params’]
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Loaded passthrough dset with 20 items
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Intersection of output and passthrough has 0 items
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Output dataset contains: [‘mscope_params’, ‘gain_ref_blob’]
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Outputting passthrough result gain_ref_blob
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Outputting passthrough result mscope_params
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Passing through outputs for output group micrographs_incomplete from input group movies
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] This job outputted results [‘micrograph_blob’]
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Loaded output dset with 20 items
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Passthrough results [‘movie_blob’, ‘gain_ref_blob’, ‘mscope_params’]
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Loaded passthrough dset with 20 items
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Intersection of output and passthrough has 20 items
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Output dataset contains: [‘mscope_params’, ‘gain_ref_blob’, ‘movie_blob’]
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Outputting passthrough result movie_blob
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Outputting passthrough result gain_ref_blob
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Outputting passthrough result mscope_params
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Checking outputs for output group micrographs
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Checking outputs for output group micrographs_incomplete
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Updating job size…
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 253 MB] Exporting job and creating csg files…
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 254 MB] ***************************************************************
[Thu, 05 Dec 2024 19:06:34 GMT] [CPU RAM used: 254 MB] Job complete. Total time 23.55s

6

Thanks @AllanChinNTU for posting additional information.
We have not tested CryoSPARC on Windows Subsystem for Linux and are not sure whether CUDA-dependent CryoSPARC jobs can run on that platform.

1 Like