cuLaunchKernel during full-frame motion correction

Troubleshooting
1

Hello, I’m running into the following error during full-frame motion correction of K3 movies. This error occurs whether “Low GPU memory” is checked or unchecked.

[CPU: 1.92 GB]   Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/cryosparc/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 165, in thread_work
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py", line 146, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py", line 149, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py", line 150, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py", line 274, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py", line 284, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
  File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/cuda_kernels.py", line 153, in cryosparc2_compute.jobs.motioncorrection.cuda_kernels.do_extract_and_weight_gpu (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/cuda_kernels.c:4028)
  File "/home/cryosparc/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/driver.py", line 418, in function_call
    func._launch_kernel(grid, block, arg_buf, shared, stream)
LogicError: cuLaunchKernel failed: invalid argument

Below is information about the movies:

                    This is a TIFF file.

 Number of columns, rows, sections .....   11520    8184      50
 Map mode ..............................    0   (byte)
 Start cols, rows, sects, grid x,y,z ...    0     0     0   11520   8184     50
 Pixel spacing (Angstroms)..............  0.5705     0.5705     0.5705
 Cell angles ...........................   90.000   90.000   90.000
 Fast, medium, slow axes ...............    X    Y    Z
 Origin on x,y,z .......................    0.000       0.000       0.000
 Minimum density .......................   0.0000
 Maximum density .......................   108.00
 Mean density ..........................   54.000
 tilt angles (original,current) ........   0.0   0.0   0.0   0.0   0.0   0.0
 Space group,# extra bytes,idtype,lens .        0        0        0        0

     2 Titles :
SerialEMCCD: Dose frac. image, scaled by 1.00  r/f 0
  SuperRef_stack_00000.dm4

Our system has 4 RTX 2080’s, and is running NVIDIA driver version 440.64.00, CUDA 10.2

Below is more information about our cryoSPARC install. I have also installed patch 200528, which is not reflected below.

$ cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/cryosparc/cryosparc2/cryosparc2_master
Current cryoSPARC version: v2.15.0
----------------------------------------------------------------------------

cryosparcm process status:

app                              STOPPED   Not started
app_dev                          STOPPED   Not started
command_core                     RUNNING   pid 12866, uptime 0:00:49
command_proxy                    RUNNING   pid 12925, uptime 0:00:46
command_rtp                      STOPPED   Not started
command_vis                      RUNNING   pid 12912, uptime 0:00:47
database                         RUNNING   pid 12777, uptime 0:00:52
watchdog_dev                     STOPPED   Not started
webapp                           RUNNING   pid 12955, uptime 0:00:45
webapp_dev                       STOPPED   Not started

----------------------------------------------------------------------------

global config variables:

export CRYOSPARC_LICENSE_ID="xxxx"
export CRYOSPARC_MASTER_HOSTNAME="troll.ohsu.edu"
export CRYOSPARC_DB_PATH="/home/cryosparc/cryosparc2/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
2

Update: this issue persists when using full-frame motion correction on a system with similar specs except running 2080 Ti’s, making a memory issue less likely.

3

Hi, it has been two weeks since I posted this. Does anyone at cryoSPARC know what causes this issue? @hsnyder?

4

Hi @posertinlab,

I apologize for not seeing this sooner. I haven’t encountered this error, but it looks like there might be a version mismatch between two different parts of cryoSPARC (perhaps something went wrong when the patch was applied). I’ll do some experiments and get back to you about this tomorrow. Thanks for your patience.

Harris

5

Hi @posertinlab,

I’ve sent you a direct message related to this issue.

Harris

6

what was the problem, and solution?

7

Unfortunately no solution as of yet, but please let us know if this issue persists in version 3.1.

Harris

8

Hi. I’m experiencing the same error when attempting to do full-frame motion correction on cryosparc version 4.2.1. Has there been any progress on this issue?

9

Hi @avd38 , if you haven’t already, you might try updating to CryoSPARC v4.4, which includes it’s own cuda toolkit. We’ve found that this resolves toolkit version problems in most cases.