Error during calling GPU

Troubleshooting
1

Hi,

I recently installed cryosparc2 and I found that all four GPU can be recognized by cryosparcw gpulist.
When I called the GPU during the particle extraction, I’ve got this error.

When I turned off GPU and use only CPU, it worked fine.

Please let me know what is the problem.

Best, Uhnsoo Cho

Starting multithreaded pipeline …

Started pipeline

GPU 0 using a batch size of 4096

– 0.0: processing J1/imported/18jun19a_4MLL_01_00012gr_00038sq_v02_00002hl_v01_00002en.frames_96_0.mrc

– 0.1: processing J1/imported/18jun19a_4MLL_01_00010gr_00049sq_v02_00007hl_v01_00002en.frames_92_0.mrc

Traceback (most recent call last):
File “cryosparc2_compute/jobs/runcommon.py”, line 705, in run_with_except_hook
run_old(*args, **kw)
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py”, line 754, in run
self.__target(*self.__args, **self.__kwargs)
File “cryosparc2_compute/jobs/pipeline.py”, line 53, in stage_target
work = processor.process(item)
File “cryosparc2_compute/jobs/extract/run.py”, line 268, in process
cuda_dev = self.cuda_dev, ET=self.ET, timer=timer, mic_idx=mic_idx, batch_size = self.batch_size)
File “cryosparc2_compute/jobs/extract/extraction_gpu.py”, line 208, in do_extract_particles_single_mic_gpu
skcuda.fft.ifft( ET.output_f_gpu[:batch_size], ET.output_gpu[:batch_size], ifft_plan ,scale=True)
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py”, line 298, in ifft
return _fft(x_gpu, y_gpu, plan, cufft.CUFFT_INVERSE, y_gpu.size/plan.batch)
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py”, line 198, in _fft
int(y_gpu.gpudata))
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py”, line 301, in cufftExecC2R
cufftCheckStatus(status)
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py”, line 110, in cufftCheckStatus
raise cufftExceptions[status]
cufftExecFailed

2

Hi Uhnsoo,

What were the box and bin sizes of the particle extraction job?

Best,
Ali

3

Hi @uhnsoo,

Is it possible if you can give us an update on this issue?

4

Hi sarulthasan,

Thanks for checking.
We are still having an issue with it.

It seems like there is a problem in communication between cryosparcV2 and GPU.
We reinstalled cryosparcV2, updated to V2.3, set the path to cuda 8.0, and downgrade the nvidia drive to version 390.
However, we still couldn’t solve the issue.

Here is a new error message that we have now.

Processed 0 of 659 movies in 0.01s

Loading raw movie data from J15/imported/17may06a_b_00010gr_00018sq_v01_00002hl16_00002edhii.frames.mrc …

Done in 11.21s

Loading gain data from J15/imported/17may06a_b_00010gr_00018sq_v01_00002hl16_00002edhii.frames.mrc …

Done in 1.16s

Processing …

Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 387, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 388, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 111, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 114, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py”, line 516, in fill
func = elementwise.get_fill_kernel(self.dtype)
File “”, line 2, in get_fill_kernel
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/tools.py”, line 430, in context_dependent_memoize
result = func(*args)
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/elementwise.py”, line 492, in get_fill_kernel
“fill”)
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/elementwise.py”, line 161, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs)
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/elementwise.py”, line 147, in get_elwise_kernel_and_types
keep, options, **kwargs)
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/elementwise.py”, line 75, in get_elwise_module
options=options, keep=keep)
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py”, line 291, in init
arch, code, cache_dir, include_dirs)
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py”, line 255, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py”, line 78, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode(“utf-8”))
File “/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py”, line 55, in preprocess_source
cmdline, stderr=stderr)
CompileError: nvcc preprocessing of /tmp/tmp148TCJ.cu failed
[command: nvcc --preprocess -arch sm_61 -I/mnt/data2/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/cuda /tmp/tmp148TCJ.cu --compiler-options -P]
[stderr:
In file included from /usr/local/cuda-8.0/bin/…//include/cuda_runtime.h:78:0,
from :0:
/usr/local/cuda-8.0/bin/…//include/host_config.h:119:2: error: #error – unsupported GNU version! gcc versions later than 5 are not supported!
#error – unsupported GNU version! gcc versions later than 5 are not supported!
^~~~~
]

Outputs

5

We are using Ubuntu 18.04. Can it be an issue?
and we installed the SBgrid suite.