Extract from micrograph doesn't work (V2.3.2)

Passing a job directly from “inspect particles” job to “extract from micrograph” gives the following error (GPU settings underneath; it’s in default processing mode). This doesn’t seem to be a problem if the extracting job happens in the “local motion correction” job.

Starting multithreaded pipeline …

Started pipeline

GPU 0 using a batch size of 1024

– 0.0: processing J1/imported/Frame2_0041.mrc

– 0.1: processing J1/imported/Frame2_0042.mrc

Traceback (most recent call last):
File “cryosparc2_compute/jobs/runcommon.py”, line 738, in run_with_except_hook
run_old(*args, **kw)
File “/home/dgl/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py”, line 754, in run
self.__target(*self.__args, **self.__kwargs)
File “cryosparc2_compute/jobs/pipeline.py”, line 53, in stage_target
work = processor.process(item)
File “cryosparc2_compute/jobs/extract/run.py”, line 268, in process
cuda_dev = self.cuda_dev, ET=self.ET, timer=timer, mic_idx=mic_idx, batch_size = self.batch_size)
File “cryosparc2_compute/jobs/extract/extraction_gpu.py”, line 208, in do_extract_particles_single_mic_gpu
skcuda.fft.ifft( ET.output_f_gpu[:batch_size], ET.output_gpu[:batch_size], ifft_plan ,scale=True)
File “/home/dgl/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py”, line 298, in ifft
return _fft(x_gpu, y_gpu, plan, cufft.CUFFT_INVERSE, y_gpu.size/plan.batch)
File “/home/dgl/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py”, line 198, in _fft
int(y_gpu.gpudata))
File “/home/dgl/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py”, line 301, in cufftExecC2R
cufftCheckStatus(status)
File “/home/dgl/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py”, line 110, in cufftCheckStatus
raise cufftExceptions[status]
cufftExecFailed

±----------------------------------------------------------------------------+
| NVIDIA-SMI 396.54 Driver Version: 396.54 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro P4000 Off | 00000000:73:00.0 On | N/A |
| 46% 33C P8 6W / 105W | 765MiB / 8111MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1321 G /usr/lib/xorg/Xorg 396MiB |
| 0 2139 G compiz 190MiB |
| 0 2394 G …quest-channel-token=6741064447546151985 176MiB |
±----------------------------------------------------------------------------+

Hey @Bruk :slight_smile:

Is it possible if you can confirm which version of CUDA you are running?

nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17

I had to direct the cryosparc2_worker installation to a different CUDA path (v9.1) than the master installation as per this issue fix: AttributeError: undefined symbol: cufftSetCompatibilityMode (V2)

I also just reran the exract particles job from the proteosome tutorial directly (without local motion correction, just passing the job from inspect pics to extract from micrographs) and it seemed to have worked. Could there be some issue related to one of the .mrc files in my own data that is somehow not problematic in any of the preceding jobs (upload, ctf estimation, 2D class averaging, selecting 2D classes and using the template picker/inspect pics)? Excluding “Frame2_0042.mrc” from analysis results in identical error, just with the next image over…