Template Picker error: cufftAllocFailed

Hi,

When I select 3+ 2D classes to use for the Template Picker I get this error:

Traceback (most recent call last): File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main File “cryosparc2_worker/cryosparc2_compute/jobs/template_picker_gpu/run.py”, line 112, in cryosparc2_compute.jobs.template_picker_gpu.run.run File “cryosparc2_worker/cryosparc2_compute/jobs/template_picker_gpu/run.py”, line 218, in cryosparc2_compute.jobs.template_picker_gpu.run.run File “/CryoSparc/CryoSparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py”, line 126, in init onembed, ostride, odist, self.fft_type, self.batch) File “/CryoSparc/CryoSparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py”, line 741, in cufftMakePlanMany cufftCheckStatus(status) File “/CryoSparc/CryoSparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py”, line 116, in cufftCheckStatus raise e cufftAllocFailed

I do not get this error when I use 3 or less 2D classes for the Template picker.

What can I do to fix this issue?

Hi @chana this is unfortunate - we routinely use the particle picker with dozens of classes. Can you report the size of the micrographs, the size of the particles, the pixel size, your GPU hardware info, etc?
We recently fixed a bug (out in the last release) where K3 data was causing out-of-memory errors that were avoidable. I think in another thread you reported that you’re already on the latest version?

Similar issue. K3 data cannot be processed unless binned. Also have an older data set with
px of 8964 13176 that has the same error. Even 2x binning this data has the same error. Everthing else appears fine. Running latest 2.5 with CUDA 10

I observed that images that have this problem give the following info
“Computing cc with FFT size 4096” just before memory error. Might explain the problem
This is regardless of the particle size given or the actual image size. Even if binned 3 fold the same FFT size of 4096 shows up.
All other template picking routines have FFT sizes listed in the 100s.

Hi all, this has been fixed - we had an optimization that was kicking in too aggressively (trying to process multiple templates at a time regardless of the FFT sizes and GPU memory) that has been turned off. To be released in the next release (soon)

Thanks…hoping to use soon

I still observe this error after the v2.8 update.

Hi @MHB,

What GPU(s) are you using?

We are using Geforce GTX 1080

Also still same problem in 2.9.

I still have this error in V2.12.2 in both template and blob picker.

Hi @MHB, we are investigating

OK. I was able to pick particles using the Topaz wrapper with no issue.

@MHB can you let us know the raw pixel size of the data where you were seeing this error (eg. the 8964x13176 sized data)?

2.372

let me know if you need any other info

K3 images; 2x2 binned; v2.12.4. I have tried a single template and reducing the number of micrographs and still recieved the error.

Topaz, however, is running fine with up to 50 templates and >100 micrographs. But, with Topaz specifying cores can be used as a job parameter.

I noticed on my workstation that although 16 cores and 2 GPUs are specified as resources for template picker, the job only utilizes 4 cores and one GPU. Unlike ‘Extract from Micrographs’, one cannot specify the number of GPUs to use for ‘Template Picker’. Wouldn’t this contribute to the lack of memory error? Is there no way to specify in the parameters for ‘Template Picker’ to use a greater number of cores/GPUs?

Hey @MHB did you get anywhere with this? we are experiencing this error as well, and the other I posted here… Memory error during full-frame motion correction
LogicError: cuMemHostAlloc failed: invalid argument during Heterogeneous Refinement

I feel like this is a ‘not enough CPUs per GPU’ problem because for the first time I got the ‘waiting for CPUs’ message for a job but I’m not sure

Cheers,
Alex

This problem still persists for me. For that project I just used the Topaz wrapper no problem. Sorry i cant help.

Thanks for your reply! Glad that Topaz worked for you, I might have to check it out.

Template picker now works in version 3.0 for this data.

2 Likes