Particle Extraction failed: cufftInternalError

Troubleshooting
1

Hi, I have a question about Extraction particle after template pick. I am running 4.1.1 version. I have half of the images which got an error message:

Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 60, in exec
    return self.process(item)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/extract/run.py", line 498, in process
    result = extraction_gpu.do_extract_particles_single_mic_gpu(mic=mic, bg_bin=bg_bin,
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/extract/extraction_gpu.py", line 136, in do_extract_particles_single_mic_gpu
    fft_plan = skcuda_fft.Plan(shape=(patch_size, patch_size),
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/fft.py", line 132, in __init__
    self.worksize = cufft.cufftMakePlanMany(
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cufft.py", line 749, in cufftMakePlanMany
    cufftCheckStatus(status)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cufft.py", line 124, in cufftCheckStatus
    raise e
cryosparc_compute.skcuda_internal.cufft.cufftInternalError

That never happened before.

Thanks,
Hanah

2

HI Hanah,

Given that it is happening on half your images, my guess is that you are running with 2 GPUs and one is misbehaving for some reason. Running picking on a smaller subset, and trying each GPU in turn, may help to isolate the problem.

Cheers
Oli

1 Like
3

I second this suggestion in case this job ran on multiple GPUs.

May I also ask you to post the following info: