Error during refinement with C4 symmetry

3D Refinement
1

Hi all,

One of our users gets the following error message when trying to refine a structure with a C4 symmetry:

Traceback (most recent call last): File “cryosparc2_compute/jobs/runcommon.py”, line 747, in run_with_except_hook run_old(*args, **kw) File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 101, in cryosparc2_compute.engine.cuda_core.GPUThread.run File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 102, in cryosparc2_compute.engine.cuda_core.GPUThread.run File “cryosparc2_worker/cryosparc2_compute/engine/engine.py”, line 1076, in cryosparc2_compute.engine.engine.process.work File “cryosparc2_worker/cryosparc2_compute/engine/engine.py”, line 643, in cryosparc2_compute.engine.engine.EngineThread.backproject File “cryosparc2_worker/cryosparc2_compute/engine/cuda_kernels.py”, line 1656, in cryosparc2_compute.engine.cuda_kernels.backproject File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 274, in cryosparc2_compute.engine.cuda_core.context_dependent_memoize.wrapper File “cryosparc2_worker/cryosparc2_compute/engine/cuda_kernels.py”, line 1562, in cryosparc2_compute.engine.cuda_kernels.get_backproject_kernels File “/gpfs/share/apps/cryosparc/2.0.27/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py”, line 294, in init self.module = module_from_buffer(cubin) LogicError: cuModuleLoadDataEx failed: device kernel image is invalid - error : Binary format for key=‘0’, ident=’’ is not recognized

The input of the jobs are the volume and particles from a homogeneous refinement in C1. Instead of doing the alignment through cryosparc, we also tried to first align the volume on the C4 axis with Relion, but it didn’t help. We also tried C2, and C2 was successfully done.
We finally tried C4 with the non-uniform refinement job instead (both with GPU and CPU), but these jobs failed too with the same error message.

Does anyone know what can cause this error when the C4 symmetry is used?

Thanks,
Best,
Nicolas

2

Hi @ncoudray,

Can you tell us the CUDA version of the node in question, as well as the box size, particle size, pixel size, etc? Any other non-default parameters?

3

Hi @apunjani

thanks for getting back to us so quickly.
We are using CUDA 9.1
Box size is 280 pix
pixelsize is 1.03 Ang/px (from super resolution that were taken at 0.515 Å/px)
expected particle size is around 150 Ang and there was 87,169 particles

Thanks

4

Any progress on this problem? (I’m the user @ncoudray referred to.) Thanks

5

Hi @irkvelo,

Unfortunately the farthest we have been able to trace this is that it is most likely a CUDA /NVIDIA driver installation issue, or a cryoSPARC worker installation issue (any chance this was on a cluster node, or you changed the CUDA version since the initial install, or changed/added GPUs since the initial install of cryoSPARC worker?)
We are at a bit of a loss because we have never seen this error before and it’s hard to explain why the job works with some settings but not others (C4 symmetry is not handled much differently than C2 under the hood)

6

Is there maybe another node you could run these jobs on? You don’t need to install cryosparc2_master again, only the worker on the other node, and connect it to your existing master node

7

Hi @apunjani,

I got the same error after I reinstall v2.14 when running 2D jobs, I tried update my version 2.11 but it failed, then I reinstall everything in the other folder of v2.14 and I UPDATE the connect between master and worker , then I have this issue, any suggestion?

Thanks!

Jinru