CS error during 2D classification

Troubleshooting
1

Hi,
I’m recently using a new server for data processing which is giving only me this very odd error.

[CPU: 555.5 MB]  Traceback (most recent call last):
  File "/app/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 1791, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
  File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 1048, in cryosparc_compute.engine.engine.process.work
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 192, in cryosparc_compute.engine.engine.EngineThread.setup_current_data_and_ctf
  File "cryosparc_worker/cryosparc_compute/engine/cuda_kernels.py", line 1693, in cryosparc_compute.engine.cuda_kernels.compute_ctf
  File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 410, in cryosparc_compute.engine.cuda_core.context_dependent_memoize.wrapper
  File "cryosparc_worker/cryosparc_compute/engine/cuda_kernels.py", line 1679, in cryosparc_compute.engine.cuda_kernels.get_compute_ctf_kernel
  File "/app/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 294, in __init__
    self.module = module_from_buffer(cubin)
pycuda._driver.LogicError: cuModuleLoadDataEx failed: device kernel image is invalid - error   : Binary format for key='0', ident='' is not recognized

Now the strange thing is that I didn’t get this error in previous steps while using gpu (my initial guess is some sort of GPU error). And more strange part is other users are having no trouble with 2D classification.
(NB: I tried extracting particle from one patch-motion corrected micrographs and used that single micrograph extracted particles for 2D classification without scratch dir and with 10 classes and rest default params, still the error persists)

Can you please advise what is going on.
I never seen this before in our other server.

Thank you.

2

@diffracteD Just to confirm: Have other users succeeded with 2D classification on the same worker?
Please can you post

  • cryoSPARC version and patch level
  • gpu models and driver version (from nvidia-smi)
  • linux kernel version (from uname -a)
  • cpu version (lscpu | grep name)
  • worker cuda version (grep CUDA /app/cryosparc_worker/config.sh)
  • worker pycuda version
eval $(/app/cryosparc_worker/bin/cryosparcw env)
python -c "import pycuda.driver; print(pycuda.driver.get_version())"
3

Hi @wtempel , Unfortunately I’m not the administrator of this server.
I have conveyed your message to them. They will probably get back on this.

Thanks.