I’m recently using a new server for data processing which is giving only me this very odd error.
[CPU: 555.5 MB] Traceback (most recent call last): File "/app/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 1791, in run_with_except_hook run_old(*args, **kw) File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run File "cryosparc_master/cryosparc_compute/engine/engine.py", line 1048, in cryosparc_compute.engine.engine.process.work File "cryosparc_master/cryosparc_compute/engine/engine.py", line 192, in cryosparc_compute.engine.engine.EngineThread.setup_current_data_and_ctf File "cryosparc_worker/cryosparc_compute/engine/cuda_kernels.py", line 1693, in cryosparc_compute.engine.cuda_kernels.compute_ctf File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 410, in cryosparc_compute.engine.cuda_core.context_dependent_memoize.wrapper File "cryosparc_worker/cryosparc_compute/engine/cuda_kernels.py", line 1679, in cryosparc_compute.engine.cuda_kernels.get_compute_ctf_kernel File "/app/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 294, in __init__ self.module = module_from_buffer(cubin) pycuda._driver.LogicError: cuModuleLoadDataEx failed: device kernel image is invalid - error : Binary format for key='0', ident='' is not recognized
Now the strange thing is that I didn’t get this error in previous steps while using gpu (my initial guess is some sort of GPU error). And more strange part is other users are having no trouble with 2D classification.
(NB: I tried extracting particle from one patch-motion corrected micrographs and used that single micrograph extracted particles for 2D classification without scratch dir and with 10 classes and rest default params, still the error persists)
Can you please advise what is going on.
I never seen this before in our other server.