Compiling problems

I’m lost…
Suddenly, every time cryoSPARC trying to using GPU, It crashed like this:

Traceback (most recent call last):
File “cryosparc2_compute/jobs/runcommon.py”, line 747, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 101, in cryosparc2_compute.engine.cuda_core.GPUThread.run
File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 102, in cryosparc2_compute.engine.cuda_core.GPUThread.run
File “cryosparc2_worker/cryosparc2_compute/engine/engine.py”, line 1076, in cryosparc2_compute.engine.engine.process.work
File “cryosparc2_worker/cryosparc2_compute/engine/engine.py”, line 643, in cryosparc2_compute.engine.engine.EngineThread.backproject
File “cryosparc2_worker/cryosparc2_compute/engine/cuda_kernels.py”, line 1656, in cryosparc2_compute.engine.cuda_kernels.backproject
File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 274, in cryosparc2_compute.engine.cuda_core.context_dependent_memoize.wrapper
File “cryosparc2_worker/cryosparc2_compute/engine/cuda_kernels.py”, line 1562, in cryosparc2_compute.engine.cuda_kernels.get_backproject_kernels
File “cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py”, line 291, in init
arch, code, cache_dir, include_dirs)
File “cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py”, line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File “cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py”, line 137, in compile_plain
stderr=stderr.decode(“utf-8”, “replace”))
CompileError: nvcc compilation of /tmp/tmp_1HgJA/kernel.cu failed
[command: nvcc --cubin -arch sm_30 -I/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/cuda kernel.cu]
[stderr:
kernel.cu(48): error: identifier “__ldg” is undefined

kernel.cu(165): error: identifier “__ldg” is undefined

2 errors detected in the compilation of “/tmp/tmpxft_0000d79a_00000000-7_kernel.cpp1.ii”.
]

Finally, we found that the GPUs we are using were pulled out of machine by the managers and causing the compiling problems:sweat: