Unfortunately got a new error when I started a completely new NU Refinement job:
Traceback (most recent call last):
File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 3007, in add_ptx
driver.cuLinkAddData(self.handle, input_ptx, ptx, len(ptx),
File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 352, in safe_cuda_api_call
return self._check_cuda_python_error(fname, libfn(*args))
File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 412, in _check_cuda_python_error
raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [CUresult.CUDA_ERROR_UNSUPPORTED_PTX_VERSION] Call to cuLinkAddData results in CUDA_ERROR_UNSUPPORTED_PTX_VERSION
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 2192, in run_with_except_hook
run_old(*args, **kw)
File "cryosparc_master/cryosparc_compute/gpu/gpucore.py", line 117, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
File "cryosparc_master/cryosparc_compute/gpu/gpucore.py", line 118, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
File "cryosparc_master/cryosparc_compute/engine/engine.py", line 1123, in cryosparc_master.cryosparc_compute.engine.engine.process.work
File "cryosparc_master/cryosparc_compute/engine/engine.py", line 350, in cryosparc_master.cryosparc_compute.engine.engine.EngineThread.compute_resid_pow
File "/data/cryosparc/cryosparc_worker/cryosparc_compute/gpu/gpuarray.py", line 110, in fill
from .elementwise.fill import fill
File "/data/cryosparc/cryosparc_worker/cryosparc_compute/gpu/elementwise/fill.py", line 24, in <module>
def fill(arr, x, out):
File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/np/ufunc/decorators.py", line 203, in wrap
guvec.add(fty)
File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/np/ufunc/deviceufunc.py", line 475, in add
kernel = self._compile_kernel(fnobj, sig=tuple(outertys))
File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/vectorizers.py", line 241, in _compile_kernel
return cuda.jit(sig)(fnobj)
File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/decorators.py", line 133, in _jit
disp.compile(argtypes)
File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/dispatcher.py", line 928, in compile
kernel.bind()
File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/dispatcher.py", line 207, in bind
self._codelibrary.get_cufunc()
File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/codegen.py", line 184, in get_cufunc
cubin = self.get_cubin(cc=device.compute_capability)
File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/codegen.py", line 159, in get_cubin
linker.add_ptx(ptx.encode())
File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 3010, in add_ptx
raise LinkerError("%s\n%s" % (e, self.error_log))
numba.cuda.cudadrv.driver.LinkerError: [CUresult.CUDA_ERROR_UNSUPPORTED_PTX_VERSION] Call to cuLinkAddData results in CUDA_ERROR_UNSUPPORTED_PTX_VERSION
ptxas application ptx input, line 9; fatal : Unsupported .version 7.8; current version is '7.5'