Hi,
The new version of homogeneous refinements jobs fail for me when they try to start the iterations, please see below. I am running the latest CryoSPARC update, and the older version of homogeneous refinement (legacy) works fine.
Is this a CUDA issue?
Thank you for your help
Nadav
[CPU: 1.31 GB] ====== Starting Refinement Iterations ======
[CPU: 1.31 GB] ----------------------------- Start Iteration 0
[CPU: 1.31 GB] Using Max Alignment Radius 8.590 (30.000A)
[CPU: 1.31 GB] Auto batchsize: 2880 in each split
[CPU: 1.37 GB] Engine Started.
[CPU: 2.38 GB] Traceback (most recent call last):
File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
run_old(*args, **kw)
File "/home/nadav/cryoSPARC_v2/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "cryosparc2_worker/cryosparc2_compute/engine/newengine.py", line 1691, in cryosparc2_compute.engine.newengine.process.work
File "cryosparc2_worker/cryosparc2_compute/engine/newengine.py", line 1731, in cryosparc2_compute.engine.newengine.process.work
File "cryosparc2_worker/cryosparc2_compute/engine/newengine.py", line 879, in cryosparc2_compute.engine.newengine.EngineThread.compute_resid_pow
File "cryosparc2_worker/cryosparc2_compute/engine/newcuda_kernels.py", line 3326, in cryosparc2_compute.engine.newcuda_kernels.compute_resid_pow
File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 362, in cryosparc2_compute.engine.cuda_core.context_dependent_memoize.wrapper
File "cryosparc2_worker/cryosparc2_compute/engine/newcuda_kernels.py", line 3279, in cryosparc2_compute.engine.newcuda_kernels.get_compute_resid_pow_kernel
File "/home/nadav/cryoSPARC_v2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py", line 291, in __init__
arch, code, cache_dir, include_dirs)
File "/home/nadav/cryoSPARC_v2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py", line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/home/nadav/cryoSPARC_v2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py", line 137, in compile_plain
stderr=stderr.decode("utf-8", "replace"))
CompileError: nvcc compilation of /tmp/tmpsQ7gQT/kernel.cu failed
[command: nvcc --cubin -arch sm_61 -I/home/nadav/cryoSPARC_v2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/cuda kernel.cu]
[stderr:
kernel.cu(131): error: identifier "__shfl_down_sync" is undefined
kernel.cu(256): error: identifier "__shfl_down_sync" is undefined
kernel.cu(318): error: identifier "__shfl_down_sync" is undefined
kernel.cu(382): error: identifier "__shfl_down_sync" is undefined
kernel.cu(569): error: identifier "__ballot_sync" is undefined
5 errors detected in the compilation of "/tmp/tmpxft_000011a0_00000000-7_kernel.cpp1.ii".]