New homogeneous refinement fails

3D Refinement
#1

Hi,
The new version of homogeneous refinements jobs fail for me when they try to start the iterations, please see below. I am running the latest CryoSPARC update, and the older version of homogeneous refinement (legacy) works fine.
Is this a CUDA issue?

Thank you for your help
Nadav

[CPU: 1.31 GB]   ====== Starting Refinement Iterations ======
[CPU: 1.31 GB]   ----------------------------- Start Iteration 0
[CPU: 1.31 GB]     Using Max Alignment Radius 8.590 (30.000A)
[CPU: 1.31 GB]     Auto batchsize: 2880 in each split
[CPU: 1.37 GB]   Engine Started.
[CPU: 2.38 GB]   Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/nadav/cryoSPARC_v2/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_worker/cryosparc2_compute/engine/newengine.py", line 1691, in cryosparc2_compute.engine.newengine.process.work
  File "cryosparc2_worker/cryosparc2_compute/engine/newengine.py", line 1731, in cryosparc2_compute.engine.newengine.process.work
  File "cryosparc2_worker/cryosparc2_compute/engine/newengine.py", line 879, in cryosparc2_compute.engine.newengine.EngineThread.compute_resid_pow
  File "cryosparc2_worker/cryosparc2_compute/engine/newcuda_kernels.py", line 3326, in cryosparc2_compute.engine.newcuda_kernels.compute_resid_pow
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 362, in cryosparc2_compute.engine.cuda_core.context_dependent_memoize.wrapper
  File "cryosparc2_worker/cryosparc2_compute/engine/newcuda_kernels.py", line 3279, in cryosparc2_compute.engine.newcuda_kernels.get_compute_resid_pow_kernel
  File "/home/nadav/cryoSPARC_v2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py", line 291, in __init__
    arch, code, cache_dir, include_dirs)
  File "/home/nadav/cryoSPARC_v2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py", line 254, in compile
    return compile_plain(source, options, keep, nvcc, cache_dir, target)
  File "/home/nadav/cryoSPARC_v2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py", line 137, in compile_plain
    stderr=stderr.decode("utf-8", "replace"))
CompileError: nvcc compilation of /tmp/tmpsQ7gQT/kernel.cu failed
[command: nvcc --cubin -arch sm_61 -I/home/nadav/cryoSPARC_v2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/cuda kernel.cu]
[stderr:
kernel.cu(131): error: identifier "__shfl_down_sync" is undefined
kernel.cu(256): error: identifier "__shfl_down_sync" is undefined
kernel.cu(318): error: identifier "__shfl_down_sync" is undefined
kernel.cu(382): error: identifier "__shfl_down_sync" is undefined
kernel.cu(569): error: identifier "__ballot_sync" is undefined
5 errors detected in the compilation of "/tmp/tmpxft_000011a0_00000000-7_kernel.cpp1.ii".]
#2

HI @nadav,

This error arises because the version of CUDA you are running cryoSPARC with doesn’t support the new commands in the Refinement job you’re running. Is it possible if you can download and install a newer version of CUDA (9.2+)? You can then re-compile cryoSPARC using the command detailed below. Make sure to run this command on your worker node that has a GPU.

https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring/cryosparcw#cryosparcw-newcuda-less-than-path-greater-than

#3

OK, I will try to update CUDA once other processes are done
thank you