Nvidia rtx 3090

Troubleshooting
#1

Hi,

I just got the 3090 GPU card and want to burn it with cryoSPARC.
After finishing the driver update to 455.28 and re-install cryoSPARC (CUDA 10.2),
there’s an error message block the process:

Blockquote
[CPU: 4.36 GB] Traceback (most recent call last):
File “cryosparc2_compute/jobs/runcommon.py”, line 1685, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 110, in cryosparc2_compute.engine.cuda_core.GPUThread.run
File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 111, in cryosparc2_compute.engine.cuda_core.GPUThread.run
File “cryosparc2_worker/cryosparc2_compute/engine/engine.py”, line 991, in cryosparc2_compute.engine.engine.process.work
File “cryosparc2_worker/cryosparc2_compute/engine/engine.py”, line 87, in cryosparc2_compute.engine.engine.EngineThread.load_image_data_gpu
File “/home/linhsinhung/cryosparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py”, line 549, in fill
func = elementwise.get_fill_kernel(self.dtype)
File “”, line 2, in get_fill_kernel
File “/home/linhsinhung/cryosparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/tools.py”, line 432, in context_dependent_memoize
result = func(*args)
File “/home/linhsinhung/cryosparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/elementwise.py”, line 496, in get_fill_kernel
“fill”)
File “/home/linhsinhung/cryosparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/elementwise.py”, line 161, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs)
File “/home/linhsinhung/cryosparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/elementwise.py”, line 147, in get_elwise_kernel_and_types
keep, options, **kwargs)
File “/home/linhsinhung/cryosparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/elementwise.py”, line 75, in get_elwise_module
options=options, keep=keep)
File “/home/linhsinhung/cryosparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py”, line 291, in init
arch, code, cache_dir, include_dirs)
File “/home/linhsinhung/cryosparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py”, line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File “/home/linhsinhung/cryosparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py”, line 78, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode(“utf-8”))
File “/home/linhsinhung/cryosparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py”, line 55, in preprocess_source
cmdline, stderr=stderr)
CompileError: nvcc preprocessing of /tmp/tmpvew70x.cu failed
[command: nvcc --preprocess -arch sm_86 -I/home/linhsinhung/cryosparcV2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/cuda /tmp/tmpvew70x.cu --compiler-options -P]
[stderr:
nvcc fatal : Value ‘sm_86’ is not defined for option ‘gpu-architecture’

Seems like I’m facing the problem like the one previous 2080 also had.
NVIDIA RTX 2080: nvcc fatal : Value ‘sm_75’

So…the only way is to wait for the update of CUDA11 support ??
or there is any other way to force GPU card recognize old sm version?

#2

Hi @linhsinhung,

You’re correct, this error is because this GPU needs to be run with CUDA 11.0, which we unfortunately don’t have support for as of v2.15.0 and v2.15.2-live_privatebeta. Currently there isn’t a way around this that I know of- you’re going to have to wait until the next release of cryoSPARC to use your new GPU with cryoSPARC. We’re working super hard to get the next release out as soon as possible- the wait shouldn’t be too long now.

1 Like
#3

Hi Stephan,

Any idea when that could be?

Best,

Jean-Paul

#4

Hi @jparmache_psu, CUDA 11 support is available in v3.0, released yesterday.

1 Like
#5

Seems like CUDA11 is not latest enough for RTX3090
It should be upgrade to CUDA11.1 (due to sm_86)

#6

I have a 3090, which seems to be working now (cryosparc v3.0.1; cuda 11.1.1_455.32; nvidia-455.45.01 driver). Both the 3090’s and the two Quadro P6000’s in this computer are reporting as 1080Ti’s for some reason now, but all are working.

$bin/cryosparcw gpulist
Detected 3 CUDA devices.

id pci-bus name

   0      0000:3B:00.0  Quadro P6000
   1      0000:86:00.0  Quadro P6000
   2      0000:DB:00.0  GeForce RTX 3090

$bin/cryosparcw connect --worker xxx --master xxx --ssdpath xxx --lane xxx -gpus 0,1,2 --update
…
gpus : [{‘id’: 0, ‘mem’: 11721506816, ‘name’: ‘GeForce GTX 1080 Ti’}, {‘id’: 1, ‘mem’: 11721506816, ‘name’: ‘GeForce GTX 1080 Ti’}, {‘id’: 2, ‘mem’: 11721506816, ‘name’: ‘GeForce GTX 1080 Ti’}]
…

#7

Hi @thomaspv,

GPU info is handled by a different function, which gets called whenever you restart cryoSPARC. To update the GPU info without having to restart cryoSPARC, run the command:
cryosparcm cli "get_gpu_info()"

Adding GPU to system
#8

Thank you @stephan. Everything is working now and labeled correctly.

gpus : [{‘id’: 0, ‘mem’: 25637224448, ‘name’: ‘Quadro P6000’}, {‘id’: 1, ‘mem’: 25637224448, ‘name’: ‘Quadro P6000’}, {‘id’: 2, ‘mem’: 25447170048, ‘name’: ‘GeForce RTX 3090’}]

2 Likes
#9

Thanks for your awesome work!