CompileError: nvcc preprocessing failed: gcc versions later than 6 are not supported!

Troubleshooting
1

We have an Ubuntu 16.04 master and an Ubuntu 18.04 standalone worker installed with Cryosparc 2.2. I started out with CUDA 9.2 on the worker, but Cryosparc is incompatible with CUDA 9.2 as per this ticket:

AttributeError: undefined symbol: cufftSetCompatibilityMode (V2)

So I downgraded CUDA to 9.1 and re-installed the worker. Now when we attempt to perform full frame motion correction, we get the following error:

CompileError: nvcc preprocessing of /tmp/tmp35pFNe.cu failed
[command: nvcc --preprocess -arch sm_61 -I/data1/local/home/cryosparc_user/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/cuda /tmp/tmp35pFNe.cu --compiler-options -P]
[stderr:
In file included from /usr/local/cuda/bin/…//include/host_config.h:50:0,
from /usr/local/cuda/bin/…//include/cuda_runtime.h:78,
from :0:
/usr/local/cuda/bin/…//include/crt/host_config.h:121:2: error: #error – unsupported GNU version! gcc versions later than 6 are not supported!
#error – unsupported GNU version! gcc versions later than 6 are not supported!
^~~~~
]

Unfortunately CUDA 9.1 only supports versions of gcc 6 or below, and Ubuntu 18.04 ships with gcc 7.3 (which itself is ancient – the current version of gcc is 8.2).

I can install gcc 6 on the standalone worker, but i can’t understand why compiles are happening in the first place (shouldn’t everything be pre-compiled on install?). Also, if I install gcc, how will the Cryosparc system know to use that rather than the system default gcc?

2

You can force CUDA to work with a specific gcc version by symlinking it to the cuda ‘bin’ subdirectory, e.g.

sudo ln -s /usr/bin/gcc-5 /usr/local/cuda-9.1/bin/gcc
sudo ln -s /usr/bin/g+±5 /usr/local/cuda-9.1/bin/g++

Hope this helps.

[Edit: Fixed the typo @pgoetz pointed out]

3

Thanks very much for pointing me in the right direction, and I wish I’d known about this trick months ago; it solves other problems as well. However, for the benefit of someone else looking at this post, I think what you meant to say was this:

sudo ln -s /usr/bin/gcc-5 /usr/local/cuda-9.1/bin/gcc
sudo ln -s /usr/bin/g++-5 /usr/local/cuda-9.1/bin/g++
1 Like
4

Indeed, that’s what I meant to type. Thanks!