Install OK, fails abinitio test run

RHEL7, CUDA 8, paths all OK, nvcc OK, Install all OK. Fails with test dataset at abinitio. Tried both v0.5.3 and v.0.5.6, same error. Checked CUDA paths again, reinstalled pycuda, still same error.

Computing Ab-Initio Structure:
Volume Size: 64 (voxel size 4.86A)
Final Output Volume Size: 128
Data Size: 256 (pixel size 1.22A)
Resolution Range: 35.00A to 12.00A
Fourier Radius Range: 10.0 to 25.9 with steps of 0.040000
Generating Initial Densities (seed = 1245111445)
Generating Random Initial Density for Class 0
Traceback (most recent call last):
File “/progs/cryosparc/cryosparc-compute/”, line 530, in
cuda_devs = current_experiment_params[‘cuda_devs’])
File “/progs/cryosparc/cryosparc-compute/”, line 225, in abinit
premult3 = premult3)
File “/progs/cryosparc/cryosparc-compute/”, line 57, in generate_initial_density_from_projections
ignore_dc = False)
File “/progs/cryosparc/cryosparc-compute/”, line 172, in reconstruction_kernel
compute_gradient = compute_gradient)
File “/progs/cryosparc/cryosparc-compute/objectives/”, line 997, in set_density
self.gHv = gpurealloc_zeros(self.gHv,self.gfMr.shape,n.float32,stream=stream)
File “/progs/cryosparc/cryosparc-compute/objectives/”, line 34, in gpurealloc_zeros
File “/progs/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/”, line 516, in fill
func = elementwise.get_fill_kernel(self.dtype)
File “”, line 2, in get_fill_kernel
File “/progs/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/”, line 430, in context_dependent_memoize
result = func(*args)
File “/progs/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/”, line 492, in get_fill_kernel
File “/progs/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/”, line 161, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs)
File “/progs/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/”, line 147, in get_elwise_kernel_and_types
keep, options, **kwargs)
File “/progs/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/”, line 75, in get_elwise_module
options=options, keep=keep)
File “/progs/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/”, line 291, in init
arch, code, cache_dir, include_dirs)
File “/progs/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/”, line 255, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File “/progs/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/”, line 78, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode(“utf-8”))
File “/progs/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/”, line 55, in preprocess_source
cmdline, stderr=stderr)
CompileError: nvcc preprocessing of /tmp/ failed
[command: nvcc --preprocess -arch sm_61 -I/progs/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/cuda /tmp/ --compiler-options -P]
In file included from :0:0:
/usr/include/stdc-predef.h:40:1: fatal error: cuda_runtime.h: No such file or directory
compilation terminated.

New development: on a similar workstation cryosparc 0.5.3 stopped working with the same error. It was working fine until a couple of days ago, ran ~18 jobs with no errors. What changed?

Update: reinstalled cryosparc from scratch on a RHEL7+cuda8 box, and tried to run abinitio with the example dataset. Still getting the above error. Nothing wrong with CUDA, works great with relion and other programs.
We got this powerful workstation and still cannot get ANY version of cryosparc to run on it.
Any advice? Is the new installer completely broken?
I noticed that the default pycuda is 2017 now not 2016, but reverting to the 2016 pycuda doesn’t work now (it used to).

On the other box I went through a lot of combinations and still no luck:
CUDA9 + cryosparc 0.5.6 --> Unicode error
CUDA9 + 0.5.3 --> a different error
CUDA 8 + 0.5.6 or 0.5.3 --> cuda error (as above)
CUDA 8 + previous pycuda + 0.5.3 --> cuda error (it used to work in this configuration)

It’s been a week since I posted the error…

HI @istv01,

Sorry for the delay in getting back to you. Hopefully we can clear up the issue quickly.
Nothing has changed in v0.5.3 and v0.5.6 installer has not changed either (with the exception of supporting CUDA 9). I think perhaps the difficulty may be happening because the installer is automatically looking for nvcc to find the cuda libraries, but it must be finding the wrong version.

I’m going to email you to continue this thread because I’ll need to ask you to paste the output from some commands which would be better not to post on the internet.