RuntimeError: cuInit failed in CentOS 6 system

Hi, guys,

Thank you for providing such good software! Thanks for your efforts!
But I met some errors when running example dataset. Could you help me to find some solutions?

The version of cryoSPARC is 0.6.1. And the target platform is Linux CentOS 6 x86_64 with kernel 2.6.32-642.3.1.
We have several Tesla K80 GPUs and the version of CUDA is V9.0.176. (just updated from CUDA 8.0).

Here are problems:
As we known, the web app based on Nodejs requires gcc to support C++ 11 standard, which means It’s too old to run web app with default gcc of CentOS 6 which is still in 4.4.7. So I install a newer version gcc-5.4.0 which support C++11 to run cryoSPARC. It works. But I failed to lanuch the Ab-initio Reconstruction for example data (everything is fine before this step.). Please see the follow traceback.

Launched job 7XPLxKNZFfwc8rmKY
License is valid.
Traceback (most recent call last):
  File "/home/lqhuang/Software/cryosparc/cryosparc-compute/sparcjob.py", line 85, in <module>
    cuda_dev_info = engine.cuda_core.get_all_device_infos()
  File "/home/lqhuang/Software/cryosparc/cryosparc-compute/engine/cuda_core.py", line 52, in get_all_device_infos
    cudrv.init()
RuntimeError: cuInit failed: no CUDA-capable device is detected

I’m quite sure I have installed CUDA correctly. In fact, the problem is if I deprecated gcc-5.4.0 from environment variables PATH and LD_LIBRARY_PATH, it works well including cryosparc configure gpu all or else tests under ipython directly.

Hence, I list a table.

I tried something:
(1) Reinstall pycuda compiled with gcc-5.4.0. It’s the same.
(2) Reinstall CUDA and pycuda under gcc-5.4.0. Still doesn’t work.
(3) Run python under gcc-5.4.0 and remove gcc-5.4.0 from os.environ['PATH'] and os.environ['LD_LIBRARY_PATH']. Failed. Obviously, python has loaded libraries when initializing.

And I want to reset the environment before Webapp call corresponding python inference. Because I’m not good at nodejs, I don’t know where to change codes.

It’s embarrassed that if I degrade into gcc-4.4.7 I even cannot make cryosparc start. LOL.
I tried search some issues reported under CentOS 6 in discussion forum, but no one happens this. It’s weird and strange. :sweat_smile:

@apunjani
I’m looking for any suggestions or any comments if I am doing something wrong.

Thanks a lot!

Regards,
Lanqing

Hi @lqhuang,

Thanks for posting such a detailed explanation of the problem, it really helps.

One extra piece of information that perhaps you can provide is the end of the logfile
cryosparc/run/meteor.stdout
This will contain the errors that the webapp is having when you use gcc 4.4.7

I think the solution will be that you should stick with gcc 4.4.7 for actually running cryosparc, but we’ll just have to reinstall nodejs and its dependencies with the newer gcc in the path. As far as I understand, gcc should not be needed at all after node is installed and dependencies compiled.

Hope we can figure it out,
Ali

Hi @apunjani ,

Thanks for your reply!

Here is the content of meter.stdout:

------------------------------------
 ___ __     __   __ __  __  __  ___
|   |__)\ /|  | (_ |__)|__||__)|
|___|  \ | |__| __)|   |  ||  \|___

------------------------------------
------------------------------------
/home/lqhuang/Software/cryosparc/cryosparc-webapp/bundle/programs/server/node_modules/fibers/future.js:280
                                                throw(ex);
                                                ^

Error: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /home/lqhuang/Software/cryosparc/cryosparc-webapp/bundle/programs/server/npm/node_modules/grpc/src/node/extension_binary/grpc_node.node)
    at Error (native)
    at Object.Module._extensions..node (module.js:434:18)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/home/lqhuang/Software/cryosparc/cryosparc-webapp/bundle/programs/server/npm/node_modules/grpc/src/node/src/grpc_extension.js:38:15)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)

FYI.
That’s a very good solution.
However, I’m still a little interested why other users under CentOS 6 didn’t report this. LOL.

I’m looking forward for your upgraded version. Thanks very much.

Regards,
Lanqing

Hi @lqhuang, does that mean you were able to sort out the issue?
I’m curious too why others CentOS 6 users didn’t have the same problem - but sometimes the inner workings of Linux can get so murky.

Hi Ali @apunjani ,

I solved this by upgrading CentOS 6 to CentOS 7. Fortunately, I’m one of the adminstrators and I got approved to reinstall system in this workstation. CentOS 6 sometimes is also too old for other applications. Though it cost me a lot of time to migrate, it should be worthy.

It seems it’s an inevitable problem in CentOS 6, I don’t how them could skip this problem with default system configurations.

Thanks for your help!

Regards,
Lanqing