Raise OSError('CUDA driver library not found')

Troubleshooting
1

Hi all,

I am a newcomer to cryoSPARC and have made a new standalone installation on a machine with two NVIDIA RTX 2080 Ti GPUs. I installed CUDA 10.1 and compiled cryoSPARC from the installer script in cryosparc2_master. Everything seemed to go fine and the very nice interface came up. I then tried to run the proteasome tutorial. Importing movies goes fine but when I try to run the patch motion correction job it dies immediately upon trying to access the GPUs. The output from “CryoSPARC job log P1 J4” is attached below. The culprit seems to be “raise OSError(‘CUDA driver library not found’)”. Any idea how I can fix this? As I wrote, cryoSPARC seemed to compile just fine.

Thanks
Derek

================= CRYOSPARCW =======  2019-11-22 16:19:01.488512  =========
Project P1 Job J4
Master cryoem-desktop Port 39002
===========================================================================
========= monitor process now starting main process
MAINPROCESS PID 40373
========= monitor process now waiting for main process
MAIN PID 40373
motioncorrection.run_patch cryosparc2_compute.jobs.jobregister
***************************************************************
Running job on hostname %s cryoem-desktop
Allocated Resources :  {u'lane': u'default', u'target': {u'lane': u'default', u'name': u'cryoem-desktop', u'title': u'Worker node cryoem-desktop', u'resource_slots': {u'GPU': [0, 1], u'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], u'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}, u'hostname': u'cryoem-desktop', u'worker_bin_path': u'/home/cryoem/software/cryosparc/cryosparc2_worker/bin/cryosparcw', u'cache_path': u'/mnt/disk2/cryosparc', u'cache_quota_mb': None, u'resource_fixed': {u'SSD': True}, u'cache_reserve_mb': 10000, u'type': u'node', u'ssh_str': u'cryoem@cryoem-desktop', u'desc': None}, u'license': True, u'hostname': u'cryoem-desktop', u'slots': {u'GPU': [0, 1], u'RAM': [0, 1, 2, 3], u'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}, u'fixed': {u'SSD': False}, u'lane_type': u'node'}
Process Process-1:2:
Process Process-1:1:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
    self.run()
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 35, in process_pipeline_work
  File "cryosparc2_compute/jobs/pipeline.py", line 35, in process_pipeline_work
    process_params = process_setup(proc_idx) # do any setup you want on a per-process basis
    process_params = process_setup(proc_idx) # do any setup you want on a per-process basis
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 75, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.process_setup
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 75, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.process_setup
  File "cryosparc2_compute/engine/__init__.py", line 8, in <module>
  File "cryosparc2_compute/engine/__init__.py", line 8, in <module>
    from engine import *
    from engine import *
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 12, in init cryosparc2_compute.engine.engine
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 12, in init cryosparc2_compute.engine.engine
  File "cryosparc2_worker/cryosparc2_compute/engine/gfourier.py", line 6, in init cryosparc2_compute.engine.gfourier
  File "cryosparc2_worker/cryosparc2_compute/engine/gfourier.py", line 6, in init cryosparc2_compute.engine.gfourier
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py", line 19, in <module>
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py", line 19, in <module>
    from . import misc
    from . import misc
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/misc.py", line 24, in <module>
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/misc.py", line 24, in <module>
    from . import cuda
    from . import cuda
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cuda.py", line 10, in <module>
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cuda.py", line 10, in <module>
    from .cudadrv import *
    from .cudadrv import *
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cudadrv.py", line 32, in <module>
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cudadrv.py", line 32, in <module>
    **raise OSError('CUDA driver library not found')**
**    raise OSError('CUDA driver library not found')**
OSError: CUDA driver library not found
OSError: CUDA driver library not found
Traceback (most recent call last):
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/cryoem/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
2

Hi,

Seems no-one ran into this problem before… After a bit of searching I found that cryoSPARC was looking for libcuda.so in my $LD_LIBRARY_PATH, which is set to /usr/local/cuda-10.1/lib64 in my .bashrc, but it is not there in my installation. I found it in /usr/local/cuda-10.1/targets/x86_64-linux/lib/stubs/libcuda.so and created a soft link to it in /usr/local/cuda-10.1/lib64. With this modification the previously failing patch motion correction job now works, as do several more downstream steps in the T20S tutorial. However I’m not sure whether this was the most elegant solution. I’m unfamiliar with the language of stubs, containers etc. There is another file in /usr/local/cuda-10.1/lib64 called libcudart.so. What is the difference between this and libcuda.so? I assume the “rt” stands for “run time”.

/Derek

1 Like