LogicError: cuCtxCreate failed: invalid device ordinal

Install, Configure and Update
1

Hi, I got this error short after I started 2D classification. I searched the forum but could not find any answer.

CPU: 459.7 MB]   SSD cache : complete, all requested files are available on SSD.
[CPU: 459.7 MB]    Done.
[CPU: 459.7 MB]  Windowing particles
[CPU: 459.9 MB]    Done.
[CPU: 459.9 MB]  Using 50 classes.
[CPU: 463.0 MB]  Computing 2D class averages: 
[CPU: 463.0 MB]    Volume Size: 128 (voxel size 2.53A)
[CPU: 463.0 MB]    Zeropadded Volume Size: 256
[CPU: 463.0 MB]    Data Size: 128 (pixel size 2.53A)
[CPU: 463.0 MB]    Using Reconstruction Resolution: 6.00A (53.0 radius)
[CPU: 463.0 MB]    Using Alignment Resolution: 6.00A (53.0 radius)
[CPU: 463.0 MB]    Windowing only corners of 2D classes at each iteration.
[CPU: 463.0 MB]  Using random seed for initialization of 1236193028
[CPU: 475.3 MB]    Done in 0.870s.
[CPU: 507.8 MB]  Start of Iteration 0
[CPU: 507.8 MB]  
[CPU: 540.7 MB]  Traceback (most recent call last):
  File "cryosparc2_worker/cryosparc2_compute/run.py", line 78, in cryosparc2_compute.run.main
  File "cryosparc2_worker/cryosparc2_compute/jobs/class2D/run.py", line 317, in cryosparc2_compute.jobs.class2D.run.run_class_2D
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 871, in cryosparc2_compute.engine.engine.process
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 30, in cryosparc2_compute.engine.cuda_core.initialize
LogicError: cuCtxCreate failed: invalid device ordinal

I am using cryoSPARC v2.13.2 on CentOS Linux release 7.6.1810.
Please help. Thanks.

2

Hi @donghuachen,

What version of CUDA are you using?

3

I was using cuda/10.1.168.

4

Hey @donghuachen,

I’ve seen this happen with users that have CUDA 10.1 installed.
The context error arises because scikit-cuda first tries to look for the CUBLAS library, doesn’t find it, then tries to create a CUDA context on GPU 0 to try and find out the version of CUBLAS installed. This fails if your GPUs are set to a mode that prohibits this behaviour:

0 – Default - Shared mode available for multiple processes
1 – Exclusive Thread - Only one COMPUTE thread is allowed to run on the GPU (v260 exclusive)
2 – Prohibited - No COMPUTE contexts are allowed to run on the GPU
3 – Exclusive Process - Only one COMPUTE process is allowed to run on the GPU

To change the compute mode for your GPUs, run the command:
nvidia-smi -c <compute mode #>
e.g. nvidia-smi -c 0

I think CUDA 10.1 changed the location of the CUBLAS libraries to a common area, but this change meant that users have to do some work linking the libraries to a locations where scikit cuda is looking for them.

The easier solution is to download an older version of CUDA (8 or 9) and add the directories to cryoSPARC’s path.
e.g.,

  1. Install CUDA 9.2
  2. add the line: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-9.2/lib64 to cryosparc2_worker/config.sh
5

I am using cuda/9.2.148 now and cryoSPARC seems to be working. Thanks so much!