GPU error - Deep Picker Train

[prash]

Hi CS Team

I am getting following assertion error when I try to run Deep Picker Train. I have tried with 1 parallel thread as well as 4.
AssertionError: Input number of GPUs must be less than or equal to number of available GPUs. Please check job log for more information.

Any ideas?

[wtempel]

@prash

Please check job log for more information.

Please could you share the job log, which can be retrieved with this command:
cryosparcm joblog <project_id> <job_id>
Please also post a screenshot of the failed job’s Parameters and Inputs tab.

[prash]

Here you go:

[wtempel]

The following suggestion has worked for a user who was facing a very similar issue:

@stephan

I then created a hard link, as suggested in the issue I linked above:

cd /usr/local/cuda-11.2/lib64
sudo ln libcusolver.so.11 libcusolver.so.10 

In your case, the directory seems to be /usr/local/cuda-11.4/lib64, where you first would want to confirm that the file libcusolver.so.11 exists.

[prash]

Yes, the file libcusolver.so.11 does exist.

[wtempel]

@prash

libcusolver.so.11 does exist.

Did you try linking (requires write privileges in /usr/local/cuda-11.4)

@stephan

sudo ln libcusolver.so.11 libcusolver.so.10

and running the Deep Picker Train job again?

[prash]

Hi @wtempel

The solution worked like a charm.

Thank you for your help.

Prash

[DavidHoover]

As an alternative to hard linking libcusolver.so.11, we created a directory containing a symlink to libcusolver.so.11 and added it to $LD_LIBRARY_PATH.

export LD_LIBRARY_PATH=/path/to/directory:$LD_LIBRARY_PATH
{{ run_cmd }}