Deep Particle Picking

Install, Configure and Update
1

Dear cryoSPARC developers,

We currently use CUDA 10.2 with cryoSPARC 3.3.1 on our servers. Everything works OK except Deep Particle Picking. Do I have to install CUDA 11+ to make Deep Particle Picking work on v3.3.1?

Thank you,
Sergei

2

@Sergei Please can you provide more details on how deep particle picking is failing. Which step of the picking process failed; what error messages or other indications of failure did you observe?

3

@Sergei, as of v3.2 Deep Particle Picking jobs require CUDA 11+. There is a “newcuda” subcommand to the worker command, cryosparcw, to switch cuda versions on the worker.

4

The error was:
[CPU: 377.7 MB] Traceback (most recent call last): File “cryosparc_worker/cryosparc_compute/run.py”, line 85, in cryosparc_compute.run.main File “cryosparc_worker/cryosparc_compute/jobs/deep_picker/run_deep_picker.py”, line 67, in cryosparc_compute.jobs.deep_picker.run_deep_picker.run_deep_picker_train AssertionError: Input number of GPUs must be less than or equal to number of available GPUs. Please check job log for more information.

5

Andrew, thank you. We have to install CUDA 11+ first.

6

@Sergei Along the same vein as @andrew-niaid’s suggestion, please install cuda-11.2 and update the workers with <path-to-cryosparc>/cryosparc_worker/bin/cryosparcw newcuda <path-to-cuda-11.2>.
If the error still occurs after the cuda update, you can inspect the output of cryosparcm joblog ... for additional information.

7

Hi wtempel,

We installed cuda-11.2 and run newcuda command, however deep picker is still not working.
Below are the error messages. I would appreciate your advise on how to fix it.

Thank you,
Sergei

[CPU: 393.1 MB]  Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 85, in cryosparc_compute.run.main
  File "cryosparc_worker/cryosparc_compute/jobs/deep_picker/run_deep_picker.py", line 67, in cryosparc_compute.jobs.deep_picker.run_deep_picker.run_deep_picker_train
AssertionError: Input number of GPUs must be less than or equal to number of available GPUs. Please check job log for more information.

[pletnevs@ai-hpcgpuvrc2 bin]$ /programs/linux64/cryosparc-3.3/cryosparc_master/bin/cryosparcm joblog P6 J58
2022-01-26 00:03:54.403144: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0

2022-01-26 00:03:54.407797: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11

2022-01-26 00:03:54.407856: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11

2022-01-26 00:03:54.408843: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10

2022-01-26 00:03:54.409485: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10

2022-01-26 00:03:54.409573: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /programs/linux64/cryosparc-3.3/cryosparc_worker/cryosparc_compute/blobio:/programs/linux64/cryosparc-3.3/cryosparc_worker/cryosparc_compute/libs:/programs/linux64/cryosparc-3.3/cryosparc_worker/deps/external/cudnn/lib:/usr/local/cuda-11.2/lib64

2022-01-26 00:03:54.410218: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11

2022-01-26 00:03:54.410360: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8

2022-01-26 00:03:54.410374: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.

Skipping registering GPU devices...
8

Hey @Sergei,

Try this out:

9

Hi Stephan,

It worked! Thank you!

Sergei