Slower performances for motion correction and particles extractions

Troubleshooting
#1

Hi Everyone,

Now I am working with CS 3.0 (hasn’t updated to CS 3.1)
While I am using, I feel some strange performances in some processes.
Especially both motion correction and extraction.
I checked running times between CS 2.15 and CS 3.0.
CS 3.0 took 4~5 times longer than.
For example, 2.15 took 16 hours for 15K mic, but 3.0 takes 5 hours for 1k mics.
So performance is 5 fold less.
Also 2.15 took 6 hours for 2m particles, but 3.0 took random (faster or slower)

I am using CUDA 10.1 with driver 455.45.
but NVIDIA-smi says 11.1, also only use 2 cores( I set 4 cores for run)
Give me some advice for this

Thanks

Jinseo

#2

Hi @Jinseo can you try updating to v3.1? We made some fixes there that should improve performance of some jobs in 3.1

#3

Hi,

I am already using CS 3.1.
That issues came from CS 3.1.
Except that, the other thing are great!
Thanks,

Jinseo

#4

Hi @Jinseo,

Can you tell us your:

  • OS version (output of uname -a)
  • output of nvidia-smi
#5

Hi @apunjani,
Sorry to bother you. Although I am not Jinseo I have met similar slow performance in cryosparc v3.1.0.
Here’s our computer info:
OS version: Linux spgpu 4.15.0-133-generic #137~16.04.1-Ubuntu SMP Fri Jan 15 02:55:18 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
nvidia-smi output: (Graphic card is RTX 2080Ti)

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.40       Driver Version: 430.40       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:02:00.0 Off |                  N/A |
| 40%   67C    P2    91W / 250W |   1718MiB / 11018MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:03:00.0 Off |                  N/A |
| 47%   76C    P2   135W / 250W |   1068MiB / 11019MiB |     72%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 208...  Off  | 00000000:83:00.0  On |                  N/A |
| 45%   76C    P2   193W / 250W |   2172MiB / 11019MiB |     70%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 208...  Off  | 00000000:84:00.0 Off |                  N/A |
| 44%   72C    P2   125W / 250W |   1718MiB / 11019MiB |     66%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     24714      C   python                                      1707MiB |
|    1     24715      C   python                                      1057MiB |
|    2      2293      G   /usr/lib/firefox/firefox                       6MiB |
|    2     23295      G   /usr/lib/xorg/Xorg                           188MiB |
|    2     24155      G   /opt/teamviewer/tv_bin/TeamViewer             18MiB |
|    2     24331      G   compiz                                       233MiB |
|    2     24716      C   python                                      1707MiB |
|    2     31445      G   /usr/lib/firefox/firefox                       6MiB |
|    3     24718      C   python                                      1707MiB |
+-----------------------------------------------------------------------------+

Thanks!

#6

Hi @sunny1226, please tell me the following:

  • What kids jobs are you trying to run where you’re seeing the slower performance?
  • Is SSD caching enabled for those jobs?
  • Are the particles you’re processing extracted by cryoSPARC or by a different program?
#7

Hi @apunjani,

I think now is OK.
It may be caused by dataset imported from 2.15 processed dataset.
Since all data process through CS3.1 are seemed more faster than before.
Thanks!

Jinseo