Stream 2D slows motion correction

CryoSPARC Live
1

Version 4.2.1
We have two Cryosparc systems, one has 4xA6000 while the other has 4x2080Ti. Both systems have the same issue below:

The motion correction and CTF estimation work fine in the Live session, it can catch the collection speed, until a 2D streaming work is started. Then the motion correction takes a long time and cannot follow the collection. Even the 2D is waiting for particles it is still slow to correct motion. As soon as I killed the 2D, the pre-processing speed was back.

Anyone has an idea of what’s going on with this?

2

Welcome to the forum @monsekey.
Please can you share other details of each computers setup:

  • amount of RAM
  • amount of available RAM when motion correction slows down, using command
    free -g
  • type of storage for movies and CryoSPARC project (disk type(s), connection bandwidth)
  • number and type of CPU cores
  • transparent_hugepage settings, using command
    cat /sys/kernel/mm/transparent_hugepage/enabled
3

EER files, 7200rpm disc for storage, and SSD for 2D.
Xeon(R) Gold 6348 CPU @ 2.60GHz, 112 cores in the /proc/cpuinfo file, so should be 2 CPUs with 28 physical cores and 56 logical cores each.

With motion correction but no 2D streaming:
[kriosuser@kriosgpu ~]$ free -g
total used free shared buff/cache available
Mem: 502 291 104 0 107 207
Swap: 9 1 8

[kriosuser@kriosgpu ~]$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

With 2D and no slow down
[kriosuser@kriosgpu proc]$ free -g
total used free shared buff/cache available
Mem: 502 292 100 3 110 203
Swap: 9 1 8

2D and slowed down
[kriosuser@kriosgpu proc]$ free -g
total used free shared buff/cache available
Mem: 502 290 93 3 118 205
Swap: 9 1 8

Extracting takes much longer than before slowing down. I used 2 preprocessing workers, upsampling 1 and extraction box 400.

5

Start to suspect if this problem is limited by the I/O of SSD or CPU.

6

Here a two suggestions from the Structura team for possible remedies:

  1. Reduce, under these particular circumstances, the number of cache transfer threads by adding to cryosparc_worker/config.sh:
    export CRYOSPARC_CACHE_NUM_THREADS=1
    (guide).
  2. Disable transparent_hugepage (root access needed for this command):
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
    (details). This setting may revert to the system default after a reboot.