3D-VA running very slow

[gptomaleri]

Dear cryoSPARC community,

I am running 3D-VA for 312K particles (box size: 600pix binned 2x) with resolution cut-off ranging from 3 to 6 Å and solving for 5 models. It is taking ~3 days to run the job and it seems a bit unusual from what was described by other users am I right?

The SSD seems to be working just fine. Any suggestion on why is taking this long ?

Thanks,
Gio

[user123]

Hi Gio,
This sounds normal considering the large box size (600pix) and high resolution range (3A). Consider running with binned particles such that nyquist is ~ 1.5x lower than your desired filter resolution (bin to 1 or 1.2 A/pix for 3A cutoff). Binning helps substantially and even using a smaller box size is helpful when you don’t need the super high resolution features, which are lost in 3DVA anyways.

[olibclarke]

Not 600px - 300px - in my hands 300k particles in 320px box completes in ~3-4hrs with 5 components (on a 3090-Ti), just as a data point

[user123]

Sorry, you’re right, Gio. This is quite slow. On our 3080Ti, 340k particles at box 400pix runs in ~4hrs for 3 components. And 660k particles into 5 components at 400pix runs in ~10hrs.

[gptomaleri]

Also noticed that the NU-refinement took ~7h to run on that same amount of particles.

There is something up. Does anyone from the Cryosparc team want to give their opinion?

[wtempel]

@gptomaleri
Please can you elaborate on the hardware involved.

• GPU model
• Amount of DRAM
• CPU model
• SSD model
• some measure of SSD wear, such as wear_leveling % or Percentage Used: depending on SSD type and smartmontools or nvme-cli version

[gptomaleri]

Hi @wtempel,

Here is a bit more info:

@wtempel

• GPU model Nvidia P100 with 16GB of ram each

@wtempel

• Amount of DRAM The cluster we use ranges from 192 GB to 1.5TB DRAM

@wtempel

• CPU model
  16-core Intel Skylake (2.1 GHz)
  28-core Intel Cascadelake 8276 (2.20 GHz)
  24-core Intel Cascadelake 8276 (2.40 GHz)

@wtempel

• SSD model we don’t use a desktop system, we are using many disks on a high-performance GPFS filesystem.

@wtempel

• some measure of SSD wear, such as wear_leveling % or Percentage Used: depending on SSD type and smartmontools or nvme-cli version

For the measure of SSD, wear would be most applicable to a desktop SSD-based system, not a cluster.

Also, the cluster script was set to run with 4 CPU’s, 1 GPU, and 48GB of memory not sure if changing anything will speed the process…

[wtempel]

Thank you for the clarification.
Over what kind of network (protocol, speed rating) are worker nodes connected to GPFS?
Is the cache_path variable defined for cryoSPARC integration with your cluster and, if so, to what type of storage does it point?