Caching process

Troubleshooting
1

Hi all,

After importing the particle_stack.mrc file into cryosparc.v2, I started a 2D classification job but it stuck at the transferring step. No updates after several hours and no error messages. Even after we cleaned up the cache folder and restarted the job, there was still no outputs after the transferring line. What might be the reason for this? Any help would be appreciated.

Importing job module for job type homo_abinit…

Job ready to run

Using random seed for sgd of 1410902803

Loading a ParticleStack with 449617 items…

SSD cache : cache successfuly synced in_use

SSD cache : cache successfuly synced, found 463435.87MB of files on SSD.

SSD cache : cache successfuly requested to check 1 files.

Detected file change due to change in file size.

SSD cache : cache successfuly checked 1 files.

SSD cache : cache successfuly updated 1 hits.

SSD cache : cache requires 175631.64MB more on the SSD for files to be downloaded.

SSD cache : cache has enough available space.

Transferring J10/imported/particle_stack_4.mrc (175632MB)
Complete : 0MB
Total : 175632MB

Thanks,
Chen

2

Hi @Chen,

The text that you see in the streamlog actually only changes when a file is finished copying - usually datasets are not in a single .mrc file (though that is no problem) but in multiple smaller files, so you would see the Complete: XMB changing as each smaller file is copied, and the cache system also reports the copy speed. In your case the single .mrc file is 175GB which it seems is not copying very quickly from wherever it is stored over to your SSD. So most likely the job is working correctly but is just waiting for the file transfer of that one large file to complete.

things to try:

  1. run the same job, but monitor disk activity using iotop or nmon or just look inside the cryosparc cache directory to see that the file is being copied (its file size will grow)
  2. run the same job but turn off SSD caching for it (the last parameter)
  3. try manually copying the large file to the SSD cache location to check how fast it goes:
    rsync -ah --progress source-file destination-file
  4. make sure the cryosparc cache directory is actually located on a local SSD