Import of EER super slow

Import and Export
1

Hi,
I am trying to import eer files in cryosparc. While it works in principle but it takes for ever. For importing 7000 files it needed almost 24 hours. Is there anything that can be done? It also seems like adding more cpus does not really increase the speed. When I import mrc files it is much faster.

Best,

David

2

Hi David,

Out of curiosity, did you skip header check?

Best,
Joon

3

yes I tried it and then its much faster however but then there seems to be issues with later steps

4

@david.haselbach Please could you provide details on the issues you encountered in later steps?

5

motion correction repeatedly crashed and I thought it may had to do with not probably reading the headers and that seemed to be the case

6

@david.haselbach Ideally, one shouldn’t have to choose between rapid eer data import and error-free processing. To that end, please can you

  1. confirm that downstream processing proceeds as expected after (slow) import?
  2. provide details that could help us understand how skipping the header check causes errors in downstream processing. For example, do downstream jobs fail immediately or do errors appear to be linked to specific eer files?
  3. post error messages (after obfuscating confidential information).
7
  1. I do confirm
  2. it is later so I do think it is specific EER files.
  3. I have to reproduce.
8

I also have the same issue. It took me 24 hours to import 2000 eer files from Falcon4 (fraction=40, upsampling factor=1). And I found that using more CPUs made it even slower. I don’t know such a slow importing speed is normal or abnormal.

9

Welcome to the forum @lcgshk .
May I ask:
What are you CryoSPARC version and patch level?
On what type of device are the EER files stored?
If the EER files are served over the network, what is the data transfer rate (Gb/s)?
Does processing work after the slow import?

10

Thank you for your quick reply!

  1. My Cryosparc version and patch level is v3.3.2+220824.
  2. We tried both storing the EER files in a portable external hard drive or local disk, importing of EER in both cases were slow while the local disk was slightly faster.
  3. EER files were not served over network.
  4. All processing worked after the slow import.
    Thank you again!
11

Hi,
I have the same problem. Currently importing 7000 eer files (Falcon4, fraction = 40, upsampling factor = 2) witch v3.3.2. I did not skip the header check and it takes a very long time. The movies are stored on a JBOD that is connected via 25 GB/s.
Cheers,
Gregor

12

@lcgshk @gha
Import of EER datasets involves reading a large amount of data and may therefore be slow. For faster import, enable “Skip Header Check”. Please keep in mind that this option may delay the discovery of potential EER file corruption to subsequent processing steps.

13

Thank you! I will try to skip the header check.

14

I ran into this problem too, after we upgraded to the Falcon 4i. Reading the header takes several seconds now, sometimes as long as a minute. It used to be instantaneous. I ran into this problem using CryoSPARC, IMOD (the header command), MotionCor2, and EMAN2. Skipping the header check during import indeed speeds it up, but I don’t see such an option during motion correction.

Random observation: Once I’ve read an EER header, reading it immediately after is instantaneous again. I try to read it again hours later, it’s slow again. My guess is that something is cached in the system RAM?

15

Same here – importing 8700 eer files using 8 cores took me 25 hours.

16

@marinegor Did you disable the header check for this import?

17

no, I didn’t. I can run that later if you want to know approximate speedup.

18

I expect a significant speedup of the import if the header check is omitted. Please let us know if you

  • do not observe a significant speedup during import
  • your use case requires the header check during import
19

yep, the speedup is significant – the job finished in few minutes instead of 25 hours (I also didn’t upsample the frames this time). Thanks!