Post copied from other thread as requested by @hsnyder
I’m having a similar issue to Connor in another thread with an EER dataset, except it never gets anywhere at all (starts cross validation then dies before any progress) the job log is full of Unknown field tag 65002 warnings (standard for EER data, it would be nice if you would make a way to suppress that specifically as the logs end up being tens of gigabytes of line after line of that with EER data!)… box size is 440 pixels, 4K sampling (EER upsampling 1), 1344 total EER frames which all read successfully but then the job heartbeats twice, reports complete for both main and monitor processes and fails.
dmesg shows Python has segfaulted in bin_motion.so with an error 6. Memory usage for the process never exceeds 32GB on a server with 1TB, even when I tell it it can use 500GB+.
I’ve previously had 450 pixel boxes work fine with both 4K and 8K TIFF, this is the first time I’m trying EER data.
Only thing other than EER being the issue I can think of is perhaps it’s too few particles per micrograph? I’m actually playing with a contaminant as a test run (~0.5% of the total dataset) so for 13,000 micrographs there are less than 40,000 particles. Still hits 2.4Å, though, which both amuses and depresses me.
Data are on a network drive (symlinked to a local directory), network utilisation appears healthy (although not fast - it’s reading one frame at a time?)
The Python segfault is reproducible by one of my collaborators on a completely different dataset on a completely different system (also EER, however).
Thanks for opening a separate topic. Right off the bat I have one question: When you say there are 1344 total EER frames, is that what you have the “EER Number of Fractions” set to in your import job? Or is that just the number of frames present in the actual tiff container file?
Thanks for the quick response! It’s reporting the total number of raw EER frames in the micrograph movie. I left the number of fractions at the default of 40 during Import.
I have a job log zipped and ready to go if you let me know where to send it (although I’ve had a very brief look and nothing screams out loud what the issue is specifically)…
@rbs_sci actually if you have the dmesg output, could you paste it? Including the seemingly nonsensical hexadecimal output that it (hopefully) prints? Ideally something like this:
I think there may be a bug related to defect files, yes. It’s a bit of a pain, but you can work around this for now by not using the defect file and instead putting zeros in the gain reference at the defect pixels, which has the same effect. If you don’t want to re-run patch motion correction, you can use cryosparc tools to set the “mscope_params/defect_path” field to an empty string on the movies dataset
I’ll post this in here as it’s relevant, although it might be better as a separate topic “Tip”? At least until CryoSPARC suppresses the TIFFRead warning for EER…
…
For CryoSPARCers using EER data, if you don’t want to have your RBMC job.log spamming millions of lines of TIFFRead warnings (and bloating up appropriately) and don’t mind losing job.log, a quick symlink works wonders and doesn’t make CryoSPARC freak out (just removing it did):
navigate to job directory, e.g.: cd data/CS-project/J155 mv job.log job.log.backup && ln -s /dev/null job.log
This only affects one of the background logs (accessed via Job, Metadata, Log) which so far hasn’t contained any useful diagnostic information when EER RBMC jobs have crashed which isn’t indicated elsewhere (i.e.: main log or dmesg)
Many may not care or want the log (which hang the CS UI if opened when it gets above a certain size), but depending on filesystem or storage media in use, it might reduce load, wear and/or I/O queue. Because it’s writing a TIFFRead warning for every single EER frame read… which adds up pretty quickly.
