Error reading/processing datasets after upgrade to cryoSPARC version: v4.0.3

Install, Configure and Update
1

Hi there,

After upgrading from 3.x.x to v4.0.3 I do have problems processing any raw dataset (mainly .eer). I get various strange messages such as:

######################
Fail count: 2

Traceback (most recent call last):
File “cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py”, line 371, in cryosparc_compute.jobs.rtp_workers.run.rtp_worker
File “cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py”, line 434, in cryosparc_compute.jobs.rtp_workers.run.process_movie
File “cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py”, line 580, in cryosparc_compute.jobs.rtp_workers.run.do_patch_motion
File “cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py”, line 585, in cryosparc_compute.jobs.rtp_workers.run.do_patch_motion
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 255, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 302, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
AssertionError: Get rid of patches that are too close to edges

######################
or

######################

Traceback (most recent call last):
File “cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py”, line 371, in cryosparc_compute.jobs.rtp_workers.run.rtp_worker
File “cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py”, line 434, in cryosparc_compute.jobs.rtp_workers.run.process_movie
File “cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py”, line 467, in cryosparc_compute.jobs.rtp_workers.run.do_check
File “cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py”, line 170, in cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read
File “cryosparc_worker/cryosparc_compute/blobio/prefetch.py”, line 48, in cryosparc_compute.blobio.prefetch.Prefetch.get
RuntimeError: Error ocurred (Input/output error) at line 903 in open
Could not open file. Make sure the path exists.

IO request details:
filename: /Athena_Offload/EM_STORAGE2/userXX/projectX/data1/eer/FoilHole_25791913_Data_25787546_25787548_20221125_121729_EER.eer
filetype: 2
header_only: 0
idx_start: 0
idx_limit: -1
eer_upsampfactor: 1
eer_numfractions: 40
num_threads: 6
buffer: (nil)
nx, ny, nz: 0 0 0
dtype: 0
total_time: -1.000000
######################

Main Error: Could not open file. Make sure the path exists. → The .eer files do exist, but are write protected.

I never had any problems with Cryosparc Version 3.xx before. If I change to non-protected, I’ll get the first massage.

I also reinstall the master and worker instance of cryosparc without any errors. Normal jobs e.g. homogeneous refinement etc. do work, so I do not think it is a GPU related problem.

If I import the movie-stack I get following error:

######################
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 93, in cryosparc_compute.run.main
File “/cryosparc/cryosparc/cryosparc_master/cryosparc_compute/jobs/imports/run.py”, line 861, in run_import_movies_or_micrographs
eer_upsamp_factor=eer_upsamp_factor)
File “/cryosparc/cryosparc/cryosparc_master/cryosparc_compute/jobs/imports/run.py”, line 687, in read_movie_header
nz, ny, nx = eer.read_eer_shape(abs_path)
File “/cryosparc/cryosparc/cryosparc_master/cryosparc_compute/blobio/eer.py”, line 18, in read_eer_shape
eer = open_eer(fname)
File “/cryosparc/cryosparc/cryosparc_master/cryosparc_compute/blobio/eer.py”, line 15, in open_eer
return eerdecompressor.ElectronCountedFramesDecompressor(str(fname))
RuntimeError: unable to open tiff file
######################

???

Thanks for helping,

eddi

2

Did you successfully process this dataset using an older version of CryoSPARC? If so, which version of CryoSPARC was that?
Please can you post the outputs of these two commands:

stat /Athena_Offload/EM_STORAGE2/userXX/projectX/data1/eer/FoilHole_25791913_Data_25787546_25787548_20221125_121729_EER.eer
stat -f /Athena_Offload/EM_STORAGE2/userXX/projectX/data1/eer/FoilHole_25791913_Data_25787546_25787548_20221125_121729_EER.eer
3

Hi,

I haven’t tried this data set with the older version of CryoSPARC
Is it possible to run a different version of CryoSPARC on the same system with the same database. I would be afraid to make it inconsistent.

I run a stat and stat –f on the files.
#######################
stat FoilHole_25791913_Data_25787546_25787548_20221125_121729_EER.eer
File: FoilHole_25791913_Data_25787546_25787548_20221125_121729_EER.eer
Size: 623019016 Blocks: 1216848 IO Block: 4096 regular file
Device: fe0bh/65035d Inode: 321654652 Links: 1
Access: (0660/-rw-rw----) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2022-11-29 14:26:40.103545436 +0100
Modify: 2022-11-25 11:57:16.026331900 +0100
Change: 2022-11-29 14:40:01.607537177 +0100
Birth: -

#######################

stat -f FoilHole_25791913_Data_25787546_25787548_20221125_121729_EER.eer
File: “FoilHole_25791913_Data_25787546_25787548_20221125_121729_EER.eer”
ID: f8bc14e57ca37e81 Namelen: 255 Type: ext2/ext3
Block size: 4096 Fundamental block size: 4096
Blocks: Total: 11434383795 Free: 8002958509 Available: 7428937710
Inodes: Total: 717520896 Free: 717504071

#######################

I don’t get any error message when opening those movies with TF Imageinspector…

At the same time of the CryoSparc Update we got an Upgrade to the Falcon 4 camera server/software. I checked the metadata on a file prior the update and new acquired data, and it seems that the new movies have additional metadata field. Could that be the problem?

EER Metadata

#######################
old metadata
#######################

[EER Decoder Metadata]
EER Version = 1
Size X (pixels) = 4096
Size Y (pixels) = 4096
Number of frames = 1778
Pixel format = 16-bit unsigned integer
Dose (e/pixel) = 44.82224154
Decoding time (s) = 4.832
Number of threads = 7

[EER Acquisition Metadata]
ExposureTime (s) = 7.386577
MeanDoseRate (e/pixel/s) = 6.269425
NumberOfFrames = 1778
SensorImageHeight (pixel) = 4096
SensorImageWidth (pixel) = 4096
TotalDose (e/pixel) = 44.822242

#######################
new metadata
#######################

[EER Decoder Metadata]
EER Version = 1
Size X (pixels) = 4096
Size Y (pixels) = 4096
Number of frames = 1351
Pixel format = 16-bit unsigned integer
Dose (e/pixel) = 25.44193572
Decoding time (s) = 18.314
Number of threads = 7

[EER Acquisition Metadata]
AcquisitionID = 20221125.190616.039
CameraName = BM-Falcon
CommercialName = Falcon 4
EerGainReference = ImagesForProcessing/BM-Falcon/200kV/20220715_150644_EER_GainReference.gain
ExposureTime (s) = 5.617337
MeanDoseRate (e/pixel/s) = 5.6231264099152112
NumberOfFrames = 1351
SensorImageHeight (pixel) = 4096
SensorImageWidth (pixel) = 4096
SensorPixelSize.height (m) = 7.33616362e-11
SensorPixelSize.width (m) = 7.33616362e-11
SerialNumber = 21-10-A18-BDF
Timestamp = 2022-11-25T19:06:16.623645+01:00
TotalDose (e/pixel) = 31.541774855505345

#######

Last night I was able processing the data with WARP (Real-time cryo-electron microscopy data preprocessing with Warp | Nature Methods).

Warp did not have any problems to read those movies…

Thanks again

4

Thank you for this additional information. Would you be willing to share with us:

  • job.json from that failed job’s directory
  • FoilHole_25791913_Data_25787546_25787548_20221125_121729_EER.eer
  • any other input file(s) used in this job as applicable (such as gain reference?)

If needed, we can make arrangements for you to upload the data to our server.

5

Hi,

I’m sure I found the error(s). It seems that there were several small issues, which lead to the inconsistent error messages.
First it seems that the used gain reference must be not write protected. I had to change the owner to “cryosparcuser” as well the folder it was saved in.

Next I found that I used the wrong pixel size (a factor 10 to low (:frowning:, did just realize), hence the strange error of:

“” AssertionError: Get rid of patches that are too close to edges""

I played arount with permission settings on the movies, but now they are back on reading only (chmod 644 .eer) and it is still working and the data set is being processed right now. I will report back if I’ll have issues with the next *.eer data.

Thanks again for your help!!!

6

Just to clarify: have you now confirmed that, while the gain reference must obviously be readable, it does not need to be writable?