Hi,
I am getting below error when running Local Motion Correction job. These movies were imported into cryoSPARC via symbolic link.
Thank you in advance for any suggestions and inputs.
@joonpark Please can you
- post the error message as text
- ensure that the actual eer file to which the symbolic link inside
J325/imported/points is available under the path referenced by the symbolic link
Hi @wtempel
I checked to make sure that the movie.mrc that is shown in the error message is present under J325/import as well as in the directory where it symbolic links are (except for the random numbers in prefix).
Here is the text:
[CPU: 541.5 MB Avail: 140.06 GB]
Error occurred while processing J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_local.py", line 237, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_local.run_local_motion_correction_multi.motionworker.process
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/motioncorrection.py", line 132, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.motioncorrection.motion_correction
File "/mnt/ape2/cryosparc/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/motioncorrection/mic_utils.py", line 117, in replace_hot_mov
mov[:,hotmask] = n.random.poisson(avg_dose_per_frame, size=(N_Z, numhot)).astype(mov.dtype)
File "mtrand.pyx", line 3592, in numpy.random.mtrand.RandomState.poisson
File "_common.pyx", line 865, in numpy.random._common.disc
File "_common.pyx", line 412, in numpy.random._common.check_constraint
ValueError: lam < 0 or lam is NaN
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/ape2/cryosparc/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 61, in exec
return self.process(item)
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_local.py", line 223, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_local.run_local_motion_correction_multi.motionworker.process
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_local.py", line 235, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_local.run_local_motion_correction_multi.motionworker.process
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_local.py", line 271, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_local.run_local_motion_correction_multi.motionworker.process
AttributeError: 'NoneType' object has no attribute 'dtype'
Marking as incomplete and continuing...
[CPU: 455.7 MB Avail: 140.81 GB]
Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/run.py", line 95, in cryosparc_master.cryosparc_compute.run.main
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_local.py", line 374, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_local.run_local_motion_correction_multi
AssertionError: First exposure failed with exception: <cryosparc_compute.jobs.pipeline.PipelineExceptionResult object at 0x7fbad4c8ffd0>
Thank you
Please can you post additional information:
- Does the same error occur on movies other than 017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc. To test this, you might need to create a particle subset that excludes that movie, which can be achieved with a Curate Exposures job:
- connect “micrographs” from an upstream motion correction or CTF estimation job
- connect particles
- reject exposures with with low indices
- retry Local Motion Correction with the previous step’s Particles accepted
- Does the same error occur with other datasets from the same microscope?
- Outputs of the commands
project_dir=/absolute/path/to/project_directory # substitute actual project directory stat ${project_dir}/J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc stat -f ${project_dir}/J325/imported/ stat $(readlink -e ${project_dir}/J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc) stat -f $(readlink -e ${project_dir}/J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc)
Hi,
-
Same error still occurs on different movies. I have tried taking out just 017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc. I also tried a few different numbers of micrograph using the index threshold in Curate Exposure job.
-
This does happen to one other dataset collected a day before. However, this does not happen to any other dataset from the same microscope collected at different time.
stat /data2/cryosparc/P204/J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
File: /data2/cryosparc/P204/J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc -> /mnt/ywyin2/jopark/polgamma_20220611_ssDNA-ddC_Ca2+_20220611_111641/symlink/FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
Size: 146 Blocks: 0 IO Block: 4096 symbolic link
Device: 851h/2129d Inode: 122406568455 Links: 1
Access: (0777/lrwxrwxrwx) Uid: (59900/cryosparc) Gid: (59900/cryosparc)
Access: 2023-11-28 16:54:23.171081028 -0600
Modify: 2023-11-17 13:41:46.966374831 -0600
Change: 2023-11-17 13:41:46.966374831 -0600
Birth: 2023-11-17 13:41:46.966374831 -0600
stat -f /data2/cryosparc/P204/J325/imported
File: "/data2/cryosparc/P204/J325/imported"
ID: 85100000000 Namelen: 255 Type: xfs
Block size: 4096 Fundamental block size: 4096
Blocks: Total: 46875543803 Free: 5186932593 Available: 5186932593
Inodes: Total: 3750085184 Free: 3744032184
readlink -e /data2/cryosparc/P204/J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
/data2/jopark/polgamma_20220611_ssDNA-ddC_Ca2+_20220611_111641/Images-Disc1/GridSquare_9849803/Data/FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
stat /data2/jopark/polgamma_20220611_ssDNA-ddC_Ca2+_20220611_111641/Images-Disc1/GridSquare_9849803/Data/FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
File: /data2/jopark/polgamma_20220611_ssDNA-ddC_Ca2+_20220611_111641/Images-Disc1/GridSquare_9849803/Data/FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
Size: 1308623872 Blocks: 2555912 IO Block: 4096 regular file
Device: 851h/2129d Inode: 278512438321 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1805/ jopark) Gid: ( 1797/ ywyin)
Access: 2023-11-28 16:54:23.226082570 -0600
Modify: 2022-06-11 12:10:00.000000000 -0500
Change: 2023-11-16 18:12:15.769709501 -0600
Birth: 2023-11-16 18:11:00.494587428 -0600
stat -f /data2/jopark/polgamma_20220611_ssDNA-ddC_Ca2+_20220611_111641/Images-Disc1/GridSquare_9849803/Data/FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
File: "/data2/jopark/polgamma_20220611_ssDNA-ddC_Ca2+_20220611_111641/Images-Disc1/GridSquare_9849803/Data/FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc"
ID: 85100000000 Namelen: 255 Type: xfs
Block size: 4096 Fundamental block size: 4096
Blocks: Total: 46875543803 Free: 5186932084 Available: 5186932084
Inodes: Total: 3750085184 Free: 3744032184
Do this dataset and
share the same file storage “arrangement” whereas
has another storage “arrangement”?
I am trying to imagine what may lead the command
revealing a link target
whereas
points to a different path.
Is /mnt/ywyin2/ a link to /data2/?
They both share the same file storage along with all the other datasets. All of my datasets are stored at /data2/jopark/. You are correct. /mnt/ywyin2 is a link to /data2.
@joonpark Thanks for providing these details. Given
the most plausible explanation to us is some sort of disruption of data collection or data management during those two days.
Hi @wtempel
Just a follow up. I tested a few different things since it was confusing that the corrupted movie files would have no problem with all the other data processing (i.e. patch motion correction, ctf estimation, particle picking, 3D refinement, etc.) but only with local motion correction and reference based motion correction. It turned out that when I enable “save in 16-bit” option on the patch motion correction as well as particle extraction for quick 2D classification, it gives me above issue. When I disable those option, local motion and reference based motion correction jobs all run without errors. Could this be a universal bug or just something specific to my dataset?
Thank you,
Joon
1 Like
Hi @joonpark. Good catch re: 16-bit! My first thought is that if that’s the case, there’s probably something unusual about the background estimate saved by patch motion correction. In the project folder, these files would be located like motioncorrection/[something]_background.mrc. Are you able to visualize example background images from the 16-bit and 32-bit runs? I might be able to assist if you’re not sure how to go about doing that - let me know.
–Harris
Hi @hsnyder
I tried to visualize the background.mrc for both 32-bit and 16-bit of the same micrograph using eman2 project manager. I can successfully view 32-bit, but when I try to view 16-bit, project manager crashes with following errors.
32-bit
16-bit
$ Traceback (most recent call last):
File “/opt/apps/EMAN2/2.31/lib/python2.7/site-packages/eman2_gui/embrowser.py”, line 3347, in
self.wbutmisc[-1].clicked[bool].connect(lambda x, v = i*2+j :self.buttonMisc(v))
File “/opt/apps/EMAN2/2.31/lib/python2.7/site-packages/eman2_gui/embrowser.py”, line 3613, in buttonMisc
self.curactions[num]2 # This calls the action method
File “/opt/apps/EMAN2/2.31/lib/python2.7/site-packages/eman2_gui/embrowser.py”, line 542, in show2dSingleNew
self.show2dSingle(brws, new=True)
File “/opt/apps/EMAN2/2.31/lib/python2.7/site-packages/eman2_gui/embrowser.py”, line 466, in show2dSingle
else : data = EMData(self.path)
File “/opt/apps/EMAN2/2.31/lib/python2.7/site-packages/EMAN2db.py”, line 386, in db_emd_init
self.__initc(*parms)
RuntimeError: UnexpectedBehaviorException at /home/eman2/anaconda2/conda-bld/eman2_1569858012581/work/libEM/em
util.cpp:923: ‘The mode size was 0?’ caught
[1]+ Aborted (core dumped) e2projectmanager.py
Thanks,
Joon
In case anyone else encouters this problem, the issue was that in 16bit mode, the background estimate mrc was full of NaNs. I think this dataset has a very high average background level, and it caused the 16 bit float to overflow. We can do something to address this in the future, but in the meantime, a workaround is to use 32 bit