Local Motion Correction with symlink

joonpark · November 28, 2023, 10:52pm

Hi,

I am getting below error when running Local Motion Correction job. These movies were imported into cryoSPARC via symbolic link.

Thank you in advance for any suggestions and inputs.

wtempel · November 28, 2023, 11:40pm

@joonpark Please can you

post the error message as text
ensure that the actual eer file to which the symbolic link inside J325/imported/ points is available under the path referenced by the symbolic link

joonpark · November 29, 2023, 12:03am

Hi @wtempel

I checked to make sure that the movie.mrc that is shown in the error message is present under J325/import as well as in the directory where it symbolic links are (except for the random numbers in prefix).

Here is the text:

[CPU:  541.5 MB  Avail: 140.06 GB]
Error occurred while processing J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_local.py", line 237, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_local.run_local_motion_correction_multi.motionworker.process
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/motioncorrection.py", line 132, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.motioncorrection.motion_correction
  File "/mnt/ape2/cryosparc/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/motioncorrection/mic_utils.py", line 117, in replace_hot_mov
    mov[:,hotmask] = n.random.poisson(avg_dose_per_frame, size=(N_Z, numhot)).astype(mov.dtype)
  File "mtrand.pyx", line 3592, in numpy.random.mtrand.RandomState.poisson
  File "_common.pyx", line 865, in numpy.random._common.disc
  File "_common.pyx", line 412, in numpy.random._common.check_constraint
ValueError: lam < 0 or lam is NaN

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/ape2/cryosparc/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 61, in exec
    return self.process(item)
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_local.py", line 223, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_local.run_local_motion_correction_multi.motionworker.process
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_local.py", line 235, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_local.run_local_motion_correction_multi.motionworker.process
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_local.py", line 271, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_local.run_local_motion_correction_multi.motionworker.process
AttributeError: 'NoneType' object has no attribute 'dtype'

Marking as incomplete and continuing...

[CPU:  455.7 MB  Avail: 140.81 GB]
Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 95, in cryosparc_master.cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_local.py", line 374, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_local.run_local_motion_correction_multi
AssertionError: First exposure failed with exception: <cryosparc_compute.jobs.pipeline.PipelineExceptionResult object at 0x7fbad4c8ffd0>

Thank you

wtempel · December 4, 2023, 5:29pm

Please can you post additional information:

Does the same error occur on movies other than 017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc. To test this, you might need to create a particle subset that excludes that movie, which can be achieved with a Curate Exposures job:
- connect “micrographs” from an upstream motion correction or CTF estimation job
- connect particles
- reject exposures with with low indices
  
  image1016×776 33.3 KB
- retry Local Motion Correction with the previous step’s Particles accepted
Does the same error occur with other datasets from the same microscope?

Outputs of the commands

project_dir=/absolute/path/to/project_directory # substitute actual project directory
stat ${project_dir}/J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
stat -f ${project_dir}/J325/imported/
stat $(readlink -e ${project_dir}/J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc)
stat -f $(readlink -e ${project_dir}/J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc)

joonpark · December 7, 2023, 7:38pm

Hi,

Same error still occurs on different movies. I have tried taking out just 017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc. I also tried a few different numbers of micrograph using the index threshold in Curate Exposure job.
This does happen to one other dataset collected a day before. However, this does not happen to any other dataset from the same microscope collected at different time.

stat /data2/cryosparc/P204/J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc

  File: /data2/cryosparc/P204/J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc -> /mnt/ywyin2/jopark/polgamma_20220611_ssDNA-ddC_Ca2+_20220611_111641/symlink/FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
  Size: 146             Blocks: 0          IO Block: 4096   symbolic link
Device: 851h/2129d      Inode: 122406568455  Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (59900/cryosparc)   Gid: (59900/cryosparc)
Access: 2023-11-28 16:54:23.171081028 -0600
Modify: 2023-11-17 13:41:46.966374831 -0600
Change: 2023-11-17 13:41:46.966374831 -0600
 Birth: 2023-11-17 13:41:46.966374831 -0600

stat -f /data2/cryosparc/P204/J325/imported

  File: "/data2/cryosparc/P204/J325/imported"
    ID: 85100000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 46875543803 Free: 5186932593 Available: 5186932593
Inodes: Total: 3750085184 Free: 3744032184

readlink -e /data2/cryosparc/P204/J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
/data2/jopark/polgamma_20220611_ssDNA-ddC_Ca2+_20220611_111641/Images-Disc1/GridSquare_9849803/Data/FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc

stat /data2/jopark/polgamma_20220611_ssDNA-ddC_Ca2+_20220611_111641/Images-Disc1/GridSquare_9849803/Data/FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc

  File: /data2/jopark/polgamma_20220611_ssDNA-ddC_Ca2+_20220611_111641/Images-Disc1/GridSquare_9849803/Data/FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
  Size: 1308623872      Blocks: 2555912    IO Block: 4096   regular file
Device: 851h/2129d      Inode: 278512438321  Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1805/  jopark)   Gid: ( 1797/   ywyin)
Access: 2023-11-28 16:54:23.226082570 -0600
Modify: 2022-06-11 12:10:00.000000000 -0500
Change: 2023-11-16 18:12:15.769709501 -0600
 Birth: 2023-11-16 18:11:00.494587428 -0600

stat -f /data2/jopark/polgamma_20220611_ssDNA-ddC_Ca2+_20220611_111641/Images-Disc1/GridSquare_9849803/Data/FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc

  File: "/data2/jopark/polgamma_20220611_ssDNA-ddC_Ca2+_20220611_111641/Images-Disc1/GridSquare_9849803/Data/FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc"
    ID: 85100000000 Namelen: 255     Type: xfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 46875543803 Free: 5186932084 Available: 5186932084
Inodes: Total: 3750085184 Free: 3744032184

wtempel · December 7, 2023, 10:46pm

Do this dataset and

share the same file storage “arrangement” whereas

has another storage “arrangement”?
I am trying to imagine what may lead the command

revealing a link target

whereas

joonpark:

readlink -e /data2/cryosparc/P204/J325/imported/017784668293326064237_FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc
/data2/jopark/polgamma_20220611_ssDNA-ddC_Ca2+_20220611_111641/Images-Disc1/GridSquare_9849803/Data/FoilHole_10949138_Data_10947228_10947230_20220611_120852_Fractions.mrc

points to a different path.
Is /mnt/ywyin2/ a link to /data2/?

joonpark · December 7, 2023, 11:12pm

They both share the same file storage along with all the other datasets. All of my datasets are stored at /data2/jopark/. You are correct. /mnt/ywyin2 is a link to /data2.

wtempel · December 19, 2023, 4:18pm

@joonpark Thanks for providing these details. Given

the most plausible explanation to us is some sort of disruption of data collection or data management during those two days.

joonpark · January 19, 2024, 5:26pm

Hi @wtempel

Just a follow up. I tested a few different things since it was confusing that the corrupted movie files would have no problem with all the other data processing (i.e. patch motion correction, ctf estimation, particle picking, 3D refinement, etc.) but only with local motion correction and reference based motion correction. It turned out that when I enable “save in 16-bit” option on the patch motion correction as well as particle extraction for quick 2D classification, it gives me above issue. When I disable those option, local motion and reference based motion correction jobs all run without errors. Could this be a universal bug or just something specific to my dataset?

Thank you,
Joon

hsnyder · February 13, 2024, 9:35pm

Hi @joonpark. Good catch re: 16-bit! My first thought is that if that’s the case, there’s probably something unusual about the background estimate saved by patch motion correction. In the project folder, these files would be located like motioncorrection/[something]_background.mrc. Are you able to visualize example background images from the 16-bit and 32-bit runs? I might be able to assist if you’re not sure how to go about doing that - let me know.

–Harris

joonpark · February 14, 2024, 5:59pm

Hi @hsnyder

I tried to visualize the background.mrc for both 32-bit and 16-bit of the same micrograph using eman2 project manager. I can successfully view 32-bit, but when I try to view 16-bit, project manager crashes with following errors.

32-bit

16-bit
$ Traceback (most recent call last):
File “/opt/apps/EMAN2/2.31/lib/python2.7/site-packages/eman2_gui/embrowser.py”, line 3347, in
self.wbutmisc[-1].clicked[bool].connect(lambda x, v = i*2+j :self.buttonMisc(v))
File “/opt/apps/EMAN2/2.31/lib/python2.7/site-packages/eman2_gui/embrowser.py”, line 3613, in buttonMisc
self.curactions[num]2 # This calls the action method
File “/opt/apps/EMAN2/2.31/lib/python2.7/site-packages/eman2_gui/embrowser.py”, line 542, in show2dSingleNew
self.show2dSingle(brws, new=True)
File “/opt/apps/EMAN2/2.31/lib/python2.7/site-packages/eman2_gui/embrowser.py”, line 466, in show2dSingle
else : data = EMData(self.path)
File “/opt/apps/EMAN2/2.31/lib/python2.7/site-packages/EMAN2db.py”, line 386, in db_emd_init
self.__initc(*parms)
RuntimeError: UnexpectedBehaviorException at /home/eman2/anaconda2/conda-bld/eman2_1569858012581/work/libEM/em
util.cpp:923: ‘The mode size was 0?’ caught

[1]+ Aborted (core dumped) e2projectmanager.py

Thanks,
Joon

hsnyder · March 4, 2024, 5:42pm

In case anyone else encouters this problem, the issue was that in 16bit mode, the background estimate mrc was full of NaNs. I think this dataset has a very high average background level, and it caused the 16 bit float to overflow. We can do something to address this in the future, but in the meantime, a workaround is to use 32 bit