Ref Based Motion Corr: Unrecognized data type - backwards compatibility bug

Motion Correction
1

When running reference based motion correction on a dataset processed with a previous version of cryosparc (not sure which, data was motion corrected on 2023-10-09) I ran into the following error:

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 115, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py”, line 372, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/refmotion.py”, line 785, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.refmotion.mainfn_param_optimize
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/refmotion.py”, line 701, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.refmotion.prepare_datasets
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/refmotion.py”, line 145, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.refmotion.read_gain_and_defects
File “cryosparc_master/cryosparc_compute/blobio/prefetch.py”, line 82, in cryosparc_master.cryosparc_compute.blobio.prefetch.synchronous_native_read
OSError:

IO request details:
Error ocurred (Invalid argument) at line 687 in dtype_lookup

Unrecognized data type.

filename: /mnt/DATA/[redacted]/J531/imported/20231009_K3-20050033GainRef.x1.m1_bin2.mrc
filetype: 0
header_only: 0
idx_start: 0
idx_limit: -1
eer_upsampfactor: 2
eer_numfractions: 40
num_threads: 6
buffer: (nil)
buffer_sz: 0
nx, ny, nz: 0 0 0
dtype: 0
total_time: -1.000000
io_time: 0.000000

This error is caused by the fact that in newer versions of cryosparc, the file:

[…]/J531/imported/20231009_K3-20050033GainRef.x1.m1_bin2.mrc

is a symlink to the actual image (in this case the reference correction, but this holds true also for the movies), whereas in the old version of cryosparc that was used to process the dataset, it is a text file containing the following information.

$> head /mnt/DATA/[redacted]/J531/imported/20231009_K3-20050033GainRef.x1.m1_bin2.mrc
XSym
0095
27c57fcd40b8e695ce6d66190ce2c181
/em-images/2023/[redacted]/20231009_K3-20050033GainRef.x1.m1_bin2.mrc

Temporarily substituting appropriate symlinks for the text files allows reference based motion correction to proceed.

Maybe this issue can be solved by checking if the files in the import folders are symlinks (new version) or text files (old version) and act accordingly.

2

Even in older versions of CryoSPARC, such files are expected to be symbolic links.

could be a “simulated” link as described here that may have been supported on your system previously, but not by your current system configuration.

3

Thanks @wtempel.

That might be it. I remember at a certain point we had to convert the partition to SMB for some reason or another, so the “simulated” link might have happened then. We then went back to NFS.
A historical accident then.

cheers.