Value error with patch CTF estimation and manual picker

Hi
I’ve been experiencing value error after updating v2.14.2
CTFFIND4 has no problem but when running patch CTF estimation results in the following error

[CPU: 900.1 MB]  Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/cryosparc_user/cryosparc2-40000/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 153, in thread_work
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/ctf_estimation/run.py", line 108, in cryosparc2_compute.jobs.ctf_estimation.run.run.ctfworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/ctf_estimation/run.py", line 111, in cryosparc2_compute.jobs.ctf_estimation.run.run.ctfworker.process
  File "cryosparc2_compute/blobio/mrc.py", line 115, in read_mrc
    data = read_mrc_data(file_obj, header, start_page, end_page, out)
  File "cryosparc2_compute/blobio/mrc.py", line 78, in read_mrc_data
    data = n.fromfile(file_obj, dtype=dtype, count= num_pages * ny * nx).reshape(num_pages, ny, nx)
ValueError: total size of new array must be unchanged

[CPU: 390.8 MB]  Outputting partial results now...

[CPU: 389.6 MB]  Traceback (most recent call last):
  File "cryosparc2_worker/cryosparc2_compute/run.py", line 82, in cryosparc2_compute.run.main
  File "cryosparc2_worker/cryosparc2_compute/jobs/ctf_estimation/run.py", line 258, in cryosparc2_compute.jobs.ctf_estimation.run.run
AssertionError: Child process with PID 5556 has terminated unexpectedly!

Also running manual picker after CTFFIND4, results in the similar following error

[CPU: 245.8 MB]  Traceback (most recent call last):
  File "cryosparc2_master/cryosparc2_compute/run.py", line 82, in cryosparc2_compute.run.main
  File "cryosparc2_compute/jobs/manual_picker/run.py", line 734, in run_manual_picker
    _, mrcdata = mrc.read_mrc(path_abs, return_psize=True)
  File "cryosparc2_compute/blobio/mrc.py", line 115, in read_mrc
    data = read_mrc_data(file_obj, header, start_page, end_page, out)
  File "cryosparc2_compute/blobio/mrc.py", line 78, in read_mrc_data
    data = n.fromfile(file_obj, dtype=dtype, count= num_pages * ny * nx).reshape(num_pages, ny, nx)
ValueError: total size of new array must be unchanged

I would appreciate it if anyone could let me know what to do.
Thanks

Hi @KSJ,

What are the original sizes of your images? Did you perform motion correction in cryoSPARC? If you did, what size did the Import Movies job say the movies are?

Original image size is (3838, 3870)

It is EMPIAR-10059 data and it is already motion corrected.

So I did ‘Import micrographs’ - ‘CTFFIND4’

If I run ‘Patch CTF’ at this point, I got an error.

And after CTFFIND4, when running ‘manual picking’, an error occurs again.

Hey @KSJ,

Thanks for getting back with that info, it really helps.
I was able to import EMPIAR-10059 and run Patch Motion Correction successfully. Is it possible if you can tell me what parameters you used for the import micrographs job and if you changed any parameters in the Patch CTF Correction job? Also, where did you save them to? Do you use a special file system, or is it just saved on a hard drive thats mounted locally to your workstation?

I used this parameter for import micrographs.
pixel size : 1.22
Accelerating Voltage : 300
Spherical Abberation : 2
Total exposure dose : 41

and I run ‘Patch CTF estimation’ with default parameters.

My all cryosparc data are saved in mounted hard drive.

I tried again from the start (import micrographs) and still get the same error :cry:

Hey @KSJ,

Is it possible if you can run this code for me, and attach the file it creates?

First, get together 2 paths

  1. dir_path: The path to the directory where you’ve saved the micrographs from EMPIAR 10059
  2. output_path: The path to the file where the stats will be written to

Then, in a shell, run cryosparcm icli, which will open up an interactive python shell. You can then copy and paste this code, with the two variables modified

import os
from cryosparc2_compute.blobio import mrc
dir_path = '/EMPIAR/10059/data/micrographs'
output_path = '/u/cryosparcuser/header_stats.txt'
for path in os.listdir(dir_path):
    filepath = os.path.join(dir_path, path)
    with open(output_path, 'a') as file_out:
        with open(filepath, 'rb') as file_obj:
            header = mrc.read_mrc_header(file_obj)
            file_out.write(filepath + ' : ')
            file_out.write(str(header) + ', ')
            file_out.write("File size: {}".format(os.path.getsize(filepath)))
            file_out.write('\n')

Thank you very much for your help!

Thank you @stephan
Is output_path the path to the file where the results are stored?
I put output_path with
/media/Data/ksj/cryosparc/nanodisc/P4
and an error occured
[Errno 21] Is a directory: ‘/media/Data/ksj/cryosparc/nanodisc/P4’

Can you tell me again what output_path is?
Sorry for bothering you :cry:

Hey @KSJ,

Not a problem, the output path has to be a text file.
For example /media/Data/ksj/cryosparc/nanodisc/P4/header_stats.txt, similar to my example paths

@stephan
Thank you for your kinid explanation

But I don’t know which text file to set as output_path.
I try to search stats.txt file but I didn’t find any.
Do I have to make temporary text file?

Hey @KSJ,

The file you set in the output_path variable will be created when the script run. So for example, when I ran the code I posted above where the output_path variable was /u/cryosparcuser/header_stats.txt, header_stats.txt didn’t exist. It was created in the /u/cryosparcuser directory when I ran the code snippet. I hope that makes sense!

Thank you very much for your help @stephan !

I ran that code and it made this file

/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0002_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0001_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0003_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0004_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0005_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0006_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0007_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0008_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0009_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0061_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0062_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0063_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0064_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0065_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0066_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0067_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0068_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0069_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0070_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0071_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0072_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0073_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0074_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0075_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0076_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0077_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0078_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0079_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0080_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0081_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0082_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0083_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0084_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0085_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0086_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0087_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072
/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0088_2x_dfpt.mrc : {'origin': array([ 0.,  0.,  0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 56957072

Hey @KSJ,

Thank you for running this!
Is this the entire file? It’s only reporting about 88 files, while the 10059 should have 1200. Is it possible that the text got truncated when you copy+pasted? Maybe you can try attaching the .txt file itself.

Oh sorry @stephan you are right it was truncated.

As you say, I need to attach a file because of the character limit.
But it seems that only the photo file can be uploaded to the post.
Can you tell me where should I do it?
Thank you!

Hey @KSJ,

Sorry, I didn’t realize. Please email me at sarulthasan@structura.bio, thanks!

@stephan
I mailed you!
Please check your e-mail :blush:

Hey @KSJ,

Thanks for sending this over!
So I went over the file, and theres one line in particular that may explain the ValueError:

/media/Data/ksj/EMPIAR10059/ftp.ebi.ac.uk/empiar/world_availability/10059/data/micrographs/stack_0340_2x_dfpt.mrc : {'origin': array([ 0., 0., 0.], dtype=float32), 'zlen': 0.0, 'xlen': 0.0, 'datatype': 2, 'nsymbt': 128, 'nx': 3838, 'ny': 3710, 'nz': 1, 'ylen': 0.0}, File size: 45374872

You can calculate the correct file size of the image based on the the information printed out by the script:
filesize = 4 * (nx * ny * nz*) + nsymbt + 1024
Therefore, the file size should be 56957072 bytes. File stack_0340_2x_dfpt.mrc seems to be corrupted. When the Patch Motion job tries to read the file based on the information from the header, it complains since the .mrc file on disk doesn’t have the right number of bytes.

Is it possible if you can try re-downloading stack_0340_2x_dfpt.mrc and try again?

Hi Everyone,

One potential cause for this error is a bug in cryoSPARC, and it has been fixed as of v2.15.0: