TIF movie import error with 4-bit values packed in bytes

Import and Export
1

Hello,

I’m trying to import K2 superres movies and it has been throwing me the following error.

[CPU: 795.0 MB]  Traceback (most recent call last):
  File "cryosparc2_master/cryosparc2_compute/run.py", line 82, in cryosparc2_compute.run.main
  File "cryosparc2_compute/jobs/imports/run.py", line 651, in run_import_movies_or_micrographs
    imgdata = tiff.read_tiff(abs_path).sum(axis=0) * gainref
ValueError: operands could not be broadcast together with shapes (7676,3710) (7676,7420)

The movies need to be gain corrected, rotated and flipped according to the person who collected the data. The r/f in the header is also 7 suggesting the same (flip and rotate). The gainref file is in mrc format and is provided. From what I can decipher, the dimensions of the gainref in the Y are double that of the datafiles. The header for the tif movie says

 RO image file on unit   1 : Mix4_00002_Jan28.tif     Size=     230464 K
                This is a TIFF file.
  This file actually has 4-bit values packed in bytes, with 3710 bytes in X
 Number of columns, rows, sections .....    7420    7676      30
 Map mode ..............................    0   (byte)
 Start cols, rows, sects, grid x,y,z ...    0     0     0    7420   7676     30
 Pixel spacing (Angstroms)..............  0.5480     0.5480     0.5480
 Cell angles ...........................   90.000   90.000   90.000
 Fast, medium, slow axes ...............    X    Y    Z
 Origin on x,y,z .......................    0.000       0.000       0.000
 Minimum density .......................   0.0000
 Maximum density .......................   4.0000
 Mean density ..........................   2.0000
 tilt angles (original,current) ........   0.0   0.0   0.0   0.0   0.0   0.0
 Space group,# extra bytes,idtype,lens .        0        0        0        0
 3 Titles :
SerialEMCCD: Dose frac. image, 4 bits packed  r/f 7
  SuperRef_Mix4_00001_Jan28.dm4
  defects_Mix4_00001_Jan28.txt

The most important piece of information I can gather from this is
This file actually has 4-bit values packed in bytes, with 3710 bytes in X
That’s where the mismatch is and I’m not sure how I can correct it in cryosparc. I can correct it outside using the

clip unpack/normalize

command but that takes forever when you have 1000s of movies to process. Is there a way to do it on the fly in cryosparc? I feel like there isn’t or maybe I’m missing a trick here. Help?

2

Hi @vamsee,

Thanks for reporting.
Can you confirm which software was used for the data collection and creation of the tiff files?

Unfortunately right now cryoSPARC does not support 4-bit packed TIFF files. It is possible to unpack these on the fly but would require some changes in the software. So for the time being you will have to unpack/convert them externally unfortunately!

3

SerialEM.
Yeah, I didn’t think there was support for it. I have an alternate way of doing it although it is cumbersome and takes up huge amounts of space. Would be good to know if there is a way of implementing it using the CLI.

Thanks

4

@vamsee could you convert each movie one at a time (or maybe a bunch in parallel at a time) so that you only need enough extra space to store one or a few uncompressed images? Presumably the file size per movie is nearly the same after converting to LZW-compressed-TIFF from the 4-bit version you have now?

5

@apunjani - The file sizes approximately triple from tif to mrc (ex: 136MB to 450MB) and it takes a while to convert >2k images, not to mention the space requirements. The read/write speeds for the converted mrcs obviously takes a big hit as we have to store these files on larger, slower SATA drives (200Mbps) instead of the smaller, much faster, NVMe drives (1.5Gbps). Cryosparc also is not currently able to handle unbinned super res files well (pixel size - 0.5A). So, I include a binning step also while the tif to mrc conversion happens. All of this makes it very cumbersome and takes up almost a full day and then the read/write speeds become a pain. Overall, our processing speeds have been affected by this one conversion step. Hence, my original question about converting on the fly.