I have a question about the collected data!
I’ve only collected the .mrc type movies with Falcon 3/4 detectors and, recently, I got the gain-normalized K3 tiff movies with EPU
File size
I found that the Falcon detectors made movies .mrc type with 16-bit/pixel gain-normalized. so for example, 40 frame .mrc data from Falcon has 4096 x 4096 x 40 x 16 bits (~ 1.25 GB, yes I usually get this movie size)
But, I found that the K2 detector makes the .mrc data with 8-bit non-normalized and 32-bit floating points gain-normalized (Movie Compression — RELION documentation)
Is it also the same in K3 gain-normalized (32-bit, mrc) ?
.tiff LZW compression fold on K3
if K2 and K3 store data in the same way, 50 frames gain normalized K3 movie has 4092 x 5760 x 50 x 32 bits (4.39 GB ! in a simple calculation…). And then how .tiff LZW compression make its size ?
I got the K3 gain-normalized 50 frames .tiff movie data with ~560 MB file size. Is it ok…?
.dm4 gain-reference
I requested wtih non-gain normalized .tiff movie, and I got the .tiff movies about ~ 240 MB (yes, non-gain normalized K3 .mrc format has 8-bits and about ~25 % ? compression by .tiff, 4092 x 5760 x 50 x 8 bit x 0.25 ~ 270 MB, almost as expected)
Unfortunately, when I put the gain-reference on cryoSPARC movie import (gain file .dm4 to .mrc conversion by eman2) I got this error
I don’t know the exact detector and collection setting because I haven’t operated Titan Krios myself, but I didn’t request and they didn’t use super-resolultion mode.
Yes, you are right @wtempel
I asked about this problem at another Institute, The Gatan K3 system originally creates a gain reference in super-resolution mode. If I obtained the movies in gain-normalized mode with Gatan/EPU, the gain-reference is processed automatically in proper shrinks.
If in non-gain normalized mode, the gain-reference needs to shrink by eman2 or IMOD, etc
@Jadyn There’s usually 2 gain files made by DM. One has M0 at the end and the other has M1. One is the ‘normal’ gain ref and the other is the SuperRes gain ref. Looks like you might have just gotten the wrong gain ref on import
K3-18290204 Dark Ref. x1.m0.dm4
K3-18290204 Gain Ref. x1.m1.kv[300].dm4
K3-18290204 Pixel Defect Map.m1.kv[300].dm4
K3-18290204 User Pixel Defect Map.dm4
I asked the EM Institute about the two types of files (M0, M1), and they said that the “K3-18290204 Gain Ref. x1.m1.kv[300].dm4” is the normal counting mode gain ref files.
also,
m0 = linear mode gain-ref
m2 = superRes gain-ref
But it didn’t work like you mentioned.
You mean, “K3-18290204 Gain Ref. x1.m0.kv[300].dm4” was also generated automatically with m1 but, I didn’t get the file?
Or,
they should have chosen other gain options when exporting the gain-ref?
The K3 only has two modes of operation–linear and super resolution. That’s the reason the gain reference .dm4 is at 11Kx8K. Outputting frames at physical pixel size involves a secondary operation performed post hoc, e.g. binning or fourier reduction. The gain reference requires a similar treatment before it can be applied to the movie frames.
Where EPU is concerned, there are two ways of writing out K3 frames at physical pixel size. One can collect either in “Counted bin 1” or in “Super resolution bin 2”. In the case of the latter, EPU writes out a gain reference, binned and transformed in the same way as the movie frames. For convenience, this can be used as-is without further manipulation. Alternatively, and in the case of the former, the .dm4 file can be retrieved from GMS and then binned/downsampled by a factor of 2 with e.g. IMOD’s newstack or e2proc2d.
… or you can choose to process the gain-corrected movies and not worry about the gain reference.
K3 MRC output from EPU are either 8-bit unsigned (normalized or non-normalized) or 4-bit unsigned (non-normalized packed). TIFF output are 8-bit unsigned. The Falcon data is 32-bit largely down to the complexity of the “blob” convolution that the detectors perform.
Hi @Jadyn the K3-18290204 Gain Ref. x1.m0 .kv[300].dm4 should have also been written at the same time the gain was collected. They output to the same folder for us and I just grab the correct one for the imaging mode that the data was collected in.
