Libtiff error: Using code not in table

[rbs_sci]

Hi CryoSPARC team,

I’ve had two TIFF based datasets recently throw the same type of error up:

[libtiff error] /cryosparc_blobio_56433_56456: Using code not yet in table

One dataset I can’t share, the other I was using as a sanity test; it’s EMPIAR 11285, the first 250 micrographs.

Nothing else seems affected: micrographs which throw this error exist, output looks OK, contain usable particles, and RBMC fine as well but the red shriek on relevant jobs warning about an error is a little disconcerting.

This has happened on three different systems now (Epyc with ECC RAM + A5000s, Xeon with ECC RAM + A5000s and Ryzen 3900X with 3200MHz DDR4 + GTX1080Tis) although it doesn’t always report the blobio part…

No errors in dmesg.

Can anyone shed any light on this?

Thanks,
R

edit: both datasets are super res, one K2, one K3. Fourier cropping or not, low memory mode or not appear to make no difference (although I need LMM+Fourier cropping on the 1080Tis for Patch Motion to succeed…)

[rposert]

Hi @rbs_sci! I’ll look into this for you.

Could you give a bit more detail on when/where you see these errors? I downloaded the first 300 movies from EMPIAR 11285 and am getting normal log messages during Patch Motion Correction.

[rbs_sci]

Cheers, @rposert! I’ll check and get back to you.

[rbs_sci]

On EMPIAR 11285, on a re-run of the full dataset, it triggered on exposure stack_00031, but if I re-run just that exposure it doesn’t happen.

As I said, doesn’t appear to cause any problems. I’ll keep an eye open for more examples.

[olibclarke]

We see the same thing in EMPIAR 10828 - see link to movie that throws the same error here:

Sporadic warning during motion correction

Hi, I see this error occasionally during motion correction using v4.4: [libtiff error] /home/user/processing/cryosparc_projects/empiar/CS-empiar/J218/imported/011387528610215138820_200721_dLIT_QF_krios_00109.tiff: Using code not yet in table It doesn’t seem to impact the result (all micrographs are in output), but if you want to reproduce, it happens with EMPIAR-10828. If I import the single problematic movie, it gives the same error during import. You can find an example movie that gives th…

Also K3 super res gain corrected movies

[rposert]

Hi @olibclarke and @rbs_sci —

Thanks @olibclarke for uploading your bad micrograph to Dropbox. I wonder if you could do me a favor and re-download just that micrograph from EMPIAR and see if the warning persists.

As I said in the other thread, with your micrograph from Dropbox I am able to reproduce that warning during Import Movies and Patch Motion Correction. However, if I download that same movie from EMPIAR and import it, I do not get the warning. Additionally, the files’ MD5 sums are different:

$ md5sum *.tiff
3752a72ff9f7db39d60e790dbf2c23c1  200721_dLIT_QF_krios_00109.tiff
378230847fa172b4f5e79f493c6ffe96  bad-mic-from-oli.tiff

If something is happening to the TIFFs in transmission, it may explain why @rbs_sci observed this problem from an EMPIAR-11285 dataset and we did not.

[olibclarke]

Can confirm your observation that something must be going on in transmission - I redownloaded it using the same command (wget) and now it has the correct MD5sum and imports with no warnings.

The weird thing is, it is in the middle of the dataset - it’s not like it was the last file to be downloaded with the original wget command - so I’m not sure quite what could be going wrong…

[rbs_sci]

I know the Japan EMPIAR mirror was down for a while recently as they suffered from issues - multi-drive failure IIRC.

If it’s a transmission issue there isn’t much we can do except re-download the flagged mics.

I use FTP to get datasets from EMPIAR - it’s routinely a lot faster and simpler. Do you do the same, @olibclarke? As much as EMPIAR push Aspera Connect, I’ve had extremely poor prior experiences with it…

[olibclarke]

Yep I use FTP too (post must be at least 20 characters)

[rbs_sci]

If I feel like wasting a few hours over the weekend I’ll see if I can get Aspera to play nicely, but as it happens fairly infrequently I feel like it probably will be a waste of time.

[rposert]

Hello again @olibclarke and @rbs_sci. I just want to update you on our findings regarding this error message.

First (and most importantly), I want to let you know that although this error does not cause the movie import to fail, the data in the movie is corrupted and should not be trusted.

We are currently looking into a way to fail when this type of error is experienced. In the meantime, if you observe this libtiff error we strongly recommend you re-download or at least exclude that movie(s).

It may also be worth reporting this problem to EMPIAR (perhaps using their email on this page?) when/if you observe it again, as it is not clear how this corruption is happening. Once loaded into CryoSPARC’s memory, the TIFFs only differ in a band of pixels in the middle of one of the middle frames of the movie.

Thanks again to both of you for reporting this problem!

[rbs_sci]

I just had this happen again with EMPIAR 10948 (first 59 movies for a training course).

If you get a chance, @rposert, would you check it out yourself to double check that I’m not going crazy? Three calls of the error but all mics look OK to me…

edit: RBMC later threw an error explicitly on 201109GroEL_1.5mgml_0xECH_96_000.tif three times, but I’m not sure if that’s the same micrograph as the errors during Patch Motion.

[rposert]

Hi @rbs_sci — did you see the error during import or motion correction? I was able to import all of 10948 with no error.

[rbs_sci]

Error in motion correction. In patch motion there was no indication which movie was defective, but RBMC complained about 96.

Thanks for testing; I’ll re-download 10948 (again!) and see if hashes differ…