I/O error -- CryoSPARC Live v4.6.1

CryoSPARC Live
1

I had an issue yesterday after upgrading to the latest version of CryoSPARC (v4.6.1, previously on v4.6.0) with Live Motion Correction. Importing the same movies/gain outside of CryoSPARC Live and running motion correction did not have any issues. The issue was resolved after downgrading to v4.6.0.

Here is a snapshot from the eventlog from one of the Live Preprocessing Workers. Please let me know if I can provide any additional details to help troubleshoot.

[Fri, 15 Nov 2024 01:24:27 GMT] [CPU RAM used: 1704 MB] PROCESSING EXPOSURE        845     ===========================================================
[Fri, 15 Nov 2024 01:24:27 GMT] [CPU RAM used: 1704 MB] Reading exposure /fsx/data-folder/FoilHole_14414631_Data_14428504_24_20241114_134811_Fractions.tiff and initializing .cs file...
[Fri, 15 Nov 2024 01:24:27 GMT] [CPU RAM used: 1704 MB] Asynchronously read file (requested header) in 1.404s
[Fri, 15 Nov 2024 01:24:27 GMT] [CPU RAM used: 1711 MB] Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py", line 381, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.rtp_worker
  File "cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py", line 450, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.process_movie
  File "cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py", line 514, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.do_check
  File "cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py", line 149, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read_gainref_header
  File "cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py", line 139, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.read_header_sync
  File "cryosparc_master/cryosparc_compute/blobio/prefetch.py", line 58, in cryosparc_master.cryosparc_compute.blobio.prefetch.Prefetch.get
  File "/shared/cryosparc/cryosparc_worker/cryosparc_compute/ioengine/cmdbuf.py", line 130, in await_async_file_read
    iocb.wait()
  File "/shared/cryosparc/cryosparc_worker/cryosparc_compute/ioengine/cmdbuf.py", line 87, in wait
    raise IOError('\n\n'.join(errs))
OSError: I/O error, tif_readmic (2)  line 1113: Invalid argument
Please report this as a bug.

I/O request details:
	filename:  /fsx/data-folder/20241112_135716_EER_GainReference.gain
	data type: 0x0
	frames:    [0:0]
	eer upsample factor: 0
	eer number of fractions: 0
2

@emgreen Thanks for reporting this issue. A fix was just released in CryoSPARC v4.6.2.

3

Thank you for the quick fix @wtempel! I just wanted to confirm that CryoSPARC Live in v4.6.2 seems to be running without issues on our system now.

2 Likes
4

Hello,
After updating to 6.4.2 I am still getting
[CPU: 1.01 GB]
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py”, line 381, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.rtp_worker
File “cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py”, line 450, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.process_movie
File “cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py”, line 483, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.do_check
File “cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py”, line 172, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read
File “cryosparc_master/cryosparc_compute/blobio/prefetch.py”, line 58, in cryosparc_master.cryosparc_compute.blobio.prefetch.Prefetch.get
File “/login/home/cryosparc_user/software/cryosparc3/cryosparc_worker/cryosparc_compute/ioengine/cmdbuf.py”, line 130, in await_async_file_read
iocb.wait()
File “/login/home/cryosparc_user/software/cryosparc3/cryosparc_worker/cryosparc_compute/ioengine/cmdbuf.py”, line 87, in wait
raise IOError(‘\n\n’.join(errs))
OSError: I/O error, TIFFReadDirectory line 1099: Input/output error
The file is probably corrupt. If this is a movie, try deleting it and re-importing the movie set. If this is a particle stack, try the ‘check for corrupt particles’ job (if corrupt particles are found, they will be excluded from the job’s output).

I/O request details:
filename: /data03/npplab/20241126CW_MSKCC_G2/tifs/R59_4_0002.tif
data type: 0x10
frames: [0:-1]
eer upsample factor: 2
eer number of fractions: 40

[CPU: 1.01 GB]
WARNING: Caught exception while trying to purge exposure cache: .get() called more than once on a prefetch object.

[CPU: 1.01 GB]
Waiting 5 seconds before continuing…

[CPU: 77.9 MB]
[libtiff error] TIFFFetchDirectory: Can not read TIFF directory count

[CPU: 77.9 MB]
[libtiff error] TIFFReadDirectory: Failed to read directory at offset 168617810

[CPU: 1.01 GB]
No new exposure received since 361 seconds ago. Searching again in 10 seconds…

5

Hi @Yehuda,

Try running tiffinfo /data03/npplab/20241126CW_MSKCC_G2/tifs/R59_4_0002.tif. That may tell you if the file is actually corrupt. If the file is bad, it’s probably best to delete it and continue processing. If the file is fine, and it’s just CryoSPARC that’s having trouble with it, then please let us know.

Also, if you’re using CryoSPARC live in real time (i.e. the data is being created while the session is running), sometimes there can be issues where CryoSPARC tries to read an image that hasn’t fully been written yet, and simply processing the file later will work. We take some steps to avoid that, but certain network filesystem configurations, in particular, can occasionally do mischief.

Harris