Has anyone noticed that cryoSPARC Live v4.0.1 does not process the last movies of the collection? I ran cryoSPARC Live during a collection of 13,000 movies and everything worked well. However, now that the collection is finished, the final 45 movies are not entering the processing stream and are listed as queued. The worker jobs dedicated to preprocessing are still listed as running. I’ve tried pausing and restarting the session to no avail.
Any help much appreciated.
Thanks,
Andrew
Hi @Andrew,
Can you send us a screenshot of what you’re seeing?
Did you happen to modify the Exposure Processing Priority for the session?
Hi @stephan,
Thanks for your prompt reply. When, I noticed the movies were queued long after the collection was finished, I tried changing the Exposure Processing Priority to “Latest” but this did not seem to help.
This instance of cryoSPARC is running on
Operating System: CentOS Linux 7 (Core) CPE OS Name: cpe:/o:centos:centos:7 Kernel: Linux 3.10.0-693.21.1.el7.x86_64 Architecture: x86-64
Thanks a lot for your help,
Andrew
Hi @Andrew,
Thanks for the additional information. Can we take a look at the contents of the Event Logs for J87 and J88?
Can you also paste the output of the following:
cryosparcm icli
db.exposures.find_one({'project_uid': 'P40', 'session_uid': 'S3', 'stage': 'found'})
Here are the event logs.
[2022-10-12 15:17:23.30]
License is valid.
[2022-10-12 15:17:23.31]
Launching job on lane default target wslocgpu6 ...
[2022-10-12 15:17:23.37]
Request to export P40 J87
[2022-10-12 15:17:23.37]
Running job on master node hostname wslocgpu6
[2022-10-12 15:17:23.37]
Requested to dump P40 J87, but job is in launched status, so it cannot be exported
[2022-10-12 15:17:26.51]
[CPU: 84.0 MB]
Job J87 Started
[2022-10-12 15:17:26.53]
[CPU: 84.1 MB]
Master running v4.0.1, worker running v4.0.1
[2022-10-12 15:17:26.55]
[CPU: 84.1 MB]
Working in directory: /data/Andrew/rest-of-path-here/J87
[2022-10-12 15:17:26.57]
[CPU: 84.1 MB]
Running on lane default
[2022-10-12 15:17:26.57]
[CPU: 84.1 MB]
Resources allocated:
[2022-10-12 15:17:26.58]
[CPU: 84.1 MB]
Worker: wslocgpu6
[2022-10-12 15:17:26.60]
[CPU: 84.1 MB]
CPU : [0, 1, 6, 7, 8, 9]
[2022-10-12 15:17:26.61]
[CPU: 84.1 MB]
GPU : [0]
[2022-10-12 15:17:26.61]
[CPU: 84.1 MB]
RAM : [0, 1]
[2022-10-12 15:17:26.62]
[CPU: 84.1 MB]
SSD : False
[2022-10-12 15:17:26.62]
[CPU: 84.1 MB]
--------------------------------------------------------------
[2022-10-12 15:17:26.62]
[CPU: 84.1 MB]
Importing job module for job type rtp_worker...
[2022-10-12 15:17:37.71]
[CPU: 267.8 MB]
Job ready to run
[2022-10-12 15:17:37.71]
[CPU: 267.8 MB]
***************************************************************
[2022-10-12 15:17:38.25]
[CPU: 268.3 MB]
No new exposure received since 294 seconds ago. Searching again in 10 seconds...
[2022-10-12 15:22:43.97]
[CPU: 268.4 MB]
Signal to Stop!
[2022-10-12 15:22:43.97]
[CPU: 268.4 MB]
--------------------------------------------------------------
[2022-10-12 15:22:44.00]
[CPU: 268.4 MB]
Compiling job outputs...
[2022-10-12 15:22:44.01]
[CPU: 268.4 MB]
Updating job size...
[2022-10-12 15:22:44.02]
[CPU: 268.4 MB]
Exporting job and creating csg files...
[2022-10-12 15:22:44.06]
[CPU: 268.4 MB]
***************************************************************
[2022-10-12 15:22:44.07]
[CPU: 268.4 MB]
Job complete. Total time 306.36s
[2022-10-12 15:17:24.14]
License is valid.
[2022-10-12 15:17:24.15]
Launching job on lane default target wslocgpu6 ...
[2022-10-12 15:17:24.18]
Running job on master node hostname wslocgpu6
[2022-10-12 15:17:24.26]
Request to export P40 J88
[2022-10-12 15:17:24.26]
Requested to dump P40 J88, but job is in launched status, so it cannot be exported
[2022-10-12 15:17:26.53]
[CPU: 84.0 MB]
Job J88 Started
[2022-10-12 15:17:26.54]
[CPU: 84.1 MB]
Master running v4.0.1, worker running v4.0.1
[2022-10-12 15:17:26.55]
[CPU: 84.1 MB]
Working in directory: /data/Andrew/rest-of-path/J88
[2022-10-12 15:17:26.57]
[CPU: 84.1 MB]
Running on lane default
[2022-10-12 15:17:26.57]
[CPU: 84.1 MB]
Resources allocated:
[2022-10-12 15:17:26.58]
[CPU: 84.1 MB]
Worker: wslocgpu6
[2022-10-12 15:17:26.60]
[CPU: 84.1 MB]
CPU : [10, 11, 12, 13, 14, 15]
[2022-10-12 15:17:26.62]
[CPU: 84.1 MB]
GPU : [2]
[2022-10-12 15:17:26.62]
[CPU: 84.1 MB]
RAM : [2, 5]
[2022-10-12 15:17:26.63]
[CPU: 84.1 MB]
SSD : False
[2022-10-12 15:17:26.63]
[CPU: 84.1 MB]
--------------------------------------------------------------
[2022-10-12 15:17:26.64]
[CPU: 84.1 MB]
Importing job module for job type rtp_worker...
[2022-10-12 15:17:37.71]
[CPU: 267.9 MB]
Job ready to run
[2022-10-12 15:17:37.71]
[CPU: 267.9 MB]
***************************************************************
[2022-10-12 15:17:38.25]
[CPU: 268.3 MB]
No new exposure received since 294 seconds ago. Searching again in 10 seconds...
[2022-10-12 15:22:43.97]
[CPU: 268.4 MB]
Signal to Stop!
[2022-10-12 15:22:44.00]
[CPU: 268.4 MB]
--------------------------------------------------------------
[2022-10-12 15:22:44.00]
[CPU: 268.4 MB]
Compiling job outputs...
[2022-10-12 15:22:44.01]
[CPU: 268.4 MB]
Updating job size...
[2022-10-12 15:22:44.02]
[CPU: 268.4 MB]
Exporting job and creating csg files...
[2022-10-12 15:22:44.05]
[CPU: 268.4 MB]
***************************************************************
[2022-10-12 15:22:44.06]
[CPU: 268.4 MB]
Job complete. Total time 306.36s
And here is the output of cryosparcm icli
db.exposures.find_one({'project_uid': 'P40', 'session_uid': 'S3', 'stage': 'found'}):
(base) [exx@wslocgpu6 exx]# cryosparcm icli
Python 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:53)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.33.0 -- An enhanced Interactive Python. Type '?' for help.
connecting to wslocgpu6:39002 ...
cli, rtp, db, gfs and tools ready to use
In [1]: db.exposures.find_one({'project_uid': 'P40', 'session_uid': 'S3', 'stage': 'found'})
Out[1]:
{'_id': ObjectId('6344359ca699542dd30f7116'),
'uid': 622,
'project_uid': 'P40',
'session_uid': 'S3',
'abs_file_path': '/mnt/path/GridSquare_10362044/Data/FoilHole_10383042_Data_10374193_10374195_20221010_135410_fractions.tiff',
'discovered_at': 1665414556.0310528,
'picker_type': None,
'size': 594542592,
'deleted': False,
'parameter_version': 4,
'stage': 'found',
'fail_count': 2,
'failed': True,
'fail_reason': 'Traceback (most recent call last):\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 371, in cryosparc_compute.jobs.rtp_workers.run.rtp_worker\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 434, in cryosparc_compute.jobs.rtp_workers.run.process_movie\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 467, in cryosparc_compute.jobs.rtp_workers.run.do_check\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 170, in cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read\n File "cryosparc_worker/cryosparc_compute/blobio/prefetch.py", line 48, in cryosparc_compute.blobio.prefetch.Prefetch.get\nRuntimeError: [libtiff] TIFFFetchDirectory: Can not read TIFF directory count\n[libtiff] TIFFReadDirectory: Failed to read directory at offset 608977542\nError ocurred (Input/output error) at line 749 in TIFFReadDirectory\nThe file is probably corrupt. If this is a movie, try deleting it and re-importing the movie set. If this is a particle stack, try the \'check for corrupt particles\' job (if corrupt particles are found, they will be excluded from the job\'s output).\n\nIO request details:\nfilename: /mnt/path/GridSquare_10362044/Data/FoilHole_10383042_Data_10374193_10374195_20221010_135410_fractions.tiff\nfiletype: 1\nheader_only: 0\nidx_start: 0\nidx_limit: -1\neer_upsampfactor: 2\neer_numfractions: 40\nnum_threads: 6\nbuffer: (nil)\nnx, ny, nz: 0 0 0\ndtype: 0\ntotal_time: -1.000000\n\n',
'in_progress': False,
'manual_reject': False,
'threshold_reject': False,
'test': False,
'priority': 0,
'exp_group_id': 1,
'groups': {'exposure': {},
'particle_manual': {},
'particle_blob': {},
'particle_template': {},
'particle_deep': {},
'particle_extracted': {},
'particle_manual_extracted': {}},
'attributes': {'found_at': 1665419514.0055494,
'check_at': 0,
'motion_at': 0,
'thumbs_at': 0,
'ctf_at': 0,
'pick_at': 0,
'extract_at': 0,
'manual_extract_at': 0,
'ready_at': 0,
'total_motion_dist': 0,
'max_intra_frame_motion': 0,
'average_defocus': 0,
'defocus_range': 0,
'astigmatism_angle': 0,
'astigmatism': 0,
'phase_shift': 0,
'ctf_fit_to_A': 0,
'ice_thickness_rel': 0,
'df_tilt_angle': 0,
'total_manual_picks': 0,
'total_blob_picks': 0,
'blob_pick_score_median': 0,
'total_template_picks': 0,
'template_pick_score_median': 0,
'total_extracted_particles': 0,
'total_extracted_particles_manual': 0,
'total_extracted_particles_blob': 0,
'total_extracted_particles_template': 0},
'preview_img_1x': [],
'preview_img_2x': [],
'worker_juid': None}
Thanks a lot for your help,
Andrew
Hey @Andrew,
Nice, getting somewhere. Lets try this one:
db.exposures.find_one({'project_uid': 'P40', 'session_uid': 'S3', 'stage' : { '$ne' : 'ready' }, 'failed' : False, 'fail_count' : { '$lte' : 3 }, 'in_progress' : False,})
Hi @stephan,
That doesn’t appear to give an output.
Cheers,
Andrew
(base) [exx@wslocgpu6 exx]# cryosparcm icli
Python 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:53)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.33.0 -- An enhanced Interactive Python. Type '?' for help.
connecting to wslocgpu6:39002 ...
cli, rtp, db, gfs and tools ready to use
In [1]: db.exposures.find_one({'project_uid': 'P40', 'session_uid': 'S3', 'stage' : { '$ne' : 'ready' }, 'failed' : False, 'fail_count' : { '$lte' : 3 }, 'in_progress' : False,})
In [2]:
Hi @Andrew,
Apologies, can you try this updated query:
db.exposures.find({'project_uid': 'P40', 'session_uid': 'S3', 'stage' : 'found', 'in_progress' : False})
Hi @stephan
No problem. Here’s the output:
In [1]: db.exposures.find({'project_uid': 'P40', 'session_uid': 'S3', 'stage' : 'found', 'in_progress' : False})
Out[1]: <pymongo.cursor.Cursor at 0x7f959c3ef290>
Hey @Andrew,
Can you try this:
list(db.exposures.find({'project_uid': 'P40', 'session_uid': 'S3', 'stage' : 'found', 'in_progress' : False}))
Hi @stephan,
Here is the first part of the output as I ran up against the character limit. Let me know if you need the rest and I can send it to you as text file.
In [8]: list(db.exposures.find({'project_uid': 'P40', 'session_uid': 'S3', 'stage' : 'found', 'in_progress' : False}))
Out[8]:
[{'_id': ObjectId('6344359ca699542dd30f7116'),
'uid': 622,
'project_uid': 'P40',
'session_uid': 'S3',
'abs_file_path': '/mnt/path/GridSquare_10362044/Data/FoilHole_10383042_Data_10374193_10374195_20221010_135410_fractions.tiff',
'discovered_at': 1665414556.0310528,
'picker_type': None,
'size': 594542592,
'deleted': False,
'parameter_version': 4,
'stage': 'found',
'fail_count': 2,
'failed': True,
'fail_reason': 'Traceback (most recent call last):\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 371, in cryosparc_compute.jobs.rtp_workers.run.rtp_worker\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 434, in cryosparc_compute.jobs.rtp_workers.run.process_movie\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 467, in cryosparc_compute.jobs.rtp_workers.run.do_check\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 170, in cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read\n File "cryosparc_worker/cryosparc_compute/blobio/prefetch.py", line 48, in cryosparc_compute.blobio.prefetch.Prefetch.get\nRuntimeError: [libtiff] TIFFFetchDirectory: Can not read TIFF directory count\n[libtiff] TIFFReadDirectory: Failed to read directory at offset 608977542\nError ocurred (Input/output error) at line 749 in TIFFReadDirectory\nThe file is probably corrupt. If this is a movie, try deleting it and re-importing the movie set. If this is a particle stack, try the \'check for corrupt particles\' job (if corrupt particles are found, they will be excluded from the job\'s output).\n\nIO request details:\nfilename: /mnt/path/GridSquare_10362044/Data/FoilHole_10383042_Data_10374193_10374195_20221010_135410_fractions.tiff\nfiletype: 1\nheader_only: 0\nidx_start: 0\nidx_limit: -1\neer_upsampfactor: 2\neer_numfractions: 40\nnum_threads: 6\nbuffer: (nil)\nnx, ny, nz: 0 0 0\ndtype: 0\ntotal_time: -1.000000\n\n',
'in_progress': False,
'manual_reject': False,
'threshold_reject': False,
'test': False,
'priority': 0,
'exp_group_id': 1,
'groups': {'exposure': {},
'particle_manual': {},
'particle_blob': {},
'particle_template': {},
'particle_deep': {},
'particle_extracted': {},
'particle_manual_extracted': {}},
'attributes': {'found_at': 1665419514.0055494,
'check_at': 0,
'motion_at': 0,
'thumbs_at': 0,
'ctf_at': 0,
'pick_at': 0,
'extract_at': 0,
'manual_extract_at': 0,
'ready_at': 0,
'total_motion_dist': 0,
'max_intra_frame_motion': 0,
'average_defocus': 0,
'defocus_range': 0,
'astigmatism_angle': 0,
'astigmatism': 0,
'phase_shift': 0,
'ctf_fit_to_A': 0,
'ice_thickness_rel': 0,
'df_tilt_angle': 0,
'total_manual_picks': 0,
'total_blob_picks': 0,
'blob_pick_score_median': 0,
'total_template_picks': 0,
'template_pick_score_median': 0,
'total_extracted_particles': 0,
'total_extracted_particles_manual': 0,
'total_extracted_particles_blob': 0,
'total_extracted_particles_template': 0},
'preview_img_1x': [],
'preview_img_2x': [],
'worker_juid': None},
{'_id': ObjectId('6344cd6da699542dd30f7efd'),
'uid': 4072,
'project_uid': 'P40',
'session_uid': 'S3',
'abs_file_path': '/mnt/path/GridSquare_10362019/Data/FoilHole_10432285_Data_10374240_10374242_20221011_034924_fractions.tiff',
'discovered_at': 1665453421.2513576,
'picker_type': None,
'size': 550274666,
'deleted': False,
'parameter_version': 4,
'stage': 'found',
'fail_count': 2,
'failed': True,
'fail_reason': 'Traceback (most recent call last):\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 371, in cryosparc_compute.jobs.rtp_workers.run.rtp_worker\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 434, in cryosparc_compute.jobs.rtp_workers.run.process_movie\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 467, in cryosparc_compute.jobs.rtp_workers.run.do_check\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 170, in cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read\n File "cryosparc_worker/cryosparc_compute/blobio/prefetch.py", line 48, in cryosparc_compute.blobio.prefetch.Prefetch.get\nRuntimeError: Error ocurred (Input/output error) at line 666 in sendfile\n\nIO request details:\nfilename: /mnt/path/GridSquare_10362019/Data/FoilHole_10432285_Data_10374240_10374242_20221011_034924_fractions.tiff\nfiletype: 1\nheader_only: 0\nidx_start: 0\nidx_limit: -1\neer_upsampfactor: 2\neer_numfractions: 40\nnum_threads: 6\nbuffer: (nil)\nnx, ny, nz: 0 0 0\ndtype: 0\ntotal_time: -1.000000\n\n',
'in_progress': False,
'manual_reject': False,
'threshold_reject': False,
'test': False,
'priority': 0,
'exp_group_id': 1,
'groups': {'exposure': {},
'particle_manual': {},
'particle_blob': {},
'particle_template': {},
'particle_deep': {},
'particle_extracted': {},
'particle_manual_extracted': {}},
'attributes': {'found_at': 1665453545.815927,
'check_at': 0,
'motion_at': 0,
'thumbs_at': 0,
'ctf_at': 0,
'pick_at': 0,
'extract_at': 0,
'manual_extract_at': 0,
'ready_at': 0,
'total_motion_dist': 0,
'max_intra_frame_motion': 0,
'average_defocus': 0,
'defocus_range': 0,
'astigmatism_angle': 0,
'astigmatism': 0,
'phase_shift': 0,
'ctf_fit_to_A': 0,
'ice_thickness_rel': 0,
'df_tilt_angle': 0,
'total_manual_picks': 0,
'total_blob_picks': 0,
'blob_pick_score_median': 0,
'total_template_picks': 0,
'template_pick_score_median': 0,
'total_extracted_particles': 0,
'total_extracted_particles_manual': 0,
'total_extracted_particles_blob': 0,
'total_extracted_particles_template': 0},
'preview_img_1x': [],
'preview_img_2x': [],
'worker_juid': None},
{'_id': ObjectId('6344ce89a699542dd30f7f16'),
'uid': 4097,
'project_uid': 'P40',
'session_uid': 'S3',
'abs_file_path': '/mnt/path/GridSquare_10362021/Data/FoilHole_10434257_Data_10374250_10374252_20221011_035602_fractions.tiff',
'discovered_at': 1665453705.1439629,
'picker_type': None,
'size': 200278016,
'deleted': False,
'parameter_version': 4,
'stage': 'found',
'fail_count': 2,
'failed': True,
'fail_reason': 'Traceback (most recent call last):\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 371, in cryosparc_compute.jobs.rtp_workers.run.rtp_worker\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 434, in cryosparc_compute.jobs.rtp_workers.run.process_movie\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 467, in cryosparc_compute.jobs.rtp_workers.run.do_check\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 170, in cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read\n File "cryosparc_worker/cryosparc_compute/blobio/prefetch.py", line 48, in cryosparc_compute.blobio.prefetch.Prefetch.get\nRuntimeError: [libtiff] TIFFFetchDirectory: Can not read TIFF directory count\n[libtiff] TIFFReadDirectory: Failed to read directory at offset 213082966\nError ocurred (Input/output error) at line 749 in TIFFReadDirectory\nThe file is probably corrupt. If this is a movie, try deleting it and re-importing the movie set. If this is a particle stack, try the \'check for corrupt particles\' job (if corrupt particles are found, they will be excluded from the job\'s output).\n\nIO request details:\nfilename: /mnt/path/GridSquare_10362021/Data/FoilHole_10434257_Data_10374250_10374252_20221011_035602_fractions.tiff\nfiletype: 1\nheader_only: 0\nidx_start: 0\nidx_limit: -1\neer_upsampfactor: 2\neer_numfractions: 40\nnum_threads: 6\nbuffer: (nil)\nnx, ny, nz: 0 0 0\ndtype: 0\ntotal_time: -1.000000\n\n',
'in_progress': False,
'manual_reject': False,
'threshold_reject': False,
'test': False,
'priority': 0,
'exp_group_id': 1,
'groups': {'exposure': {},
'particle_manual': {},
'particle_blob': {},
'particle_template': {},
'particle_deep': {},
'particle_extracted': {},
'particle_manual_extracted': {}},
'attributes': {'found_at': 1665453781.8223348,
'check_at': 0,
'motion_at': 0,
'thumbs_at': 0,
'ctf_at': 0,
'pick_at': 0,
'extract_at': 0,
'manual_extract_at': 0,
'ready_at': 0,
'total_motion_dist': 0,
'max_intra_frame_motion': 0,
'average_defocus': 0,
'defocus_range': 0,
'astigmatism_angle': 0,
'astigmatism': 0,
'phase_shift': 0,
'ctf_fit_to_A': 0,
'ice_thickness_rel': 0,
'df_tilt_angle': 0,
'total_manual_picks': 0,
'total_blob_picks': 0,
'blob_pick_score_median': 0,
'total_template_picks': 0,
'template_pick_score_median': 0,
'total_extracted_particles': 0,
'total_extracted_particles_manual': 0,
'total_extracted_particles_blob': 0,
'total_extracted_particles_template': 0},
'preview_img_1x': [],
'preview_img_2x': [],
'worker_juid': None},
{'_id': ObjectId('6344d0baa699542dd30f7f48'),
'uid': 4147,
'project_uid': 'P40',
'session_uid': 'S3',
'abs_file_path': '/mnt/path/GridSquare_10362021/Data/FoilHole_10434285_Data_10374240_10374242_20221011_040358_fractions.tiff',
'discovered_at': 1665454266.162165,
'picker_type': None,
'size': 560698018,
'deleted': False,
'parameter_version': 4,
'stage': 'found',
'fail_count': 2,
'failed': True,
'fail_reason': 'Traceback (most recent call last):\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 371, in cryosparc_compute.jobs.rtp_workers.run.rtp_worker\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 434, in cryosparc_compute.jobs.rtp_workers.run.process_movie\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 467, in cryosparc_compute.jobs.rtp_workers.run.do_check\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 170, in cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read\n File "cryosparc_worker/cryosparc_compute/blobio/prefetch.py", line 48, in cryosparc_compute.blobio.prefetch.Prefetch.get\nRuntimeError: Error ocurred (Input/output error) at line 666 in sendfile\n\nIO request details:\nfilename: /mnt/path/GridSquare_10362021/Data/FoilHole_10434285_Data_10374240_10374242_20221011_040358_fractions.tiff\nfiletype: 1\nheader_only: 0\nidx_start: 0\nidx_limit: -1\neer_upsampfactor: 2\neer_numfractions: 40\nnum_threads: 6\nbuffer: (nil)\nnx, ny, nz: 0 0 0\ndtype: 0\ntotal_time: -1.000000\n\n',
'in_progress': False,
'manual_reject': False,
'threshold_reject': False,
'test': False,
'priority': 0,
'exp_group_id': 1,
'groups': {'exposure': {},
'particle_manual': {},
'particle_blob': {},
'particle_template': {},
'particle_deep': {},
'particle_extracted': {},
'particle_manual_extracted': {}},
'attributes': {'found_at': 1665454380.0165384,
'check_at': 0,
'motion_at': 0,
'thumbs_at': 0,
'ctf_at': 0,
'pick_at': 0,
'extract_at': 0,
'manual_extract_at': 0,
'ready_at': 0,
'total_motion_dist': 0,
'max_intra_frame_motion': 0,
'average_defocus': 0,
'defocus_range': 0,
'astigmatism_angle': 0,
'astigmatism': 0,
'phase_shift': 0,
'ctf_fit_to_A': 0,
'ice_thickness_rel': 0,
'df_tilt_angle': 0,
'total_manual_picks': 0,
'total_blob_picks': 0,
'blob_pick_score_median': 0,
'total_template_picks': 0,
'template_pick_score_median': 0,
'total_extracted_particles': 0,
'total_extracted_particles_manual': 0,
'total_extracted_particles_blob': 0,
'total_extracted_particles_template': 0},
'preview_img_1x': [],
'preview_img_2x': [],
'worker_juid': None},
{'_id': ObjectId('6344db99a699542dd30f804b'),
'uid': 4406,
'project_uid': 'P40',
'session_uid': 'S3',
'abs_file_path': '/mnt/path/GridSquare_10362021/Data/FoilHole_10434386_Data_10374234_10374236_20221011_045155_fractions.tiff',
'discovered_at': 1665457049.7583923,
'picker_type': None,
'size': 319815680,
'deleted': False,
'parameter_version': 4,
'stage': 'found',
'fail_count': 2,
'failed': True,
'fail_reason': 'Traceback (most recent call last):\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 371, in cryosparc_compute.jobs.rtp_workers.run.rtp_worker\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 434, in cryosparc_compute.jobs.rtp_workers.run.process_movie\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 467, in cryosparc_compute.jobs.rtp_workers.run.do_check\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 170, in cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read\n File "cryosparc_worker/cryosparc_compute/blobio/prefetch.py", line 48, in cryosparc_compute.blobio.prefetch.Prefetch.get\nRuntimeError: Error ocurred (Input/output error) at line 666 in sendfile\n\nIO request details:\nfilename: /mnt/path/GridSquare_10362021/Data/FoilHole_10434386_Data_10374234_10374236_20221011_045155_fractions.tiff\nfiletype: 1\nheader_only: 0\nidx_start: 0\nidx_limit: -1\neer_upsampfactor: 2\neer_numfractions: 40\nnum_threads: 6\nbuffer: (nil)\nnx, ny, nz: 0 0 0\ndtype: 0\ntotal_time: -1.000000\n\n',
'in_progress': False,
'manual_reject': False,
'threshold_reject': False,
'test': False,
'priority': 0,
'exp_group_id': 1,
'groups': {'exposure': {},
'particle_manual': {},
'particle_blob': {},
'particle_template': {},
'particle_deep': {},
'particle_extracted': {},
'particle_manual_extracted': {}},
'attributes': {'found_at': 1665457161.7803006,
'check_at': 0,
'motion_at': 0,
'thumbs_at': 0,
'ctf_at': 0,
'pick_at': 0,
'extract_at': 0,
'manual_extract_at': 0,
'ready_at': 0,
'total_motion_dist': 0,
'max_intra_frame_motion': 0,
'average_defocus': 0,
'defocus_range': 0,
'astigmatism_angle': 0,
'astigmatism': 0,
'phase_shift': 0,
'ctf_fit_to_A': 0,
'ice_thickness_rel': 0,
'df_tilt_angle': 0,
'total_manual_picks': 0,
'total_blob_picks': 0,
'blob_pick_score_median': 0,
'total_template_picks': 0,
'template_pick_score_median': 0,
'total_extracted_particles': 0,
'total_extracted_particles_manual': 0,
'total_extracted_particles_blob': 0,
'total_extracted_particles_template': 0},
'preview_img_1x': [],
'preview_img_2x': [],
'worker_juid': None},
{'_id': ObjectId('6344df21a699542dd30f806f'),
'uid': 4442,
'project_uid': 'P40',
'session_uid': 'S3',
'abs_file_path': '/mnt/path/GridSquare_10362021/Data/FoilHole_10434354_Data_10374193_10374195_20221011_050654_fractions.tiff',
'discovered_at': 1665457953.2557948,
'picker_type': None,
'size': 253755392,
'deleted': False,
'parameter_version': 4,
'stage': 'found',
'fail_count': 2,
'failed': True,
'fail_reason': 'Traceback (most recent call last):\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 371, in cryosparc_compute.jobs.rtp_workers.run.rtp_worker\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 434, in cryosparc_compute.jobs.rtp_workers.run.process_movie\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 467, in cryosparc_compute.jobs.rtp_workers.run.do_check\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 170, in cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read\n File "cryosparc_worker/cryosparc_compute/blobio/prefetch.py", line 48, in cryosparc_compute.blobio.prefetch.Prefetch.get\nRuntimeError: [libtiff] TIFFFetchDirectory: Can not read TIFF directory count\n[libtiff] TIFFReadDirectory: Failed to read directory at offset 264697786\nError ocurred (Input/output error) at line 749 in TIFFReadDirectory\nThe file is probably corrupt. If this is a movie, try deleting it and re-importing the movie set. If this is a particle stack, try the \'check for corrupt particles\' job (if corrupt particles are found, they will be excluded from the job\'s output).\n\nIO request details:\nfilename: /mnt/path/GridSquare_10362021/Data/FoilHole_10434354_Data_10374193_10374195_20221011_050654_fractions.tiff\nfiletype: 1\nheader_only: 0\nidx_start: 0\nidx_limit: -1\neer_upsampfactor: 2\neer_numfractions: 40\nnum_threads: 6\nbuffer: (nil)\nnx, ny, nz: 0 0 0\ndtype: 0\ntotal_time: -1.000000\n\n',
'in_progress': False,
'manual_reject': False,
'threshold_reject': False,
'test': False,
'priority': 0,
'exp_group_id': 1,
'groups': {'exposure': {},
'particle_manual': {},
'particle_blob': {},
'particle_template': {},
'particle_deep': {},
'particle_extracted': {},
'particle_manual_extracted': {}},
'attributes': {'found_at': 1665458078.9792573,
'check_at': 0,
'motion_at': 0,
'thumbs_at': 0,
'ctf_at': 0,
'pick_at': 0,
'extract_at': 0,
'manual_extract_at': 0,
'ready_at': 0,
'total_motion_dist': 0,
'max_intra_frame_motion': 0,
'average_defocus': 0,
'defocus_range': 0,
'astigmatism_angle': 0,
'astigmatism': 0,
'phase_shift': 0,
'ctf_fit_to_A': 0,
'ice_thickness_rel': 0,
'df_tilt_angle': 0,
'total_manual_picks': 0,
'total_blob_picks': 0,
'blob_pick_score_median': 0,
'total_template_picks': 0,
'template_pick_score_median': 0,
'total_extracted_particles': 0,
'total_extracted_particles_manual': 0,
'total_extracted_particles_blob': 0,
'total_extracted_particles_template': 0},
'preview_img_1x': [],
'preview_img_2x': [],
'worker_juid': None},
{'_id': ObjectId('6344e637a699542dd30f810d'),
'uid': 4600,
'project_uid': 'P40',
'session_uid': 'S3',
'abs_file_path': '/mnt/path/GridSquare_10362011/Data/FoilHole_10436365_Data_10374250_10374252_20221011_053707_fractions.tiff',
'discovered_at': 1665459767.3401365,
'picker_type': None,
'size': 384827392,
'deleted': False,
'parameter_version': 4,
'stage': 'found',
'fail_count': 2,
'failed': True,
'fail_reason': 'Traceback (most recent call last):\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 371, in cryosparc_compute.jobs.rtp_workers.run.rtp_worker\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 434, in cryosparc_compute.jobs.rtp_workers.run.process_movie\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 467, in cryosparc_compute.jobs.rtp_workers.run.do_check\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 170, in cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read\n File "cryosparc_worker/cryosparc_compute/blobio/prefetch.py", line 48, in cryosparc_compute.blobio.prefetch.Prefetch.get\nRuntimeError: Error ocurred (Input/output error) at line 666 in sendfile\n\nIO request details:\nfilename: /mnt/path/GridSquare_10362011/Data/FoilHole_10436365_Data_10374250_10374252_20221011_053707_fractions.tiff\nfiletype: 1\nheader_only: 0\nidx_start: 0\nidx_limit: -1\neer_upsampfactor: 2\neer_numfractions: 40\nnum_threads: 6\nbuffer: (nil)\nnx, ny, nz: 0 0 0\ndtype: 0\ntotal_time: -1.000000\n\n',
'in_progress': False,
'manual_reject': False,
'threshold_reject': False,
'test': False,
'priority': 0,
'exp_group_id': 1,
'groups': {'exposure': {},
'particle_manual': {},
'particle_blob': {},
'particle_template': {},
'particle_deep': {},
'particle_extracted': {},
'particle_manual_extracted': {}},
'attributes': {'found_at': 1665459864.0998833,
'check_at': 0,
'motion_at': 0,
'thumbs_at': 0,
'ctf_at': 0,
'pick_at': 0,
'extract_at': 0,
'manual_extract_at': 0,
'ready_at': 0,
'total_motion_dist': 0,
'max_intra_frame_motion': 0,
'average_defocus': 0,
'defocus_range': 0,
'astigmatism_angle': 0,
'astigmatism': 0,
'phase_shift': 0,
'ctf_fit_to_A': 0,
'ice_thickness_rel': 0,
'df_tilt_angle': 0,
'total_manual_picks': 0,
'total_blob_picks': 0,
'blob_pick_score_median': 0,
'total_template_picks': 0,
'template_pick_score_median': 0,
'total_extracted_particles': 0,
'total_extracted_particles_manual': 0,
'total_extracted_particles_blob': 0,
'total_extracted_particles_template': 0},
'preview_img_1x': [],
'preview_img_2x': [],
'worker_juid': None},
{'_id': ObjectId('6344eaada699542dd30f8177'),
'uid': 4706,
'project_uid': 'P40',
'session_uid': 'S3',
'abs_file_path': '/mnt/path/GridSquare_10362011/Data/FoilHole_10436401_Data_10374234_10374236_20221011_055432_fractions.tiff',
'discovered_at': 1665460909.4672048,
'picker_type': None,
'size': 496720252,
'deleted': False,
'parameter_version': 4,
'stage': 'found',
'fail_count': 2,
'failed': True,
'fail_reason': 'Traceback (most recent call last):\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 371, in cryosparc_compute.jobs.rtp_workers.run.rtp_worker\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 434, in cryosparc_compute.jobs.rtp_workers.run.process_movie\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 467, in cryosparc_compute.jobs.rtp_workers.run.do_check\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 170, in cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read\n File "cryosparc_worker/cryosparc_compute/blobio/prefetch.py", line 48, in cryosparc_compute.blobio.prefetch.Prefetch.get\nRuntimeError: Error ocurred (Input/output error) at line 666 in sendfile\n\nIO request details:\nfilename: /mnt/path/GridSquare_10362011/Data/FoilHole_10436401_Data_10374234_10374236_20221011_055432_fractions.tiff\nfiletype: 1\nheader_only: 0\nidx_start: 0\nidx_limit: -1\neer_upsampfactor: 2\neer_numfractions: 40\nnum_threads: 6\nbuffer: (nil)\nnx, ny, nz: 0 0 0\ndtype: 0\ntotal_time: -1.000000\n\n',
'in_progress': False,
'manual_reject': False,
'threshold_reject': False,
'test': False,
'priority': 0,
'exp_group_id': 1,
'groups': {'exposure': {},
'particle_manual': {},
'particle_blob': {},
'particle_template': {},
'particle_deep': {},
'particle_extracted': {},
'particle_manual_extracted': {}},
'attributes': {'found_at': 1665461021.254879,
'check_at': 0,
'motion_at': 0,
'thumbs_at': 0,
'ctf_at': 0,
'pick_at': 0,
'extract_at': 0,
'manual_extract_at': 0,
'ready_at': 0,
'total_motion_dist': 0,
'max_intra_frame_motion': 0,
'average_defocus': 0,
'defocus_range': 0,
'astigmatism_angle': 0,
'astigmatism': 0,
'phase_shift': 0,
'ctf_fit_to_A': 0,
'ice_thickness_rel': 0,
'df_tilt_angle': 0,
'total_manual_picks': 0,
'total_blob_picks': 0,
'blob_pick_score_median': 0,
'total_template_picks': 0,
'template_pick_score_median': 0,
'total_extracted_particles': 0,
'total_extracted_particles_manual': 0,
'total_extracted_particles_blob': 0,
'total_extracted_particles_template': 0},
'preview_img_1x': [],
'preview_img_2x': [],
'worker_juid': None},
{'_id': ObjectId('6344ebb7a699542dd30f8192'),
'uid': 4733,
'project_uid': 'P40',
'session_uid': 'S3',
'abs_file_path': '/mnt/path/GridSquare_10362011/Data/FoilHole_10436410_Data_10374193_10374195_20221011_060041_fractions.tiff',
'discovered_at': 1665461175.234276,
'picker_type': None,
'size': 261095424,
'deleted': False,
'parameter_version': 4,
'stage': 'found',
'fail_count': 2,
'failed': True,
'fail_reason': 'Traceback (most recent call last):\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 371, in cryosparc_compute.jobs.rtp_workers.run.rtp_worker\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 434, in cryosparc_compute.jobs.rtp_workers.run.process_movie\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 467, in cryosparc_compute.jobs.rtp_workers.run.do_check\n File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 170, in cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read\n File "cryosparc_worker/cryosparc_compute/blobio/prefetch.py", line 48, in cryosparc_compute.blobio.prefetch.Prefetch.get\nRuntimeError: [libtiff] TIFFFetchDirectory: Can not read TIFF directory count\n[libtiff] TIFFReadDirectory: Failed to read directory at offset 272782030\nError ocurred (Input/output error) at line 749 in TIFFReadDirectory\nThe file is probably corrupt. If this is a movie, try deleting it and re-importing the movie set. If this is a particle stack, try the \'check for corrupt particles\' job (if corrupt particles are found, they will be excluded from the job\'s output).\n\nIO request details:\nfilename: /mnt/path/GridSquare_10362011/Data/FoilHole_10436410_Data_10374193_10374195_20221011_060041_fractions.tiff\nfiletype: 1\nheader_only: 0\nidx_start: 0\nidx_limit: -1\neer_upsampfactor: 2\neer_numfractions: 40\nnum_threads: 6\nbuffer: (nil)\nnx, ny, nz: 0 0 0\ndtype: 0\ntotal_time: -1.000000\n\n',
'in_progress': False,
'manual_reject': False,
'threshold_reject': False,
'test': False,
'priority': 0,
'exp_group_id': 1,
'groups': {'exposure': {},
'particle_manual': {},
'particle_blob': {},
'particle_template': {},
'particle_deep': {},
Thanks!
Andrew
Hey @Andrew,
It seems like those “Queued” movies are actually movies that CryoSPARC failed to read, but are mislabeled because they didn’t get to a stage where we consider them to be “processed”. This caused them to be labelled as ‘failed’ and ‘queued’. If you’d like to know exactly which exposures failed and how, you can click on the “Failed” stat under the Details section of the left-hand panel. As for the exposures being double-counted as being “Queued”, this is a bug, so thank you for reporting this!