Cryosparc live fails on all newly incoming exposures

CryoSPARC Live
1

My standalone installation of Cryosparc live (4.6.0) works perfectly fine, except for newly incoming exposures which get tagged as “Failed”.

When I clear and restart the session, the very same, and prevously failing, exposures process flawlessly. However all newly incoming exposures again fail.

Error message:

Traceback (most recent call last):

  • File “cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py”, line 381, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.rtp_worker*
  • File “cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py”, line 450, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.process_movie*
  • File “cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py”, line 483, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.do_check*
  • File “cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py”, line 172, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.RTPExposureCache.cache_read*
  • File “cryosparc_master/cryosparc_compute/blobio/prefetch.py”, line 58, in cryosparc_master.cryosparc_compute.blobio.prefetch.Prefetch.get*
  • File “/home/cryosparc/cryosparc_worker/cryosparc_compute/ioengine/cmdbuf.py”, line 125, in await_async_file_read*
  • iocb.wait()*
  • File “/home/cryosparc/cryosparc_worker/cryosparc_compute/ioengine/cmdbuf.py”, line 82, in wait*
  • raise IOError(‘\n\n’.join(errs))*
    OSError: I/O error, tif_mem_open line 1112: Input/output error
    Ran out of memory

I/O request details:

  • filename: /home/cryosparc/data/Clemens_20241022_SJ13_1/eers/FoilHole_23216673_Data_23198911_11_20241022_222028_EER.eer*
  • data type: 0x10*
  • frames: [0:-1]*
  • eer upsample factor: 1*
  • eer number of fractions: 40*
2

Please can you post the contents of the icli_out.txt file created with these commands:

cd $(mktemp -d) # create and enter a temporary directory
pwd # record the directory's path for file retrieval
cryosparcm icli | tee icli_out.txt # enter the interactive CLI
project_uid = 'P99' # replace with actual CryoSPARC project id
session_uid = 'S19' # replace with actual Live session id
list(db.jobs.find({'job_type': 'rtp_worker', 'project_uid': project_uid, 'params_spec.session_uid.value': session_uid}, {'_id': 0, 'version': 1, 'instance_information': 1, 'status': 1, 'project_uid': 1, 'uid': 1, 'started_at': 1}))
exit()
3

Thanks a lot for your reply. Here comes the output of the script:

Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0]
Type ‘copyright’, ‘credits’ or ‘license’ for more information
IPython 8.26.0 – An enhanced Interactive Python. Type ‘?’ for help.

connecting to wbbc106:61002 …
cli, rtp, db, gfs and tools ready to use

Update: It seems that actually about 100 of 3000 total exposures have processed correctly, the remaining 2900 are tagged as “Failed”.

4

The code was not intended to be run as a script; the commands

project_uid = 'P99' # replace with actual CryoSPARC project id
session_uid = 'S19' # replace with actual Live session id
list(db.jobs.find({'job_type': 'rtp_worker', 'project_uid': project_uid, 'params_spec.session_uid.value': session_uid}, {'_id': 0, 'version': 1, 'instance_information': 1, 'status': 1, 'project_uid': 1, 'uid': 1, 'started_at': 1}))
exit()

need to be run at the icli prompt, and their outputs recorded.

5

OK, I now entered the commands manually into the icli interface:

In [1]: project_uid = ‘P1’

In [2]: session_uid = ‘S1’

In [3]: list(db.jobs.find({‘job_type’: ‘rtp_worker’, ‘project_uid’: project_uid,
…: ‘params_spec.session_uid.value’: session_uid}, {‘_id’: 0, ‘version’: 1,
…: ‘instance_information’: 1, ‘status’: 1, ‘project_uid’: 1, ‘uid’: 1, ‘st
…: arted_at’: 1}))
…:
Out[3]:
[{‘project_uid’: ‘P1’,
‘uid’: ‘J11’,
‘instance_information’: {},
‘started_at’: None,
‘status’: ‘completed’,
‘version’: ‘v4.6.0’},
{‘project_uid’: ‘P1’,
‘uid’: ‘J10’,
‘instance_information’: {},
‘started_at’: None,
‘status’: ‘completed’,
‘version’: ‘v4.6.0’},
{‘project_uid’: ‘P1’,
‘uid’: ‘J9’,
‘instance_information’: {},
‘started_at’: None,
‘status’: ‘completed’,
‘version’: ‘v4.6.0’},
{‘project_uid’: ‘P1’,
‘uid’: ‘J8’,
‘instance_information’: {},
‘started_at’: None,
‘status’: ‘completed’,
‘version’: ‘v4.6.0’},
{‘project_uid’: ‘P1’,
‘uid’: ‘J7’,
‘instance_information’: {},
‘started_at’: None,
‘status’: ‘completed’,
‘version’: ‘v4.6.0’},
{‘project_uid’: ‘P1’,
‘uid’: ‘J6’,
‘instance_information’: {},
‘started_at’: None,
‘status’: ‘completed’,
‘version’: ‘v4.6.0’},
{‘project_uid’: ‘P1’,
‘uid’: ‘J13’,
‘instance_information’: {},
‘started_at’: None,
‘status’: ‘killed’,
‘version’: ‘v4.6.0’},
{‘project_uid’: ‘P1’,
‘uid’: ‘J12’,
‘instance_information’: {},
‘started_at’: None,
‘status’: ‘killed’,
‘version’: ‘v4.6.0’},
{‘project_uid’: ‘P1’,
‘uid’: ‘J5’,
‘instance_information’: {},
‘started_at’: None,
‘status’: ‘killed’,
‘version’: ‘v4.6.0’},
{‘project_uid’: ‘P1’,
‘uid’: ‘J4’,
‘instance_information’: {},
‘started_at’: None,
‘status’: ‘killed’,
‘version’: ‘v4.6.0’},
{‘project_uid’: ‘P1’,
‘uid’: ‘J3’,
‘instance_information’: {},
‘started_at’: None,
‘status’: ‘killed’,
‘version’: ‘v4.6.0’},
{‘project_uid’: ‘P1’,
‘uid’: ‘J2’,
‘instance_information’: {},
‘started_at’: None,
‘status’: ‘killed’,
‘version’: ‘v4.6.0’},
{‘project_uid’: ‘P1’,
‘uid’: ‘J15’,
‘instance_information’: {‘platform_node’: ‘wbbc106’,
‘platform_release’: ‘6.8.0-48-generic’,
‘platform_version’: ‘#48-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 14:04:52 UTC 2024’,
‘platform_architecture’: ‘x86_64’,
‘cpu_model’: ‘Intel(R) Xeon(R) Gold 5318Y CPU @ 2.10GHz’,
‘physical_cores’: 48,
‘total_memory’: ‘1007.37GB’,
‘available_memory’: ‘992.78GB’,
‘used_memory’: ‘8.77GB’,
‘ofd_soft_limit’: 1024,
‘ofd_hard_limit’: 1048576,
‘driver_version’: ‘12.2’,
‘CUDA_version’: ‘11.8’,
‘gpu_info’: [{‘id’: 0,
‘name’: ‘NVIDIA GeForce RTX 4090’,
‘mem’: 25393692672,
‘pcie’: ‘0000:17:00’},
{‘id’: 1,
‘name’: ‘NVIDIA GeForce RTX 4090’,
‘mem’: 25393692672,
‘pcie’: ‘0000:31:00’},
{‘id’: 2,
‘name’: ‘NVIDIA GeForce RTX 4090’,
‘mem’: 25393692672,
‘pcie’: ‘0000:b1:00’},
{‘id’: 3,
‘name’: ‘NVIDIA GeForce RTX 4090’,
‘mem’: 25393692672,
‘pcie’: ‘0000:ca:00’}]},
‘started_at’: datetime.datetime(2024, 11, 11, 9, 1, 31, 236000),
‘status’: ‘running’,
‘version’: ‘v4.6.0’},
{‘project_uid’: ‘P1’,
‘uid’: ‘J14’,
‘instance_information’: {‘platform_node’: ‘wbbc106’,
‘platform_release’: ‘6.8.0-48-generic’,
‘platform_version’: ‘#48-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 14:04:52 UTC 2024’,
‘platform_architecture’: ‘x86_64’,
‘cpu_model’: ‘Intel(R) Xeon(R) Gold 5318Y CPU @ 2.10GHz’,
‘physical_cores’: 48,
‘total_memory’: ‘1007.37GB’,
‘available_memory’: ‘993.16GB’,
‘used_memory’: ‘8.38GB’,
‘ofd_soft_limit’: 1024,
‘ofd_hard_limit’: 1048576,
‘driver_version’: ‘12.2’,
‘CUDA_version’: ‘11.8’,
‘gpu_info’: [{‘id’: 0,
‘name’: ‘NVIDIA GeForce RTX 4090’,
‘mem’: 25393692672,
‘pcie’: ‘0000:17:00’},
{‘id’: 1,
‘name’: ‘NVIDIA GeForce RTX 4090’,
‘mem’: 25393692672,
‘pcie’: ‘0000:31:00’},
{‘id’: 2,
‘name’: ‘NVIDIA GeForce RTX 4090’,
‘mem’: 25393692672,
‘pcie’: ‘0000:b1:00’},
{‘id’: 3,
‘name’: ‘NVIDIA GeForce RTX 4090’,
‘mem’: 25393692672,
‘pcie’: ‘0000:ca:00’}]},
‘started_at’: datetime.datetime(2024, 11, 11, 9, 1, 28, 947000),
‘status’: ‘running’,
‘version’: ‘v4.6.0’}]

In [4]: