Restack particles I/O error

Troubleshooting
1

I am trying to restack some particle sets in v4.7.0, but the jobs keep failing after running for a while and finishing multiple batches. This happens with different particle sets from different datasets. Te error message is as follows:

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 129, in cryosparc_master.cryosparc_compute.run.main
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/jobs/extract/run.py", line 1090, in run_restack_particles
    pool.map(restack_batch, enumerate(batches))
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/jobs/extract/run.py", line 1076, in restack_batch
    data[idx] = particle.rawdata.view() # reads data
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py", line 145, in view
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py", line 140, in get
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/ioengine/cmdbuf.py", line 148, in sync_file_read
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/ioengine/cmdbuf.py", line 130, in await_async_file_read
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/ioengine/cmdbuf.py", line 87, in wait
OSError: I/O error, iosys_process_cached_fd_reads line 2311: Remote I/O error
The file is probably corrupt. If this is a movie, try deleting it and re-importing the movie set. If this is a particle stack, try the 'check for corrupt particles' job (if corrupt particles are found, they will be excluded from the job's output).

I/O request details:
	filename:  /home/path/xx_fractions_patch_aligned_doseweighted_particles.mrc
	data type: 0x10
	frames:    [19:20]
	eer upsample factor: 2
	eer number of fractions: 40

====== Job process terminated abnormally.

The rows mentioning eer parameters are suspicious, as my micrographs are tiff files from a K3 detector. I have tried the Check for corrupt particles job (“Fail if corrupt particles are detected” turned OFF), but it fails with the same error message. I can run other jobs (Ab initio/refinements) with the problematic particle sets with no issue.
Is this a known issue/user error?

2

To confirm that the indicated `_particles.mrc file is not in fact corrupted, please can you post the outputs of these commands for both

  1. the failing a restack job and
  2. a refinement job to which the same particle inputs (matching job ID and slot names) had been connected and that was run after the failed restack attempt
csprojectid="P99"; csjobid="J199" # replace with actual project and job ids of failed job
cryosparcm cli "get_job('$csprojectid', '$csjobid', 'type', 'version', 'errors_run', 'input_slot_groups', 'params_spec', 'started_at')"
3

These errors are likely related to file I/O issues we are currently having with the instance due to too many open files.