Ab-initio was terminated abnormally

Troubleshooting
1

Hi Cryosparc,

Recently I tried to run ab-initio on a fairly large particle stack (~3M particles, box size 192) with multiple classes (I tried 4 and 6). Unfortunately, ab-initio always stopped in the middle of running with no error info (just general error info “Job process terminated abnormally”); sometimes even the whole cryosparc stopped responding. My workstation has a 32 cores CPU, 4 2080Ti, and 256 GB RAM. The version of cryosparc is v3.3.1+220215. Does anyone have any suggestions on this? Thank you so much!

2

Are there additional hints in the job log, which can be displayed with this shell command:
cryosparcm joblog <project_uid> <job_uid>
?

3

Hello,

I am not sure if this is a similar issue but my ab-initio job is also terminating abnormally with 1.2M particles. I am not sure what the error is as I checked in J29 and the file does exist, and I checked for corrupt particles and the job did not find any corrupt particles in the stack. Any guidance on what this error is would be great!

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 96, in cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/abinit/run.py”, line 305, in cryosparc_compute.jobs.abinit.run.run_homo_abinit
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1142, in cryosparc_compute.engine.engine.process
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1143, in cryosparc_compute.engine.engine.process
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1028, in cryosparc_compute.engine.engine.process.work
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 87, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/particles.py”, line 33, in get_original_real_data
return self.blob.view().copy()
File “/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/blobio/mrc.py”, line 143, in view
return self.get()
File “/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/blobio/mrc.py”, line 138, in get
_, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
File “cryosparc_master/cryosparc_compute/blobio/prefetch.py”, line 70, in cryosparc_compute.blobio.prefetch.synchronous_native_read
OSError:

IO request details:
filename: /data/EmilyC/20230807/CS-project/J29/extract/017714230682662797794_20230807_1133_EC_41_3_A006_G001_H062_D001_patch_aligned_doseweighted_particles.mrc
filetype: 0
header_only: 0
idx_start: 15
idx_limit: 16
eer_upsampfactor: 2
eer_numfractions: 40
num_threads: 6
buffer: 0x5619b8f0b6d0
nx, ny, nz: 384 384 1
dtype: 2
total_time: -1.000000
io_time: 0.000000

4

Welcome to the forum @emcanni1.
Please can you post:

  • CryoSPARC version and patch
  • lines in the Event Log immediately preceding the Traceback
  • the output of the command
    stat /data/EmilyC/20230807/CS-project/J29/extract/017714230682662797794_20230807_1133_EC_41_3_A006_G001_H062_D001_patch_aligned_doseweighted_particles.mrc
5

Hello,

The cryoSPARC version and patch are: 4.3.0+230816

Here is the event log:

image
image1228×1000 152 KB

Here is the output of the command:

stat /data/EmilyC/20230807/CS-project/J29/extract/017714230682662797794_20230807_1133_EC_41_3_A006_G001_H062_D001_patch_aligned_doseweighted_particles.mrc
  File: '/data/EmilyC/20230807/CS-project/J29/extract/017714230682662797794_20230807_1133_EC_41_3_A006_G001_H062_D001_patch_aligned_doseweighted_particles.mrc'
  Size: 15336448  	Blocks: 30252      IO Block: 131072 regular file
Device: 2dh/45d	Inode: 1451751     Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1001/  spuser)   Gid: ( 1001/  spuser)
Access: 2023-09-06 00:00:10.486387624 -0400
Modify: 2023-08-24 09:07:33.588160254 -0400
Change: 2023-08-24 09:07:33.588160254 -0400
 Birth: -
6

@emcanni1 In case you retried the job, did it fail at the same mrc file?
Is this a Single Workstation CryoSPARC instance?

v4.3.1 fixes a memory leak that may or may not be linked to the OSError you observed.

7

CryoSPARC v4.4.1 includes a fix for missing details related to IO errors.