Strange error after extraction in 4.5.1

Hi,

I encountered a very strange error that never happened before the update to 4.5.1. After extraction, which finishes successfully, I run 2D classification, and after various amounts of iterations, I get:

 [CPU:   4.16 GB  Avail: 244.36 GB]

Traceback (most recent call last):
  File "/home/cryosparc_user/cryosparc_gpu1/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 2294, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc_master/cryosparc_compute/gpu/gpucore.py", line 134, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
  File "cryosparc_master/cryosparc_compute/gpu/gpucore.py", line 135, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
  File "cryosparc_master/cryosparc_compute/jobs/class2D/newrun.py", line 619, in cryosparc_master.cryosparc_compute.jobs.class2D.newrun.class2D_engine_run.work
  File "cryosparc_master/cryosparc_compute/engine/newengine.py", line 550, in cryosparc_master.cryosparc_compute.engine.newengine.EngineThread.read_image_data
  File "/home/cryosparc_user/cryosparc_gpu1/cryosparc_worker/cryosparc_compute/particles.py", line 34, in get_original_real_data
    data = self.blob.view()
  File "/home/cryosparc_user/cryosparc_gpu1/cryosparc_worker/cryosparc_compute/blobio/mrc.py", line 145, in view
    return self.get()
  File "/home/cryosparc_user/cryosparc_gpu1/cryosparc_worker/cryosparc_compute/blobio/mrc.py", line 140, in get
    _, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
  File "cryosparc_master/cryosparc_compute/blobio/prefetch.py", line 82, in cryosparc_master.cryosparc_compute.blobio.prefetch.synchronous_native_read
OSError: 

IO request details:
Error ocurred (Invalid argument) at line 680 in mrc_readmic (1) 

The requested frame/particle cannot be accessed. The file may be corrupt, or there may be a mismatch between the file and its associated metadata (i.e. cryosparc .cs file).

filename:    /mnt/scratch/cryosparc/instance_10.0.90.83:39001/links/P4-J1327-1715432654/f18c26f6c0c700292a4ee93b729ae367572998e0.mrc
filetype:    0
header_only: 0
idx_start:   83
idx_limit:   84
eer_upsampfactor: 2
eer_numfractions: 40
num_threads: 6
buffer:      (nil)
buffer_sz:   0
nx, ny, nz:  0 0 0
dtype:       0
total_time:  -1.000000
io_time:     0.000000

I also ran the check for the corrupt particles job, but it did not return any errors. I used this dataset before, and everything went well. The short investigation shows that it is rather the problem with extraction than the SSD copy, as the non-SSD run also fails. I would appreciate some ideas on how to fix it.

PS: The file itself in not empty:

-rw-rw-r-- 1 cryosparc_user domain_users 2.6M May 11 00:41 /mnt/scratch/cryosparc/instance_10.0.90.83:39001//store-v2/f1/f18c26f6c0c700292a4ee93b729ae367572998e0

and seems to have correct MRC data in it:

mrcfile.open('/mnt/scratch/cryosparc/instance_10.0.90.83:39001//store-v2/f1/f18c26f6c0c700292a4ee93b729ae367572998e0').data
array([[[-0.545   ,  0.6787  , -0.906   , ...,  0.8677  ,  0.2632  ,
         -0.171   ],
        [ 0.00933 , -0.1229  , -0.429   , ..., -0.6714  , -0.2866  ,
         -0.03214 ],
        [ 0.373   ,  0.2352  ,  0.4573  , ...,  0.9834  , -0.656   ,
          0.2996  ],
        ...,
        [-0.1368  , -0.7676  ,  0.0905  , ...,  0.2947  ,  0.1769  ,
          0.2515  ],
        [-1.277   ,  0.01749 , -0.1985  , ..., -0.1833  ,  0.02    ,
         -0.1722  ],
        [-0.568   , -0.7456  , -0.0936  , ...,  0.1059  ,  0.4585  ,
          0.3828  ]],

       [[-0.1036  , -0.00622 ,  1.105   , ...,  1.425   ,  0.4885  ,
         -0.437   ],
        [ 0.2231  ,  0.5845  , -0.8555  , ..., -0.5776  , -0.6157  ,
          0.06976 ],
        [-0.5815  ,  0.1168  , -0.2161  , ..., -0.02008 ,  0.3801  ,
         -0.1897  ],
        ...,
        [-0.569   , -0.08795 , -0.4204  , ..., -0.4531  , -0.03506 ,
         -0.01866 ],
        [ 0.784   , -0.682   , -0.0538  , ..., -0.433   ,  0.1461  ,
         -0.2615  ],
        [-0.7603  ,  0.11676 , -0.10974 , ...,  0.3801  , -0.1593  ,
          0.4434  ]],

       [[ 0.5557  , -1.033   ,  0.5127  , ...,  0.3188  ,  0.115   ,
         -0.2216  ],
        [-0.5664  , -0.001858,  0.2634  , ...,  0.1373  , -0.1454  ,
          0.7183  ],
        [ 0.0503  ,  0.2732  , -1.192   , ..., -0.3132  , -0.1333  ,
          0.5625  ],
        ...,
        [ 1.128   ,  0.0885  , -0.1423  , ...,  0.3215  , -0.5073  ,
          0.539   ],
        [-0.264   , -0.653   ,  0.3455  , ...,  0.3254  ,  0.869   ,
          0.2727  ],
        [-0.412   ,  0.3765  , -0.10205 , ...,  0.6997  ,  0.0564  ,
          0.77    ]],

       ...,

       [[ 0.0524  ,  0.01288 ,  0.714   , ..., -0.1938  , -0.9307  ,
          0.633   ],
        [ 0.3123  ,  0.2301  , -0.2915  , ...,  0.405   , -0.2133  ,
          0.4438  ],
        [ 0.3967  ,  0.5156  , -0.3208  , ...,  0.1609  ,  0.516   ,
         -0.0394  ],
        ...,
        [ 0.03418 ,  0.245   , -0.2365  , ...,  0.4922  , -0.343   ,
          0.014336],
        [-0.9146  , -0.02989 , -0.01746 , ..., -0.4229  ,  0.1776  ,
          0.1112  ],
        [ 0.361   ,  0.7344  , -0.4753  , ...,  0.462   , -0.3662  ,
          0.5327  ]],

       [[ 1.13    , -0.02214 ,  1.055   , ..., -0.5215  ,  0.442   ,
          0.3564  ],
        [-0.527   ,  0.587   , -0.3025  , ..., -0.5864  , -0.1426  ,
         -0.2998  ],
        [ 0.08997 ,  0.3374  , -0.3252  , ...,  0.407   ,  0.2761  ,
         -0.003357],
        ...,
        [-0.007282, -0.3916  ,  0.2644  , ..., -0.12396 ,  0.2969  ,
          0.3967  ],
        [-0.05515 ,  0.4001  ,  0.1183  , ...,  0.1626  ,  0.321   ,
          0.8013  ],
        [-0.292   ,  0.6875  , -0.00401 , ...,  0.2888  ,  0.16    ,
          0.02405 ]],

       [[ 0.318   , -0.6143  , -0.4092  , ..., -0.307   ,  1.173   ,
          0.1466  ],
        [ 0.01765 ,  0.381   ,  1.221   , ..., -0.1182  , -0.03516 ,
          0.4124  ],
        [-0.0908  , -0.1321  , -0.2008  , ...,  0.3044  ,  0.651   ,
          0.293   ],
        ...,
        [ 0.4333  ,  0.2445  , -0.3245  , ..., -0.4429  , -0.2135  ,
          0.1926  ],
        [-0.4136  ,  0.2413  ,  0.5254  , ...,  0.2388  ,  0.8677  ,
          0.00493 ],
        [-0.4097  ,  0.563   , -0.2517  , ...,  0.345   , -0.5933  ,
          0.05396 ]]], dtype=float16)
>>> mrcfile.open('/mnt/scratch/cryosparc/instance_10.0.90.83:39001//store-v2/f1/f18c26f6c0c700292a4ee93b729ae367572998e0').data.shape
(83, 128, 128)

PS2: Removing the faulty particle mrc file from CS files fixed the problem. Interestingly, the same error happened 3 times (2X restarted extract job using CPU, 1X GPU extraction). Previously, the same micrographs never showed an error after extraction.

Best,
Dawid

Thanks @dzyla for reporting this problem.
Would you be willing to share some information with the CryoSPARC developers:

  1. the faulty particle file
  2. the *.cs file from particle extraction (before removing the reference to the faulty mrc file
  3. available job reports of the particle extraction jobs that produced the faulty mrc file
    Please let us know by email if you can share these files with us using your own (or own institution’s) file sharing platform or whether we should make arrangements for you to upload the files to our server.

I have the particle file and CS files; however, the job report is missing (which is essentially what I have already attached). Could you please provide me with an email address I could use to share the files? Thank you!

@dzyla I sent you a private message two days ago. Did you receive it?