Error while 2D Classification

There is a problem in 2D classification.

Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 84, in cryosparc_compute.run.main
  File "cryosparc_worker/cryosparc_compute/jobs/class2D/run.py", line 159, in cryosparc_compute.jobs.class2D.run.run_class_2D
  File "/Share/app/cryosparc/cryosparc2_worker/cryosparc_compute/particles.py", line 123, in get_prepared_fspace_data
    return fourier.resample_fspace(fourier.fft(self.get_prepared_real_data()), self.dataset.N)
  File "/Share/app/cryosparc/cryosparc2_worker/cryosparc_compute/particles.py", line 118, in get_prepared_real_data
    return (self.dataset.prepare_normalization_factor * self['blob/sign']) * (self.dataset.prepare_real_window * (self.get_original_real_data()))
  File "/Share/app/cryosparc/cryosparc2_worker/cryosparc_compute/particles.py", line 113, in get_original_real_data
    return self.blob.view().copy()
  File "/Share/app/cryosparc/cryosparc2_worker/cryosparc_compute/blobio/mrc.py", line 126, in view
    return self.get()
  File "/Share/app/cryosparc/cryosparc2_worker/cryosparc_compute/blobio/mrc.py", line 121, in get
    _, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
  File "cryosparc_worker/cryosparc_compute/blobio/prefetch.py", line 64, in cryosparc_compute.blobio.prefetch.synchronous_native_read
RuntimeError: mrc_readmic (1) 447: Invalid argument

cryosparc v3.2.0 and inputs from Extract from Micrographs

*.mrc files are generated by MotionCor2 and uploaded by Import Micrographs. Then using Patch CTF estimation (multi), Manually Curate Exposures and Template picker.

Hi @hsmkxyj,

This could be data corruption. Could you run the new “check particles” job (released in last week’s patch), and confirm that the particle files are valid? See the following link for instructions on how to apply patches: https://guide.cryosparc.com/setup-configuration-and-management/software-updates#apply-patches

–Harris

Hello,

I’m also running into this issue with 2D classification error, and this appears to be the only open ticket about this issue. I patched my CS instance and ran the new particle check job, however the error still occurs while using the particle check output stack. The particle check job seems to have ran to completion and flagged no errors in the stack.

Best,
Justas

Hi @jr10,

Thanks for the additonal report. A few people have experienced this now, and we’re investigating in more detail. I will provide an update once our investigation yields some information.

–Harris

1 Like

Hi @jr10,

Would you be able to post screenshots of your entire workflow (i.e. the job tree) leading up to the job that failed with this messge? Or, alternatively, list the sequence of jobs, like hsmkxyj did ?

Thanks

@hsmkxyj,

were you using the CPU or GPU version of extract from micrographs?

Thanks

Hi @hsnyder,

Of course, I can explain the workflow briefly and I’ll also include a screenshot (the tree is a little messy).

The movies were processed in Relion 3.1 for motion correction, and the micrographs (~9000) moved to our processing workstation. Imported mics in 2 batches (2 different grids), patch CTF multi corrected in 2 batches, combined in curate exposures, and a subset of 100 went to blob picker. That picked subset was extracted, 2D classed, 3D ab-initio, and then refined. 20 templates were made for template picking, all micrographs were picked, the inspect picks job reduced the count by ~3/4, and then the output stack from that job has been giving the error. I’ve reextracted a different subset of the same micrographs, and also a different subset using a slightly different selection of micrographs, but none work. Screenshots attached.

1 Like

Hi,
I’m looking forward to the reply. A similar error occurred when I did 2D classification. The particle stacks were obtained from CTF estimation(CTFFIND4), template picker job. I have installed the new patch but “Check for corrupt particles” job ran smoothly and flagged no errors in the particle stacks, too.
Thanks

Blockquote
Traceback (most recent call last):
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 1791, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 1027, in cryosparc_compute.engine.engine.process.work
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 87, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/particles.py”, line 113, in get_original_real_data
return self.blob.view().copy()
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 126, in view
return self.get()
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 121, in get
_, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
File “cryosparc_worker/cryosparc_compute/blobio/prefetch.py”, line 64, in cryosparc_compute.blobio.prefetch.synchronous_native_read
RuntimeError: mrc_readmic (1) 447: Invalid argument

Hi @hsnyder
After applying the patches release yesterday(Patch 210601, Patch 210601 is available for cryoSPARC v3.2.0), I reran the job. The error message became like this:
It seems that the argument {‘nx’,‘ny’} in this particle stack is invalid. I wonder how this heppened. Is there any way to solve this problem, or quickly identify and remove these invalid particle stacks?
Thanks!

Traceback (most recent call last):
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 1791, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 1027, in cryosparc_compute.engine.engine.process.work
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 87, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/particles.py”, line 15, in get_original_real_data
return self.blob.view().copy()
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 115, in view
return self.get()
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 110, in get
_, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
File “cryosparc_worker/cryosparc_compute/blobio/prefetch.py”, line 64, in cryosparc_compute.blobio.prefetch.synchronous_native_read
RuntimeError: mrc_readmic (1) 447: Invalid argument
[arguments] /bakup/cryo_cache/instance_login01:39001/projects/P3/J97/extract/FoilHole_28282941_Data_28275789_28275791_20210428_165950_fractions_particles.mrc, 0, 0, 431, 432, 2, 40

Update: issue was solved after re-extracting particles. The number of particles is different after re-extraction, which may be the cause of the error.

Previous post:
I have experienced a similar error for a 3D Variability Display (cluster) job. 3D Variability Display (simple) job works fine. Particles were exported from cryoSPARC live session.

Traceback (most recent call last):
File “cryosparc_worker/cryosparc_compute/run.py”, line 84, in cryosparc_compute.run.main
File “cryosparc_worker/cryosparc_compute/jobs/var3D/run_disp.py”, line 340, in cryosparc_compute.jobs.var3D.run_disp.run
File “cryosparc_worker/cryosparc_compute/jobs/var3D/run_disp.py”, line 255, in cryosparc_compute.jobs.var3D.run_disp.run.do_backproject
File “cryosparc_worker/cryosparc_compute/engine/newengine.py”, line 394, in cryosparc_compute.engine.newengine.EngineThread.read_image_data
File “/home/xxx/apps/cryosparcV2/cryosparc2_worker/cryosparc_compute/particles.py”, line 15, in get_original_real_data
return self.blob.view().copy()
File “/home/xxx/apps/cryosparcV2/cryosparc2_worker/cryosparc_compute/blobio/mrc.py”, line 115, in view
return self.get()
File “/home/xxx/apps/cryosparcV2/cryosparc2_worker/cryosparc_compute/blobio/mrc.py”, line 110, in get
_, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
File “cryosparc_worker/cryosparc_compute/blobio/prefetch.py”, line 64, in cryosparc_compute.blobio.prefetch.synchronous_native_read
RuntimeError: mrc_readmic (1) 447: Invalid argument
[arguments] /ssd/instance_xxx:39001/projects/P39/S2/extract/template/128/198_005_Jun03_16.42.55_patch_aligned_doseweighted_particles.mrc, 0, 0, 615, 616, 2, 40

Replying to myself here to provide an update following patch 210601. I applied the patch and reran particle extraction using the original (J25) particle curation job output. As previous posters have mentioned, my error changed slightly, providing a specific path to an MRC file that is in the workstation cache. However, the re-extraction of the particles didn’t fix the error as some have mentioned, and the check particles job finished without issue, and output a stack of particles the same size as the extraction job/particle curation job.

Hi @jr10, @bing, @duyx20,

I have an update on this issue. If micrographs with the same filenames are imported in two separate jobs, and then those micrographs are combined into a single picking or extraction job, this error can result. We are going to put out a patch that will fix the issue in the sense of making the filenames unique even if they’re the same at import time, but in the meantime it should be possible to avoid this issue by not combining micrograph outputs into the same picker job when those micrograph outputs have repeated file names.

Also perhaps worth mentioning: the repeated names are probably a sign that that the actual data is also repeated (i.e. identical files are being imported in multiple separate import jobs and then combined). We don’t recommend this workflow, as cryoSPARC’s reconstruction algorithms assume that each particle is a unique image.

Please let me know if this matches up with what you’re seeing and if the workaround I’m mentioning works for you. This was a surprisingly subtle issue and we’d definitely like to know if there are other workflows that trigger the same symptoms.

Thanks everyone for their help in tracking this down!

Harris

5 Likes

Thank you for your help! I looked at our data and this seems to check out on our end, the two grids were processed by the same Relion process, so the output images were numbered the same (everything was stack_XXXXX.mrc). The other data that we saw this error on also used identical data names (in that case particle stacks that were both particles.mrcs that were outputs from Scipion). It may be worth noting that the particle stack jobs were different imports in different jobs/workspaces, but were under the same project, and I think what triggered the error was having both stacks loaded into the cache. Looking forward to the patch!

2 Likes

Hi @hsnyder, I imported all particles in a job and ran 2D and Select 2D classes, then ran ab-initio reconstruction. It worked, but when I reran ab-initio with the same input, the similar error happened (see below).

[CPU:
352.7 MB] Traceback (most recent call last): File “/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py”,
line 1790, in run_with_except_hook run_old(*args, **kw) File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 132, in
cryosparc_compute.engine.cuda_core.GPUThread.run File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 1027, in cryosparc_compute.engine.engine.process.work File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 87, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/particles.py”, line 113, in get_original_real_data return self.blob.view().copy() File “/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/blobio/mrc.py”, line 126, in view return self.get() File “/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/blobio/mrc.py”,
line 121, in get _, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1) File “cryosparc_worker/cryosparc_compute/blobio/prefetch.py”, line 64, in cryosparc_compute.blobio.prefetch.synchronous_native_read
RuntimeError: mrc_readmic (1) 447: Invalid argument