Hi @jr10,
Thanks for the additonal report. A few people have experienced this now, and we’re investigating in more detail. I will provide an update once our investigation yields some information.
–Harris
1 Like
Hi @jr10,
Would you be able to post screenshots of your entire workflow (i.e. the job tree) leading up to the job that failed with this messge? Or, alternatively, list the sequence of jobs, like hsmkxyj did ?
Thanks
Hi @hsnyder,
Of course, I can explain the workflow briefly and I’ll also include a screenshot (the tree is a little messy).
The movies were processed in Relion 3.1 for motion correction, and the micrographs (~9000) moved to our processing workstation. Imported mics in 2 batches (2 different grids), patch CTF multi corrected in 2 batches, combined in curate exposures, and a subset of 100 went to blob picker. That picked subset was extracted, 2D classed, 3D ab-initio, and then refined. 20 templates were made for template picking, all micrographs were picked, the inspect picks job reduced the count by ~3/4, and then the output stack from that job has been giving the error. I’ve reextracted a different subset of the same micrographs, and also a different subset using a slightly different selection of micrographs, but none work. Screenshots attached.
1 Like
Hi,
I’m looking forward to the reply. A similar error occurred when I did 2D classification. The particle stacks were obtained from CTF estimation(CTFFIND4), template picker job. I have installed the new patch but “Check for corrupt particles” job ran smoothly and flagged no errors in the particle stacks, too.
Thanks
Blockquote
Traceback (most recent call last):
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 1791, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 1027, in cryosparc_compute.engine.engine.process.work
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 87, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/particles.py”, line 113, in get_original_real_data
return self.blob.view().copy()
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 126, in view
return self.get()
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 121, in get
_, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
File “cryosparc_worker/cryosparc_compute/blobio/prefetch.py”, line 64, in cryosparc_compute.blobio.prefetch.synchronous_native_read
RuntimeError: mrc_readmic (1) 447: Invalid argument
Hi @hsnyder，
After applying the patches release yesterday(Patch 210601, Patch 210601 is available for cryoSPARC v3.2.0), I reran the job. The error message became like this:
It seems that the argument {‘nx’,‘ny’} in this particle stack is invalid. I wonder how this heppened. Is there any way to solve this problem, or quickly identify and remove these invalid particle stacks?
Thanks!
Traceback (most recent call last):
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 1791, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 1027, in cryosparc_compute.engine.engine.process.work
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 87, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/particles.py”, line 15, in get_original_real_data
return self.blob.view().copy()
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 115, in view
return self.get()
File “/home/lliulab/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 110, in get
_, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
File “cryosparc_worker/cryosparc_compute/blobio/prefetch.py”, line 64, in cryosparc_compute.blobio.prefetch.synchronous_native_read
RuntimeError: mrc_readmic (1) 447: Invalid argument
[arguments] /bakup/cryo_cache/instance_login01:39001/projects/P3/J97/extract/FoilHole_28282941_Data_28275789_28275791_20210428_165950_fractions_particles.mrc, 0, 0, 431, 432, 2, 40
Update: issue was solved after re-extracting particles. The number of particles is different after re-extraction, which may be the cause of the error.
Previous post:
I have experienced a similar error for a 3D Variability Display (cluster) job. 3D Variability Display (simple) job works fine. Particles were exported from cryoSPARC live session.
Traceback (most recent call last):
File “cryosparc_worker/cryosparc_compute/run.py”, line 84, in cryosparc_compute.run.main
File “cryosparc_worker/cryosparc_compute/jobs/var3D/run_disp.py”, line 340, in cryosparc_compute.jobs.var3D.run_disp.run
File “cryosparc_worker/cryosparc_compute/jobs/var3D/run_disp.py”, line 255, in cryosparc_compute.jobs.var3D.run_disp.run.do_backproject
File “cryosparc_worker/cryosparc_compute/engine/newengine.py”, line 394, in cryosparc_compute.engine.newengine.EngineThread.read_image_data
File “/home/xxx/apps/cryosparcV2/cryosparc2_worker/cryosparc_compute/particles.py”, line 15, in get_original_real_data
return self.blob.view().copy()
File “/home/xxx/apps/cryosparcV2/cryosparc2_worker/cryosparc_compute/blobio/mrc.py”, line 115, in view
return self.get()
File “/home/xxx/apps/cryosparcV2/cryosparc2_worker/cryosparc_compute/blobio/mrc.py”, line 110, in get
_, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
File “cryosparc_worker/cryosparc_compute/blobio/prefetch.py”, line 64, in cryosparc_compute.blobio.prefetch.synchronous_native_read
RuntimeError: mrc_readmic (1) 447: Invalid argument
[arguments] /ssd/instance_xxx:39001/projects/P39/S2/extract/template/128/198_005_Jun03_16.42.55_patch_aligned_doseweighted_particles.mrc, 0, 0, 615, 616, 2, 40
Replying to myself here to provide an update following patch 210601. I applied the patch and reran particle extraction using the original (J25) particle curation job output. As previous posters have mentioned, my error changed slightly, providing a specific path to an MRC file that is in the workstation cache. However, the re-extraction of the particles didn’t fix the error as some have mentioned, and the check particles job finished without issue, and output a stack of particles the same size as the extraction job/particle curation job.
I have an update on this issue. If micrographs with the same filenames are imported in two separate jobs, and then those micrographs are combined into a single picking or extraction job, this error can result. We are going to put out a patch that will fix the issue in the sense of making the filenames unique even if they’re the same at import time, but in the meantime it should be possible to avoid this issue by not combining micrograph outputs into the same picker job when those micrograph outputs have repeated file names.
Also perhaps worth mentioning: the repeated names are probably a sign that that the actual data is also repeated (i.e. identical files are being imported in multiple separate import jobs and then combined). We don’t recommend this workflow, as cryoSPARC’s reconstruction algorithms assume that each particle is a unique image.
Please let me know if this matches up with what you’re seeing and if the workaround I’m mentioning works for you. This was a surprisingly subtle issue and we’d definitely like to know if there are other workflows that trigger the same symptoms.
Thanks everyone for their help in tracking this down!
Harris
6 Likes
Thank you for your help! I looked at our data and this seems to check out on our end, the two grids were processed by the same Relion process, so the output images were numbered the same (everything was stack_XXXXX.mrc). The other data that we saw this error on also used identical data names (in that case particle stacks that were both particles.mrcs that were outputs from Scipion). It may be worth noting that the particle stack jobs were different imports in different jobs/workspaces, but were under the same project, and I think what triggered the error was having both stacks loaded into the cache. Looking forward to the patch!
2 Likes
Hi @hsnyder, I imported all particles in a job and ran 2D and Select 2D classes, then ran ab-initio reconstruction. It worked, but when I reran ab-initio with the same input, the similar error happened (see below).
[CPU:
352.7 MB] Traceback (most recent call last): File “/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py”,
line 1790, in run_with_except_hook run_old(*args, **kw) File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 132, in
cryosparc_compute.engine.cuda_core.GPUThread.run File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 1027, in cryosparc_compute.engine.engine.process.work File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 87, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/particles.py”, line 113, in get_original_real_data return self.blob.view().copy() File “/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/blobio/mrc.py”, line 126, in view return self.get() File “/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/blobio/mrc.py”,
line 121, in get _, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1) File “cryosparc_worker/cryosparc_compute/blobio/prefetch.py”, line 64, in cryosparc_compute.blobio.prefetch.synchronous_native_read
RuntimeError: mrc_readmic (1) 447: Invalid argument
Hi @hsnyder,
thank you all for this thread.
Like @jr10, also for me re-extracting the particles solved the problem. I had encountered the same issue described by the other users when doing either ab initio or hetero refinement. This happened after merging 2 different “extract from micrographs” jobs located in two different workspaces of the same project. But as I said re-extracting the particles was the solution.
By any chance is there now a way to avoid this problem all together, so to be able to go back from the merged to the unmerged particle stacks?
For the record, this was the original error:
[CPU: 1.61 GB] Traceback (most recent call last):
File “/data/cryosparcuser/CRYOSPARC/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 1811, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 1028, in cryosparc_compute.engine.engine.process.work
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 87, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “/data/cryosparcuser/CRYOSPARC/cryosparc_worker/cryosparc_compute/particles.py”, line 22, in get_original_real_data
return self.blob.view().copy()
File “/data/cryosparcuser/CRYOSPARC/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 127, in view
return self.get()
File “/data/cryosparcuser/CRYOSPARC/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 122, in get
_, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
File “cryosparc_worker/cryosparc_compute/blobio/prefetch.py”, line 67, in cryosparc_compute.blobio.prefetch.synchronous_native_read
RuntimeError: mrc_readmic (1) 454: Invalid argument
[arguments] /data/cryosparcuser/test/P15/S1/extract/template/300/FoilHole_30842771_Data_30843964_30843966_20211002_190141_Fractions_patch_aligned_doseweighted_particles.mrc, 0, 0, 479, 480, 2, 40
I am also encountering the problem mentioned here. I’m using cryosparc v3.3.1, and despite re-extracting the particles in a single job as suggested I keep getting the same error during ab initio reconstruction jobs. The error also seems to be for the same particles in every job. Has anyone faced this issue before and is there any way to fix this?
Traceback (most recent call last):
File “cryosparc_worker/cryosparc_compute/run.py”, line 85, in cryosparc_compute.run.main
File “cryosparc_worker/cryosparc_compute/jobs/abinit/run.py”, line 304, in cryosparc_compute.jobs.abinit.run.run_homo_abinit
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 1150, in cryosparc_compute.engine.engine.process
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 1151, in cryosparc_compute.engine.engine.process
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 1028, in cryosparc_compute.engine.engine.process.work
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 87, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “/ssd0/cryosparc_sep2022/cryosparc_worker/cryosparc_compute/particles.py”, line 22, in get_original_real_data
return self.blob.view().copy()
File “/ssd0/cryosparc_sep2022/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 127, in view
return self.get()
File “/ssd0/cryosparc_sep2022/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 122, in get
_, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
File “cryosparc_worker/cryosparc_compute/blobio/prefetch.py”, line 67, in cryosparc_compute.blobio.prefetch.synchronous_native_read
RuntimeError: mrc_readmic (1) 454: Invalid argument
[arguments] /ssd0/cryosparc_sep2022/ssd_dir/instance_seraphin:39101/projects/P5/J2183/extract/FoilHole_11873160_Data_11857765_11857767_20221209_155826_fractions_patch_aligned_doseweighted_particles.mrc, 0, 0, 89, 90, 2, 40
edit: The re-extraction of particles was to extract the ‘good’ particles without downsampling. An ab-initio job with the same particles from my original downsampled set runs totally fine. A check particles job also does not show any corruption of particles.
Hi @swatib,
If this is the same issue that others experienced, then some of the upstream micrographs have the same names/paths. Is that right? Could you describe the job chain before the ab-initio job?
Thanks,
Harris
Hi @hsnyder,
This is what confused me, there should not be any upstream micrographs which have the same names or paths.
I imported the micrographs through cryosparc live on one workspace, and used blob picker to do an initial reconstruction. I used this to generate templates, and moved to a second workspace. In this workspace 2, I created an exposure sets job, took a small set of micrographs and used template picking to optimize a topaz model. I used a second exposure sets job to split my original micrograph set into stacks of 5000, and ran each set through Topaz extract and then extract micrographs, with downsampling by 4. I did 2D classification and ab initio reconstruction with this set of particles, and all of this worked fine.
I then re-extracted the final particle set (micrographs linked from the exposure sets job and particles from final select 2D jobs) without downsampling and ran ab initio again. Thats when this error popped up. I cleared and re-ran the extract job, this did not resolve the error. I don’t know how duplicate names or paths are possible if the source of everything is the same import job from cryosparc live. And also why this didn’t happen with the downsampled particles, but only with the reextracted ones.
Hi @swatib,
Very interesting. So if I understand correctly, you have
topaz extract (job "A") → extract from mics (1/4 downsample) → 2D class → select 2D → ab initio (success)
Then you also have a chain that uses the particles from select 2D but ultimately fails.
Perhaps try this:
Clone the extract from mics (1/4 downsample) job, change the downsampling (to no downsample), clear the particles input, and connect your final particles from select 2D. Run the result through ab-initio and see what happens.
—Harris
Hi @hsnyder,
So this is slightly complicated by the fact that I ran three topaz extract jobs and three corresponding extract from micrographs jobs, but then combined the particles after 2 rounds of 2D classification. But since all of those worked cloning any one of them and adding the micrograph inputs from the other topaz extract jobs should be fine? I’ll try this and let you know how it goes.
Hi @swatib ,
Thanks for clarifying. Yes I think trying what you proposed is a good next step. It might also be worth trying extracting the three upstream topaz jobs seperately, and combining downstream. I have a strong suspicion that this problem has something to do with combining multiple inputs, though I’m not sure yet exactly what the cause is. Hopefully with some permutations of the experiment, we can nail down the exact step that causes this to happen…
–Harris
Hi @hsnyder,
I tried re-extracting using the cloned job, adding in the micrograph outputs from the other topaz extract jobs and and the particles from select 2D, and the ab initio job was giving the exact same error. Since it kept mentioning the same micrograph in the error, I decided to just do a curate exposures job linking both the particles and the micrographs, and remove that particular micrograph from the dataset. I also checked for duplicate filenames but I didn’t catch anything. This workaround worked, but I have no idea why this issue happened in the first place.
Hi @swatib,
Well, I’m glad you found a solution! If you wouldn’t mind, could you run the following command on the offending particle stack (the one that gets printed out in the error message) and post the output?
xxd -e -g 4 /path/to/offending-particles.mrc | head
Edit: could you also post the output of
stat /path/to/offending-particles.mrc
Thanks,
Harris