Hi,everyone. I’m trying to use non-uniform refinement on my data set. However,the job fails with the error “ValueError: cannot reshape array of size 0 into shape (320,320)” I tried to import the binned particle during the prelimiary refinement.Then I tried to re-extract the unbinned particles to do non-uniform refinement job but with the error.Thank you for your help.
2 Likes
Hi @Linlin,
Could you post the full traceback that you saw in the streamlog?
Also, did you have a mask connected? And was the input volume that you had connected at the same or different box size from the particles?
Thank you for your reply.Here is the traceback:
Traceback (most recent call last):
File "/home/lllab/cryoSparc/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py", line 1722, in run_with_except_hook
run_old(*args, **kw)
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 129, in cryosparc_compute.engine.cuda_core.GPUThread.run
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 130, in cryosparc_compute.engine.cuda_core.GPUThread.run
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 997, in cryosparc_compute.engine.engine.process.work
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 87, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File "/home/lllab/cryoSparc/cryosparc2_worker/cryosparc_compute/particles.py", line 108, in get_original_real_data
return self.blob.view().copy()
File "/home/lllab/cryoSparc/cryosparc2_worker/cryosparc_compute/blobio/mrc.py", line 124, in view
return self.get()
File "/home/lllab/cryoSparc/cryosparc2_worker/cryosparc_compute/blobio/mrc.py", line 121, in get
data = n.fromfile(file_obj, dtype=self.dtype, count= n.prod(self.shape)).reshape(self.shape)
ValueError: cannot reshape array of size 0 into shape (320,320)
And,I do not have a mask connected.The input model has the same box size(from previous homogeneous refinement) as the particles(320).
I’m having the same issue. I believe it’s actually a caching bug.
I’ve been able to get past it by deleting the cache, and restarting the job. I also have another job, which uses the same inputs as the first, but with additional particles. After caching the additional particles, this second job fails with the reshape error, however clearing (or cloning) and restarting the job always leads to the particles being cached again before resulting in the same error. This time, clearing the cache manually didn’t help.
Seems like there is an edge case that leads to the error, or else it’s a result of some kind of hardware failure, but I have no other evidence for that.
Update
I imported these two sets of particles from a single .star file and started a new job, with a clean cache. Although the number of particles and stack files are the same, previously the caching step reported
SSD cache : cache successfuly requested to check 9872 files.
but now reports
SSD cache : cache successfuly requested to check 5097 files.
The two sets were two different classes from Class3D in Relion. I think they both have some of the same stacks referenced (though different individual particles of course). The new job appears to be proceeding normally into refinement.
TL;DR is apparently an issue when there are two input particle sets with some shared stack files (though no shared particles).
2 Likes
I believe I am experiencing the same issue during Local Refinement:
Traceback (most recent call last):
File "/home/hiter/software/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py", line 1722, in run_with_except_hook
run_old(*args, **kw)
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 129, in cryosparc_compute.engine.cuda_core.GPUThread.run
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 130, in cryosparc_compute.engine.cuda_core.GPUThread.run
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 997, in cryosparc_compute.engine.engine.process.work
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 87, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File "/home/hiter/software/cryosparc/cryosparc2_worker/cryosparc_compute/particles.py", line 108, in get_original_real_data
return self.blob.view().copy()
File "/home/hiter/software/cryosparc/cryosparc2_worker/cryosparc_compute/blobio/mrc.py", line 124, in view
return self.get()
File "/home/hiter/software/cryosparc/cryosparc2_worker/cryosparc_compute/blobio/mrc.py", line 121, in get
data = n.fromfile(file_obj, dtype=self.dtype, count= n.prod(self.shape)).reshape(self.shape)
ValueError: cannot reshape array of size 0 into shape (384,384)
These are two independently imported data sets from Relion, combined at NU-Refinement (New) and Symmetry Expanded prior to Local Refinement.
1 Like
Update: I’m more confident it’s the same error reported here and by others as a caching issue. The cached files are reported to have changed size and are re-uploaded. Pre-caching the files using the “Cache particle stack” job will not fix the problem. The job will run if I split the particles into the independently imported stacks for two Local Refinements, or if I disable “Cache to SSD” in the Local Refinement job. Looking forward to a fix. Thanks!
Hi @DanielAsarnow, @navid-paknejad @Navid, @Linlin,
Thanks a lot for reporting this. We are investigating a solution and this is a high priority issue. We believe we have reproduced the issue ourselves, but any more details or information (or new occurrences) that you can report would be super helpful.
2 Likes
I believe I just ran into the same issue in 2d classification.
[CPU: 7.40 GB] Traceback (most recent call last):
File “/home/mhanson/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 1722, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 129, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 130, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 997, in cryosparc_compute.engine.engine.process.work
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 87, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “/home/mhanson/cryosparc/cryosparc_worker/cryosparc_compute/particles.py”, line 108, in get_original_real_data
return self.blob.view().copy()
File “/home/mhanson/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 124, in view
return self.get()
File “/home/mhanson/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py”, line 121, in get
data = n.fromfile(file_obj, dtype=self.dtype, count= n.prod(self.shape)).reshape(self.shape)
ValueError: cannot reshape array of size 0 into shape (240,240)
1 Like
Hi everyone, thanks for reporting this and sending over the traceback information. We’ve found a bug with the caching system in v3 that appears to cause this issue. We’ll be releasing the fix in a few days.
As a work around until then, I suggest re-running the job with SSD caching disabled - let me know this works or causes any further issues.
1 Like
Okay, the fix is now out in cryoSPARC v3.1, let me know here if you run into further issues.
Nick
Hi,
I am getting this error “cannot reshape array of size 0 into shape (200,200)” in 2D classification when I use a combination of micrographs that are in separate folders but have identical names. Giving each micrograph a unique name resolves the issue.
I am doing NU refinement (NEW) using v3.1.0 and still got this problem:
Traceback (most recent call last):
File "/app/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 1726, in run_with_except_hook
run_old(*args, **kw)
File "/app/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "cryosparc_worker/cryosparc_compute/engine/newengine.py", line 2109, in cryosparc_compute.engine.newengine.process.work
File "cryosparc_worker/cryosparc_compute/engine/newengine.py", line 2127, in cryosparc_compute.engine.newengine.process.work
File "cryosparc_worker/cryosparc_compute/engine/newengine.py", line 394, in cryosparc_compute.engine.newengine.EngineThread.read_image_data
File "/app/cryosparc_worker/cryosparc_compute/particles.py", line 113, in get_original_real_data
return self.blob.view().copy()
File "/app/cryosparc_worker/cryosparc_compute/blobio/mrc.py", line 124, in view
return self.get()
File "/app/cryosparc_worker/cryosparc_compute/blobio/mrc.py", line 121, in get
data = n.fromfile(file_obj, dtype=self.dtype, count= n.prod(self.shape)).reshape(self.shape)
ValueError: cannot reshape array of size 0 into shape (384,384)