Class 2D - ValueError: total size of new array must be unchanged

[belousoff]

Hey there,

I have been experiencing this error with a 2D refinement.

Sometimes it runs for a few iterations, then fails, sometimes it doesn’t even get to the first iteration.

It is a particle stack of ~1M particles, boxsize 288x288, collected in CTEM mode.

The input was an imported stack from RELION.

Details are below.

----------------------------------------------------------------------------
CryoSPARC System master node installed at
/apps/cryosparc2_master
Current cryoSPARC version: v2.9.0
----------------------------------------------------------------------------

cryosparcm process status:

command_core                     RUNNING   pid 24591, uptime 3 days, 7:05:49
command_proxy                    RUNNING   pid 24629, uptime 3 days, 7:05:45
command_vis                      RUNNING   pid 24614, uptime 3 days, 7:05:47
database                         RUNNING   pid 24510, uptime 3 days, 7:05:51
watchdog_dev                     STOPPED   Not started
webapp                           RUNNING   pid 24638, uptime 3 days, 7:05:43
webapp_dev                       STOPPED   Not started

----------------------------------------------------------------------------

global config variables:

export CRYOSPARC_LICENSE_ID=""
export CRYOSPARC_MASTER_HOSTNAME="LithgowGPU"
export CRYOSPARC_DB_PATH="/apps/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false

> 
export CRYOSPARC_LICENSE_ID=""
export CRYOSPARC_USE_GPU=true
export CRYOSPARC_CUDA_PATH="/usr/local/cuda"
export CRYOSPARC_DEVELOP=false

> 
Cuda 10.1
OS: Linux Mint 18.1 Serena

>
========= sending heartbeat
***************************************************************
Running job  J3  of type  class_2D
Running job on hostname %s LithgowGPU
Allocated Resources :  {u'lane': u'default', u'target': {u'lane': u'default', u'name': u'LithgowGPU', u'title': u'Worker node LithgowGPU', u'resource_slots': {u'GPU': [0, 1, 2, 3], u'RAM': [0, 1, 2, 3, 4, 5, 6, 7], u'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}, u'hostname': u'LithgowGPU', u'worker_bin_path': u'/apps/cryosparc2_worker/bin/cryosparcw', u'cache_path': u'/scratch/sparc', u'cache_quota_mb': None, u'resource_fixed': {u'SSD': True}, u'cache_reserve_mb': 10000, u'type': u'node', u'ssh_str': u'mbelouso@LithgowGPU', u'desc': None}, u'license': True, u'hostname': u'LithgowGPU', u'slots': {u'GPU': [1], u'RAM': [3, 4, 5], u'CPU': [2, 3]}, u'fixed': {u'SSD': True}, u'lane_type': u'node'}
========= sending heartbeat
cryosparc2_compute/sigproc.py:754: RuntimeWarning: invalid value encountered in divide
  frc[k, :copylen] = (AB / n.sqrt(AA*BB))[:copylen]
cryosparc2_compute/sigproc.py:821: RuntimeWarning: invalid value encountered in greater
  crossings = n.where((fsc[:-1] > thresh) * (fsc[1:] < thresh))[0]
cryosparc2_compute/sigproc.py:821: RuntimeWarning: invalid value encountered in less
  crossings = n.where((fsc[:-1] > thresh) * (fsc[1:] < thresh))[0]
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
cryosparc2_compute/util/logsumexp.py:40: RuntimeWarning: divide by zero encountered in log
  return n.log(wa * n.exp(a - vmax) + wb * n.exp(b - vmax) ) + vmax
========= sending heartbeat
========= sending heartbeat
/apps/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/matplotlib/pyplot.py:516: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
  max_open_warning, RuntimeWarning)
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
/apps/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)
/apps/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/numpy/core/_methods.py:70: RuntimeWarning: invalid value encountered in true_divide
  ret = ret.dtype.type(ret / rcount)
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
**custom thread exception hook caught something
**** handle exception rc
set status to failed
Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 830, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 110, in cryosparc2_compute.engine.cuda_core.GPUThread.run (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:4625)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 111, in cryosparc2_compute.engine.cuda_core.GPUThread.run (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:4576)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 991, in cryosparc2_compute.engine.engine.process.work (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:27291)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 90, in cryosparc2_compute.engine.engine.EngineThread.load_image_data_gpu (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:5342)
  File "cryosparc2_compute/particles.py", line 107, in get_original_real_data
    return self.blob.view().copy() 
  File "cryosparc2_compute/blobio/mrc.py", line 101, in view
    return self.get()
  File "cryosparc2_compute/blobio/mrc.py", line 98, in get
    data = n.fromfile(file_obj, dtype=self.dtype, count= n.prod(self.shape)).reshape(self.shape)
ValueError: total size of new array must be unchanged
========= main process now complete.
========= monitor process now complete.

[prangley]

Did you figure out how to resolve this error? I seem to be having the same issue.
Thanks, Eliza

[belousoff]

I managed only to fix it by reextracting the offending particle stack again in RELION. There seemed to be some unknown issue that I could never figure out.

[prangley]

Thanks for responding so quickly! I have tried that as well, but unfortunately it has not worked for me.

[stephan]

Hi @prangley,

There was a bug in earlier versions of cryoSPARC where if you imported movies from separate folders that had non-unique names, cryoSPARC Patch Motion correction will incorrectly overwrite the files, which caused the particles to be also overwritten incorrectly, potentially causing the ValueError you see here in 2D Classification (the first time you would use the extracted particles).
The fix for this bug has been released in cryoSPARC v2.15.0. Unfortunately, you will need to re-run Patch Motion Correction and Extract From Micrographs (and all steps in between) in order to re-write the micrographs and particles that were incorrectly overwritten.

On the other hand, this bug also may arise from a corrupt data file, which was most likely the case why re-extracting the particle stack worked for @belousoff. If you know for certain the movie files you imported have unique file names (prior to v2.15.0), then it is worth checking the data integrity of the files you are trying to process.

To check the integrity of data files, see some helper scripts I’ve posted here:
For Raw .mrc Movies:

Value error with patch CTF estimation and manual picker

Hey @KSJ, Is it possible if you can run this code for me, and attach the file it creates? First, get together 2 paths dir_path: The path to the directory where you’ve saved the micrographs from EMPIAR 10059 output_path: The path to the file where the stats will be written to Then, in a shell, run cryosparcm icli, which will open up an interactive python shell. You can then copy and paste this code, with the two variables modified import os from cryosparc2_compute.blobio import mrc dir_pa…

For Particle .mrc stacks:

2D classification ValueError: total size of new array must be unchanged

Hey @Guillaume, Is it possible if you can run this code for me, and email me the file it creates (sarulthasan@structura.bio)? First, identify 3 variables, including the path to where the output of this script will be written to: project_uid : the uid of the project where the import particles job exists (e.g., P1) job_uid : the uid of the import particles job (e.g., J1) output_path : The path to the file where the stats will be written to (file will be created when the script is run) Then, …