Class 2D Error= ValueError: total size of new array must be unchanged

open

(Matthew Belousoff) #1

Hey there,

I have been experiencing this error with a 2D refinement.

Sometimes it runs for a few iterations, then fails, sometimes it doesn’t even get to the first iteration.

It is a particle stack of ~1M particles, boxsize 288x288, collected in CTEM mode.

The input was an imported stack from RELION.

Details are below.

----------------------------------------------------------------------------
CryoSPARC System master node installed at
/apps/cryosparc2_master
Current cryoSPARC version: v2.9.0
----------------------------------------------------------------------------

cryosparcm process status:

command_core                     RUNNING   pid 24591, uptime 3 days, 7:05:49
command_proxy                    RUNNING   pid 24629, uptime 3 days, 7:05:45
command_vis                      RUNNING   pid 24614, uptime 3 days, 7:05:47
database                         RUNNING   pid 24510, uptime 3 days, 7:05:51
watchdog_dev                     STOPPED   Not started
webapp                           RUNNING   pid 24638, uptime 3 days, 7:05:43
webapp_dev                       STOPPED   Not started

----------------------------------------------------------------------------

global config variables:

export CRYOSPARC_LICENSE_ID=""
export CRYOSPARC_MASTER_HOSTNAME="LithgowGPU"
export CRYOSPARC_DB_PATH="/apps/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false

> 
export CRYOSPARC_LICENSE_ID=""
export CRYOSPARC_USE_GPU=true
export CRYOSPARC_CUDA_PATH="/usr/local/cuda"
export CRYOSPARC_DEVELOP=false

> 
Cuda 10.1
OS: Linux Mint 18.1 Serena

>
========= sending heartbeat
***************************************************************
Running job  J3  of type  class_2D
Running job on hostname %s LithgowGPU
Allocated Resources :  {u'lane': u'default', u'target': {u'lane': u'default', u'name': u'LithgowGPU', u'title': u'Worker node LithgowGPU', u'resource_slots': {u'GPU': [0, 1, 2, 3], u'RAM': [0, 1, 2, 3, 4, 5, 6, 7], u'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}, u'hostname': u'LithgowGPU', u'worker_bin_path': u'/apps/cryosparc2_worker/bin/cryosparcw', u'cache_path': u'/scratch/sparc', u'cache_quota_mb': None, u'resource_fixed': {u'SSD': True}, u'cache_reserve_mb': 10000, u'type': u'node', u'ssh_str': u'mbelouso@LithgowGPU', u'desc': None}, u'license': True, u'hostname': u'LithgowGPU', u'slots': {u'GPU': [1], u'RAM': [3, 4, 5], u'CPU': [2, 3]}, u'fixed': {u'SSD': True}, u'lane_type': u'node'}
========= sending heartbeat
cryosparc2_compute/sigproc.py:754: RuntimeWarning: invalid value encountered in divide
  frc[k, :copylen] = (AB / n.sqrt(AA*BB))[:copylen]
cryosparc2_compute/sigproc.py:821: RuntimeWarning: invalid value encountered in greater
  crossings = n.where((fsc[:-1] > thresh) * (fsc[1:] < thresh))[0]
cryosparc2_compute/sigproc.py:821: RuntimeWarning: invalid value encountered in less
  crossings = n.where((fsc[:-1] > thresh) * (fsc[1:] < thresh))[0]
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
cryosparc2_compute/util/logsumexp.py:40: RuntimeWarning: divide by zero encountered in log
  return n.log(wa * n.exp(a - vmax) + wb * n.exp(b - vmax) ) + vmax
========= sending heartbeat
========= sending heartbeat
/apps/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/matplotlib/pyplot.py:516: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
  max_open_warning, RuntimeWarning)
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
/apps/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)
/apps/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/numpy/core/_methods.py:70: RuntimeWarning: invalid value encountered in true_divide
  ret = ret.dtype.type(ret / rcount)
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
**custom thread exception hook caught something
**** handle exception rc
set status to failed
Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 830, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 110, in cryosparc2_compute.engine.cuda_core.GPUThread.run (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:4625)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 111, in cryosparc2_compute.engine.cuda_core.GPUThread.run (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:4576)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 991, in cryosparc2_compute.engine.engine.process.work (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:27291)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 90, in cryosparc2_compute.engine.engine.EngineThread.load_image_data_gpu (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:5342)
  File "cryosparc2_compute/particles.py", line 107, in get_original_real_data
    return self.blob.view().copy() 
  File "cryosparc2_compute/blobio/mrc.py", line 101, in view
    return self.get()
  File "cryosparc2_compute/blobio/mrc.py", line 98, in get
    data = n.fromfile(file_obj, dtype=self.dtype, count= n.prod(self.shape)).reshape(self.shape)
ValueError: total size of new array must be unchanged
========= main process now complete.
========= monitor process now complete.