Error during Heterogeneous Refinement

Troubleshooting
1

Hello all,

I am trying to run a Heterogeneous refinement for the first time, but I keep getting an error near the end of iteration 0 that leads to the job failing. Does anyone have a suggestion as to why this happens and how to fix it? Here is the logfile from the job, should it help.

Running job  J180  of type  hetero_refine
Running job on hostname %s cmproc1
Allocated Resources :  {u'lane': u'default', u'target': {u'lane': u'default', u'name': u'cmproc1', u'title': u'Worker node cmproc1', u'resource_slots': {u'GPU': [0, 1, 2], u'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31], u'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]}, u'hostname': u'cmproc1', u'worker_bin_path': u'/mntdirect/_users/sharp/cryosparc/cryosparc2_worker/bin/cryosparcw', u'cache_path': u'/tmp/cryosparc', u'cache_quota_mb': None, u'resource_fixed': {u'SSD': True}, u'cache_reserve_mb': 10000, u'type': u'node', u'ssh_str': u'sharp@cmproc1', u'desc': None}, u'license': True, u'hostname': u'cmproc1', u'slots': {u'GPU': [1], u'RAM': [3, 4], u'CPU': [4, 5, 6, 7]}, u'fixed': {u'SSD': True}, u'lane_type': u'node'}
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
FSC No-Mask...        ========= sending heartbeat
 0.143 at 12.427 radwn. 0.5 at 10.558 radwn. Took 5.273s.
FSC Spherical Mask... ========= sending heartbeat
========= sending heartbeat
 0.143 at 13.392 radwn. 0.5 at 11.603 radwn. Took 8.378s.
FSC Loose Mask...     ========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
 0.143 at 16.584 radwn. 0.5 at 13.344 radwn. Took 35.741s.
========= sending heartbeat
========= sending heartbeat
FSC No-Mask...         0.143 at 12.276 radwn. 0.5 at 10.496 radwn. Took 5.399s.
FSC Spherical Mask... ========= sending heartbeat
========= sending heartbeat
 0.143 at 13.355 radwn. 0.5 at 11.498 radwn. Took 8.324s.
FSC Loose Mask...     ========= sending heartbeat
========= sending heartbeat
 0.143 at 17.351 radwn. 0.5 at 12.919 radwn. Took 21.841s.
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
**** handle exception rc
set status to failed
Traceback (most recent call last):
  File "cryosparc2_worker/cryosparc2_compute/run.py", line 78, in cryosparc2_compute.run.main
  File "cryosparc2_worker/cryosparc2_compute/jobs/hetero_refine/run.py", line 476, in cryosparc2_compute.jobs.hetero_refine.run.run_hetero_refine
TypeError: 'NoneType' object is not subscriptable
========= main process now complete.
========= monitor process now complete.

Every other job type, including homogeneous refinement, has worked with no issues until now. This error also happens when I change settings or memory allocation.

Thank you in advance.

Best,
David

2

Hi @DavidF,

Thanks for reporting this. We are looking into it. Can you give some more details about the use case? Box size, pixel size, number of particles, number of classes, where did the initial models come from, etc?

Thanks,
Ali

3

Hi @DavidF, could you also let us know the parameters of the job? Specifically, are intermediate plots disabled?

Ali H.

4

Thank you both for your answer!

So it did happen with many different box sizes (400, 256, 128), particles (20000 to 160000) and models.
However it is true that I had intermediate plots turned off.

Turning them on seems to have solved this.

Thanks a lot.

Best,
David

5

This error still happens in v2.13.2. In addition the error listed below happens if box size is left empty. I just noticed 2.14 is available, so I will update and re-test.

Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main
File “cryosparc2_worker/cryosparc2_compute/jobs/hetero_refine/run.py”, line 111, in cryosparc2_compute.jobs.hetero_refine.run.run_hetero_refine
AttributeError: ‘module’ object has no attribute ‘get_N_input’

6

The error seems to persist even in v2.14.2. Seems switching off intermedediate plots reproducibly lead to job crashing with error.

7

Thanks for reporting @ChrisK - we will investigate and try to fix in the next release

8

Hi everyone, this is now fixed in the latest cryoSPARC v2.15. After you update, you may disable intermediate plots without issue!