Hi,
running an ab initio with 10 classes by using about 120,000 particle images I got, at iteration 591 the following crash:
…
(Average: 12.7%)
– Class 4 – lr: 0.20 eps: inf step ratio : nan ESS R: 1.000 S: 1.000 Class Size: 0.0% (Average: 11.3%)
– Class 5 – lr: 0.20 eps: inf step ratio : nan ESS R: 1.000 S: 1.000 Class Size: 0.0% (Average: 10.7%)
– Class 6 – lr: 0.20 eps: inf step ratio : nan ESS R: 1.000 S: 1.000 Class Size: 0.0% (Average: 10.7%)
– Class 7 – lr: 0.20 eps: inf step ratio : nan ESS R: 1.000 S: 1.000 Class Size: 43.4% (Average: 8.2%)
– Class 8 – lr: 0.20 eps: inf step ratio : nan ESS R: 1.000 S: 1.000 Class Size: 22.7% (Average: 6.4%)
– Class 9 – lr: 0.20 eps: inf step ratio : nan ESS R: 1.000 S: 1.000 Class Size: 33.9% (Average: 7.0%)
Done iteration 00591 of 01643 in 28.345s. Total time 11805.9s. Est time remaining 31214.6s.
----------- Iteration 592 (epoch 0.887). radwn 30.55 resolution 17.04A minisize 300 beta 0.00
Traceback (most recent call last):
File “/home/blaasd3/cryosparc/cryosparc-compute/sparcjob.py”, line 288, in
cuda_devs = current_experiment_params[‘cuda_devs’])
File “/home/blaasd3/cryosparc/cryosparc-compute/abinit.py”, line 391, in abinit
**search_params)
File “/home/blaasd3/cryosparc/cryosparc-compute/search.py”, line 359, in heuristicsearches
doscale = doscale, ignore_dc = ignore_dc, cuda_workers = cuda_workers)
File “/home/blaasd3/cryosparc/cryosparc-compute/search.py”, line 245, in compute_Us
results.append(work(arg))
File “/home/blaasd3/cryosparc/cryosparc-compute/search.py”, line 214, in work
IW.load(D, C, 1.0/N_R, 1.0/N_S, 1.0,
ZeroDivisionError: float division by zero
Hmm…
Thanks for reporting this issue. This looks like a bug we’ve seen before but haven’t been able to reproduce or fix yet, we’re not sure the root cause. Can you try with a different number of classes (9?) and see what happens?
Also can you do the following:
On the launch page for this experiment, find the input box called ‘Flags’ and select the ‘searchinfo’ flag. Then on the left, click on ‘Show from top’. You should see something like this:
Same dataset as in the run repoprted before. Whereas it run to completion with 1 class it crashes with 10 or - as below - with 8 classes
This is the output of 'Flags > searchinfo > Show from top:
=== Start of BnB marginalization at iteration 260 ==
=== Start of BnB maximization at iteration 504 ==
I started ab initio again, this time with 8 classes, and had immediately a crash with:
Launched job J3vynQnRDv8s7yn7H
License is valid.
Job Type: abinit Job ID: J3vynQnRDv8s7yn7H
Assigned Slots: RAM: [0] GPU: [0]
Starting Ab-Initio Reconstruction , Experiment ID BPKK3TQ38Wr9rduNC
Loading Dataset …
---- Loading File Set. Type: star Path: /home/blaasd3/cryosparc/run/bulk/./local_fs/home/blaasd3/DATA/HRV2/all_HRV2.star
Loaded meta data for 129598 images.
Found raw data for 129598 images.
Loading image data…
Found references to 1 unique data files.
Checking SSD cache for data files…
Dataset image size is 256.
Traceback (most recent call last):
File “/home/blaasd3/cryosparc/cryosparc-compute/sparcjob.py”, line 166, in
dataset = dataio.dataset_factory(current_dataset_params, do_cache=current_experiment_params.get(‘do_cache’, True))
File “/home/blaasd3/cryosparc/cryosparc-compute/dataio/init.py”, line 18, in dataset_factory
do_cache=do_cache)
File “/home/blaasd3/cryosparc/cryosparc-compute/dataio/load.py”, line 75, in filesets_into_dataset
load_all_referenced_mrcs(dataset, dataset, load_type, do_cache)
File “/home/blaasd3/cryosparc/cryosparc-compute/dataio/load.py”, line 158, in load_all_referenced_mrcs
mm = mrcdata[img.data_input_abspath][img.data_input_idx]
IndexError: list index out of range
I am afraid that something is wrong with the data set but what?
Hi!
I have had the same problem and then repeated the same job with half the number of classes and that worked. So I guess I simply had too many classes. In case it helps, here is the Flags> Searchinfo for the job that didn’t go through.
=== Start of BnB marginalization at iteration 473 ==
=== Start of BnB maximization at iteration 500 ==
