Ab-initio has 90% unused particles

[stjiafle]

As said in the title, my ab-initio will only take 80-90k particles and has around ~800k unused particles when it is done. I kept my setting as default and I’m not sure why it keep doing this (the homogeneous refinement resolution is really bad because of this, but even if I used the “unused particles” they don’t look good either).
There is no error message I can see:

[CPU: 1.41 GB]
----------- Iteration   483 (epoch 0.104).  radwn  9.32  resolution 12.00A  minisize  300  beta 0.00 

[CPU: 1.41 GB]
   -- Class  0 -- lr: 0.10 eps:  0.01 step ratio : 0.0123 ESS R: 1098.959 S:  4.730 Class Size: 100.0% (Average: 100.0%)

[CPU: 1.45 GB]
  Done iteration 00483 of 00484 in  5.961s. Total time 1068.3s. Est time remaining    0.0s.

[CPU: 1.45 GB]
Full abinit run took 1068.285s

[CPU: 1.14 GB]
--------------------------------------------------------------

[CPU: 1.14 GB]
Compiling job outputs...

[CPU: 1.14 GB]
Passing through outputs for output group particles_all_classes from input group particles

[CPU: 1.18 GB]
This job outputted results ['blob', 'ctf', 'alignments_class_0']

[CPU: 1.18 GB]
  Loaded output dset with 82200 items

[CPU: 1.18 GB]
Passthrough results ['alignments2D', 'pick_stats', 'location']

[CPU: 1.52 GB]
  Loaded passthrough dset with 790250 items

[CPU: 1.42 GB]
  Intersection of output and passthrough has 82200 items

[CPU: 1.42 GB]
Passing through outputs for output group particles_class_0 from input group particles

[CPU: 1.46 GB]
This job outputted results ['blob', 'ctf', 'alignments3D']

[CPU: 1.46 GB]
  Loaded output dset with 82200 items

[CPU: 1.46 GB]
Passthrough results ['alignments2D', 'pick_stats', 'location']

[CPU: 1.57 GB]
  Loaded passthrough dset with 790250 items

[CPU: 1.46 GB]
  Intersection of output and passthrough has 82200 items

[CPU: 1.46 GB]
Passing through outputs for output group particles_unused from input group particles

[CPU: 1.69 GB]
This job outputted results ['blob', 'ctf']

[CPU: 1.69 GB]
  Loaded output dset with 708050 items

[CPU: 1.69 GB]
Passthrough results ['alignments2D', 'pick_stats', 'location']

[CPU: 1.76 GB]
  Loaded passthrough dset with 790250 items

[CPU: 1.76 GB]
  Intersection of output and passthrough has 708050 items

[CPU: 1.57 GB]
Checking outputs for output group particles_all_classes

[CPU: 1.57 GB]
Checking outputs for output group particles_class_0

[CPU: 1.58 GB]
Checking outputs for output group particles_unused

[CPU: 1.68 GB]
Updating job size...

[CPU: 1.68 GB]
Exporting job and creating csg files...

[CPU: 1.68 GB]
***************************************************************

[CPU: 1.68 GB]
Job complete. Total time 1122.71s

Is there a setting somewhere that need to be set or does that mean something is wrong with my data?
(my coworker who has been 3.9 version has no problem processing this data - using the same setting)
I’m a pretty new user so if anyone can give me pointers in detail that would be helpful!

[wtempel]

Please see this related discussion:

3D ab initio job does not load entire particle stack

@jenchem A single-class ab initio reconstruction job might indeed not output all input particles. We are planning to implement a notification to that effect. After single-class ab initio reconstruction, one can still include all particles in a subsequent job by connecting as inputs all particles input to ab initio reconstruction the volume output by ab initio reconstruction

[stjiafle]

Hi wtempel,
Thank you for pointing out this thread since I missed it from my initial search.
Similar to them, I have my input as

[CPU:  191.7 MB]
Importing job module for job type homo_abinit...

[CPU:  293.7 MB]
Job ready to run

[CPU:  293.7 MB]
***************************************************************

[CPU:  729.9 MB]
Using random seed for sgd of 1044547418

[CPU:  814.5 MB]
Loading a ParticleStack with 790250 items...

I already did this step as described by jenchem:

@jenchem

Looking back, it appears I loaded all of the particles from all stacks (ie all particles that I wanted for further processing) into one 2D-classification job, then selected all of the output from that one 2D-classification job to start a 3D ab initio. That seems to have worked for my case.

My dataset come from two different import movies, but it combined into one Patch CTF, and one template picker, extract mics, and 2D class.

I tried doing homogeneous refinement using the unused particles and it doesn’t go so hot.
So right now I’m importing this job to the other computer (which currently using cryosparc v3.9) and try over there and see if the problem still persist.