Hi,
I ran a non-uniform refinement job with particles from reference based motion correction, and it failed with the message “Job process terminated abnormally”. The box size is 576 px and a pixel size of 0.648 A. I tried running in “Low-memory mode” and restricting “GPU batch size of images” to 1. We use RTX A6000 GPUs. I have also tried homogenous refinement with the same error message and it stops at the exact point as the non-uniform refinement during processing . Please suggest a way around. Thanks!
Sharing the entire log output-
License is valid.
Launching job on lane default target superrack …
Running job on master node hostname superrack
[CPU: 92.0 MB Avail: 971.68 GB]
Job J152 Started
[CPU: 92.0 MB Avail: 971.55 GB]
Master running v4.7.0, worker running v4.7.0
[CPU: 92.0 MB Avail: 971.39 GB]
Working in directory: /data/connlab/suparno/cryosparc/rsmf/CS-rsmf-pncc-data-jan25/J152
[CPU: 92.0 MB Avail: 971.39 GB]
Running on lane default
[CPU: 92.0 MB Avail: 971.42 GB]
Resources allocated:
[CPU: 92.0 MB Avail: 971.46 GB]
Worker: superrack
[CPU: 92.0 MB Avail: 971.49 GB]
CPU : [0, 1, 2, 3]
[CPU: 92.0 MB Avail: 971.54 GB]
GPU : [0]
[CPU: 92.0 MB Avail: 971.59 GB]
RAM : [0, 1, 5]
[CPU: 92.0 MB Avail: 971.66 GB]
SSD : False
[CPU: 92.0 MB Avail: 971.72 GB]
--------------------------------------------------------------
[CPU: 92.0 MB Avail: 971.80 GB]
Importing job module for job type nonuniform_refine_new…
[CPU: 334.5 MB Avail: 957.36 GB]
Job ready to run
[CPU: 334.5 MB Avail: 957.35 GB]
***************************************************************
[CPU: 334.5 MB Avail: 957.26 GB]
Transparent hugepages are enabled. You may encounter stalls or performance problems with CryoSPARC jobs.
[CPU: 1.11 GB Avail: 954.92 GB]
Using random seed of 1456569334
[CPU: 1.12 GB Avail: 954.86 GB]
Loading a ParticleStack with 57296 items…
[CPU: 1.15 GB Avail: 943.53 GB]
Done.
[CPU: 1.15 GB Avail: 943.58 GB]
====== Gold Standard Split ======
[CPU: 1.15 GB Avail: 943.60 GB]
Particles have input alignments3D connected, so reusing pre-existing split
[CPU: 1.15 GB Avail: 943.63 GB]
Set A is greater than set B by 66 particles (0.115 percent difference relative to the total dataset).
[CPU: 1.16 GB Avail: 945.38 GB]
Split A has 28681 particles
[CPU: 1.16 GB Avail: 945.34 GB]
Split B has 28615 particles
[CPU: 1.16 GB Avail: 945.30 GB]
====== Refinement ======
[CPU: 1.16 GB Avail: 945.26 GB]
Input particles have box size 1152
[CPU: 1.16 GB Avail: 945.24 GB]
Input particles have pixel size 0.3240
[CPU: 1.16 GB Avail: 945.21 GB]
Particles will be zeropadded/truncated to size 1152 during alignment
[CPU: 1.16 GB Avail: 945.18 GB]
Volume refinement will be done with effective box size 1152
[CPU: 1.16 GB Avail: 945.15 GB]
Volume refinement will be done with pixel size 0.3240
[CPU: 1.16 GB Avail: 945.12 GB]
Particles will be zeropadded/truncated to size 1152 during backprojection
[CPU: 1.16 GB Avail: 945.09 GB]
Particles will be backprojected with box size 1152
[CPU: 1.16 GB Avail: 945.07 GB]
Volume will be internally cropped and stored with box size 1152
[CPU: 1.16 GB Avail: 945.03 GB]
Volume will be interpolated with box size 1152 (zeropadding factor 1.00)
[CPU: 1.16 GB Avail: 945.00 GB]
DC components of images will be ignored and volume will be floated at each iteration.
[CPU: 1.16 GB Avail: 944.96 GB]
Spherical windowing of maps is enabled
[CPU: 1.16 GB Avail: 944.93 GB]
Refining with C1 symmetry enforced
[CPU: 1.16 GB Avail: 944.90 GB]
Refining with pose and shift marginalization during backprojection enabled.
[CPU: 1.16 GB Avail: 944.87 GB]
Resetting input per-particle scale factors to 1.0
[CPU: 1.16 GB Avail: 944.83 GB]
Starting at initial resolution 12.000A (radwn 31.104).
[CPU: 1.16 GB Avail: 944.80 GB]
====== Non-Uniform Refinement ======
[CPU: 1.16 GB Avail: 944.76 GB]
Non-Uniform Refinement is enabled.
[CPU: 1.16 GB Avail: 944.72 GB]
Using AWF of 3.00.
[CPU: 1.16 GB Avail: 944.69 GB]
Using butterworth filter with order 8.
[CPU: 1.16 GB Avail: 944.65 GB]
====== Masking ======
[CPU: 25.96 GB Avail: 923.09 GB]
No mask input was connected, so dynamic masking will be enabled.
[CPU: 25.96 GB Avail: 923.05 GB]
Dynamic mask threshold: -1.0000
[CPU: 25.96 GB Avail: 923.03 GB]
Dynamic mask near (A): 6.00
[CPU: 25.96 GB Avail: 923.01 GB]
Dynamic mask far (A): 14.00
[CPU: 25.96 GB Avail: 922.98 GB]
====== Initial Model ======
[CPU: 25.96 GB Avail: 922.96 GB]
Resampling initial model to specified volume representation size and pixel-size…
[CPU: 42.79 GB Avail: 884.70 GB]
Estimating scale of initial reference.
[CPU: 89.2 MB Avail: 915.16 GB]
WARNING: io_uring support disabled (not supported by kernel), I/O performance may degrade
[CPU: 43.01 GB Avail: 902.05 GB]
Rescaling initial reference by a factor of 0.491
[CPU: 43.02 GB Avail: 861.56 GB]
Estimating scale of initial reference.
[CPU: 43.01 GB Avail: 821.95 GB]
Rescaling initial reference by a factor of 0.988
[CPU: 43.01 GB Avail: 848.56 GB]
Estimating scale of initial reference.
[CPU: 43.01 GB Avail: 852.33 GB]
Rescaling initial reference by a factor of 1.007
[CPU: 173.8 MB Avail: 910.09 GB]
====== Job process terminated abnormally.
