Hi Oli,
Thank you for your reply.
I binned the micrographs by Patch Motion Cor Job, then the Topaz Extract works, but I would like to downsample more, such as bin8, bin16, but the CS Patch Motion Cor F-crop option only provides as many as bin4, if you know how can I downsample more in CS?
This shouldn’t be necessary to run Topaz, as Topaz downsamples on the fly. Have you tried adjusting the downsampling factor in Topaz? What settings are you using?
Do you require down-sampling beyond factor 4 before picking, or would a downstream Downsample Particles job work for you?
@CleoShen
you may want to use some other package to downsample of desired value. What stage of processing you are in ?
@olibclarke @wtempel
I am taking this thread to ask about the helical processing questions that I have posted 2 months ago ? I think a separate thread is made but have not received any response. Can you help.
I have no idea, I have not worked much with helical samples, sorry…
Yes, I’d like to down-sample 8 before picking. I thought I could do two rounds of bin when running two times of patch motion cor, but it doesn’t work.
I’m running the Topaz extract job on bin4 micrograph now.
Hi @CleoShen you do not need to downsample your micrographs for Topaz. This happens automatically during preprocessing in Topaz, and is controlled by the downsampling factor that you set in Topaz train. What settings are you using for Topaz train/extract?
Also can you post the error you are seeing in Topaz extract? If you are trying to run Topaz extract on a lot of micrographs (>5k), then you will need to split into smaller groups for Topaz extract
Hi Oli,
I set downsample4 at the beginning, but the jobs are killed multiple times (I assume it’s because of the big memory request); then I tested downsampling by patch motion cor first, then the Topaz extra works.
Can you post the error, and the settings you are using
Sure! The error is
OSError: [Errno 12] Cannot allocate memory.
The settings are:
Starting micrograph preprocessing by running command /home/groups/brunger/ccw0820/topaz/bin/topaz preprocess --scale 4 --niters 200 --num-workers 8 -o /oak/stanford/groups/brunger/vATPase/SPA/SPA_SupRes/P3/J417/preprocessed [MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY]
thanks - by settings I mean a screenshot of the GUI panel showing exactly what settings you used - downsampling factor, number of threads, etc (because what you posted only has the preprocess command)… it does say you are using 8 workers, so if you are running out memory you might try reducing that (using e.g. 1 or 2 workers, and one or two threads, instead of 8 and 8 which is the default
Oh, I’m sorry. Here it is:
You have 20 threads listed here, with 8 cpus - on my system that would result in system lockup, because it will spawn a lot of processes at once. I would try with 2 and 2 and see if that helps.
Cheers
Oli
Hi Oli,
The error becomes like this; any idea?
Traceback (most recent call last):
File "cryosparc_worker/cryosparc_compute/run.py", line 85, in cryosparc_compute.run.main
File "/home/groups/brunger/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 1109, in run_topaz_wrapper_extract
utils.run_process(extract_command)
File "/home/groups/brunger/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 98, in run_process
assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})"
AssertionError: Subprocess exited with status 1 (/home/groups/brunger/ccw0820/topaz/bin/topaz extract --radius 10 --threshold -6 --up-scale 4 --assignment-radius -1 --min-radius 5 --max-radius 100 --step-radius 5 --num-workers 8 --device 0 --model /oak/stanford/groups/brunger/vATPase/SPA/SPA_SupRes/P3/J3…)
Not sure. ‘Subprocess exited with status 1’ means that the topaz command failed, but does not say how. Try running using CPUs only. Or try making the same command outside of cryosparc (but in the same environment) and seeing what the actual error is.
Best,
-Alex
Some contributors to this topic have mentioned alternatives to additional, explicit down-sampling. Nevertheless, should the need arise, a recent cryoSPARC patch adds support for additional down-sampling ratios.