@Guillaume @DanielAsarnow We noted support in the wrapper for Topaz 0.3+ as a feature request.
3 Likes
Hello Jay,
thank you for the tips! I tried splitting the dataset with Exposure sets, but the Topaz train job would fail with ‘Argument list too long’ error even with 3800 micrographs. I splitted also the particles and used 10 000 only, but with the same result. Increasing the number of threads to 6 and CPUs to 8 did not help either.
I am using Cryosparc version 4.6.2 and topaz version 0.2.5a.
Should I reduce the number of particles even further or do some other trick to make topaz working, please?
With regards,
Ammenoel
Usually anywhere under 3000 mics is safe. Possibly 3800 is still too many - try with a smaller set?
The limit depends on your OS configuration, more specifically a configuration option called ARG_MAX. If you have access to a Linux shell account on the same system your CryoSPARC instance is running, the command xargs --show-limits will display info about system limits (and it will pause; you then need to hit Ctrl+D to exit the command and get your prompt back). The relevant line is Maximum length of command we could actually use: 2086444. The number is the maximum length allowed for command argument lists. This is the number I got on my cluster, it might differ on your system.
I believe this number is in characters or bytes (though I could not confirm this), which means that the number of micrographs you can list before hitting the limit also depends on the length of the file names. If your micrograph files have very long names, you will hit the ARG_MAX limit with fewer of them than if they had shorter names.
More info here: The maximum length of arguments for a new process
I hope this helps!
2 Likes
Ok thanks for the advice. Finally I needed to limit the number of micrographs to ~2500 for Topaz train job and to ~516 micrographs for Topaz extract.
A.
Interesting, thank you for further clarification! My micrograph names are unfortunately very long so this might well explain my troubles.
A.