Hi all,
Recently, we’ve had users fail to run Topaz through CryoSPARC.
Traceback (most recent call last): File "cryosparc_master/cryosparc_compute/run.py", line 115, in cryosparc_master.cryosparc_compute.run.main File "/service_account/core_cryo_sparc/software/cryosparc_cluster/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 300, in run_topaz_wrapper_train utils.run_process(convert_command) File "/service_account/software/cryosparc_cluster/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 99, in run_process assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})" AssertionError: Subprocess exited with status 127 (/path/to/topaz/runscript/topaz_run convert --down-scale 11 --threshold 0 -o /path/to/cryosparc/project/J104/topaz_particles_processed.txt /path/path/path/…)
This failure seems to be coming from the topaz convert ... step. We had issues with “version” before, but I added a couple bash lines from another thread that solved this issue.
I also tried reinstalling Topaz and pointing to that, but same issue persists.
Any help would be appreciated. Thanks!
Walid
@walidabualafia Please can you post the outputs of the following commands:
cat /path/to/topaz/runscript/topaz_run
cryosparcm eventlog P99 J104 | tail -n 30
Hi @wtempel,
We were able to solve this issue by employing a combination of changes. We set cryosparc -s --version to produce the correct output instead of scripting this. This helped us get a little bit further and stopped the job from exiting.
Regarding the issue above, we just upgraded CryoSPARC in-place, and it solved the problem. I have a suspicion that some of the master/worker’s code for running Topaz got corrupted or modified. So reinstalling CryoSPARC reset the command back to the original form.
Things are working as usual. Thank you!
On another note, is it possible to schedule a brief meeting with one of your engineers to discuss the best set up for (centralized) CryoSPARC? We are running more than 10 instances here, and would like to put them on appropriate compute/storage to improve performance. Please let me know if this is something your team can help with.
Thank you again,
Walid
Hi @walidabualafia,
For academic users, we do our best to support CryoSPARC via the Discussion Forum so that our team and other users can chime in especially around optimizing hardware configurations. Other users who may be reading can also benefit. If you can create a new post with the details of your setup and questions, then our team and others will hopefully be able to provide some helpful responses.
The CryoSPARC guide is a good starting point to become familiar with the CryoSPARC
application architecture and corresponding infrastructure requirements.
Thank you!