I’ve been running CS Live v2.15.0 for a while (haven’t gotten a chance to update yet) but all of a sudden the GPUs on the worker node have become unavailable. The RTP_worker jobs start but nothing is logged in the job log or running on the worker node.
I also checked command_core log and not information about it there.
The gpus show up correctly in nvidia-smi and can be used in other programs without issue.
This is odd- can you try first restarting cryoSPARC? cryosparcm restart
Once that’s done, you can try starting the RTP worker jobs again. If they still don’t show any logs in the job cards, check the running processes on the machine the jobs are queued to so we can see what the processes are actually doing.
I figured it out, you will get this response if paths to the cryosparc folder are not the same on all nodes. So if a user links a project on a path that is longer then what shows on the worker nodes, then it just does not start the job.
I was hoping the cryosparcm log command_core would be more apparent but I didnt notice it until I saw the slight difference in the path.
So if anyone ever has this problem, double check the path are available on the work nodes.
