We have a zombie job listed as “launched” but it is not running on any of the workers. After rebooting the complete system this job reappears again as “launched”. Trying to kill it from the GUI or the command line (cryosparcm job kill <PROJECT_UID> <JOB_UID>) yields an error:
✕ APIError: *** [API] (POST http://cryo-master.local:39002/projects/P7/jobs/J5111:kill, code 500) received error response
Does anybody know how to kill the job and remove it?
@Pielhaas Please can you try again
and, after that, run the command
cryosparcm snaplogs. Please send us the resulting tgz file. I will send you a PM about the email address.
The log file has been send.
We have not received the file that you mentioned had been sent. Please can you check whether you received notification of a rejected message or similar?
It has been send again
We still cannot see the message. Was it sent from the email address with which you registered for this forum?
Have you received it this time?
Unfortunately not. Would you be able to share the file privately on your organization’s file sharing platform, and send me the link in a personal message?
Mail with link has been sent
Thanks. Fetched the file.
@Pielhaas The job in question may have ended up in this inconsistent state due to an unlikely combination of circumstances. You may try the following command to adjust the state of the job in question, after substituting the actual project and job UIDs in place of PX and JY:
cryosparcm cli 'db.jobs.update_one({"project_uid": "PX", "uid": "JY"}, {"$set": {"status": "killed"}}).modified_count'
That fixed it,thanks!