All jobs are in queue forever, The new build job failed to run

Install, Configure and Update
1

We are running cryosparc in our computer set, but when we create a new job, the error will be displayed in the operation interface, and than the job will queue forever rather than going to the next. The new job failed to run, we want to know how to fix the problem.

Unable to queue P7 J62: ServerError: bad operand type for unary -: ‘NoneType’

2

Welcome to the forum @ecwww .

Were you able to run jobs using this CryoSPARC instance before?
Please can you post the output of the command

cryosparcm cli "get_scheduler_targets()"
3

Hi,
We can run jobs using this Cryosparc before, many jobs have been done. However, one day the job we were running suddenly failed. Since then, all new jobs are in queue forever and failed to run.

There is the output of the command:

image
image808×221 11.2 KB

4

Thanks for posting this information. Please can you also post the outputs of these commands:

cryosparcm status | grep -v LICENSE
hostname -f
host cn5
host $(hostname -f)
ssh fbh@cn5 hostname -f
6

@ecwww
Your previous post, which I removed from public view (see below), indicates a failure to connect to our license server. Please confirm connectivity with the command
curl -L get.cryosparc.com,
which should print

{"message":"Missing Authentication Token"}

to the screen.
There may be additional problems, such as DNS resolution of the cluster hostname by the worker host. You may want to reach out to your IT support regarding these issues.

I removed the post as it contained your license id. For future posts, please

7

Thanks for your guidance. When I try the command it doesn’t print the same as your post. There is the output

<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> <meta http-equiv="refresh" content="0;url=/srun_portal_pc?ac_id=1&amp;theme=pro">

Maybe there is something wrong with our hardware, thanks for your help. I will pay attention to it next time I post.

8

Is this the complete output?

Not sure. I suspect the configuration of your network intercepts/blocks the outgoing curl traffic.

9

When we try it again now it print the same as you post

Then we restart cryosparc, it didn’t report an error, but when we start a new job, the same error will still pop up in our operation interface.

10

Please can you email us the tgz file that is created by the command

cryosparcm snaplogs

I will send you a direct message with the email address.

11

@ecwww It seems that somehow, P7 J62 has been queued without priority. We are not sure how this happened. To recover, you may try these commands on the CryoSPARC master computer:

cryosparcm icli # enter the interactive command line interface
db.jobs.update_many({'priority': None}, {'$set': {'priority': 1}})
exit