2.14 fails receiving jobs from command line

Troubleshooting
1

Cryosparc 2.14.seem not to like the commands we were sending for 2.11.

Not even a simple import particle works.

I’ve downgraded master to 2.11 and import worked , but the 2d class failed cause the worker was 2.14.

I’ve updated again to 2.14 and got the same result back.

Some logs:

command_core:

-----------------------------------------------------
[JSONRPC ERROR  2020-03-26 19:56:23.461546  at  wait_job_complete ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 114, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 4321, in wait_job_complete
    while get_job_status(project_uid, job_uid) != 'completed':
  File "cryosparc2_command/command_core/__init__.py", line 123, in wrapper
    raise e
TypeError: argument of type 'NoneType' is not iterable

I do not see weird stuff on the other logs but can send them if needed.

OS: CentOS7

Note: I’ve open the ports in the firewall as suggested in on another post, but it did not make any difference.

the command is:

/usr/local/cryosparc/cryosparc2_master/bin/cryosparcm cli 'make_job("import_particles","P2","W4", "/usr/local/scipion2/Administrator", "None", {"particle_blob_path": "/home/administ
rator/ScipionUserData/projects/TestCryosparcClassify2D/Runs/000132_ProtCryo2D/tmp", "particle_meta_path": "/home/administrator/ScipionUserData/projects/TestCryosparcClassify2D/Runs/0001
32_ProtCryo2D/tmp/input_particles.star", "psize_A": "4.0"}, {})'
/usr/local/cryosparc/cryosparc2_master/bin/cryosparcm cli 'enqueue_job("P2","J4","default")'

Wait! “/usr/local/scipion2/Administrator” is suspicious. I’ll be back…

Could be related to the user in the command line?. Has it being improved/secured somehow?. In 2.11 we were able to put anything as the user, and I guess None is the password

2

I’ve just downgraded to v2.12.4 (that works for me on my laptop) and it worked fine. It must be something between 2.12.4+ and latest.

3

I’m probably wrong about my assumptions.

Now in a second machine, I was getting the same errors at command_core with 2.11 and 2.12.4.

Current status is: 2.12.4, showing those errors in the log, but scipion sends jobs successfully.

4

Could all this happen is you’ve never accepted the T&C in the GUI?

I’m starting to think is related.

5

Hi @pconesa1,
This is not related to T&C. The function signature for “make_job” may have been changed by one of our developers in a recent version - he’s currently away but we will get back to you next week.

6

Thanks @apunjani! I was becoming paranoic about it. :wink:

7

Will it be possible to specify GPU/GPUs from the command line?

8

Hi @pconesa1, the make_job signature did indeed change in 2.14. Here’s the full signature:

make_job(job_type, project_uid, workspace_uid, user_id, created_by_job_uid=None, title=None, params={}, input_group_connects={})

This should make the job that you want:

cryosparcm cli 'make_job("import_particles","P2","W4", "/usr/local/scipion2/Administrator", None, None, {"particle_blob_path": "/home/administrator/ScipionUserData/projects/TestCryosparcClassify2D/Runs/000132_ProtCryo2D/tmp", "particle_meta_path": "/home/administrator/ScipionUserData/projects/TestCryosparcClassify2D/Runs/0001
32_ProtCryo2D/tmp/input_particles.star", "psize_A": "4.0"})'

Will it be possible to specify GPU/GPUs from the command line?

Yes, you can do this with enqueue_job, which has this signature:

enqueue_job(project_uid, job_uid, lane=<default lane>, hostname=<default worker hostname>, gpus=<list of indexes>)

For example, to queue to the first two GPUs on hostname cryoem.structura.bio, run

cryosparcm cli "enqueue_job('P2', 'J4', hostname='cryoem.structura.bio', gpus=[0,1])"

Note: Specifying the hostname argument is the only way you can queue jobs to a specific GPU.

Hope that helps!

Nick

9

Great @nfrasser! I think we have enough to try it out. Will report back in case of we find any issue.

Thank you very much

10

I can confirm now, and realize that the output I posted are a consequence of us (Scipion), sending wrong syntaxt commands that worked for 2.11 but not compatible with 2.12, and probably higher.