Cryosparc-tools can't find 'import movies' jobs

Scripting
1

Hi, I’m running into additional cryosparc-tools trouble.

cryosparc-tools is unable to find ‘import_movies’ jobs. When I create an ‘import_movies’ job, it successfully runs and imports the movies. I can then work with that job in web interface as if I had created it manually. However, if I try to use the job’s wait_for_status method, then I get an error (not shown). If I attempt to connect the output of the ‘import_movies’ job to a patch motion job like this:

patch_motion_job = workspace.create_job(
            type="patch_motion_correction_multi",
            connections={'movies': (import_job.uid, "imported_movies")},
            params=config['preprocessing']['patch_motion_correction_multi'],
            title="Run motion correction",
            desc="Automatically perform beam-induced motion correction")
    patch_motion_job.queue(config['project_options']['lane_name'])

then I get the following error:

Traceback (most recent call last):
  File "/home/ubuntu/repos/cryo_flows/basic_workflow.py", line 78, in <module>
    main(parser.parse_args())
  File "/home/ubuntu/repos/cryo_flows/basic_workflow.py", line 68, in main
    preprocessing(cs, config)
  File "/home/ubuntu/repos/cryo_flows/basic_workflow.py", line 52, in preprocessing
    patch_motion_job = workspace.create_job(
  File "/opt/conda/envs/pyem/lib/python3.9/site-packages/cryosparc/workspace.py", line 89, in create_job
    return self.cs.create_job(
  File "/opt/conda/envs/pyem/lib/python3.9/site-packages/cryosparc/tools.py", line 350, in create_job
    job.connect(parent_job, output_name, input_name, refresh=False)
  File "/opt/conda/envs/pyem/lib/python3.9/site-packages/cryosparc/job.py", line 279, in connect
    result: bool = self.cs.cli.job_connect_group(  # type: ignore
  File "/opt/conda/envs/pyem/lib/python3.9/site-packages/cryosparc/command.py", line 112, in func
    assert "error" not in res, f'Error for "{key}" with params {params}:\n' + format_server_error(res["error"])
AssertionError: Error for "job_connect_group" with params {'project_uid': 'P3', 'source_group': 'imported_movies.movies', 'dest_group': 'J70.J69'}:
ServerError: 'NoneType' object is not subscriptable
Traceback (most recent call last):
  File "/home/ubuntu/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 200, in wrapper
    res = func(*args, **kwargs)
  File "/home/ubuntu/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 266, in wrapper
    return func(*args, **kwargs)
  File "/home/ubuntu/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 6222, in job_connect_group
    builder.connect_group(input_group_name  = input_group_name,
  File "/home/ubuntu/cryosparc/cryosparc_master/cryosparc_compute/jobs/buildcommon.py", line 112, in connect_group
    output_group = com.query(outputjob['output_result_groups'], lambda g: g['name'] == output_group_name, error="No match for %s in job %s" % (output_group_name, outputjob['uid']))
TypeError: 'NoneType' object is not subscriptable

This is unexpected, because the import_movies job in cryosparc-tools has a UID that matches the UID I observe in the web interface (i.e. the job definitely exists). The patch motion job is also successfully created, but the connection to the ‘import movies’ job is not made.

If it helps to know, this seems to not be true for every job type. In the following code, I am able to find a patch-motion correction job but not an import movies job:

In [9]: cs.find_job('P3', 'J70') # A patch motion correction job, found successfully
Out[9]: <cryosparc.job.Job at 0x7faa6b076610>

In [10]: cs.find_job('P3', 'J69') # An import movies job, causing error
*** CommandClient: (http://localhost:39002/api) HTTP Error 500 INTERNAL SERVER ERROR; please check cryosparcm log command_core for additional information.
Response from server: b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>500 Internal Server Error</title>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>\n'
---------------------------------------------------------------------------
Error                                     Traceback (most recent call last)
File /opt/conda/envs/pyem/lib/python3.9/site-packages/cryosparc/command.py:104, in CommandClient._get_callable.<locals>.func(*args, **kwargs)
    103 try:
--> 104     with make_json_request(self, "/api", data=data) as request:
    105         res = json.loads(request.read())

File /opt/conda/envs/pyem/lib/python3.9/contextlib.py:119, in _GeneratorContextManager.__enter__(self)
    118 try:
--> 119     return next(self.gen)
    120 except StopIteration:

File /opt/conda/envs/pyem/lib/python3.9/site-packages/cryosparc/command.py:191, in make_request(client, method, url, query, data, headers)
    189         break
--> 191 raise CommandClient.Error(client, error_reason, url=url)

Error: *** CommandClient: (http://localhost:39002/api) HTTP Error 500 INTERNAL SERVER ERROR; please check cryosparcm log command_core for additional information.
Response from server: b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>500 Internal Server Error</title>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>\n'

The above exception was the direct cause of the following exception:

Error                                     Traceback (most recent call last)
Cell In [10], line 1
----> 1 cs.find_job('P3', 'J69') # A patch motion correction job

File /opt/conda/envs/pyem/lib/python3.9/site-packages/cryosparc/tools.py:253, in CryoSPARC.find_job(self, project_uid, job_uid)
    242 """
    243 Get a job by its unique project and job ID.
    244 
   (...)
    250     Job: job instance
    251 """
    252 job = Job(self, project_uid, job_uid)
--> 253 job.refresh()
    254 return job

File /opt/conda/envs/pyem/lib/python3.9/site-packages/cryosparc/job.py:94, in Job.refresh(self)
     87 def refresh(self):
     88     """
     89     Reload this job from the CryoSPARC database.
     90 
     91     Returns:
     92         Job: self
     93     """
---> 94     self._doc = self.cs.cli.get_job(self.project_uid, self.uid)  # type: ignore
     95     return self

File /opt/conda/envs/pyem/lib/python3.9/site-packages/cryosparc/command.py:107, in CommandClient._get_callable.<locals>.func(*args, **kwargs)
    105         res = json.loads(request.read())
    106 except CommandClient.Error as err:
--> 107     raise CommandClient.Error(
    108         self, f'Did not receive a JSON response from method "{key}" with params {params}', url=self._url
    109     ) from err
    111 assert res, f'JSON response not received for method "{key}" with params {params}'
    112 assert "error" not in res, f'Error for "{key}" with params {params}:\n' + format_server_error(res["error"])

Error: *** CommandClient: (http://localhost:39002) Did not receive a JSON response from method "get_job" with params ('P3', 'J69')

No clue what to do about this error, but I would love it to get some attention, as cryosparc-tools is effectively unusable for me if it can’t import movies for other jobs to use.

2

@epalovcak if you change

connections={'movies': (import_job.uid, "imported_movies")},

to

connections={'exposures': (import_job.uid, "imported_movies")},

does that work?

3

Hey @kyestachowski, unfortunately this did not fix the error.

4

@epalovcak can you queue a patch motion correction job from the icli using the submission scheme similar to the benchmark workflow instead of cryoSPARC tools?

5

Yes, using the icli strategy works fine moving the results of import_movies to patch_motion_correction_multi, as expected:

In [13]: cli.make_job(
    ...:     job_type='patch_motion_correction_multi',
    ...:     title='test_cli',
    ...:     project_uid='P3',
    ...:     workspace_uid='W3',
    ...:     user_id=uuid,
    ...:     input_group_connects={'movies': 'J10.imported_movies'})
Out[13]: 'J74'

In [14]: cli.enqueue_job('P3', 'J74', 'default', uuid)
Out[14]: 'launched'

I think the error must be coming from the way cryosparc-tools deals with ‘import_movies’ jobs

6

I think so too. I have taken to using the benchmark workflow style as its pretty simple and has been able to do what I need as of now.

7

Thanks, yeah, I guess that’s the best solution currently.

Too bad, because queuing/working with jobs in a non-cryoSPARC python environment would be really useful for compatibility with proprietary internal LIMS systems and monitoring tools. That’s gonna be a pretty desirable use-case for pharma companies. It’d be great to hear from the cryoSPARC dev team if/when they’ll be able to address bugs in cryosparc-tools to better support automated workflows.

8

Hi @epalovcak, thanks for reporting this. This should now be working in the latest release of cryosparc-tools. Please update and let me know if you run into any more trouble.

1 Like
9

This appears to be fixed! Thank you for the quick turn-around!!! Very much appreciated.

If I run:

In [14]: patch_motion_job = workspace.create_job(
    ...:             type="patch_motion_correction_multi",
    ...:             connections={'movies': (import_juid, "imported_movies")})

In [15]: patch_motion_job.queue('default')

then the motion correction job launches without issue.

One lingering note: when I create an import_movies job, the Job object that is returned has an appropriate job uid which I can use to connect it to subsequent steps. However, if I try to obtain an import_movies job that already exists with cs.find_job('P3', 'J81'), I still get a similar error to the one reported above (basically, no JSON response received). Similarly, job methods like wait_for_status and refresh throw an error on import_movies jobs, which presumably are expecting a JSON message that they don’t receive. Because queued patch_motion jobs wait for the import_movies job to finish and don’t have an error, this doesn’t prevent me from using cryosparc-tools in any way, but it’s not the expected behavior either.

10

@epalovcak glad to hear it’s working! The errors you’re seeing in import_movies is issue in CryoSPARC itself and will be addressed in a future CryoSPARC release. Will update here when that’s available.

11

That makes sense, I appreciate the help!

12

Hi @epalovcak, the Import Movies issue has been fixed in the latest CryoSPARC v4.3. After updating CryoSPARC, please also update your Python environment to the latest cryosparc-tools version:

pip install -U cryosparc-tools