Error loading particles using cryosparc-tools

Scripting
#1

Hi,

I am trying to load a particle set in a cryposparc-tools script. The script is incomplete, but I am trying to debug what I have so far.

When I run it (python3 select_defocus_change.py) it dies with an error (attached) at the point where I try to load the first particle set. J434 is a Homogeneous Reconstruction job, so it has a particles output group.

Script:

from cryosparc.tools import CryoSPARC

cs = CryoSPARC(host="c112384", base_port=39000, license="xxx" ,email="xxx", password="xxx")
assert cs.test_connection()

project = cs.find_project("P23")
job1 = cs.find_job("P23", "J434")
job2 = cs.find_job("P23", "J448")
particle_set1 = job1.load_output("particles")
particle_set2 = job2.load_output("particles")
particle_set1['ctf/def_abs'] = (particle_set1['ctf/df1_A']+particle_set1['ctf/df1_A'])/2
particle_set2['ctf/def_abs'] = (particle_set2['ctf/df1_A']+particle_set2['ctf/df1_A'])/2
defocus_change = abs(particle_set1['ctf/def_abs']-particle_set2['ctf/def_abs'])
particle_set2['ctf/def_change'] = defocus_change

project.save_external_result(
  workspace_uid = "W4",
  dataset = particle_set2,
  type = "particle",
  name = "defocus_change_particles",
  slots = ["ctf"],
  passsthrough = (job1.uid, "particles"),
  title = "particles with changed defocus",
)

Error:

user@c112384:~$ python3 select_defocus_change.py
Connection succeeded to CryoSPARC command_core at http://c112384:39002
Connection succeeded to CryoSPARC command_vis at http://c112384:39003
Connection succeeded to CryoSPARC command_rtp at http://c112384:39005
Traceback (most recent call last):
  File "select_defocus_change.py", line 9, in <module>
    particle_set1 = job1.load_output("particles")
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/job.py", line 390, in load_output
    metafiles = set(r["metafiles"][0 if r["passthrough"] else version] for r in results)
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/job.py", line 390, in <genexpr>
    metafiles = set(r["metafiles"][0 if r["passthrough"] else version] for r in results)
IndexError: list index out of range
Select particles/mics based on defocus change?
#2

If I modify the script, adding job0 which is a homogeneous refinement job to see if it is the job type that is the issue, I get a different error (and I have confirmed that the job is definitely in the specified project, in the same workspace as the other two jobs):

Script:

from cryosparc.tools import CryoSPARC
import numpy as np

cs = CryoSPARC(host="c112384", base_port=39000, license="xxx" ,email="xxx", password="xxx")
assert cs.test_connection()
project = cs.find_project("P23")
job0 = cs.find_job("P23", "J426")
job1 = cs.find_job("P23", "J434")
job2 = cs.find_job("P23", "J448")
particle_set0 = job0.load_output("particles")
particle_set1 = job1.load_output("particles")
particle_set2 = job2.load_output("particles")
particle_set1['ctf/def_abs'] = (particle_set1['ctf/df1_A']+particle_set1['ctf/df1_A'])/2
particle_set2['ctf/def_abs'] = (particle_set2['ctf/df1_A']+particle_set2['ctf/df1_A'])/2
defocus_change = abs(particle_set1['ctf/def_abs']-particle_set2['ctf/def_abs'])
mask = defocus_change >= 1000
defocus_change = defocus_change[mask]
particle_set2['ctf/def_change'] = defocus_change
project.save_external_result(
  workspace_uid = "W4",
  dataset = particle_set2,
  type = "particle",
  name = "defocus_change_particles",
  slots = ["ctf"],
  passsthrough = (job1.uid, "particles"),
  title = "particles with changed defocus",
)

Error:

(cryosparc_tools) user@c112384:~$ python3 select_defocus_change.py
Connection succeeded to CryoSPARC command_core at http://c112384:39002
Connection succeeded to CryoSPARC command_vis at http://c112384:39003
Connection succeeded to CryoSPARC command_rtp at http://c112384:39005
*** CommandClient: (http://c112384:39003/get_project_file) HTTP Error 422 UNPROCESSABLE ENTITY; please check cryosparcm log command_vis for additional information.
Response from server: b'Invalid file path J426/J426_passthrough_particles.cs; must be in project directory for P23'
Traceback (most recent call last):
  File "select_defocus_change.py", line 10, in <module>
    particle_set0 = job0.load_output("particles")
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/job.py", line 391, in load_output
    datasets = [self.cs.download_dataset(self.project_uid, f) for f in metafiles]
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/job.py", line 391, in <listcomp>
    datasets = [self.cs.download_dataset(self.project_uid, f) for f in metafiles]
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/tools.py", line 595, in download_dataset
    with self.download(project_uid, path_rel) as response:
  File "/usr/local/envs/cryosparc_tools/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/command.py", line 191, in make_request
    raise CommandClient.Error(client, error_reason, url=url)
cryosparc.command.Error: *** CommandClient: (http://c112384:39003/get_project_file) HTTP Error 422 UNPROCESSABLE ENTITY; please check cryosparcm log command_vis for additional information.
Response from server: b'Invalid file path J426/J426_passthrough_particles.cs; must be in project directory for P23'
#3

@olibclarke the list index out of range error is happening because one of the output slots is missing a dataset file entry. This typically happens when a job did not fully complete. Could you DM me the job document from P23-J434 > Metadata > Data > Copy JSON? As a workaround, you may be able to specify only the slots you need instead of every particles slot, e.g.,

particle_set1 = job1.load_output("particles", slots=["ctf"])
particle_set2 = job2.load_output("particles", slots=["ctf"])

The second error is likely happening because the project directly is a symbolic link. This is a CryoSPARC bug that will be fixed in the next release. To confirm, could you enter the following terminal commands and send me the output?

cryosparcm cli "get_project_dir_abs('P23')"
readlink -f $(cryosparcm cli "get_project_dir_abs('P23')")

One last thing, note that your script has an extra s in the passthrough argument (should be passthrough = ..., not passsthrough = ...)

1 Like
#4

Thanks @nfrasser!! DMed you the requested info :grin:

#5

If I use the workaround (just specifying CTF slot), then all particle sets now fail with the other error (which as you say I think is due to the project DIR being a symlink):

(cryosparc_tools) user@c112384:~$ python3 select_defocus_change.py
Connection succeeded to CryoSPARC command_core at http://c112384:39002
Connection succeeded to CryoSPARC command_vis at http://c112384:39003
Connection succeeded to CryoSPARC command_rtp at http://c112384:39005
*** CommandClient: (http://c112384:39003/get_project_file) HTTP Error 422 UNPROCESSABLE ENTITY; please check cryosparcm log command_vis for additional information.
Response from server: b'Invalid file path J434/J434_passthrough_particles.cs; must be in project directory for P23'
Traceback (most recent call last):
  File "select_defocus_change.py", line 9, in <module>
    particle_set1 = job1.load_output("particles",slots=['ctf'])
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/job.py", line 391, in load_output
    datasets = [self.cs.download_dataset(self.project_uid, f) for f in metafiles]
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/job.py", line 391, in <listcomp>
    datasets = [self.cs.download_dataset(self.project_uid, f) for f in metafiles]
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/tools.py", line 595, in download_dataset
    with self.download(project_uid, path_rel) as response:
  File "/usr/local/envs/cryosparc_tools/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/command.py", line 191, in make_request
    raise CommandClient.Error(client, error_reason, url=url)
cryosparc.command.Error: *** CommandClient: (http://c112384:39003/get_project_file) HTTP Error 422 UNPROCESSABLE ENTITY; please check cryosparcm log command_vis for additional information.
Response from server: b'Invalid file path J434/J434_passthrough_particles.cs; must be in project directory for P23'