Error loading particles using cryosparc-tools

olibclarke · March 19, 2023, 3:27pm

Hi,

I am trying to load a particle set in a cryposparc-tools script. The script is incomplete, but I am trying to debug what I have so far.

When I run it (python3 select_defocus_change.py) it dies with an error (attached) at the point where I try to load the first particle set. J434 is a Homogeneous Reconstruction job, so it has a particles output group.

Script:

from cryosparc.tools import CryoSPARC

cs = CryoSPARC(host="c112384", base_port=39000, license="xxx" ,email="xxx", password="xxx")
assert cs.test_connection()

project = cs.find_project("P23")
job1 = cs.find_job("P23", "J434")
job2 = cs.find_job("P23", "J448")
particle_set1 = job1.load_output("particles")
particle_set2 = job2.load_output("particles")
particle_set1['ctf/def_abs'] = (particle_set1['ctf/df1_A']+particle_set1['ctf/df1_A'])/2
particle_set2['ctf/def_abs'] = (particle_set2['ctf/df1_A']+particle_set2['ctf/df1_A'])/2
defocus_change = abs(particle_set1['ctf/def_abs']-particle_set2['ctf/def_abs'])
particle_set2['ctf/def_change'] = defocus_change

project.save_external_result(
  workspace_uid = "W4",
  dataset = particle_set2,
  type = "particle",
  name = "defocus_change_particles",
  slots = ["ctf"],
  passsthrough = (job1.uid, "particles"),
  title = "particles with changed defocus",
)

Error:

user@c112384:~$ python3 select_defocus_change.py
Connection succeeded to CryoSPARC command_core at http://c112384:39002
Connection succeeded to CryoSPARC command_vis at http://c112384:39003
Connection succeeded to CryoSPARC command_rtp at http://c112384:39005
Traceback (most recent call last):
  File "select_defocus_change.py", line 9, in <module>
    particle_set1 = job1.load_output("particles")
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/job.py", line 390, in load_output
    metafiles = set(r["metafiles"][0 if r["passthrough"] else version] for r in results)
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/job.py", line 390, in <genexpr>
    metafiles = set(r["metafiles"][0 if r["passthrough"] else version] for r in results)
IndexError: list index out of range

olibclarke · March 20, 2023, 1:21am

If I modify the script, adding job0 which is a homogeneous refinement job to see if it is the job type that is the issue, I get a different error (and I have confirmed that the job is definitely in the specified project, in the same workspace as the other two jobs):

Script:

from cryosparc.tools import CryoSPARC
import numpy as np

cs = CryoSPARC(host="c112384", base_port=39000, license="xxx" ,email="xxx", password="xxx")
assert cs.test_connection()
project = cs.find_project("P23")
job0 = cs.find_job("P23", "J426")
job1 = cs.find_job("P23", "J434")
job2 = cs.find_job("P23", "J448")
particle_set0 = job0.load_output("particles")
particle_set1 = job1.load_output("particles")
particle_set2 = job2.load_output("particles")
particle_set1['ctf/def_abs'] = (particle_set1['ctf/df1_A']+particle_set1['ctf/df1_A'])/2
particle_set2['ctf/def_abs'] = (particle_set2['ctf/df1_A']+particle_set2['ctf/df1_A'])/2
defocus_change = abs(particle_set1['ctf/def_abs']-particle_set2['ctf/def_abs'])
mask = defocus_change >= 1000
defocus_change = defocus_change[mask]
particle_set2['ctf/def_change'] = defocus_change
project.save_external_result(
  workspace_uid = "W4",
  dataset = particle_set2,
  type = "particle",
  name = "defocus_change_particles",
  slots = ["ctf"],
  passsthrough = (job1.uid, "particles"),
  title = "particles with changed defocus",
)

Error:

(cryosparc_tools) user@c112384:~$ python3 select_defocus_change.py
Connection succeeded to CryoSPARC command_core at http://c112384:39002
Connection succeeded to CryoSPARC command_vis at http://c112384:39003
Connection succeeded to CryoSPARC command_rtp at http://c112384:39005
*** CommandClient: (http://c112384:39003/get_project_file) HTTP Error 422 UNPROCESSABLE ENTITY; please check cryosparcm log command_vis for additional information.
Response from server: b'Invalid file path J426/J426_passthrough_particles.cs; must be in project directory for P23'
Traceback (most recent call last):
  File "select_defocus_change.py", line 10, in <module>
    particle_set0 = job0.load_output("particles")
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/job.py", line 391, in load_output
    datasets = [self.cs.download_dataset(self.project_uid, f) for f in metafiles]
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/job.py", line 391, in <listcomp>
    datasets = [self.cs.download_dataset(self.project_uid, f) for f in metafiles]
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/tools.py", line 595, in download_dataset
    with self.download(project_uid, path_rel) as response:
  File "/usr/local/envs/cryosparc_tools/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/command.py", line 191, in make_request
    raise CommandClient.Error(client, error_reason, url=url)
cryosparc.command.Error: *** CommandClient: (http://c112384:39003/get_project_file) HTTP Error 422 UNPROCESSABLE ENTITY; please check cryosparcm log command_vis for additional information.
Response from server: b'Invalid file path J426/J426_passthrough_particles.cs; must be in project directory for P23'

nfrasser · March 20, 2023, 3:58pm

@olibclarke the list index out of range error is happening because one of the output slots is missing a dataset file entry. This typically happens when a job did not fully complete. Could you DM me the job document from P23-J434 > Metadata > Data > Copy JSON? As a workaround, you may be able to specify only the slots you need instead of every particles slot, e.g.,

particle_set1 = job1.load_output("particles", slots=["ctf"])
particle_set2 = job2.load_output("particles", slots=["ctf"])

The second error is likely happening because the project directory* is a symbolic link. This is a CryoSPARC bug that will be fixed in the next release. To confirm, could you enter the following terminal commands and send me the output?

cryosparcm cli "get_project_dir_abs('P23')"
readlink -f $(cryosparcm cli "get_project_dir_abs('P23')")

One last thing, note that your script has an extra s in the passthrough argument (should be passthrough = ..., not passsthrough = ...)

olibclarke · March 20, 2023, 4:09pm

Thanks @nfrasser!! DMed you the requested info

olibclarke · March 20, 2023, 4:12pm

If I use the workaround (just specifying CTF slot), then all particle sets now fail with the other error (which as you say I think is due to the project DIR being a symlink):

(cryosparc_tools) user@c112384:~$ python3 select_defocus_change.py
Connection succeeded to CryoSPARC command_core at http://c112384:39002
Connection succeeded to CryoSPARC command_vis at http://c112384:39003
Connection succeeded to CryoSPARC command_rtp at http://c112384:39005
*** CommandClient: (http://c112384:39003/get_project_file) HTTP Error 422 UNPROCESSABLE ENTITY; please check cryosparcm log command_vis for additional information.
Response from server: b'Invalid file path J434/J434_passthrough_particles.cs; must be in project directory for P23'
Traceback (most recent call last):
  File "select_defocus_change.py", line 9, in <module>
    particle_set1 = job1.load_output("particles",slots=['ctf'])
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/job.py", line 391, in load_output
    datasets = [self.cs.download_dataset(self.project_uid, f) for f in metafiles]
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/job.py", line 391, in <listcomp>
    datasets = [self.cs.download_dataset(self.project_uid, f) for f in metafiles]
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/tools.py", line 595, in download_dataset
    with self.download(project_uid, path_rel) as response:
  File "/usr/local/envs/cryosparc_tools/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/exx/.local/lib/python3.8/site-packages/cryosparc/command.py", line 191, in make_request
    raise CommandClient.Error(client, error_reason, url=url)
cryosparc.command.Error: *** CommandClient: (http://c112384:39003/get_project_file) HTTP Error 422 UNPROCESSABLE ENTITY; please check cryosparcm log command_vis for additional information.
Response from server: b'Invalid file path J434/J434_passthrough_particles.cs; must be in project directory for P23'

gabor · March 28, 2023, 12:18pm

Dear both,

I have the same problem with the symlinks. A quick workaround was that I changed the project path to the real path removing the symlink. However, I don’t know yet whether this will result in something else not to work.
Gabor

wtempel · March 28, 2023, 2:46pm

Am I understanding correctly:
You first (using what method?)

then removed the symbolic link using rm?

gabor · March 28, 2023, 3:06pm

Sorry, I didn’t detailed it - I used the
cryosparcm cli “update_project(‘PXX’, {‘project_dir’ : ‘/real/path/to/project’})”
command to update the project directory with a path without symlinks.

nfrasser · August 9, 2023, 3:20pm

Hello all, this issue has been fixed in the latest CryoSPARC v4.3 and corresponding cryosparc-tools release. Accessing project directories that are symbolic links no longer fails with the “Invalid file path” error.

After updating CryoSPARC, use this command in your Python environment to update to the latest tools version:

pip install -U cryosparc-tools