Cryosparctools failed to connect

Scripting
1

I have used cryosparctools before and it worked well. Recently, we reinstalled all the cryosparctools and cryosparc (still same license-ID) but somehow we have problem to connect.

The test script is:

from cryosparc.tools import CryoSPARC

cs = CryoSPARC(
    license="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    host="localhost",
    base_port=39000,
    email="ali@example.com",
    password="password123"
)

It worked before with the correct license, host, base_port, email and password info.

Now, when I run it, after a while, it complains:

**data/user/.conda/envs/cryosparctools/lib/python3.9/site-packages/cryosparc/command.py:135: UserWarning: *** CommandClient: command (http://cajal:61003/api) did not reply within timeout of 300 seconds, attempt 1 of 3**

2

Please elaborate:

  1. Do you define the host parameter with "localhost" or "cajal"?
  2. Are you running the script on the CryoSPARC master computer, or another computer?
  3. Does the computer where the script runs have continuous access to the relevant ports on the CryoSPARC master
    i. directly
    ii. via port forwarding or similar
  4. Does the script only “complain” after a while, but not immediately?
  5. What does the script do, or is supposed to do, between when it starts and when it complains?
  6. What is the output (on the computer where the script runs) of the command
    curl cajal:61003
  7. Are there any errors in the command_vis log. You can browse the log on CryoSPARC master host with the command
    cryosparcm log command_vis | less
3
  1. I put in cajal, which worked before.
  2. I ran it on the master computer.
  3. I assumed yes since I could connect and run scripts a few months ago. There should not be a drastic change in port. But I have to say I don’t know.
  4. It only complained after 5-6 minutes but not immediately. It probably corresponds to the 300 s time out.
  5. The main script is for symmetry_expand_filter but I ran a test script, which only contained the portion to connect to the cryosparc instance to test first. I did this in the past and it worked well.
  6. Apparently, it took at least 10 mins without any response.
  7. There is no error with the command_vis log that is related to any recent events.
4

In this case, please can you try the commands

hostname
host cajal
curl localhost:61003

and post the outputs.

5

Sorry for the delay. Here’s the answer

$hostname
cajal
$host cajal
cajal.campus.mcgill.ca has address 132.xxx.xx.xxx
$curl localhost:61003
Doesn’t response for at least 5 minutes

What should I ask my admin for open up port for cryosparc tools?

6

There are several possible explanations, such as

  • a firewall may be blocking the port, but I think this is unlikely in this case. What is the output of the command
    host localhost
    ?
  • CryoSPARC may not be running/
  • CryoSPARC may be running, but on a different port range

If you post the output of these commands (run on cajal), we may be able to say more:

cryosparcm status | grep -v LICENSE
ps -eo pid,ppid,start,command | grep cryosparc_
7

There are some ports blocked for sure. I asked the sys admin, and he enabled 61003 but cryosparctool still did not connect.

$host localhost
localhost has address 127.0.0.1

$cryosparcm status | grep -v LICENSE
Doesn't work

So I located the cryosparcm location and type:

$/data2/apps/cryosparc/master/bin/cryosparcm status | grep -v LICENSE
ERROR: This command can only be called by the owner of this script: cryosparc. You are xxxxxx!
Alternatively, set CRYOSPARC_FORCE_USER=true in cryosparc_master/config.sh to suppress this error.

That was run after the admin enabled the port.

curl localhost:61003

Still did not respond like before.

But if I ran (61000 is the default on used for port forwarding in our case instead of the default 39000)

curl localhost:61000

It responded immediately.

8

Please can you provide details how port forwarding is used in the current scenario, showing the full command(s) and describing the computers involved.

If the cryosparc-tools script is to be run on the CryoSPARC master computer, no port forwarding should be necessary to run the script.
Within
cs = CryoSPARC(...),
host= could be defined as "localhost" and base_port= could be defined using the value of the CRYOSPARC_BASE_PORT variable of the CryoSPARC instance.

Please can you re-run the cryosparcm command under the Linux account under which CryoSPARC was installed and also post the outputs of the commands

ps -eo pid,ppid,start,command | grep cryosparc_
curl localhost:61000
9

I use this command to connect:

ssh -N -f -L localhost:9999:localhost:61000 xxx@cajal.campus.mcgill.ca

To connect using cryosparctools, I logged in to cajal

ssh xxx@cajal.campus.mcgill.ca

and run either the Python script directly or through Jupyter Notebook.

I ran the script and within

cs = CryoSPARC(...)

with host = “localhost”, base_port=61000

It didn’t connect. Basically, it was stuck at the step of connecting.
I can ask the admin to run it again to see but I can ask him to open the port. But I need more information to give him since port 61000 is already open.

10

Apparently, after asking the sys admin to restart cryosparc, everything works. So I guess it is the easiest way to try to fix the problem. Thanks a lot for your help.

11

@builab Another user reported a problem that may be related to the problem you reported. Please can you try suggestions in Volume viewer error - #3 by wtempel and let us know whether that setting helps in your case?

12

Thanks @wtempel . I was able to connect again after restarting a few days. We use 4.5.1 though.

13

The GUNICORN_CMD_ARGS= suggestion would apply to v4.5.1 also, in case you encounter the issue again.

14

Patch 240807 for CryoSPARC v4.5.3 includes a fix for this issue.
Before applying the patch, ensure the CryoSPARC version is 4.5.3 and remove from or comment out in

/path/to/cryosparc_master/config.sh

any

export GUNICORN_CMD_ARGS=

line.
After patching, please restart CryoSPARC.