Problem with ssh tunneling

Install, Configure and Update
1

My remote connection to the cryosparc web server from my home laptop has stopped working. I am using the ssh tunneling command:

ssh -N -f -L localhost:39000:localhost:39000 @<remote_hostname>
and
http://localhost:39000

and the tunneling seems to be active (see below). However, now I am getting the message that the page isn’t working and localhost didn’t send any data.

I don’t understand why it isn’t working anymore. The only thing is that the connection tends to drop and I need to reissue the tunneling command a few times a day. I’d be grateful for any advice.

Luca

% lsof -i :39000
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
Google 3673 lp212 24u IPv4 0x241c5c77be2f82cf 0t0 TCP user-32-41.vpn.net.private.cam.ac.uk:53836->lp212-pc13238.bioc.cam.ac.uk:39000 (SYN_SENT)
ssh 3686 lp212 5u IPv6 0x241c5c77c0f827b7 0t0 TCP localhost:39000 (LISTEN)
ssh 3686 lp212 6u IPv4 0x241c5c77be2fe19f 0t0 TCP localhost:39000 (LISTEN)

% nc -z localhost 39000
Connection to localhost port 39000 [tcp/*] succeeded!

2

@Luca Did you already

  • check the output of this command on your local laptop
    curl localhost:39000
  • confirm that the CryoSPARC page is up on <remote_hostname> by running that same command on the CryoSPARC master host
    curl localhost:39000
  • try terminating and re-establishing the ssh channel
  • try forwarding (and pointing your browser to) a different local port
    ssh -L 61000:localhost:39000 remoteuser@remotehost
3

Hi @wtempel, thank you for your help.

On both laptop and remote host (with cryoSPARC running as checked with ‘cryosparcm status’), when I run the curl command I get: “curl: (52) Empty reply from server”.

I tried re-establishing the ssh tunnel several times, and I always get ‘connection refused’. Same when I try the 61000 port.

Luca

4

There might be a problem independent from the ssh tunnel.
Please can you run these commands on the remote host and post all terminal contents:

cryosparcm status | grep -v CRYOSPARC_LICENSE
curl localhost:39000 # as before
5

Here it is:

Last login: Fri Jan 5 13:44:49 2024 from 172.16.32.197
cryosparc_user@lp212-System-Product-Name:~$ cryosparcm status | grep -v CRYOSPARC_LICENCE

CryoSPARC System master node installed at
/home/cryosparc_user/software/cryosparc/cryosparc2_master
Current cryoSPARC version: v4.4.1

CryoSPARC process status:

app EXITED Dec 25 08:55 PM
app_api RUNNING pid 28111, uptime 10 days, 18:45:07
app_api_dev STOPPED Not started
command_core RUNNING pid 27982, uptime 10 days, 18:45:21
command_rtp RUNNING pid 28054, uptime 10 days, 18:45:13
command_vis RUNNING pid 28026, uptime 10 days, 18:45:14
database RUNNING pid 27859, uptime 10 days, 18:45:25

License is valid

global config variables:
export CRYOSPARC_LICENSE_ID=“XXXXXX”
export CRYOSPARC_MASTER_HOSTNAME=“lp212-System-Product-Name”
export CRYOSPARC_DB_PATH=“/home/cryosparc_user/software/cryosparc/cryosparc2_database”
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true

cryosparc_user@lp212-System-Product-Name:~$ curl localhost:39000
curl: (52) Empty reply from server

6

app seems not to be running. Are there any indications of a reason inside the app or supervisord log?
cryosparcm log app
and
cryosparcm log supervisord | grep '2023-12-25'
?

7

cryosparcm log app:

2023-12-25 20:55:43,088 ERROR | at Vb.listen (/home/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_app/custom-server/dist/server/index.js:891:2244)

2023-12-25 20:55:43,088 ERROR | at A5r (/home/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_app/custom-server/dist/server/index.js:909:2735)

2023-12-25 20:55:43,088 ERROR | at processTicksAndRejections (node:internal/process/task_queues:96:5)

2023-12-25 20:55:43,108 INFO | [StatsManager:singleProject] P13
2023-12-25 20:55:43,120 INFO | [StatsManager:singleProject] P21
2023-12-25 20:55:43,130 INFO | [StatsManager:singleProject] P27
2023-12-25 20:55:43,141 INFO | [StatsManager:singleProject] P29
2023-12-25 20:55:43,151 INFO | [StatsManager:singleProject] P30
2023-12-25 20:55:43,160 INFO | [StatsManager:singleProject] P32
2023-12-25 20:55:43,191 INFO | [StatsManager:singleProject] P34
2023-12-25 20:55:43,200 INFO | [StatsManager:singleProject] P36
2023-12-25 20:55:43,212 INFO | [StatsManager:singleProject] P37
2023-12-25 20:55:43,221 INFO | [StatsManager:singleProject] P38
Computed stats for 11 projects 249.947 ms
2023-12-25 20:55:43,245 INFO | [StatsManager:allWorkspaces]
2023-12-25 20:55:43,258 INFO | [StatsManager:singleWorkspace] P8 W1
2023-12-25 20:55:43,275 INFO | [StatsManager:singleWorkspace] P8 W2
2023-12-25 20:55:43,284 INFO | [StatsManager:singleWorkspace] P8 W3
2023-12-25 20:55:43,293 INFO | [StatsManager:singleWorkspace] P8 W4
2023-12-25 20:55:43,304 INFO | [StatsManager:singleWorkspace] P13 W1
2023-12-25 20:55:43,311 INFO | [StatsManager:singleWorkspace] P21 W1
2023-12-25 20:55:43,317 INFO | [StatsManager:singleWorkspace] P27 W1
2023-12-25 20:55:43,325 INFO | [StatsManager:singleWorkspace] P29 W3
2023-12-25 20:55:43,333 INFO | [StatsManager:singleWorkspace] P30 W2
2023-12-25 20:55:43,341 INFO | [StatsManager:singleWorkspace] P32 W1
2023-12-25 20:55:43,357 INFO | [StatsManager:singleWorkspace] P34 W1
2023-12-25 20:55:43,366 INFO | [StatsManager:singleWorkspace] P32 W2
2023-12-25 20:55:43,386 INFO | [StatsManager:singleWorkspace] P32 W4
2023-12-25 20:55:43,410 INFO | [StatsManager:singleWorkspace] P32 W5
2023-12-25 20:55:43,435 INFO | [StatsManager:singleWorkspace] P36 W1
2023-12-25 20:55:43,444 INFO | [StatsManager:singleWorkspace] P32 W8
2023-12-25 20:55:43,456 INFO | [StatsManager:singleWorkspace] P32 W9
2023-12-25 20:55:43,469 INFO | [StatsManager:singleWorkspace] P38 W1
Computed stats for 18 workspaces 241.887 ms
Waiting for data… (interrupt to abort)

cryosparcm log supervisord | grep ‘2023-12-25’:
2023-12-25 20:55:03,956 INFO waiting for app_api to stop
2023-12-25 20:55:03,956 INFO waiting for command_core to stop
2023-12-25 20:55:03,956 INFO waiting for command_rtp to stop
2023-12-25 20:55:03,956 INFO waiting for command_vis to stop
2023-12-25 20:55:03,956 INFO waiting for database to stop
2023-12-25 20:55:03,967 WARN stopped: app_api (terminated by SIGTERM)
2023-12-25 20:55:04,033 WARN stopped: command_rtp (terminated by SIGQUIT (core dumped))
2023-12-25 20:55:04,047 WARN stopped: command_core (terminated by SIGQUIT (core dumped))
2023-12-25 20:55:04,064 WARN stopped: command_vis (terminated by SIGQUIT (core dumped))
2023-12-25 20:55:04,305 INFO stopped: database (exit status 0)
2023-12-25 20:55:22,141 INFO RPC interface ‘supervisor’ initialized
2023-12-25 20:55:22,141 CRIT Server ‘unix_http_server’ running without any HTTP authentication checking
2023-12-25 20:55:22,143 INFO daemonizing the supervisord process
2023-12-25 20:55:22,143 INFO supervisord started with pid 27739
2023-12-25 20:55:26,986 INFO spawned: ‘database’ with pid 27859
2023-12-25 20:55:28,798 INFO success: database entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-25 20:55:30,762 INFO spawned: ‘command_core’ with pid 27982
2023-12-25 20:55:36,625 INFO success: command_core entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2023-12-25 20:55:37,745 INFO spawned: ‘command_vis’ with pid 28026
2023-12-25 20:55:38,747 INFO success: command_vis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-25 20:55:38,891 INFO spawned: ‘command_rtp’ with pid 28054
2023-12-25 20:55:39,893 INFO success: command_rtp entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-25 20:55:42,539 INFO spawned: ‘app’ with pid 28093
2023-12-25 20:55:44,489 INFO success: app entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-25 20:55:44,663 INFO spawned: ‘app_api’ with pid 28111
2023-12-25 20:55:46,116 INFO success: app_api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-12-25 20:55:46,116 WARN exited: app (exit status 1; not expected)

8

Thanks for posting these log entries.
Please can you try

  • shutting down CryoSPARC completely, then
  • starting CryoSPARC again, then
  • confirming that, on the master host, output of the command
    curl localhost:39000
    includes the line
    <title>CryoSPARC</title>
  • reestablishing the ssh tunnel

Do these steps resolve the issue?

9

@wtempel thanks again for your help.
I stopped and started again cryosparc (with cryosparc stop/start), after which ‘curl localhost :3900’ did show the line: CryoSPARC. However, when I tried to reestablish the ssh tunnel, I got:

bind: Address already in use
channel_setup_fwd_listener_tcpip: cannot listen to port: 39000
Could not request local forwarding.

I killed the process using port 39000 on the host, and this time the ssh tunneling worked (checked with lsof -i:39000). However, when I tried to connect to cryoSPARC web interface, the host refused connection.

If I repeat the process of stopping and starting cryosparc, curl localhost:39000 gives:

curl: (52) Empty reply from server
and then:
curl: (7) Failed to connect to localhost port 39000: Connection refused

10

@Luca Please can you check whether the web app has failed again: On the remote server, what are the outputs of the commands

free -g # enough RAM?
cryosparcm status | grep -v LICENSE
curl localhost:39000
grep app /home/cryosparc_user/software/cryosparc/cryosparc2_master/run/supervisord.log