Updated from 3.3.2 to 4.1.1 and now cannot start cryoSPARC -- getting EADDRINUSE in logs

Billy · December 22, 2022, 5:16pm

I updated our cryoSPARC v3.3.2 to 4.1.1 this morning. It said it was successful. However, when I start cryoSPARC and then look at the ‘status’, for “app” it says “EXITED”. Looking at the logs for “app” (‘cryosparcm log app’) it is saying port 39000 is already in use:

------------------8<—Begin-Cut-Here—8<------------------
2022-12-22 11:40:26,127 INFO | DDP port: 39006
2022-12-22 11:40:26,130 INFO | /websocket proxy target: http://hostname.redacted.emory.edu:39006
[HPM] Proxy created: /websocket → http://hostname.redacted.emory.edu:39006
2022-12-22 11:40:26,208 INFO | [MongoDB] Connected
2022-12-22 11:40:26,210 INFO | [StatsManager:initialized]
2022-12-22 11:40:26,210 INFO | [StatsManager:start]
2022-12-22 11:40:26,211 INFO | [StatsManager:getJobTypesBySection]
2022-12-22 11:40:26,213 INFO | [ReportsManager:initialized]
2022-12-22 11:40:26,213 INFO | [MigrationManager:initialized]
2022-12-22 11:40:26,213 INFO | [MigrationManager:start]
2022-12-22 11:40:26,214 INFO | [MigrationManager:migrateNotes]
2022-12-22 11:40:26,294 INFO | [StatsManager:allProjects]
2022-12-22 11:40:26,298 INFO | [StatsManager:allWorkspaces]
2022-12-22 11:40:26,378 ERROR | uncaughtException: listen EADDRINUSE: address already in use :::39000
2022-12-22 11:40:26,378 ERROR | Error: listen EADDRINUSE: address already in use :::39000
2022-12-22 11:40:26,378 ERROR | at Server.setupListenHandle [as _listen2] (node:net:1432:16)
2022-12-22 11:40:26,378 ERROR | at listenInCluster (node:net:1480:12)
2022-12-22 11:40:26,378 ERROR | at Server.listen (node:net:1568:7)
2022-12-22 11:40:26,378 ERROR | at pb.listen (/data/home/cryosparcuser/cryosparc/cryosparc_master/cryosparc_app/custom-server/dist/server/index.js:890:2244)
2022-12-22 11:40:26,378 ERROR | at RSr (/data/home/cryosparcuser/cryosparc/cryosparc_master/cryosparc_app/custom-server/dist/server/index.js:908:2735)
2022-12-22 11:40:26,378 ERROR | at processTicksAndRejections (node:internal/process/task_queues:96:5)
-------------------8<—End-Cut-Here—8<-------------------

However, I can find nothing using that port:

------------------8<—Begin-Cut-Here—8<------------------
(base) lsof -i tcp:39000
(base) lsof -i :39000
(base) fuser 39000/tcp
-------------------8<—End-Cut-Here—8<-------------------

Searching, I found this topic ‘Web app startup error: listen EADDRINUSE’ in the forums which suggested looking for stray processes, and deleting the “socket” file in /tmp but I do not have any:

------------------8<—Begin-Cut-Here—8<------------------
(base) ps -ax | grep “supervisord”
29450 pts/0 S+ 0:00 grep --color=auto supervisord
(base) ps -ax | grep “crysosparc”
29475 pts/0 S+ 0:00 grep --color=auto crysosparc
(base) ps -ax | grep “mongod”
29506 pts/0 S+ 0:00 grep --color=auto mongod
(base) ls -l /tmp/cryo*
ls: cannot access /tmp/cryo*: No such file or directory
-------------------8<—End-Cut-Here—8<-------------------

I WILL say that before I did the update, I checked the status (‘cryosparcm status’) and it said it was NOT running. I took it at face value that it wasn’t lying to me, and tried to do a backup of the database before the upgrade. I got an error ‘Database: ERROR (spawn error)’ and in searching for that, it was again suggested that there were stray processes running – which there was 1 at that time (found with ‘ps -ax | grep supervisord’) that I killed. From there, the database backup and upgrade said they completed successfully.

I’m kind of at a loss where to go from here. Help!

wtempel · December 22, 2022, 11:22pm

“It” probably wasn’t lying to you, but “it” may not have “known better” . Under certain circumstances,
cryosparcm status may “miss” running processes.
Please can you post the output of
ps axuww | grep -e cryosparc -e mongo
and
ss -anp '( sport = :39000 or dport = :39000 )'
The latter command may require elevated privileges for complete output. If there is no output and if you can, try prefixing the command with sudo.
Could another user be running a CryoSPARC server on the same machine?

Billy · December 23, 2022, 12:48am

Yes, after I posted that I found this topic – “cryoSPARC is surely running but cryosparcm status says not” – So I think in the future I will either double-check via web browser that it is not running, or move that “sock” file to someplace else besides /tmp , as I also found a couple of our other systems where ‘cryosparcm status’ says it is not running, yet I can login to the web interface… :-/

The output to your questions:

------------------8<—Begin-Cut-Here—8<------------------

(base) ps axuww | grep -e cryosparc -e mongo
root        944  0.0  0.0 163252  6024 ?        Ss   09:27   0:00 sshd: cryosparcuser [priv]
root        996  0.0  0.0 163252  6024 ?        Ss   09:27   0:00 sshd: cryosparcuser [priv]
cryospa+    999  0.0  0.0 163672  3236 ?        S    09:27   0:03 sshd: cryosparcuser@notty
cryospa+   1019  0.0  0.0 163252  2476 ?        S    09:27   0:00 sshd: cryosparcuser@notty
root     108139  0.0  0.0 165384  6076 ?        Ss   19:32   0:00 sshd: cryosparcuser [priv]
root     108173  0.0  0.0 165384  6068 ?        Ss   19:32   0:00 sshd: cryosparcuser [priv]
cryospa+ 108177  0.0  0.0 165660  2732 ?        S    19:32   0:00 sshd: cryosparcuser@pts/4
cryospa+ 108180  0.0  0.0 165384  2404 ?        S    19:32   0:00 sshd: cryosparcuser@notty
cryospa+ 109054  0.0  0.0 112812   976 pts/4    S+   19:36   0:00 grep --color=auto -e cryosparc -e mongo

-------------------8<—End-Cut-Here—8<-------------------

And

------------------8<—Begin-Cut-Here—8<------------------

[mysudouser@hostname ~]$ sudo ss -anp '( sport = :39000 or dport = :39000 )'
[sudo] password for mysudouser:
Netid State      Recv-Q Send-Q                          Local Address:Port                                         Peer Address:Port
tcp   LISTEN     0      128                                     [::1]:39000                                                [::]:*                   users:(("ssh",pid=188759,fd=4))
[mysudouser@hostname ~]$

-------------------8<—End-Cut-Here—8<-------------------

No, nobody else is running a cryoSPARC instance on this system.

wtempel · December 23, 2022, 2:39pm

Interesting.
Please can you check if that process is still using this port:

sudo ss -anp '( sport = :39000 or dport = :39000 )'

and, if it is, find out more about it (assuming the pid has not changed, substitute new pid otherwise):

ps uxww 188759

and

ps ww --ppid 188759

wtempel · December 23, 2022, 3:00pm

For situations where the absence of active CryoSPARC processes is critical, I recommend:

ps axuww | grep -e cryosparc -e mongo

Note that this listing may include processes unrelated to the specific CryoSPARC instance and that should not be killed, such as those related to:

other CryoSPARC instances
processes unrelated to CryoSPARC of a user whose username includes cryosparc as a substring
mongod processes not related to CryoSPARC

Billy · December 23, 2022, 6:27pm

Interesting. It looks like ‘someuser’ is port forwarding ssh using 39000!

------------------8<—Begin-Cut-Here—8<------------------

[mysudouser@hostname ~]$ ps uxww 188759
USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
mysudouser   21078  0.0  0.0 165688  2804 ?        S    13:00   0:00 sshd: mysudouser@pts/0
mysudouser   21079  0.1  0.0 116892  3436 pts/0    Ss   13:00   0:00 -bash
mysudouser   21115  0.0  0.0 165384  2416 ?        S    13:00   0:00 sshd: mysudouser@notty
mysudouser   21128  0.0  0.0  72288  2872 ?        Ss   13:00   0:00 /usr/libexec/openssh/sftp-server
mysudouser   21420  0.0  0.0 155448  1852 pts/0    R+   13:01   0:00 ps uxww 188759
mysudouser   25683  0.0  0.0 163556  2956 ?        S    Dec22   0:01 sshd: mysudouser@pts/2
mysudouser   25685  0.0  0.0 116892  3436 pts/2    Ss+  Dec22   0:00 -bash
mysudouser   25705  0.0  0.0 163252  2468 ?        S    Dec22   0:00 sshd: mysudouser@notty
mysudouser   25714  0.0  0.0  72288  2880 ?        Ss   Dec22   0:00 /usr/libexec/openssh/sftp-server
someuser    188759  0.0  0.0 187408  2840 ?        Ss    2021   5:21 ssh -N -f -L localhost:39000:localhost:39000 hostname.redacated.emory.edu

[mysudouser@hostname ~]$ ps ww --ppid 18759
   PID TTY      STAT   TIME COMMAND

[mysudouser@hostname ~]$

-------------------8<—End-Cut-Here—8<-------------------

And after killing off that process by ‘someuser’, cryoSPARC now SUCCESSFULLY starts! Yay. Thank you for your help, and teaching me a few more troubleshooting tricks!