Database Exited too quickly error at startup

Troubleshooting
1

the database error occurred when starting ‘cryosparcm start’

it is displayed in the terminal under “Cryosparc process status”
database FATAL Exited too quickly (process log may have details)

“cryosparcm log database” shows four attempts at starting the database (a few ms appart):
(pasted the last out of the four startup attempts, since the log entries are identical)

"2021-09-16T13:19:47.702+0200 I CONTROL [initandlisten] MongoDB starting : pid=7115 port=39001 dbpath=/fs/home/xxxxxx/cryosparc2/cryosparc2_database 64-bit host=hpcl5006
2021-09-16T13:19:47.702+0200 I CONTROL [initandlisten] db version v3.4.10
2021-09-16T13:19:47.702+0200 I CONTROL [initandlisten] git version: 078f28920cb24de0dd479b5ea6c66c644f6326e9
2021-09-16T13:19:47.702+0200 I CONTROL [initandlisten] allocator: tcmalloc
2021-09-16T13:19:47.702+0200 I CONTROL [initandlisten] modules: none
2021-09-16T13:19:47.702+0200 I CONTROL [initandlisten] build environment:
2021-09-16T13:19:47.702+0200 I CONTROL [initandlisten] distarch: x86_64
2021-09-16T13:19:47.702+0200 I CONTROL [initandlisten] target_arch: x86_64
2021-09-16T13:19:47.702+0200 I CONTROL [initandlisten] options: { net: { port: 39001 }, replication: { oplogSizeMB: 64, replSet: “meteor” }, storage: { dbPath: “/fs/home/xxxxxx/cryosparc2/cryosparc2_database”, journal: { enabled: false }, wiredTiger: { engineConfig: { cacheSizeGB: 4.0 } } } }
2021-09-16T13:19:47.723+0200 E NETWORK [initandlisten] listen(): bind() failed Address already in use for socket: 0.0.0.0:39001
2021-09-16T13:19:47.723+0200 E NETWORK [initandlisten] addr already in use
2021-09-16T13:19:47.723+0200 E NETWORK [initandlisten] Failed to set up sockets during startup.
2021-09-16T13:19:47.723+0200 E STORAGE [initandlisten] Failed to set up listener: InternalError: Failed to set up sockets
2021-09-16T13:19:47.723+0200 I NETWORK [initandlisten] shutdown: going to close listening sockets…
2021-09-16T13:19:47.723+0200 I NETWORK [initandlisten] shutdown: going to flush diaglog…
2021-09-16T13:19:47.723+0200 I CONTROL [initandlisten] now exiting
2021-09-16T13:19:47.723+0200 I CONTROL [initandlisten] shutting down with code:48
"

there are no orphaned “supervisord”, “mongod”, “cryosparcm” processes
deleted the mongod.lock file

next I have tried restoring the database from a previous backup
cryosparcm restore --file="/fs/home/xxxxx/cryosparc2/cryosparc2_database/backup/cryosparc_backup_2021_08_09_12h14.archive"
(where username is replaced by xxxxx)

but I get the same fatal database error

the database log lists the error:
“listen(): bind() failed Address already in use for socket: 0.0.0.0:39001”

the IP 0.0.0.0 looks peculiar,
Does anyone know how to proceed?
Cryosparc is running on a server other people use as well, and scheduling a restart is not that easy.

Thanks

2

Hi @mturk, it looks like this might be process other than cryoSPARC what’s bound to the database port and preventing it from starting up. Can you check what it is with lsof like this?

lsof -i 39001
lsof -a -d txt -p <PID>

Substitute <PID> with the PID number under the output of the previous command. This will show the full path of what process is running. Then paste the output here.

If this looks like a cryoSPARC process, try terminating it with

kill <PID>

If it’s not a cryoSPARC process, determine if you can safely terminate it or move it to a different port. Otherwise, I suggest changing the base port to something else. To do this, first install the latest patch. Then open cryosparc_master/config.sh and set CRYOSPARC_BASE_PORT to a different value (e.g., 40000) that doesn’t conflict with running processes. Finally, run cryosparcm fixdbport.

Hope that helps, let me know if you need additional help.

1 Like
3

Hi @nfrasser, thank you for your help.
The port was taken by another process.

could not run “cryosparcm patch”. requires a running cryosparcm

but the other steps solved the problem

  • Changing the CRYOSPARC_BASE_PORT to a new number
  • “cryosparcm restart”
  • “cryosparcm fixdbport”

message from terminal after “cryosparc restart”

CryoSPARC is not already running.
If you would like to restart, use cryosparcm restart
Starting cryoSPARC System master process…
CryoSPARC is not already running.
database: started
ERROR: Did NOT get correct database host configuration.
Expected “localhost:39101” but instead got “localhost:39001”

To fix this, enter the following command:

xxxx/cryosparc2/cryosparc2_master/bin/cryosparcm fixdbport

Then restart cryoSPARC and check again:

xxxx/cryosparc2/cryosparc2_master/bin/cryosparcm restart
xxxx/cryosparc2/cryosparc2_master/bin/cryosparcm checkdb

thank you for your help!