CryoSPARC installation stuck at "command_core: started"

Install, Configure and Update
22

The timestamps are confusing in that the *.sock files appear to be newer than the time stamp of the grep command. Was there a gap between the ps and ls -l commands?

The presence of /tmp/mongodb-40001.sock strongly suggests a running mongod process that escaped detection.
Deletion should only be attempted if one has ensured that the *.sock files are not in use by an existing process. For an additional check before deleting a mongo socket, you may run:

fuser /tmp/mongodb-40001.sock
ps -p $(fuser /tmp/mongodb-40001.sock | awk '{print $NF}') -o user,pid,ppid,start,cmd

The fuser commands should be run by the owner of the sock file because fuser may not produce any output otherwise.

This exception explains the CryoSPARC startup failure and may have been caused by an undetected mongod process (see above). If /home/cemkguest/software/cryosparc/cryosparc_database/ is shared via the network with other computers, it is conceivable that the interfering mongod process may be running on another computer, and would not be detected by the ps command on the current computer.

At this moment, I am unconvinced a re-installation is necessary or would fix the problem.
Instead, the cause of the DBPathInUse should be determined and avoided in future CryoSPARC startup attempts.
Caution: If a mongod process needs to be terminated, never use the SIGKILL signal (aka kill -9, source).

23

Dear @wtempel - ok, it seems like we´re getting closer to the solution?

Yes, I am sorry - I ran the commands individually. After deleting the .sock files, I rebooted the computer, and re-ran the commands as a whole:

ps -eo user:16,pid,ppid,start,cmd | grep -e cryosparc_ -e mongo
ls -l /tmp/cryosparc*.sock /tmp/mongo*.sock
grep -v LICENSE /home/cemkguest/software/cryosparc/cryosparc_master/config.sh


cemkguest           5135    3548 21:11:52 python /home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/cemkguest/software/cryosparc/cryosparc_master/supervisord.conf
cemkguest           5242    5135 21:11:56 mongod --auth --dbpath /home/cemkguest/software/cryosparc/cryosparc_database --port 40001 --oplogSize 64 --replSet meteor --wiredTigerCacheSizeGB 4 --bind_ip_all
cemkguest           5346    5135 21:11:59 python /home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:40002 cryosparc_command.command_core:start() -c /home/cemkguest/software/cryosparc/cryosparc_master/gunicorn.conf.py
cemkguest           5347    5346 21:11:59 python /home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:40002 cryosparc_command.command_core:start() -c /home/cemkguest/software/cryosparc/cryosparc_master/gunicorn.conf.py
cemkguest           5413    5047 21:12:40 grep --color=auto -e cryosparc_ -e mongo
srwx------ 1 cemkguest cemkguest 0 Dec  5 21:11 /tmp/cryosparc-supervisor-ac8d7489d682c54f43fdee4a1b5d8cb2.sock
srwx------ 1 cemkguest cemkguest 0 Dec  5 21:11 /tmp/mongodb-40001.sock
# Instance Configuration
export CRYOSPARC_MASTER_HOSTNAME="scopem-phobos.ethz.ch"
export CRYOSPARC_DB_PATH="/home/cemkguest/software/cryosparc/cryosparc_database"
export CRYOSPARC_BASE_PORT=40000
export CRYOSPARC_DB_CONNECTION_TIMEOUT_MS=20000
export NO_PROXY="${CRYOSPARC_MASTER_HOSTNAME},localhost,127.0.0.1"
# Security
export CRYOSPARC_INSECURE=false
export CRYOSPARC_DB_ENABLE_AUTH=true

# Cluster Integration
export CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL=10
export CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES=1000000

# Project Configuration
export CRYOSPARC_PROJECT_DIR_PREFIX='CS-'

# Development
export CRYOSPARC_DEVELOP=false

# Other
export CRYOSPARC_CLICK_WRAP=true

Now I do not get the error message about the DBPathInUse, it’s back to being stuck at command_core:started

But I understand tha there might be a mongod process that is interfering with everything. However, I am sure there is no other machine in the local network that has access to /home/cemkguest/software/cryosparc/cryosparc_database/.

I am not sure if an “orphaned” ssh connection could cause this? Sometimes I used to run cryosparc remotely while being connected to a VPN and then then running ssh -N -f -L localhost:40000:localhost:40000 cemkguest@ipadress to access the gui.

The other command you just posted results in the following output:

fuser /tmp/mongodb-40001.sock
ps -p $(fuser /tmp/mongodb-40001.sock | awk '{print $NF}') -o user,pid,ppid,start,cmd


/tmp/mongodb-40001.sock:  5242
/tmp/mongodb-40001.sock:
USER         PID    PPID  STARTED CMD
cemkgue+    5242    5135 21:11:56 mongod --auth --dbpath /home/cemkguest/software/cryosparc/cryosparc_database --port 40001 --oplogSize

Is there something I can do to find such orphaned mongodb processes?

Thanks for all your help, I really appreciate it!

best,
Tamino

24

Both the ps -eo and fuser command identified the same mongod process, which seems legitimate in that it is the child of a CryoSPARC-related supervisord process.

The processes shown by the ps command should be newer than the reboot. Did you manually run cryosparcm start after the reboot and before the ps command, or is CryoSPARC started automatically after a reboot?

25

Dear @wtempel,

thanks for all your efforts. We saved the database and started installing a other machine as the master. I think this issue is not required anymore.

Thanks again!

Best,
Tamino