CryoSPARC installation stuck at "command_core: started"

Install, Configure and Update
1

Dear CryoSPARC team,

I have troubles to get CryoSPARC running. I installed the software following the guide for a single workstation.
Unfortunately, the installation process was stuck over night at “command_core: started”.

The same issue persists, if I use cryosparcm restart.

cryosparcm status

CryoSPARC System master node installed at
/home/cemkguest/software/cryosparc/cryosparc_master
Current cryoSPARC version: v4.7.1

CryoSPARC process status:

app STOPPED Not started
app_api STOPPED Not started
app_api_dev STOPPED Not started
command_core RUNNING pid 11969, uptime 0:07:48
command_rtp STOPPED Not started
command_vis STOPPED Not started
database RUNNING pid 11865, uptime 0:07:52

An error ocurred while checking license status
Could not get license verification status. Are all CryoSPARC processes RUNNING?

A few helpful outputs for you:
curl https://get.cryosparc.com/checklicenseexists/$LICENSE_ID
{“success”: true}

cryosparcm log database | tail -n 30

2025-07-03T11:07:30.026+0200 I REPL [replexec-0] Starting replication fetcher thread
2025-07-03T11:07:30.026+0200 I REPL [replexec-0] Starting replication applier thread
2025-07-03T11:07:30.026+0200 I REPL [replexec-0] Starting replication reporter thread
2025-07-03T11:07:30.027+0200 I REPL [rsSync] transition to SECONDARY from RECOVERING
2025-07-03T11:07:30.027+0200 I REPL [rsSync] conducting a dry run election to see if we could be elected. current term: 7
2025-07-03T11:07:30.027+0200 I REPL [replexec-0] dry election run succeeded, running for election in term 8
2025-07-03T11:07:30.027+0200 I REPL [replexec-0] election succeeded, assuming primary role in term 8
2025-07-03T11:07:30.027+0200 I REPL [replexec-0] transition to PRIMARY from SECONDARY
2025-07-03T11:07:30.027+0200 I REPL [replexec-0] Resetting sync source to empty, which was :27017
2025-07-03T11:07:30.027+0200 I REPL [replexec-0] Entering primary catch-up mode.
2025-07-03T11:07:30.027+0200 I REPL [replexec-0] Exited primary catch-up mode.
2025-07-03T11:07:31.431+0200 I NETWORK [listener] connection accepted from 129.132.174.233:43816 #1 (1 connection now open)
2025-07-03T11:07:31.431+0200 I NETWORK [conn1] received client metadata from 129.132.174.233:43816 conn1: { driver: { name: “PyMongo”, version: “4.8.0” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “6.8.0-47-generic” }, platform: “CPython 3.10.14.final.0” }
2025-07-03T11:07:31.432+0200 I NETWORK [conn1] end connection 129.132.174.233:43816 (0 connections now open)
2025-07-03T11:07:31.432+0200 I NETWORK [listener] connection accepted from 127.0.0.1:43120 #2 (1 connection now open)
2025-07-03T11:07:31.433+0200 I NETWORK [conn2] received client metadata from 127.0.0.1:43120 conn2: { driver: { name: “PyMongo”, version: “4.8.0” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “6.8.0-47-generic” }, platform: “CPython 3.10.14.final.0” }
2025-07-03T11:07:32.027+0200 I REPL [rsSync] transition to primary complete; database writes are now permitted
2025-07-03T11:07:32.435+0200 I NETWORK [listener] connection accepted from 127.0.0.1:43132 #3 (2 connections now open)
2025-07-03T11:07:32.435+0200 I NETWORK [conn3] received client metadata from 127.0.0.1:43132 conn3: { driver: { name: “PyMongo”, version: “4.8.0” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “6.8.0-47-generic” }, platform: “CPython 3.10.14.final.0” }
2025-07-03T11:07:32.438+0200 I ACCESS [conn3] Successfully authenticated as principal cryosparc_admin on admin from client 127.0.0.1:43132
2025-07-03T11:07:32.439+0200 I NETWORK [conn2] end connection 127.0.0.1:43120 (1 connection now open)
2025-07-03T11:07:32.439+0200 I NETWORK [conn3] end connection 127.0.0.1:43132 (0 connections now open)
2025-07-03T11:07:38.447+0200 I NETWORK [listener] connection accepted from 129.132.174.233:59968 #4 (1 connection now open)
2025-07-03T11:07:38.447+0200 I NETWORK [conn4] received client metadata from 129.132.174.233:59968 conn4: { driver: { name: “PyMongo”, version: “4.8.0” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “6.8.0-47-generic” }, platform: “CPython 3.10.14.final.0” }
2025-07-03T11:07:38.448+0200 I NETWORK [conn4] end connection 129.132.174.233:59968 (0 connections now open)
2025-07-03T11:07:38.448+0200 I NETWORK [listener] connection accepted from 127.0.0.1:49346 #5 (1 connection now open)
2025-07-03T11:07:38.448+0200 I NETWORK [conn5] received client metadata from 127.0.0.1:49346 conn5: { driver: { name: “PyMongo”, version: “4.8.0” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “6.8.0-47-generic” }, platform: “CPython 3.10.14.final.0” }
2025-07-03T11:07:38.448+0200 I NETWORK [listener] connection accepted from 127.0.0.1:49358 #6 (2 connections now open)
2025-07-03T11:07:38.448+0200 I NETWORK [conn6] received client metadata from 127.0.0.1:49358 conn6: { driver: { name: “PyMongo”, version: “4.8.0” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “6.8.0-47-generic” }, platform: “CPython 3.10.14.final.0” }
2025-07-03T11:07:38.450+0200 I ACCESS [conn6] Successfully authenticated as principal cryosparc_user on admin from client 127.0.0.1:49358

2

Welcome to the forum @Tamino .
Various network-related configurations could contribute to this problem.
Please can you

  1. post the output of the command
    /home/cemkguest/software/cryosparc/cryosparc_master/bin/cryosparcm call env | grep -i proxy
  2. while command_core is RUNNING, collect and post the output of these commands in a fresh command shell
    eval $(/home/cemkguest/software/cryosparc/cryosparc_master/bin/cryosparcm env)
curl 127.0.0.1:$CRYOSPARC_COMMAND_CORE_PORT
curl -v ${CRYOSPARC_MASTER_HOSTNAME}:$CRYOSPARC_COMMAND_CORE_PORT
# record outputs, then
exit
4

thank you @wtempel. Yes, it might very well be that our network configuration prevent correct running of cryosparc.

I gladly provide you with the output you requested. Please let me know if you need anything else!

cemkguest@phobos:~/software/cryosparc/cryosparc_master$ eval $(/home/cemkguest/software/cryosparc/cryosparc_master/bin/cryosparcm env)
curl 127.0.0.1:$CRYOSPARC_COMMAND_CORE_PORT
curl -v ${CRYOSPARC_MASTER_HOSTNAME}:$CRYOSPARC_COMMAND_CORE_PORT
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta type="copyright" content="Copyright (C) 1996-2021 The Squid Software Foundation and contributors">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>ERROR: The requested URL could not be retrieved</title>
<style type="text/css"><!--
 /*
 * Copyright (C) 1996-2021 The Squid Software Foundation and contributors
 *
 * Squid software is distributed under GPLv2+ license and includes
 * contributions from numerous individuals and organizations.
 * Please see the COPYING and CONTRIBUTORS files for details.
 */

/*
 Stylesheet for Squid Error pages
 Adapted from design by Free CSS Templates
 http://www.freecsstemplates.org
 Released for free under a Creative Commons Attribution 2.5 License
*/

/* Page basics */
* {
	font-family: verdana, sans-serif;
}

html body {
	margin: 0;
	padding: 0;
	background: #efefef;
	font-size: 12px;
	color: #1e1e1e;
}

/* Page displayed title area */
#titles {
	margin-left: 15px;
	padding: 10px;
	padding-left: 100px;
	background: url('/squid-internal-static/icons/SN.png') no-repeat left;
}

/* initial title */
#titles h1 {
	color: #000000;
}
#titles h2 {
	color: #000000;
}

/* special event: FTP success page titles */
#titles ftpsuccess {
	background-color:#00ff00;
	width:100%;
}

/* Page displayed body content area */
#content {
	padding: 10px;
	background: #ffffff;
}

/* General text */
p {
}

/* error brief description */
#error p {
}

/* some data which may have caused the problem */
#data {
}

/* the error message received from the system or other software */
#sysmsg {
}

pre {
}

/* special event: FTP directory listing */
#dirmsg {
    font-family: courier, monospace;
    color: black;
    font-size: 10pt;
}
#dirlisting {
    margin-left: 2%;
    margin-right: 2%;
}
#dirlisting tr.entry td.icon,td.filename,td.size,td.date {
    border-bottom: groove;
}
#dirlisting td.size {
    width: 50px;
    text-align: right;
    padding-right: 5px;
}

/* horizontal lines */
hr {
	margin: 0;
}

/* page displayed footer area */
#footer {
	font-size: 9px;
	padding-left: 10px;
}


body
:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }
:lang(he) { direction: rtl; }
 --></style>
</head><body id=ERR_ACCESS_DENIED>
<div id="titles">
<h1>ERROR</h1>
<h2>The requested URL could not be retrieved</h2>
</div>
<hr>

<div id="content">
<p>The following error was encountered while trying to retrieve the URL: <a href="http://127.0.0.1:40002/">http://127.0.0.1:40002/</a></p>

<blockquote id="error">
<p><b>Access Denied.</b></p>
</blockquote>

<p>Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect.</p>

<p>Your cache administrator is <a href="mailto:root?subject=CacheErrorInfo%20-%20ERR_ACCESS_DENIED&amp;body=CacheHost%3A%20proxybd.ethz.ch%0D%0AErrPage%3A%20ERR_ACCESS_DENIED%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Fri,%2004%20Jul%202025%2010%3A51%3A24%20GMT%0D%0A%0D%0AClientIP%3A%20129.132.174.233%0D%0A%0D%0AHTTP%20Request%3A%0D%0AGET%20%2F%20HTTP%2F1.1%0AUser-Agent%3A%20curl%2F8.5.0%0D%0AAccept%3A%20*%2F*%0D%0AProxy-Connection%3A%20Keep-Alive%0D%0AHost%3A%20127.0.0.1%3A40002%0D%0A%0D%0A%0D%0A">root</a>.</p>
<br>
</div>

<hr>
<div id="footer">
<p>Generated Fri, 04 Jul 2025 10:51:24 GMT by proxybd.ethz.ch (squid/4.15)</p>
<!-- ERR_ACCESS_DENIED -->
</div>
</body></html>
* Uses proxy env variable http_proxy == 'http://proxy.ethz.ch:3128/'
* Host proxy.ethz.ch:3128 was resolved.
* IPv6: (none)
* IPv4: 129.132.202.155
*   Trying 129.132.202.155:3128...
* Connected to proxy.ethz.ch (129.132.202.155) port 3128
> GET http://phobos:40002/ HTTP/1.1
> Host: phobos:40002
> User-Agent: curl/8.5.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 503 Service Unavailable
< Server: squid/4.15
< Mime-Version: 1.0
< Date: Fri, 04 Jul 2025 10:51:24 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3703
< X-Squid-Error: ERR_DNS_FAIL 0
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from proxybd.ethz.ch
< X-Cache-Lookup: MISS from proxybd.ethz.ch:3128
< Via: 1.1 proxybd.ethz.ch (squid/4.15)
< Connection: close
< 
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta type="copyright" content="Copyright (C) 1996-2021 The Squid Software Foundation and contributors">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>ERROR: The requested URL could not be retrieved</title>
<style type="text/css"><!-- 
 /*
 * Copyright (C) 1996-2021 The Squid Software Foundation and contributors
 *
 * Squid software is distributed under GPLv2+ license and includes
 * contributions from numerous individuals and organizations.
 * Please see the COPYING and CONTRIBUTORS files for details.
 */

/*
 Stylesheet for Squid Error pages
 Adapted from design by Free CSS Templates
 http://www.freecsstemplates.org
 Released for free under a Creative Commons Attribution 2.5 License
*/

/* Page basics */
* {
	font-family: verdana, sans-serif;
}

html body {
	margin: 0;
	padding: 0;
	background: #efefef;
	font-size: 12px;
	color: #1e1e1e;
}

/* Page displayed title area */
#titles {
	margin-left: 15px;
	padding: 10px;
	padding-left: 100px;
	background: url('/squid-internal-static/icons/SN.png') no-repeat left;
}

/* initial title */
#titles h1 {
	color: #000000;
}
#titles h2 {
	color: #000000;
}

/* special event: FTP success page titles */
#titles ftpsuccess {
	background-color:#00ff00;
	width:100%;
}

/* Page displayed body content area */
#content {
	padding: 10px;
	background: #ffffff;
}

/* General text */
p {
}

/* error brief description */
#error p {
}

/* some data which may have caused the problem */
#data {
}

/* the error message received from the system or other software */
#sysmsg {
}

pre {
}

/* special event: FTP directory listing */
#dirmsg {
    font-family: courier, monospace;
    color: black;
    font-size: 10pt;
}
#dirlisting {
    margin-left: 2%;
    margin-right: 2%;
}
#dirlisting tr.entry td.icon,td.filename,td.size,td.date {
    border-bottom: groove;
}
#dirlisting td.size {
    width: 50px;
    text-align: right;
    padding-right: 5px;
}

/* horizontal lines */
hr {
	margin: 0;
}

/* page displayed footer area */
#footer {
	font-size: 9px;
	padding-left: 10px;
}


body
:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }
:lang(he) { direction: rtl; }
 --></style>
</head><body id=ERR_DNS_FAIL>
<div id="titles">
<h1>ERROR</h1>
<h2>The requested URL could not be retrieved</h2>
</div>
<hr>

<div id="content">
<p>The following error was encountered while trying to retrieve the URL: <a href="http://phobos:40002/">http://phobos:40002/</a></p>

<blockquote id="error">
<p><b>Unable to determine IP address from host name <q>phobos</q></b></p>
</blockquote>

<p>The DNS server returned:</p>
<blockquote id="data">
<pre>Name Error: The domain name does not exist.</pre>
</blockquote>

<p>This means that the cache was not able to resolve the hostname presented in the URL. Check if the address is correct.</p>

<p>Your cache administrator is <a href="mailto:root?subject=CacheErrorInfo%20-%20ERR_DNS_FAIL&amp;body=CacheHost%3A%20proxybd.ethz.ch%0D%0AErrPage%3A%20ERR_DNS_FAIL%0D%0AErr%3A%20%5Bnone%5D%0D%0ADNS%20ErrMsg%3A%20Name%20Error%3A%20The%20domain%20name%20does%20not%20exist.%0D%0ATimeStamp%3A%20Fri,%2004%20Jul%202025%2010%3A51%3A24%20GMT%0D%0A%0D%0AClientIP%3A%20129.132.174.233%0D%0A%0D%0AHTTP%20Request%3A%0D%0AGET%20%2F%20HTTP%2F1.1%0AUser-Agent%3A%20curl%2F8.5.0%0D%0AAccept%3A%20*%2F*%0D%0AProxy-Connection%3A%20Keep-Alive%0D%0AHost%3A%20phobos%3A40002%0D%0A%0D%0A%0D%0A">root</a>.</p>
<br>
</div>

<hr>
<div id="footer">
<p>Generated Fri, 04 Jul 2025 10:51:24 GMT by proxybd.ethz.ch (squid/4.15)</p>
<!-- ERR_DNS_FAIL -->
</div>
</body></html>
* Closing connection
5

Thanks @Tamino for posting the outputs, which suggest a number of possible approaches, depending on how you intend to configure and operate CryoSPARC. Please can you provide additional information:

  1. Do you intend to run CryoSPARC in single workstation, connected workers or cluster mode?
  2. What are the outputs of these commands on the CryoSPARC master node?
    hostname -f
host phobos
grep HOSTNAME /home/cemkguest/software/cryosparc/cryosparc_master/config.sh
/home/cemkguest/software/cryosparc/cryosparc_master/bin/cryosparcm call env | grep -i proxy
6

Dear @wtempel, thanks for the quick reply!

I am planning on using it only as a single workstation. It should only run on the local machine, with no additional workers or connected to the cluster (Only ssh connection to the machine for remote work)

Happily I provide you with the output of the commands:

cemkguest@phobos:~$ hostname -f
host phobos
grep HOSTNAME /home/cemkguest/software/cryosparc/cryosparc_master/config.sh
/home/cemkguest/software/cryosparc/cryosparc_master/bin/cryosparcm call env | grep -i proxy


phobos
phobos has address 129.132.174.233
phobos has IPv6 address fe80::d0b9:fd47:e64b:d9db
export CRYOSPARC_MASTER_HOSTNAME="phobos"
https_proxy=http://proxy.ethz.ch:3128/
http_proxy=http://proxy.ethz.ch:3128/
7

Thanks @Tamino. The topic Cryosparc not starting looks similar to your situation. Assuming that the --standalone installation was disrupted just after the installation of the cryosparc_master/ package, you may consider

  1. adding the line
    export NO_PROXY="${CRYOSPARC_MASTER_HOSTNAME},localhost,127.0.0.1"
    to the file
    /home/cemkguest/software/cryosparc/cryosparc_master/config.sh
    somewhere below the definition of CRYOSPARC_MASTER_HOSTNAME
  2. restarting CryoSPARC.
    The following steps would have been performed automatically in case of a smooth
    --standalone installation.
  3. creating the first user
  4. installing the cryosparc_worker/ package
  5. adding the line
    export NO_PROXY="phobos,localhost,127.0.0.1"
    to the file
    cryosparc_worker/config.sh
    (In case the master hostname changes from phobos in the future, phobos needs to be changed accordingly in the NO_PROXY definition.)
  6. connecting the worker component with a command like
    cryosparc_worker/bin/cryosparcw connect --master phobos --worker phobos --port 99999 --ssdpath /path/to/cache
    where the --port parameter needs to be changed from the nonsensical 99999 value to the CRYOSPARC_BASE_PORT value inside
    /home/cemkguest/software/cryosparc/cryosparc_master/config.sh
    and the --ssdpath parameter points to the actual path to be used for particle caching (or is replaced altogether with --nossd)
8

Thank you very much for your extensive reply!

Adding the suggested line to the /cryosparc_master/config.sh solved the issue, I can now open cryosparc on the web browser.

Unfortunately, the installation must have gone wrong somewhere still. I do not have any cryosparc_worker/config.sh fine, even after re-running:

tar -xf cryosparc_worker.tar.gz cryosparc_worker

What do you reccomend me to do?

Thank you very much,
all the best,
Tamino

9

@Tamino Have you run the
cryosparc_worker/install.sh command with the appropriate options?

10

Sorry for the late reply @wtempel

I managed to get cryosparc running! The only issue that came up, was to manually edit the
license id in the worker config.sh to match the one in the master config.sh

Thank you so much for all your help!

1 Like
11

Dear @wtempel

I am back to the same Issue, that upon cryosparcm start the process does not go further than “command_core: started”.

I made sure that our fixes from last time are still applicable. So the line export NO_PROXY="phobos,localhost,127.0.0.1" is still in the config.sh file, as well as the license IDs between the cryosparc_worker and cryosparc_master/config.sh files match.

I tried some trouble shooting reccomended in other issues like the complete shut down by deleting zombie processes.

After stopping cryosparc, the following command shows no output:

ps -weo pid,ppid,start,cmd | grep -e cryosparc -e mongo | grep -v grep

I also realized that while it is stuck, and I run cryosparcm status i get the following output:

CryoSPARC System master node installed at
/home/cemkguest/software/cryosparc/cryosparc_master
Current cryoSPARC version: v4.7.1

CryoSPARC process status:

app STOPPED Not started
app_api STOPPED Not started
app_api_dev STOPPED Not started
command_core RUNNING pid 33253, uptime 0:00:56
command_rtp STOPPED Not started
command_vis STOPPED Not started
database RUNNING pid 33148, uptime 0:00:59

An error ocurred while checking license status
Could not get license verification status. Are all CryoSPARC processes RUNNING?

Do you think this is an issue with the proxy of our institution?

12

@Tamino Please can you post the up-to-date output of these commands on the CryoSPARC master host when CryoSPARC startup is stuck, in a fresh shell:

eval $(/home/cemkguest/software/cryosparc/cryosparc_master/bin/cryosparcm env)
env | grep -i -e CRYOSPARC_ -e proxy | grep -v LICENSE_ID
hostname -f
ip -4 -br a
cryosparcm log command_core | tail -n 20
cryosparcm log database | tail -n 20
cryosparcm log supervisord | tail -n 20
ps -eo user,pid,ppid,start,cmd | grep -e cryosparc_ -e mongo | grep -v LICENSE_ID
host localhost
host phobos
curl 127.0.0.1:$CRYOSPARC_MONGO_PORT
curl -v ${CRYOSPARC_MASTER_HOSTNAME}:$CRYOSPARC_MONGO_PORT
ls -l $CRYOSPARC_SUPERVISOR_SOCK_FILE
ls -l /tmp/mongo*.sock
# after recording output, close shell
exit

[edited 2025-12-04. modified command to omit output of license ID.]

13

Dear @wtempel,

sorry for not replying for such a long time, the issue resolved itself last time, but seems to have appeared again.

This is the output of your requested command. In one shell, it is stuck at start up, and in a new shell I ran the commands:

thank you so much for your help!

no_proxy=localhost,127.0.0.0/8,::1
CRYOSPARC_MASTER_HOSTNAME=phobos
CONDA_EXE=/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/bin/conda
CRYOSPARC_PATH=/home/cemkguest/software/cryosparc/cryosparc_master/deps/external/mongodb/bin:/home/cemkguest/software/cryosparc/cryosparc_master/bin
CRYOSPARC_MONGO_EXTRA_FLAGS=
CRYOSPARC_INSECURE=false
CRYOSPARC_DB_ENABLE_AUTH_FLAG=--auth
CRYOSPARC_DB_CONNECTION_TIMEOUT_MS=20000
CRYOSPARC_MONGO_CACHE_GB=4
CRYOSPARC_COMMAND_VIS_PORT=40003
CONDA_PREFIX=/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env
CRYOSPARC_MONGO_FCV=3.6
LD_PRELOAD=/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.10.so
CRYOSPARC_COMMAND_RTP_PORT=40005
CRYOSPARC_HTTP_APP_PORT=40000
CRYOSPARC_ROOT_DIR=/home/cemkguest/software/cryosparc/cryosparc_master
CRYOSPARC_FORCE_USER=false
CRYOSPARC_HOSTNAME_CHECK=phobos
CRYOSPARC_MONGO_PORT=40001
CRYOSPARC_DB_PATH=/home/cemkguest/software/cryosparc/cryosparc_database
CONDA_PROMPT_MODIFIER=(cryosparc_master_env)
CRYOSPARC_CLICK_WRAP=true
https_proxy=http://proxy.ethz.ch:3128/
CRYOSPARC_HTTP_LIVEAPP_LEGACY_PORT=40006
CRYOSPARC_LIVE_ENABLED=true
CRYOSPARC_COMMAND_CORE_PORT=40002
CRYOSPARC_SUPERVISOR_SOCK_FILE=/tmp/cryosparc-supervisor-ac8d7489d682c54f43fdee4a1b5d8cb2.sock
CRYOSPARC_BASE_PORT=40000
PYTHONPATH=/home/cemkguest/software/cryosparc/cryosparc_master
CRYOSPARC_LICENSE_ID=[redacted]
NO_PROXY=localhost,127.0.0.0/8,::1
CRYOSPARC_DEVELOP=false
HTTPS_PROXY=http://proxy.ethz.ch:3128/
HTTP_PROXY=http://proxy.ethz.ch:3128/
http_proxy=http://proxy.ethz.ch:3128/
CONDA_PYTHON_EXE=/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/bin/python
CONDA_DEFAULT_ENV=cryosparc_master_env
CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL=10
CRYOSPARC_DB_ENABLE_AUTH=true
PATH=/home/cemkguest/software/cryosparc/cryosparc_master/deps/external/mongodb/bin:/home/cemkguest/software/cryosparc/cryosparc_master/bin:/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin:/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/condabin:/home/cemkguest/miniforge3/bin:/home/cemkguest/miniforge3/condabin:/home/cemkguest/software/cryosparc/cryosparc_master/bin:/home/cemkguest/software/cryosparc/cryosparc_master/bin:/home/cemkguest/software/cryosparc/cryosparc_master/bin:/home/cemkguest/cryosparc/cryosparc_master/bin:/home/cemkguest/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
CRYOSPARC_HEARTBEAT_SECONDS=180
CRYOSPARC_CONDA_ENV=cryosparc_master_env
CRYOSPARC_PROJECT_DIR_PREFIX=CS-
CRYOSPARC_FORCE_HOSTNAME=false
CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES=1000000
phobos
lo               UNKNOWN        127.0.0.1/8 
net2             UP             129.132.174.233/25 
2025-12-04 08:40:24,349 interrupt            INFO     |  === EXITED === 
2025-12-04 08:40:24 info                 INFO     | Shutting down: Master
2025-12-04 08:40:33 info                 INFO     | Starting gunicorn 22.0.0
2025-12-04 08:40:33 info                 INFO     | Listening at: http://0.0.0.0:40002 (1304538)
2025-12-04 08:40:33 info                 INFO     | Using worker: gthread
2025-12-04 08:40:33 info                 INFO     | Booting worker with pid: 1304539
2025-12-04 08:40:33,778 start                INFO     |  === STARTED === 
2025-12-04 08:40:33,778 background_worker    INFO     |  === STARTED === 
2025-12-04 08:40:33,779 run                  INFO     | === STARTED TASKS WORKER ===
2025-12-04 08:48:33 info                 INFO     | Handling signal: quit
2025-12-04 08:48:33 info                 INFO     | Worker exiting (pid: 1304539)
2025-12-04 08:48:33,418 interrupt            INFO     |  === EXITED === 
2025-12-04 08:48:33 info                 INFO     | Shutting down: Master
2025-12-04 08:49:17 info                 INFO     | Starting gunicorn 22.0.0
2025-12-04 08:49:17 info                 INFO     | Listening at: http://0.0.0.0:40002 (1305162)
2025-12-04 08:49:17 info                 INFO     | Using worker: gthread
2025-12-04 08:49:17 info                 INFO     | Booting worker with pid: 1305163
2025-12-04 08:49:17,927 start                INFO     |  === STARTED === 
2025-12-04 08:49:17,927 background_worker    INFO     |  === STARTED === 
2025-12-04 08:49:17,927 run                  INFO     | === STARTED TASKS WORKER ===
2025-12-04T08:49:15.925+0100 I NETWORK  [conn1] end connection 129.132.174.233:48976 (0 connections now open)
2025-12-04T08:49:15.925+0100 I NETWORK  [listener] connection accepted from 127.0.0.1:35324 #2 (1 connection now open)
2025-12-04T08:49:15.925+0100 I NETWORK  [conn2] received client metadata from 127.0.0.1:35324 conn2: { driver: { name: "PyMongo", version: "4.8.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "6.14.0-34-generic" }, platform: "CPython 3.10.14.final.0" }
2025-12-04T08:49:16.521+0100 I REPL     [rsSync] transition to primary complete; database writes are now permitted
2025-12-04T08:49:16.927+0100 I NETWORK  [listener] connection accepted from 127.0.0.1:35328 #3 (2 connections now open)
2025-12-04T08:49:16.927+0100 I NETWORK  [conn3] received client metadata from 127.0.0.1:35328 conn3: { driver: { name: "PyMongo", version: "4.8.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "6.14.0-34-generic" }, platform: "CPython 3.10.14.final.0" }
2025-12-04T08:49:16.930+0100 I ACCESS   [conn3] Successfully authenticated as principal cryosparc_admin on admin from client 127.0.0.1:35328
2025-12-04T08:49:16.931+0100 I NETWORK  [conn2] end connection 127.0.0.1:35324 (1 connection now open)
2025-12-04T08:49:16.931+0100 I NETWORK  [conn3] end connection 127.0.0.1:35328 (0 connections now open)
2025-12-04T08:49:22.934+0100 I NETWORK  [listener] connection accepted from 129.132.174.233:49344 #4 (1 connection now open)
2025-12-04T08:49:22.934+0100 I NETWORK  [conn4] received client metadata from 129.132.174.233:49344 conn4: { driver: { name: "PyMongo", version: "4.8.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "6.14.0-34-generic" }, platform: "CPython 3.10.14.final.0" }
2025-12-04T08:49:22.934+0100 I NETWORK  [conn4] end connection 129.132.174.233:49344 (0 connections now open)
2025-12-04T08:49:22.935+0100 I NETWORK  [listener] connection accepted from 127.0.0.1:57248 #5 (1 connection now open)
2025-12-04T08:49:22.935+0100 I NETWORK  [conn5] received client metadata from 127.0.0.1:57248 conn5: { driver: { name: "PyMongo", version: "4.8.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "6.14.0-34-generic" }, platform: "CPython 3.10.14.final.0" }
2025-12-04T08:49:22.935+0100 I NETWORK  [listener] connection accepted from 127.0.0.1:57252 #6 (2 connections now open)
2025-12-04T08:49:22.935+0100 I NETWORK  [conn6] received client metadata from 127.0.0.1:57252 conn6: { driver: { name: "PyMongo", version: "4.8.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "6.14.0-34-generic" }, platform: "CPython 3.10.14.final.0" }
2025-12-04T08:49:22.938+0100 I ACCESS   [conn6] Successfully authenticated as principal cryosparc_user on admin from client 127.0.0.1:57252
2025-12-04T08:51:10.688+0100 I NETWORK  [listener] connection accepted from 127.0.0.1:37754 #7 (3 connections now open)
2025-12-04T08:51:10.688+0100 I NETWORK  [conn7] Error receiving request from client: ProtocolError: Client sent an HTTP request over a native MongoDB connection. Ending connection from 127.0.0.1:37754 (connection id: 7)
2025-12-04T08:51:10.688+0100 I NETWORK  [conn7] end connection 127.0.0.1:37754 (2 connections now open)
2025-12-04 08:40:26,003 INFO RPC interface 'supervisor' initialized
2025-12-04 08:40:26,003 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-12-04 08:40:26,004 INFO daemonizing the supervisord process
2025-12-04 08:40:26,004 INFO supervisord started with pid 1304328
2025-12-04 08:40:29,976 INFO spawned: 'database' with pid 1304434
2025-12-04 08:40:31,303 INFO success: database entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-12-04 08:40:33,285 INFO spawned: 'command_core' with pid 1304538
2025-12-04 08:40:38,784 INFO success: command_core entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2025-12-04 08:48:33,317 INFO waiting for command_core to stop
2025-12-04 08:48:33,317 INFO waiting for database to stop
2025-12-04 08:48:33,403 INFO stopped: database (exit status 0)
2025-12-04 08:48:33,941 INFO stopped: command_core (exit status 0)
2025-12-04 08:49:10,195 INFO RPC interface 'supervisor' initialized
2025-12-04 08:49:10,195 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-12-04 08:49:10,196 INFO daemonizing the supervisord process
2025-12-04 08:49:10,197 INFO supervisord started with pid 1304949
2025-12-04 08:49:14,183 INFO spawned: 'database' with pid 1305058
2025-12-04 08:49:15,523 INFO success: database entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-12-04 08:49:17,522 INFO spawned: 'command_core' with pid 1305162
2025-12-04 08:49:22,932 INFO success: command_core entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
cemkgue+ 1304919 1304072 08:49:08 bash /home/cemkguest/software/cryosparc/cryosparc_master/bin/cryosparcm start
cemkgue+ 1304949    3261 08:49:09 python /home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/cemkguest/software/cryosparc/cryosparc_master/supervisord.conf
cemkgue+ 1305058 1304949 08:49:13 mongod --auth --dbpath /home/cemkguest/software/cryosparc/cryosparc_database --port 40001 --oplogSize 64 --replSet meteor --wiredTigerCacheSizeGB 4 --bind_ip_all
cemkgue+ 1305162 1304949 08:49:16 python /home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:40002 cryosparc_command.command_core:start() -c /home/cemkguest/software/cryosparc/cryosparc_master/gunicorn.conf.py
cemkgue+ 1305163 1305162 08:49:17 python /home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:40002 cryosparc_command.command_core:start() -c /home/cemkguest/software/cryosparc/cryosparc_master/gunicorn.conf.py
cemkgue+ 1306014 1305856 08:51:27 grep --color=auto -e cryosparc_ -e mongo
localhost has address 127.0.0.1
localhost has IPv6 address ::1
phobos has address 129.132.174.233
phobos has IPv6 address fe80::d0b9:fd47:e64b:d9db
It looks like you are trying to access MongoDB over HTTP on the native driver port.
* Uses proxy env variable no_proxy == 'localhost,127.0.0.0/8,::1'
* Uses proxy env variable http_proxy == 'http://proxy.ethz.ch:3128/'
* Host proxy.ethz.ch:3128 was resolved.
* IPv6: (none)
* IPv4: 129.132.202.155
*   Trying 129.132.202.155:3128...
* Connected to proxy.ethz.ch (129.132.202.155) port 3128
> GET http://phobos:40001/ HTTP/1.1
> Host: phobos:40001
> User-Agent: curl/8.5.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 503 Service Unavailable
< Server: squid/4.15
< Mime-Version: 1.0
< Date: Thu, 04 Dec 2025 07:51:28 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3180
< X-Squid-Error: ERR_DNS_FAIL 0
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from proxybd.ethz.ch
< X-Cache-Lookup: MISS from proxybd.ethz.ch:3128
< Via: 1.1 proxybd.ethz.ch (squid/4.15)
< Connection: close
< 
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta type="copyright" content="Copyright (C) 1996-2021 The Squid Software Foundation and contributors">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>ERROR: The requested URL could not be retrieved</title>
<style type="text/css"><!-- 
 /*
 * Copyright (C) 1996-2021 The Squid Software Foundation and contributors
 *
 * Squid software is distributed under GPLv2+ license and includes
 * contributions from numerous individuals and organizations.
 * Please see the COPYING and CONTRIBUTORS files for details.
 */

/*
 Stylesheet for Squid Error pages
 Adapted from design by Free CSS Templates
 http://www.freecsstemplates.org
 Released for free under a Creative Commons Attribution 2.5 License
*/

/* Page basics */
* {
	font-family: verdana, sans-serif;
}

html body {
	margin: 0;
	padding: 0;
	background: #efefef;
	font-size: 12px;
	color: #1e1e1e;
}

/* Page displayed title area */
#titles {
	margin-left: 15px;
	padding: 10px;
	padding-left: 100px;
	background: url('/squid-internal-static/icons/SN.png') no-repeat left;
}

/* initial title */
#titles h1 {
	color: #000000;
}
#titles h2 {
	color: #000000;
}

/* special event: FTP success page titles */
#titles ftpsuccess {
	background-color:#00ff00;
	width:100%;
}

/* Page displayed body content area */
#content {
	padding: 10px;
	background: #ffffff;
}

/* General text */
p {
}

/* error brief description */
#error p {
}

/* some data which may have caused the problem */
#data {
}

/* the error message received from the system or other software */
#sysmsg {
}

pre {
}

/* special event: FTP directory listing */
#dirmsg {
    font-family: courier, monospace;
    color: black;
    font-size: 10pt;
}
#dirlisting {
    margin-left: 2%;
    margin-right: 2%;
}
#dirlisting tr.entry td.icon,td.filename,td.size,td.date {
    border-bottom: groove;
}
#dirlisting td.size {
    width: 50px;
    text-align: right;
    padding-right: 5px;
}

/* horizontal lines */
hr {
	margin: 0;
}

/* page displayed footer area */
#footer {
	font-size: 9px;
	padding-left: 10px;
}


body
:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }
:lang(he) { direction: rtl; }
 --></style>
</head><body id=ERR_DNS_FAIL>
<div id="titles">
<h1>ERROR</h1>
<h2>The requested URL could not be retrieved</h2>
</div>
<hr>

<div id="content">
<p>The following error was encountered while trying to retrieve the URL: <a href="http://phobos:40001/">http://phobos:40001/</a></p>

<blockquote id="error">
<p><b>Unable to determine IP address from host name <q>phobos</q></b></p>
</blockquote>

<p>The DNS server returned:</p>
<blockquote id="data">
<pre>Name Error: The domain name does not exist.</pre>
</blockquote>

<p>This means that the cache was not able to resolve the hostname presented in the URL. Check if the address is correct.</p>

<p>Your cache administrator is <a href="mailto:root">root</a>.</p>
<br>
</div>

<hr>
<div id="footer">
<p>Generated Thu, 04 Dec 2025 07:51:28 GMT by proxybd.ethz.ch (squid/4.15)</p>
<!-- ERR_DNS_FAIL -->
</div>
</body></html>
* Closing connection
srwx------ 1 cemkguest cemkguest 0 Dec  4 08:49 /tmp/cryosparc-supervisor-ac8d7489d682c54f43fdee4a1b5d8cb2.sock
srwx------ 1 cemkguest cemkguest 0 Dec  4 08:49 /tmp/mongodb-40001.sock
14

It seems that the CRYOSPARC_MASTER_HOSTNAME specified inside
/home/cemkguest/software/cryosparc/cryosparc_master/config.sh is not properly resolved. Does this CryoSPARC instance use separate worker nodes or is it of the single workstation (“standalone”) type?
What are the outputs of the commands

cat /etc/hosts
grep HOST /home/cemkguest/software/cryosparc/cryosparc_master/config.sh

?

15

Dear @wtempel,

at some point we tried to attach a worker, but it was never used. So it can be viewed as a single workstation :slight_smile:

grep HOST /home/cemkguest/software/cryosparc/cryosparc_master/config.sh

127.0.0.1 localhost
127.0.1.1 dalco

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
127.0.1.1 wks
129.132.116.104 mercury
export CRYOSPARC_MASTER_HOSTNAME="phobos"
export NO_PROXY="${CRYOSPARC_MASTER_HOSTNAME},localhost,127.0.0.1"
16

Thanks @Tamino. Where did the lines

come from?

  1. Was localhost ever present inside this /etc/hosts file and has been removed for a specific reason?
  2. What are the outputs of the commands
    host localhost
host 129.132.174.233
  3. Can your IT support configure the network’s DHCP and DNS such that the computer that’s currently called phobos will permanently be assigned a fully qualified domain name that can be resolved by other computers on your lab network ()? That would be helpful if you ever wanted to again attach a worker to this CryoSPARC instance.
17

Hey @wtempel,

sorry, these two lines also came from the output of the commands. I will adjust my previous post.

I don’t know if this was ever present in /etc/hosts unfortunetly!

The output of the commands are the following:

localhost has address 127.0.0.1
localhost has IPv6 address ::1
233.174.132.129.in-addr.arpa domain name pointer scopem-phobos.ethz.ch.

I can certainly ask them! I am sure they are happy to do that. Is this necessary to get cryosparc running again for the standalone version of this machine?

Best,
Tamino

18

Regarding a more comprehensive “fix”,

suggests that a permanent fqdn may have already been configured by your IT team. If your IT team confirms that the assignment of scopem-phobos.ethz.ch to this computer is “permanent”, you might:

  1. ensure that the CryoSPARC ports (apparently 40000..40009 on your installation) cannot be accessed from the internet.
  2. set the computer’s hostname to scopem-phobos.ethz.ch
  3. set CRYOSPARC_MASTER_HOSTNAME to scopem-phobos.ethz.ch
  4. thoroughly shutdown CryoSPARC
  5. start CryoSPARC
  6. post the output of the command
    cryosparcm cli "get_scheduler_targets()"
    so we may see what adjustments may be required

Preferred but not required. A preliminary fix based on the single workstation constraint might be adding the line

127.0.1.2 phobos

to /etc/hosts, followed by a thorough shutdown and restart of CryoSPARC. You should also confirm that the CryoSPARC ports cannot be accessed from the internet.
As an aside: Your /etc/hosts file currently contains two lines for the 127.0.1.1 address. I am not sure about the purpose and history of these lines or whether they have the intended effect (links to info dated more than 20 years ago).

19

Dear @wtempel

I talked to the IT department, we tried to do the best to follow your instructions.

First we made sure that the ports cannot be accessed by the internet. With the following command (no 40000 port is accessible)

sudo ufw status verbose

Then we adjusted the /etc/hosts file to the following:

127.0.0.1 localhost
#127.0.1.1 dalco (commented out now)

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
127.0.1.1 wks
127.0.1.2 scopem-phobos.ethz.ch phobos

we also renamed the computers host name and adjusted the CRYOSPARC_MASTER_HOSTNAME:

Static hostname: scopem-phobos.ethz.ch

export CRYOSPARC_MASTER_HOSTNAME="scopem-phobos.ethz.ch"

Regarding to the command you sent me, since cryosparc is still stuck at command_core: started,
I rand the command in a new shell.

cryosparcm cli "get_scheduler_targets()"

/home/cemkguest/software/cryosparc/cryosparc_master/cryosparc_tools/cryosparc/command.py:135: UserWarning: *** CommandClient: (http://scopem-phobos.ethz.ch:40002/api) HTTP Error 503 Service Unavailable; please check cryosparcm log command for additional information.
Response from server: b'<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">\n<html><head>\n<meta type="copyright" content="Copyright (C) 1996-2021 The Squid Software Foundation and contributors">\n<meta http-equiv="Content-Type" CONTENT="text/html; charset=utf-8">\n<title>ERROR: The requested URL could not be retrieved</title>\n<style type="text/css"><!-- \n /*\n * Copyright (C) 1996-2021 The Squid Software Foundation and contributors\n *\n * Squid software is distributed under GPLv2+ license and includes\n * contributions from numerous individuals and organizations.\n * Please see the COPYING and CONTRIBUTORS files for details.\n */\n\n/*\n Stylesheet for Squid Error pages\n Adapted from design by Free CSS Templates\n http://www.freecsstemplates.org\n Released for free under a Creative Commons Attribution 2.5 License\n*/\n\n/* Page basics */\n* {\n\tfont-family: verdana, sans-serif;\n}\n\nhtml body {\n\tmargin: 0;\n\tpadding: 0;\n\tbackground: #efefef;\n\tfont-size: 12px;\n\tcolor: #1e1e1e;\n}\n\n/* Page displayed title area */\n#titles {\n\tmargin-left: 15px;\n\tpadding: 10px;\n\tpadding-left: 100px;\n\tbackground: url(\'/squid-internal-static/icons/SN.png\') no-repeat left;\n}\n\n/* initial title */\n#titles h1 {\n\tcolor: #000000;\n}\n#titles h2 {\n\tcolor: #000000;\n}\n\n/* special event: FTP success page titles */\n#titles ftpsuccess {\n\tbackground-color:#00ff00;\n\twidth:100%;\n}\n\n/* Page displayed body content area */\n#content {\n\tpadding: 10px;\n\tbackground: #ffffff;\n}\n\n/* General text */\np {\n}\n\n/* error brief description */\n#error p {\n}\n\n/* some data which may have caused the problem */\n#data {\n}\n\n/* the error message received from the system or other software */\n#sysmsg {\n}\n\npre {\n}\n\n/* special event: FTP directory listing */\n#dirmsg {\n    font-family: courier, monospace;\n    color: black;\n    font-size: 10pt;\n}\n#dirlisting {\n    margin-left: 2%;\n    margin-right: 2%;\n}\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\n    border-bottom: groove;\n}\n#dirlisting td.size {\n    width: 50px;\n    text-align: right;\n    padding-right: 5px;\n}\n\n/* horizontal lines */\nhr {\n\tmargin: 0;\n}\n\n/* page displayed footer area */\n#footer {\n\tfont-size: 9px;\n\tpadding-left: 10px;\n}\n\n\nbody\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\n:lang(he) { direction: rtl; }\n --></style>\n</head><body id=ERR_CONNECT_FAIL>\n<div id="titles">\n<h1>ERROR</h1>\n<h2>The requested URL could not be retrieved</h2>\n</div>\n<hr>\n\n<div id="content">\n<p>The following error was encountered while trying to retrieve the URL: <a href="http://scopem-phobos.ethz.ch:40002/api">http://scopem-phobos.ethz.ch:40002/api</a></p>\n\n<blockquote id="error">\n<p><b>Connection to 129.132.174.233 failed.</b></p>\n</blockquote>\n\n<p id="sysmsg">The system returned: <i>(110) Connection timed out</i></p>\n\n<p>The remote host or network may be down. Please try the request again.</p>\n\n<p>Your cache administrator is <a href="mailto:root">root</a>.</p>\n\n<br>\n</div>\n\n<hr>\n<div id="footer">\n<p>Generated Fri, 05 Dec 2025 08:52:51 GMT by proxybd.ethz.ch (squid/4.15)</p>\n<!-- ERR_CONNECT_FAIL -->\n</div>\n</body></html>\n'
  system = self._get_callable("system.describe")()
Traceback (most recent call last):
  File "/home/cemkguest/software/cryosparc/cryosparc_master/cryosparc_tools/cryosparc/command.py", line 105, in func
    with make_json_request(self, "/api", data=data, _stacklevel=4) as request:
  File "/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/cemkguest/software/cryosparc/cryosparc_master/cryosparc_tools/cryosparc/command.py", line 226, in make_request
    raise CommandError(error_reason, url=url, code=code, data=resdata)
cryosparc_tools.cryosparc.errors.CommandError: *** (http://scopem-phobos.ethz.ch:40002/api, code 503) HTTP Error 503 Service Unavailable; please check cryosparcm log command for additional information.
Response from server: b'<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">\n<html><head>\n<meta type="copyright" content="Copyright (C) 1996-2021 The Squid Software Foundation and contributors">\n<meta http-equiv="Content-Type" CONTENT="text/html; charset=utf-8">\n<title>ERROR: The requested URL could not be retrieved</title>\n<style type="text/css"><!-- \n /*\n * Copyright (C) 1996-2021 The Squid Software Foundation and contributors\n *\n * Squid software is distributed under GPLv2+ license and includes\n * contributions from numerous individuals and organizations.\n * Please see the COPYING and CONTRIBUTORS files for details.\n */\n\n/*\n Stylesheet for Squid Error pages\n Adapted from design by Free CSS Templates\n http://www.freecsstemplates.org\n Released for free under a Creative Commons Attribution 2.5 License\n*/\n\n/* Page basics */\n* {\n\tfont-family: verdana, sans-serif;\n}\n\nhtml body {\n\tmargin: 0;\n\tpadding: 0;\n\tbackground: #efefef;\n\tfont-size: 12px;\n\tcolor: #1e1e1e;\n}\n\n/* Page displayed title area */\n#titles {\n\tmargin-left: 15px;\n\tpadding: 10px;\n\tpadding-left: 100px;\n\tbackground: url(\'/squid-internal-static/icons/SN.png\') no-repeat left;\n}\n\n/* initial title */\n#titles h1 {\n\tcolor: #000000;\n}\n#titles h2 {\n\tcolor: #000000;\n}\n\n/* special event: FTP success page titles */\n#titles ftpsuccess {\n\tbackground-color:#00ff00;\n\twidth:100%;\n}\n\n/* Page displayed body content area */\n#content {\n\tpadding: 10px;\n\tbackground: #ffffff;\n}\n\n/* General text */\np {\n}\n\n/* error brief description */\n#error p {\n}\n\n/* some data which may have caused the problem */\n#data {\n}\n\n/* the error message received from the system or other software */\n#sysmsg {\n}\n\npre {\n}\n\n/* special event: FTP directory listing */\n#dirmsg {\n    font-family: courier, monospace;\n    color: black;\n    font-size: 10pt;\n}\n#dirlisting {\n    margin-left: 2%;\n    margin-right: 2%;\n}\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\n    border-bottom: groove;\n}\n#dirlisting td.size {\n    width: 50px;\n    text-align: right;\n    padding-right: 5px;\n}\n\n/* horizontal lines */\nhr {\n\tmargin: 0;\n}\n\n/* page displayed footer area */\n#footer {\n\tfont-size: 9px;\n\tpadding-left: 10px;\n}\n\n\nbody\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\n:lang(he) { direction: rtl; }\n --></style>\n</head><body id=ERR_CONNECT_FAIL>\n<div id="titles">\n<h1>ERROR</h1>\n<h2>The requested URL could not be retrieved</h2>\n</div>\n<hr>\n\n<div id="content">\n<p>The following error was encountered while trying to retrieve the URL: <a href="http://scopem-phobos.ethz.ch:40002/api">http://scopem-phobos.ethz.ch:40002/api</a></p>\n\n<blockquote id="error">\n<p><b>Connection to 129.132.174.233 failed.</b></p>\n</blockquote>\n\n<p id="sysmsg">The system returned: <i>(110) Connection timed out</i></p>\n\n<p>The remote host or network may be down. Please try the request again.</p>\n\n<p>Your cache administrator is <a href="mailto:root">root</a>.</p>\n\n<br>\n</div>\n\n<hr>\n<div id="footer">\n<p>Generated Fri, 05 Dec 2025 08:52:51 GMT by proxybd.ethz.ch (squid/4.15)</p>\n<!-- ERR_CONNECT_FAIL -->\n</div>\n</body></html>\n'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/cemkguest/software/cryosparc/cryosparc_master/cryosparc_compute/client.py", line 57, in <module>
    cli = CommandClient(host=host, port=int(port))
  File "/home/cemkguest/software/cryosparc/cryosparc_master/cryosparc_compute/client.py", line 38, in __init__
    super().__init__(service, host, port, url, timeout, headers, cls=NumpyEncoder)
  File "/home/cemkguest/software/cryosparc/cryosparc_master/cryosparc_tools/cryosparc/command.py", line 97, in __init__
    self._reload()  # attempt connection immediately to gather methods
  File "/home/cemkguest/software/cryosparc/cryosparc_master/cryosparc_tools/cryosparc/command.py", line 135, in _reload
    system = self._get_callable("system.describe")()
  File "/home/cemkguest/software/cryosparc/cryosparc_master/cryosparc_tools/cryosparc/command.py", line 108, in func
    raise CommandError(
cryosparc_tools.cryosparc.errors.CommandError: *** (http://scopem-phobos.ethz.ch:40002, code 503) Encounted error from JSONRPC function "system.describe" with params ()

sorry for all the troubles and thanks for looking into this with such patience!

20

@Tamino Have you already performed the multi-step shutdown procedure and subsequent restart of CryoSPARC? Please can you post the outputs of these commands:

ps -eo user:16,pid,ppid,start,cmd | grep -e cryosparc_ -e mongo
ls -l /tmp/cryosparc*.sock /tmp/mongo*.sock
grep -v LICENSE /home/cemkguest/software/cryosparc/cryosparc_master/config.sh
21

Dear @wtempel!

yes I did a shutdown of cryosparc, but there were no zombie processes to kill. I have not touched the .sock files (until now).

Here you see the output of the first command:

ps -eo user:16,pid,ppid,start,cmd | grep -e cryosparc_ -e mongo

cemkguest          18748    5060 16:32:52 grep --color=auto -e cryosparc_ -e mongo

This is the second command output.

sudo ls -l /tmp/cryosparc*.sock /tmp/mongo*.sock

srwx------ 1 cemkguest cemkguest 0 Dec  5 16:34 /tmp/cryosparc-supervisor-ac8d7489d682c54f43fdee4a1b5d8cb2.sock
srwx------ 1 cemkguest cemkguest 0 Dec  5 16:34 /tmp/mongodb-40001.sock

I tried to delete those two .socks files and then tried a restart of cryosparc. There was a different error message, but atleast not stuck at command_core:started, haha

cryosparcm restart
CryoSPARC is not already running.
If you would like to restart, use cryosparcm restart
Starting CryoSPARC System master process...
CryoSPARC is not already running.
configuring database...
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/cemkguest/software/cryosparc/cryosparc_master/cryosparc_compute/database_management.py", line 47, in configure_mongo
    initialize_replica_set()
  File "/home/cemkguest/software/cryosparc/cryosparc_master/cryosparc_compute/database_management.py", line 84, in initialize_replica_set
    admin_db = try_get_pymongo_db(mongo_client)
  File "/home/cemkguest/software/cryosparc/cryosparc_master/cryosparc_compute/database_management.py", line 251, in try_get_pymongo_db
    admin_db.command(({'serverStatus': 1}))
  File "/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/_csot.py", line 108, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/database.py", line 897, in command
    return self._command(
  File "/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/database.py", line 737, in _command
    return conn.command(
  File "/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/helpers.py", line 342, in inner
    return func(*args, **kwargs)
  File "/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/pool.py", line 989, in command
    return command(
  File "/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/network.py", line 212, in command
    helpers._check_command_response(
  File "/home/cemkguest/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/helpers.py", line 248, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: there are no users authenticated, full error: {'operationTime': Timestamp(1764949036, 4), 'ok': 0.0, 'errmsg': 'there are no users authenticated', 'code': 13, 'codeName': 'Unauthorized', '$clusterTime': {'clusterTime': Timestamp(1764949036, 4), 'signature': {'hash': b'\xfc\x7fQ\x82\xf6dFY\t2\xfa\x06\x99?\x00\x9f\x9d\xb5\x9d\xe3', 'keyId': 7522772700847144962}}}
[2025-12-05T16:37:18+01:00] Error configuring database. Most recent database log lines:
2025-12-05T16:37:04.576+0100 I STORAGE  [initandlisten] exception in initAndListen: DBPathInUse: Unable to lock the lock file: /home/cemkguest/software/cryosparc/cryosparc_database/mongod.lock (Resource temporarily unavailable). Another mongod instance is already running on the /home/cemkguest/software/cryosparc/cryosparc_database directory, terminating
2025-12-05T16:37:04.576+0100 I CONTROL  [initandlisten] now exiting
2025-12-05T16:37:04.576+0100 I CONTROL  [initandlisten] shutting down with code:100
2025-12-05T16:37:07.553+0100 I NETWORK  [listener] connection accepted from 127.0.0.1:49024 #7 (3 connections now open)
2025-12-05T16:37:07.553+0100 I NETWORK  [conn7] received client metadata from 127.0.0.1:49024 conn7: { driver: { name: "PyMongo", version: "4.8.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "6.14.0-34-generic" }, platform: "CPython 3.10.14.final.0" }
2025-12-05T16:37:07.554+0100 I NETWORK  [listener] connection accepted from 127.0.0.1:49040 #8 (4 connections now open)
2025-12-05T16:37:07.554+0100 I NETWORK  [conn8] received client metadata from 127.0.0.1:49040 conn8: { driver: { name: "PyMongo", version: "4.8.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "6.14.0-34-generic" }, platform: "CPython 3.10.14.final.0" }
2025-12-05T16:37:17.586+0100 I ACCESS   [conn8] Unauthorized: not authorized on admin to execute command { endSessions: [ { id: UUID("f392e0e6-fe18-4e2c-ae74-a13c11b9a837") } ], $clusterTime: { clusterTime: Timestamp(1764949036, 4), signature: { hash: BinData(0, FC7F5182F66446590932FA06993F009F9DB59DE3), keyId: 7522772700847144962 } }, $db: "admin", $readPreference: { mode: "primaryPreferred" } }
2025-12-05T16:37:17.586+0100 I NETWORK  [conn7] end connection 127.0.0.1:49024 (3 connections now open)
2025-12-05T16:37:17.586+0100 I NETWORK  [conn8] end connection 127.0.0.1:49040 (2 connections now open)

And here is the last output:

grep -v LICENSE /home/cemkguest/software/cryosparc/cryosparc_master/config.sh


# Instance Configuration
export CRYOSPARC_MASTER_HOSTNAME="scopem-phobos.ethz.ch"
export CRYOSPARC_DB_PATH="/home/cemkguest/software/cryosparc/cryosparc_database"
export CRYOSPARC_BASE_PORT=40000
export CRYOSPARC_DB_CONNECTION_TIMEOUT_MS=20000
export NO_PROXY="${CRYOSPARC_MASTER_HOSTNAME},localhost,127.0.0.1"
# Security
export CRYOSPARC_INSECURE=false
export CRYOSPARC_DB_ENABLE_AUTH=true

# Cluster Integration
export CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL=10
export CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES=1000000

# Project Configuration
export CRYOSPARC_PROJECT_DIR_PREFIX='CS-'

# Development
export CRYOSPARC_DEVELOP=false

# Other
export CRYOSPARC_CLICK_WRAP=true

I was wondering if there is a way to re-install cryosparc from scratch, but without touching the database? So that when cryosparc is reinstalled and I open the GUI, all the past project are visible. Do you think it might be easier that way, than to try and fix the issues?

best,
Tamino