I updated the cryosparc from 3.3.2 to 4.1.1 and get spawn errors. I made sure that no cryosparc jobs were orphaned and no mongod or supervisor files were in /tmp.
[cryosparc_user@xingcryoem cryosparc_user]$ ls /tmp
cryosparc-supervisor-4c4a4bf2c532bf2738317c6734588d17.sock
firefox
hsperfdata_tomcat
mongodb-39001.sock
pymp-2lVaqH
pymp-DfUqc1
pymp-fbeG20
pymp-yukEwF
rclone-cron.log
systemd-private-187902e8aef241af97b368b801496c00-bolt.service-H0eVvW
systemd-private-187902e8aef241af97b368b801496c00-chronyd.service-Tm9VYo
systemd-private-187902e8aef241af97b368b801496c00-colord.service-3ZUsm9
systemd-private-187902e8aef241af97b368b801496c00-cups.service-3zUVnV
systemd-private-187902e8aef241af97b368b801496c00-rtkit-daemon.service-Flb8E0
yxing2
[cryosparc_user@xingcryoem cryosparc_user]$ rm /tmp/cryosparc-supervisor-4c4a4bf2c532bf2738317c6734588d17.sock
[cryosparc_user@xingcryoem cryosparc_user]$ rm /tmp/mongodb-39001.sock
[cryosparc_user@xingcryoem cryosparc_user]$ cryosparcm stop
CryoSPARC is not already running.
If you would like to restart, use cryosparcm restart
[cryosparc_user@xingcryoem cryosparc_user]$ cryosparcm start
Starting cryoSPARC System master process..
CryoSPARC is not already running.
configuring database
configuration complete
database: started
checkdb success
command_core: ERROR (spawn error)
[cryosparc_user@xingcryoem cryosparc_user]$ cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master
Current cryoSPARC version: v4.1.1
----------------------------------------------------------------------------
CryoSPARC process status:
app STOPPED Not started
app_api STOPPED Not started
app_api_dev STOPPED Not started
app_legacy STOPPED Not started
app_legacy_dev STOPPED Not started
command_core BACKOFF Exited too quickly (process log may have details)
command_rtp STOPPED Not started
command_vis STOPPED Not started
database RUNNING pid 107267, uptime 0:00:14
----------------------------------------------------------------------------
An error ocurred while checking license status
Could not get license verification status. Are all CryoSPARC processes RUNNING?
[cryosparc_user@xingcryoem cryosparc_user]$ curl https://get.cryosparc.com/checklicenseexists/$LICENSE_ID
{"success": true}[cryosparc_user@xingcryoem cryosparc_user]$
The database comes up but no other jobs will spawn as seen in the status. Command core exited too quickly.
The log file is too long but here is the last few lines.
2023-01-12 15:57:57,423 COMMAND.BG_WORKER background_worker INFO | === STARTED ===
2023-01-12 15:57:57,423 COMMAND.CORE run INFO | === STARTED TASKS WORKER ===
* Serving Flask app "command_core" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 211, in sta
rt
app.run(host="0.0.0.0", port=port, threaded=True, passthrough_errors=False)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/flask/app.py", line 990, in run
run_simple(host, port, self, **options)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 1052, in run_simple
inner()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 996, in inner
srv = make_server(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 847, in make_server
return ThreadedWSGIServer(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 740, in __init__
HTTPServer.__init__(self, server_address, handler)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 452, in __init__
self.server_bind()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/http/se
rver.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
2023-01-12 16:13:04,087 COMMAND.MAIN start INFO | === STARTED ===
2023-01-12 16:13:04,087 COMMAND.BG_WORKER background_worker INFO | === STARTED ===
2023-01-12 16:13:04,088 COMMAND.CORE run INFO | === STARTED TASKS WORKER ===
* Serving Flask app "command_core" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 211, in sta
rt
app.run(host="0.0.0.0", port=port, threaded=True, passthrough_errors=False)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/flask/app.py", line 990, in run
run_simple(host, port, self, **options)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 1052, in run_simple
inner()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 996, in inner
srv = make_server(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 847, in make_server
return ThreadedWSGIServer(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 740, in __init__
HTTPServer.__init__(self, server_address, handler)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 452, in __init__
self.server_bind()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/http/se
rver.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
2023-01-12 16:13:09,342 COMMAND.MAIN start INFO | === STARTED ===
2023-01-12 16:13:09,343 COMMAND.BG_WORKER background_worker INFO | === STARTED ===
2023-01-12 16:13:09,344 COMMAND.CORE run INFO | === STARTED TASKS WORKER ===
* Serving Flask app "command_core" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 211, in sta
rt
app.run(host="0.0.0.0", port=port, threaded=True, passthrough_errors=False)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/flask/app.py", line 990, in run
run_simple(host, port, self, **options)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 1052, in run_simple
inner()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 996, in inner
srv = make_server(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 847, in make_server
return ThreadedWSGIServer(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 740, in __init__
HTTPServer.__init__(self, server_address, handler)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 452, in __init__
self.server_bind()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/http/se
rver.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
2023-01-12 16:13:15,576 COMMAND.MAIN start INFO | === STARTED ===
2023-01-12 16:13:15,577 COMMAND.BG_WORKER background_worker INFO | === STARTED ===
2023-01-12 16:13:15,577 COMMAND.CORE run INFO | === STARTED TASKS WORKER ===
* Serving Flask app "command_core" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 211, in sta
rt
app.run(host="0.0.0.0", port=port, threaded=True, passthrough_errors=False)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/flask/app.py", line 990, in run
run_simple(host, port, self, **options)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 1052, in run_simple
inner()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 996, in inner
srv = make_server(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 847, in make_server
return ThreadedWSGIServer(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 740, in __init__
HTTPServer.__init__(self, server_address, handler)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 452, in __init__
self.server_bind()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/http/se
rver.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
2023-01-12 16:22:19,484 COMMAND.MAIN start INFO | === STARTED ===
2023-01-12 16:22:19,485 COMMAND.BG_WORKER background_worker INFO | === STARTED ===
2023-01-12 16:22:19,486 COMMAND.CORE run INFO | === STARTED TASKS WORKER ===
* Serving Flask app "command_core" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 211, in sta
rt
app.run(host="0.0.0.0", port=port, threaded=True, passthrough_errors=False)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/flask/app.py", line 990, in run
run_simple(host, port, self, **options)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 1052, in run_simple
inner()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 996, in inner
srv = make_server(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 847, in make_server
return ThreadedWSGIServer(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 740, in __init__
HTTPServer.__init__(self, server_address, handler)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 452, in __init__
self.server_bind()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/http/se
rver.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
2023-01-12 16:22:24,671 COMMAND.MAIN start INFO | === STARTED ===
2023-01-12 16:22:24,671 COMMAND.BG_WORKER background_worker INFO | === STARTED ===
2023-01-12 16:22:24,672 COMMAND.CORE run INFO | === STARTED TASKS WORKER ===
* Serving Flask app "command_core" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 211, in sta
rt
app.run(host="0.0.0.0", port=port, threaded=True, passthrough_errors=False)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/flask/app.py", line 990, in run
run_simple(host, port, self, **options)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 1052, in run_simple
inner()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 996, in inner
srv = make_server(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 847, in make_server
return ThreadedWSGIServer(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 740, in __init__
HTTPServer.__init__(self, server_address, handler)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 452, in __init__
self.server_bind()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/http/se
rver.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
2023-01-12 16:22:30,939 COMMAND.MAIN start INFO | === STARTED ===
2023-01-12 16:22:30,940 COMMAND.BG_WORKER background_worker INFO | === STARTED ===
2023-01-12 16:22:30,940 COMMAND.CORE run INFO | === STARTED TASKS WORKER ===
* Serving Flask app "command_core" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 211, in sta
rt
app.run(host="0.0.0.0", port=port, threaded=True, passthrough_errors=False)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/flask/app.py", line 990, in run
run_simple(host, port, self, **options)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 1052, in run_simple
inner()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 996, in inner
srv = make_server(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 847, in make_server
return ThreadedWSGIServer(
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-pa
ckages/werkzeug/serving.py", line 740, in __init__
HTTPServer.__init__(self, server_address, handler)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 452, in __init__
self.server_bind()
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/http/se
rver.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "/mnt/ssd/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/sockets
erver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
[cryosparc_user@xingcryoem cryosparc_user]$ mv t.t command-core.txt
[cryosparc_user@xingcryoem cryosparc_user]$