Command_core FATAL

Hi -

I was about to update from v 4.02 to 4.1. Before doing it, as I stopped and re-started Cryosparc, I got an “pymongo.errors.OperationFailure: there are no users authenticated, full error:” which I solved thanks to another post by stopping cryosparc and killing an orphan mongod job.

then, I managed to re-start cryosparc, but notice the status below:


$ cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master
Current cryoSPARC version: v4.0.2
----------------------------------------------------------------------------

CryoSPARC process status:

app                              EXITED    Jan 12 08:50 AM
app_api                          STARTING  
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     FATAL     Exited too quickly (process log may have details)
command_rtp                      RUNNING   pid 2948, uptime 0:00:02
command_vis                      RUNNING   pid 2862, uptime 0:00:07
database                         RUNNING   pid 2147, uptime 0:01:05

----------------------------------------------------------------------------
License is valid
----------------------------------------------------------------------------

when running cryosparcm log command_core, I see:

  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 990, in run
    run_simple(host, port, self, **options)
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/werkzeug/serving.py", line 1052, in run_simple
    inner()
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/werkzeug/serving.py", line 1005, in inner
    fd=fd,
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/werkzeug/serving.py", line 848, in make_server
    host, port, app, request_handler, passthrough_errors, ssl_context, fd=fd
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/werkzeug/serving.py", line 740, in __init__
    HTTPServer.__init__(self, server_address, handler)
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/socketserver.py", line 452, in __init__
    self.server_bind()
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/http/server.py", line 137, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/socketserver.py", line 466, in server_bind
    self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
2023-01-12 08:50:30,345 COMMAND.STARTUP      startup              INFO     | CryoSPARC instance ID: f2b48e65-83de-47b5-b69c-2dd8cc7d36ea
2023-01-12 08:50:30,345 COMMAND.SCHEDULER    get_gpu_info         INFO     | UPDATING WORKER GPU INFO
2023-01-12 08:50:30,345 COMMAND.JOBS         update_all_job_sizes INFO     | UPDATING ALL JOB SIZES IN 10s
2023-01-12 08:50:30,345 COMMAND.DATA         export_all_projects  INFO     | EXPORTING ALL PROJECTS IN 60s...
2023-01-12 08:50:32,956 COMMAND.MAIN         start                INFO     |  === STARTED === 
2023-01-12 08:50:32,956 COMMAND.BG_WORKER    background_worker    INFO     |  === STARTED === 
2023-01-12 08:50:32,956 COMMAND.CORE         run                  INFO     | === STARTED TASKS WORKER ===
 * Serving Flask app "command_core" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 209, in start
    app.run(host="0.0.0.0", port=port, threaded=True, passthrough_errors=False)
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 990, in run
    run_simple(host, port, self, **options)
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/werkzeug/serving.py", line 1052, in run_simple
    inner()
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/werkzeug/serving.py", line 1005, in inner
    fd=fd,
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/werkzeug/serving.py", line 848, in make_server
    host, port, app, request_handler, passthrough_errors, ssl_context, fd=fd
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/werkzeug/serving.py", line 740, in __init__
    HTTPServer.__init__(self, server_address, handler)
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/socketserver.py", line 452, in __init__
    self.server_bind()
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/http/server.py", line 137, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/socketserver.py", line 466, in server_bind
    self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use

Before moving forward with the update, I’d like to know what happened and if anything should / could be done to deal with this “FATAL” status of the command_core?

Thanks,
Best,
Nicolas

more info:L I followed suggestions in Regarding installation of version 3.2 - #4 by PRIYANKA
to kill processed associated with “CRYOSPARC_BASE_PORT”, but the error was still there after re-starting cryosparc.

Finally found it another jobs “supervisord.conf” in the background that needed to be killed. It looks like this solved the issue for the moment.

1 Like