Refused connection when cryosparc is running

Install, Configure and Update
1

Cryosparc encountered a failure after some time, displaying a ‘localhost refused connection’ error. I believe my internet connection is stable. Occasionally, it runs smoothly for a few days, but other times, the issue occurs approximately every a couple of hours.

single workstation
Current cryoSPARC version: v4.4.0
Linux cryo 6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 2 18:01:13 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
total used free shared buff/cache available
Mem: 503 35 81 14 385 449
Swap: 1 0 1

CRYOSPARC_PATH=/home/cryosparc/cryosparc4.2.2/cryosparc_worker/bin
XRDP_SOCKET_PATH=/run/xrdp/sockdir
PYTHONPATH=/home/cryosparc/cryosparc4.2.2/cryosparc_worker
CRYOSPARC_SSD_PATH=/mnt/zhitai
CRYOSPARC_CUDA_PATH=/usr/local/cuda
NUMBA_CUDA_INCLUDE_PATH=/home/cryosparc/cryosparc4.2.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include
LD_LIBRARY_PATH=
PATH=/home/cryosparc/cryosparc4.2.2/cryosparc_worker/bin:/home/cryosparc/cryosparc4.2.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/home/cryosparc/cryosparc4.2.2/cryosparc_worker/deps/anaconda/condabin:/home/cryosparc/anaconda3/bin:/home/cryosparc/anaconda3/condabin:/home/cryosparc/cryosparc4.2.2/cryosparc_master/bin:/home/cryosparc/cryosparc4.2.2/cryosparc_master/bin:/usr/local/relion/build/bin:/usr/local/IMOD/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/IMOD/pythonLink

CryoSPARC System master node installed at
/home/cryosparc/cryosparc4.2.2/cryosparc_master
Current cryoSPARC version: v4.4.0

CryoSPARC process status:

app RUNNING pid 201153, uptime 0:16:16
app_api RUNNING pid 201176, uptime 0:16:14
app_api_dev STOPPED Not started
command_core RUNNING pid 201050, uptime 0:16:29
command_rtp RUNNING pid 201110, uptime 0:16:21
command_vis RUNNING pid 201086, uptime 0:16:22
database RUNNING pid 200936, uptime 0:16:33

License is valid

global config variables:
export CRYOSPARC_LICENSE_ID=" "
export CRYOSPARC_MASTER_HOSTNAME=“cryo”
export CRYOSPARC_DB_PATH=“/home/cryosparc/cryosparc4.2.2/cryosparc_database”
export CRYOSPARC_BASE_PORT=61000
export CRYOSPARC_DB_CONNECTION_TIMEOUT_MS=20000
export CRYOSPARC_INSECURE=false
export CRYOSPARC_DB_ENABLE_AUTH=true
export CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL=10
export CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES=1000000
export CRYOSPARC_PROJECT_DIR_PREFIX=‘CS-’
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_CLICK_WRAP=true

2

Welcome to the forum @cornpeasant.

Please can you

  1. describe where and during which activity you encounter the errors
  2. how you connect to CryoSPARC:
    • the full url you type in the browser
    • any ssh or vpn tunnel?
    • is your browser running on the single workstation or on a separate computer?
  3. outputs of these commands on the CryoSPARC workstation
    host cryo
host localhost
  4. a screenshot of the browser window when the error occurs
3

  1. describe where and during which activity you encounter the errors
    It happens for any activity that takes long time such as 2-D classification, template picking, heterogenous refinement

  2. how you connect to CryoSPARC:

  • the full url you type in the browser

    screenshot of error
    screenshot of error1702×1276 95.1 KB

  • any ssh or vpn tunnel?
    No ssh or vpn

  • is your browser running on the single workstation or on a separate computer?
    The browser is running on the same ubuntu workstation, and accessed by XRDP remote desktop.

  1. outputs of these commands on the CryoSPARC workstation
host cryo
host localhost

微信图片_202312031001442
微信图片_2023120310014421920×1080 245 KB

  1. a screenshot of the browser window when the error occurs

This is the screenshot of refreshed browser.

screenshot of refreshed browser
screenshot of refreshed browser1702×1276 194 KB

4

Thanks @cornpeasant for posting this information.
You may want to go through the multistep procedure (simple CryoSPARC stop, discovery and confirmed termination of stale processes and then, and only then, deletion of stale socket files, details) of a complete CryoSPARC shutdown.
After the complete CryoSPARC shutdown, please attempt
cryosparcm start.
If you encounter any problems, please post

  1. error messages
  2. outputs of the commands
    ps -weopid,ppid,start,cmd | grep -e cryosparc -e mongo | grep -v grep
ls -l /tmp/cryosparc*.sock /tmp/mongodb-*.sock
free -g
  3. details about potential concurrent compute workloads on the server that may cause the system to run out of available RAM.
5

Hi Wtempel,

Thank you for the help.
Instead of upgrading Cryosparc 4.4.0 from the old version, I made a fresh installation, but the same problem still happened after cryosparc ran for ~28 hours (I tried twice).

Here is the output of your instructed procedure.

$cryosparcm stop
CryoSPARC is running.
Stopping cryoSPARC
unix:///tmp/cryosparc-supervisor-5fccf1c670aab55f9d50ce55f18e4c54.sock refused connection

$ ps -w -U user1 -opid,ppid,start,cmd | grep -e cryosparc -e mongo | grep -v grep

NO OUTPUT.

THEN I DELETED THE SOCK FILE:

$ rm /tmp/cryosparc-supervisor-5fccf1c670aab55f9d50ce55f18e4c54.sock

$ cryosparcm start
Starting cryoSPARC System master process..
CryoSPARC is not already running.
configuring database
    configuration complete
database: started
checkdb success
command_core: started
    command_core connection succeeded
    command_core startup successful
command_vis: started
command_rtp: started
    command_rtp connection succeeded
    command_rtp startup successful
app: started
app_api: started
-----------------------------------------------------

CryoSPARC master started.
 From this machine, access CryoSPARC and CryoSPARC Live at
    http://localhost:61000

 From other machines on the network, access CryoSPARC and CryoSPARC Live at
    http://cryo:61000


Startup can take several minutes. Point your browser to the address
and refresh until you see the cryoSPARC web interface.



ps -weopid,ppid,start,cmd | grep -e cryosparc -e mongo | grep -v grep



$ ps -weopid,ppid,start,cmd | grep -e cryosparc -e mongo | grep -v grep
  82204    2765 12:58:51 python /home/jz/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/jz/cryosparc/cryosparc_master/supervisord.conf
  82319   82204 12:58:57 mongod --auth --dbpath /home/jz/cryosparc/cryosparc_database --port 61001 --oplogSize 64 --replSet meteor --wiredTigerCacheSizeGB 4 --bind_ip_all
  82430   82204 12:59:01 python -c import cryosparc_command.command_core as serv; serv.start(port=61002)
  82468   82204 12:59:08 python -c import cryosparc_command.command_vis as serv; serv.start(port=61003)
  82492   82204 12:59:09 python -c import cryosparc_command.command_rtp as serv; serv.start(port=61005)
  82556   82204 12:59:14 /home/jz/cryosparc/cryosparc_master/cryosparc_app/nodejs/bin/node ./bundle/main.js
    82589   82430 12:59:18 bash /home/jz/cryosparc/cryosparc_worker/bin/cryosparcw run --project P1 --job J33 --master_hostname cryo --master_command_core_port 61002
  82604   82589 12:59:18 python -c import cryosparc_compute.run as run; run.run() --project P1 --job J33 --master_hostname cryo --master_command_core_port 61002
  82606   82604 12:59:18 python -c import cryosparc_compute.run as run; run.run() --project P1 --job J33 --master_hostname cryo --master_command_core_port 61002
  82609   82430 12:59:20 bash /home/jz/cryosparc/cryosparc_worker/bin/cryosparcw run --project P1 --job J34 --master_hostname cryo --master_command_core_port 61002
  82624   82609 12:59:20 python -c import cryosparc_compute.run as run; run.run() --project P1 --job J34 --master_hostname cryo --master_command_core_port 61002
  82626   82624 12:59:20 python -c import cryosparc_compute.run as run; run.run() --project P1 --job J34 --master_hostname cryo --master_command_core_port 61002


$ ls -l /tmp/cryosparc*.sock /tmp/mongodb-*.sock
srwx------ 1 jz jz 0 12月  6 12:58 /tmp/cryosparc-supervisor-5fccf1c670aab55f9d50ce55f18e4c54.sock
srwx------ 1 jz jz 0 12月  6 12:58 /tmp/mongodb-61001.sock


$ free -g
               total        used        free      shared  buff/cache   available
Mem:             503          13          41           0         448         485
Swap:              1           0           1

I only run cryosparc on the workstation, so there should be enough RAM.

6

I checked the command_core log file and there are some errors:

2023-12-06 13:18:27,427 run                  ERROR    | Encountered exception while running background task
2023-12-06 13:18:27,427 run                  ERROR    | Traceback (most recent call last):
2023-12-06 13:18:27,427 run                  ERROR    |   File "cryosparc_master/cryosparc_command/core.py", line 1115, in cryosparc_master.cryosparc_command.core.background_tasks_worker
2023-12-06 13:18:27,427 run                  ERROR    |   File "/home/jz/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 186, in wrapper
2023-12-06 13:18:27,427 run                  ERROR    |     return func(*args, **kwargs)
2023-12-06 13:18:27,427 run                  ERROR    |   File "/home/jz/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 232, in wrapper
2023-12-06 13:18:27,427 run                  ERROR    |     return func(*args, **kwargs)
2023-12-06 13:18:27,427 run                  ERROR    |   File "/home/jz/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 3924, in dump_job_database
2023-12-06 13:18:27,427 run                  ERROR    |     rc.dump_job_database(project_uid = project_uid, job_uid = job_uid, job_completed = job_completed, migration = migration, abs_export_dir = abs_export_dir, logger = logger)
2023-12-06 13:18:27,427 run                  ERROR    |   File "/home/jz/cryosparc/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 444, in dump_job_database
2023-12-06 13:18:27,427 run                  ERROR    |     file_object = gridfs.get(objectid.ObjectId(object_id)).read()
2023-12-06 13:18:27,427 run                  ERROR    |   File "/home/jz/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/gridfs/__init__.py", line 153, in get
2023-12-06 13:18:27,427 run                  ERROR    |     gout._ensure_file()
2023-12-06 13:18:27,427 run                  ERROR    |   File "/home/jz/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/gridfs/grid_file.py", line 484, in _ensure_file
2023-12-06 13:18:27,427 run                  ERROR    |     raise NoFile(
2023-12-06 13:18:27,427 run                  ERROR    | gridfs.errors.NoFile: no file in gridfs collection Collection(Database(MongoClient(host=['cryo:61001'], document_class=dict, tz_aware=False, connect=False, authsource='admin'), 'meteor'), 'fs.files') with _id ObjectId('656ffffcfe82afe930c2f357')
7

Thanks @cornpeasant for this info.

This command would have displayed processes owned by (hypothetical) Linux user user1, and needs to be modified to match your circumstances (details).

Please can you similarly check the database log for errors.
(cryosparcm log database)

8

@cornpeasant Please can you confirm that certain ports based on the configured CRYOSPARC_MASTER_HOSTNAME and CRYOSPARC_BASE_PORT are accessible.
For example, what is the output of the command

curl cryo:61006

?

9

Screenshot from 2023-12-20 10-38-44
Screenshot from 2023-12-20 10-38-442410×1322 303 KB

10

Thanks @cornpeasant. I cannot spot in the screenshot a reason for the UI failure. The next step would be an analysis of browser logs.
With CryoSPARC running, please

  1. enable browser debugging
  2. re-load the UI at http://localhost:61000
  3. email us the the HAR network output
11

Hi wtempel,
I found that the issue was very likely due to high cache/buffer usage and extremely low available memory on the Linux system. I now clear the cache/buffer every hour using a script, and cryoSPARC can run smoothly so far.

1 Like