Kill signal sent by CryoSPARC (ID: <Heartbeat Monitor>

2D Classification
1

Hi all,
I have been using the version: v4.3.1 and I am getting job failure with following message.
This was for a 3D classification job, but I have seen it in 2D classification as well as Homogeneous refinement. Not sure what is causing this, any suggestion for what could be the source of this issue.

many thanks
Man

[CPU: 77.47 GB]

-------------- Iteration 35 --------------
[CPU: 77.47 GB]

[Full EM] Spooled entire dataset, 7568350 particles
[CPU: 77.41 GB]

Computing expectation step (batch A), processed 593000 / 3784175 particles THR 0 TOTAL 3004002.8 ELAPSED 4759.8222 –

**** Kill signal sent by CryoSPARC (ID: ) ****

2

Please can you post the bottom 20 or so lines from the job log (under Metadata|Log of the affected job).

3

Here are the details, las lines of the meta log

return get_job(project_uid, job_uid, 'status')['status']

File “/l_mnt/scratch/d/cryosparc/cryosparc2_master/cryosparc_command/commandcommon.py”, line 186, in wrapper
return func(*args, **kwargs)
File “/l_mnt/scratch/d/cryosparc/cryosparc2_master/cryosparc_command/command_core/init.py”, line 5839, in get_job
job = com.get_job(mongo.db, project_uid, job_uid, *args, **kwargs)
File “/l_mnt/scratch/d/cryosparc/cryosparc2_master/cryosparc_compute/jobs/common.py”, line 657, in get_job
return db[‘jobs’].find_one({‘project_uid’:project_uid, ‘uid’:job_uid}, fields)
File “/l_mnt/scratch/d/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/collection.py”, line 1491, in find_one
for result in cursor.limit(-1):
File “/l_mnt/scratch/d/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/cursor.py”, line 1280, in next
if len(self.__data) or self._refresh():
File “/l_mnt/scratch/d/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/cursor.py”, line 1165, in _refresh
self.__session = self.__collection.database.client._ensure_session()
File “/l_mnt/scratch/d/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py”, line 2027, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File “/l_mnt/scratch/d/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py”, line 1976, in __start_session
server_session = self._get_server_session()
File “/l_mnt/scratch/d/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py”, line 2013, in _get_server_session
return self._topology.get_server_session()
File “/l_mnt/scratch/d/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py”, line 525, in get_server_session
session_timeout = self._check_session_support()
File “/l_mnt/scratch/d/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py”, line 507, in _check_session_support
self._select_servers_loop(
File “/l_mnt/scratch/d/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py”, line 226, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: cryosparc.cryst.bbk.ac.uk:39001: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 655205d048256eaeb50e807f, topology_type: Single, servers: [<ServerDescription (‘cryosparc.cryst.bbk.ac.uk’, 39001) server_type: Unknown, rtt: None, error=AutoReconnect(‘cryosparc.cryst.bbk.ac.uk:39001: [Errno 111] Connection refused’)>]>

4

@Man_at_Work Please can you open the affected job in the UI, select the Metadata tab, then the Log subtab.
image
I would like to see approx. 20 lines from the bottom of that log.

5

image
image1920×915 212 KB

6

@Man_at_Work Please post error messages as text to make it easier for forum visitors to find relevant topics.

7

@Man_at_Work Are you still encountering this problem?

If you do, please can you post:

  1. the output of the commands
    ps -weopid,ppid,start,cmd | grep -e cryosparc -e mongo | grep -v grep
curl cryosparc.cryst.bbk.ac.uk:39001
host cryosparc.cryst.bbk.ac.uk
curl 127.0.0.1:39001
  2. error messages you may observe in the output of the command
    cryosparcm log database