WebGUI partly unresponsive, freezing cryosparc

KiSchnelle · August 5, 2022, 2:06pm

We getting more often the problem of our cryosparc instance “freezing”. The gui gets partly unresponsive. You can still navigate inside projects and workspaces but you can create/clone/kill jobs. Also killing via resource manager doesnt work anymore. This also happens to live.

Normally a restart of cryosparc solves it, but sometimes its just happens again.

Anyone having an idea or facing the same?

In app log i see some of this

MongoTimeoutError: Server selection timed out after 10000 ms
    at Timeout._onTimeout (/home/cryosparcuser/cryosparc_master/cryosparc_app/api/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/sdam/topology.js:773:16)
    at listOnTimeout (internal/timers.js:554:17)
    at processTimers (internal/timers.js:497:7) {
  [Symbol(mongoErrorContextSymbol)]: {}
}
(node:3379814) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:3379814) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
(node:3379814) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
(node:3379814) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
/home/cryosparcuser/cryosparc_master/cryosparc_app/api/bundle/programs/server/node_modules/fibers/future.js:313
                                                throw(ex);
                                                ^

in command_rtp

2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    | Traceback (most recent call last):
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_command/command_rtp/__init__.py", line 173, in wrapper
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |     res = func(*args, **kwargs)
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_command/command_rtp/__init__.py", line 1703, in start_rtp_worker
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |     cli.enqueue_job(project_uid = project_uid, job_uid = rtp_worker_juid, lane = lane_name)
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_compute/client.py", line 57, in func
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |     r = requests.post(self.url, data = json.dumps(data, cls=NumpyEncoder), headers = header, timeout=self.timeout)
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/api.py", line 119, in post
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |     return request('post', url, data=data, json=json, **kwargs)
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/api.py", line 61, in request
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |     return session.request(method=method, url=url, **kwargs)
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |     resp = self.send(prep, **send_kwargs)
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |     r = adapter.send(request, **kwargs)
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |     raise ReadTimeout(e, request=request)
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    | requests.exceptions.ReadTimeout: HTTPConnectionPool(host='kermit103', port=39002): Read timed out. (read timeout=300)
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    | 
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    | The above exception was the direct cause of the following exception:
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    | 
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    | Traceback (most recent call last):
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_command/command_rtp/__init__.py", line 173, in wrapper
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |     res = func(*args, **kwargs)
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_command/command_rtp/__init__.py", line 1687, in build_rtp_worker
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |     start_rtp_worker(project_uid, session_uid, rtp_worker_juid, lane_name)
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_command/command_rtp/__init__.py", line 180, in wrapper
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    |     raise ServerError(s.getvalue(), code=400) from e
2022-08-05 13:40:39,280 RTP.MAIN             wrapper              ERROR    | flask_jsonrpc.exceptions.ServerError
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | JSONRPC ERROR at start_session
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | Traceback (most recent call last):
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     six.raise_from(e, None)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "<string>", line 3, in raise_from
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     httplib_response = conn.getresponse()
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/http/client.py", line 1354, in getresponse
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     response.begin()
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/http/client.py", line 306, in begin
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     version, status, reason = self._read_status()
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/http/client.py", line 267, in _read_status
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/socket.py", line 589, in readinto
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     return self._sock.recv_into(b)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | socket.timeout: timed out
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | 
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | During handling of the above exception, another exception occurred:
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | 
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | Traceback (most recent call last):
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     timeout=timeout
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/util/retry.py", line 410, in increment
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     raise six.reraise(type(error), error, _stacktrace)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     raise value
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     chunked=chunked,
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 428, in _make_request
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     self, url, "Read timed out. (read timeout=%s)" % timeout_value
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='kermit103', port=39002): Read timed out. (read timeout=300)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | 
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | During handling of the above exception, another exception occurred:
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | 
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | Traceback (most recent call last):
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_command/command_rtp/__init__.py", line 173, in wrapper
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     res = func(*args, **kwargs)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_command/command_rtp/__init__.py", line 1703, in start_rtp_worker
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     cli.enqueue_job(project_uid = project_uid, job_uid = rtp_worker_juid, lane = lane_name)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_compute/client.py", line 57, in func
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     r = requests.post(self.url, data = json.dumps(data, cls=NumpyEncoder), headers = header, timeout=self.timeout)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/api.py", line 119, in post
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     return request('post', url, data=data, json=json, **kwargs)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/api.py", line 61, in request
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     return session.request(method=method, url=url, **kwargs)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     resp = self.send(prep, **send_kwargs)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     r = adapter.send(request, **kwargs)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     raise ReadTimeout(e, request=request)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | requests.exceptions.ReadTimeout: HTTPConnectionPool(host='kermit103', port=39002): Read timed out. (read timeout=300)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | 
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | The above exception was the direct cause of the following exception:
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | 
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | Traceback (most recent call last):
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_command/command_rtp/__init__.py", line 173, in wrapper
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     res = func(*args, **kwargs)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_command/command_rtp/__init__.py", line 1687, in build_rtp_worker
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     start_rtp_worker(project_uid, session_uid, rtp_worker_juid, lane_name)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_command/command_rtp/__init__.py", line 180, in wrapper
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     raise ServerError(s.getvalue(), code=400) from e
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | flask_jsonrpc.exceptions.ServerError
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | 
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | The above exception was the direct cause of the following exception:
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | 
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | Traceback (most recent call last):
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_command/command_rtp/__init__.py", line 173, in wrapper
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     res = func(*args, **kwargs)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_command/command_rtp/__init__.py", line 893, in start_session
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     start=True)
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |   File "/home/cryosparcuser/cryosparc_master/cryosparc_command/command_rtp/__init__.py", line 180, in wrapper
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    |     raise ServerError(s.getvalue(), code=400) from e
2022-08-05 13:40:39,282 RTP.MAIN             wrapper              ERROR    | flask_jsonrpc.exceptions.ServerError
*** client.py: command (http://kermit103:39002/api) did not reply within timeout of 300 seconds, attempt 3 of 3

sdawood · August 6, 2022, 2:19pm

Hi @KiSchnelle ,

Thanks for the post - sorry to hear you’re having trouble using cryoSPARC. In order for us to debug better, could you please email us a copy of the following logs (feedback@structura.bio):

command_core.log
command_vis.log
database.log
webapp.log

You can find these files in the directory [...]/cryosparc_master/run

Regards,
Suhail

KiSchnelle · August 8, 2022, 8:13am

I send a link via mail to the university share for downloading them, since they were to big to send as mail attachment:D (151MB)

Also the normal webapp is not accessible anymore, just loading forever in browser without error, tested on multiple machines. cryosparc live is accessible though.

cheers
Kilian