Disconnecting of the database

[Elad]

Hi
I have repeating problem that the browser is not responsive and when I type “cryosparcm status” I get the all running but this:
database EXITED

any advice,

Best,

[stephan]

Hey @Elad,

Can you send us the database logs?

https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring/cryosparcm#logs

Before You Post: Troubleshooting Guidelines

[image] Dear cryoSPARC users, Thank you for communicating with us on the discussion forum. To make it easier for us to understand and troubleshoot issues and errors encountered, please use the following template to provide as much information as possible about your issue: Environment cryoSPARC version cryosparcm status CUDA version cat cryosparc2_worker/config.sh, check the path to “CUDA PATH” OS (Ubuntu, CentOS, etc) Any other relevant configurations (cluster, headless, etc) Issue T…

[Elad]

bash-4.2$ cryosparcm log database
2020-11-27T13:31:42.841-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:43740 #174 (34 connections now open)
2020-11-27T13:31:42.842-0800 I NETWORK  [conn174] received client metadata from 10.141.27.250:43740 conn174: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T13:32:02.747-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:43770 #175 (35 connections now open)
2020-11-27T13:32:02.748-0800 I NETWORK  [conn175] received client metadata from 10.141.27.250:43770 conn175: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T13:32:02.748-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:43772 #176 (36 connections now open)
2020-11-27T13:32:02.755-0800 I NETWORK  [conn176] received client metadata from 10.141.27.250:43772 conn176: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T13:32:18.376-0800 I -        [conn175] end connection 10.141.27.250:43770 (36 connections now open)
2020-11-27T13:32:18.377-0800 I -        [conn176] end connection 10.141.27.250:43772 (36 connections now open)
2020-11-27T13:32:18.376-0800 I -        [conn172] end connection 10.141.27.250:43734 (36 connections now open)
2020-11-27T13:32:18.376-0800 I -        [conn174] end connection 10.141.27.250:43740 (36 connections now open)
2020-11-27T13:32:18.922-0800 I -        [conn173] end connection 10.141.27.250:43738 (32 connections now open)
2020-11-27T13:32:18.922-0800 I -        [conn171] end connection 10.141.27.250:43732 (32 connections now open)
2020-11-27T13:32:49.928-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:43862 #177 (31 connections now open)
2020-11-27T13:32:49.928-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:43864 #178 (32 connections now open)
2020-11-27T13:32:49.928-0800 I NETWORK  [conn177] received client metadata from 10.141.27.250:43862 conn177: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T13:32:49.930-0800 I NETWORK  [conn178] received client metadata from 10.141.27.250:43864 conn178: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T13:32:49.935-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:43868 #179 (33 connections now open)
2020-11-27T13:32:49.936-0800 I NETWORK  [conn179] received client metadata from 10.141.27.250:43868 conn179: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T13:32:49.957-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:43870 #180 (34 connections now open)
2020-11-27T13:32:49.957-0800 I NETWORK  [conn180] received client metadata from 10.141.27.250:43870 conn180: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T13:33:09.228-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:43900 #181 (35 connections now open)
2020-11-27T13:33:09.234-0800 I NETWORK  [conn181] received client metadata from 10.141.27.250:43900 conn181: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T13:33:09.237-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:43902 #182 (36 connections now open)
2020-11-27T13:33:09.237-0800 I NETWORK  [conn182] received client metadata from 10.141.27.250:43902 conn182: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T13:33:24.766-0800 I -        [conn178] end connection 10.141.27.250:43864 (36 connections now open)
2020-11-27T13:33:24.766-0800 I -        [conn181] end connection 10.141.27.250:43900 (36 connections now open)
2020-11-27T13:33:24.766-0800 I -        [conn180] end connection 10.141.27.250:43870 (36 connections now open)
2020-11-27T13:33:24.766-0800 I -        [conn182] end connection 10.141.27.250:43902 (36 connections now open)
2020-11-27T13:33:25.030-0800 I -        [conn179] end connection 10.141.27.250:43868 (32 connections now open)
2020-11-27T13:33:25.030-0800 I -        [conn177] end connection 10.141.27.250:43862 (32 connections now open)
2020-11-27T13:34:02.642-0800 I COMMAND  [conn30] command meteor.events command: find { find: "events", filter: { project_uid: "P2", job_uid: "J76", type: "checkpoint" }, projection: {} } planSummary: IXSCAN { project_uid: 1, job_uid: 1, created_at: 1 } keysExamined:4107 docsExamined:4107 fromMultiPlanner:1 replanned:1 cursorExhausted:1 numYields:104 nreturned:9 reslen:1163 locks:{ Global: { acquireCount: { r: 210 } }, Database: { acquireCount: { r: 105 } }, Collection: { acquireCount: { r: 105 } } } protocol:op_query 231ms
2020-11-27T13:41:02.727-0800 I -        [conn139] end connection 10.141.27.250:42924 (30 connections now open)
2020-11-27T13:41:02.727-0800 I -        [conn141] end connection 10.141.27.250:42930 (30 connections now open)
2020-11-27T13:41:03.030-0800 I -        [conn138] end connection 10.141.27.250:42922 (28 connections now open)
2020-11-27T13:41:03.030-0800 I -        [conn140] end connection 10.141.27.250:42928 (28 connections now open)
2020-11-27T14:04:10.794-0800 I -        [conn168] end connection 10.141.27.250:43636 (26 connections now open)
2020-11-27T14:04:10.794-0800 I -        [conn169] end connection 10.141.27.250:43640 (26 connections now open)
2020-11-27T14:04:10.794-0800 I -        [conn170] end connection 10.141.27.250:43670 (26 connections now open)
2020-11-27T14:04:11.282-0800 I -        [conn167] end connection 10.141.27.250:43628 (23 connections now open)
2020-11-27T14:04:11.282-0800 I -        [conn166] end connection 10.141.27.250:43626 (22 connections now open)
2020-11-27T15:28:58.561-0800 I -        [conn161] end connection 10.141.27.250:43404 (21 connections now open)
2020-11-27T15:28:58.561-0800 I -        [conn159] end connection 10.141.27.250:43398 (21 connections now open)
2020-11-27T15:28:59.763-0800 I -        [conn160] end connection 10.141.27.250:43402 (19 connections now open)
2020-11-27T15:28:59.763-0800 I -        [conn158] end connection 10.141.27.250:43396 (19 connections now open)
2020-11-27T18:10:38.156-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:45910 #183 (18 connections now open)
2020-11-27T18:10:38.157-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:45912 #184 (19 connections now open)
2020-11-27T18:10:38.157-0800 I NETWORK  [conn183] received client metadata from 10.141.27.250:45910 conn183: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:10:38.158-0800 I NETWORK  [conn184] received client metadata from 10.141.27.250:45912 conn184: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:10:38.164-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:45916 #185 (20 connections now open)
2020-11-27T18:10:38.164-0800 I NETWORK  [conn185] received client metadata from 10.141.27.250:45916 conn185: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:10:38.187-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:45918 #186 (21 connections now open)
2020-11-27T18:10:38.188-0800 I NETWORK  [conn186] received client metadata from 10.141.27.250:45918 conn186: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:11:04.338-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:45936 #187 (22 connections now open)
2020-11-27T18:11:04.338-0800 I NETWORK  [conn187] received client metadata from 10.141.27.250:45936 conn187: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:11:46.481-0800 I -        [conn187] end connection 10.141.27.250:45936 (22 connections now open)
2020-11-27T18:11:46.481-0800 I -        [conn186] end connection 10.141.27.250:45918 (22 connections now open)
2020-11-27T18:11:46.481-0800 I -        [conn184] end connection 10.141.27.250:45912 (22 connections now open)
2020-11-27T18:11:46.863-0800 I -        [conn185] end connection 10.141.27.250:45916 (19 connections now open)
2020-11-27T18:11:46.863-0800 I -        [conn183] end connection 10.141.27.250:45910 (19 connections now open)
2020-11-27T18:14:02.259-0800 I COMMAND  [conn14] command meteor.fs.chunks command: find { find: "fs.chunks", filter: { files_id: ObjectId('5f4e814ca703dfa211e626df') }, sort: { n: 1 }, returnKey: false, showRecordId: false } planSummary: IXSCAN { files_id: 1, n: 1 } keysExamined:1 docsExamined:1 cursorExhausted:1 numYields:1 nreturned:1 reslen:63466 locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_query 101ms
2020-11-27T18:14:02.259-0800 I COMMAND  [conn16] command meteor.fs.chunks command: find { find: "fs.chunks", filter: { files_id: ObjectId('5f4e814ca703dfa211e626e4') }, sort: { n: 1 }, returnKey: false, showRecordId: false } planSummary: IXSCAN { files_id: 1, n: 1 } keysExamined:1 docsExamined:1 cursorExhausted:1 numYields:1 nreturned:1 reslen:63097 locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_query 100ms
2020-11-27T18:20:11.606-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46044 #188 (18 connections now open)
2020-11-27T18:20:11.606-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46046 #189 (19 connections now open)
2020-11-27T18:20:11.606-0800 I NETWORK  [conn189] received client metadata from 10.141.27.250:46046 conn189: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:20:11.606-0800 I NETWORK  [conn188] received client metadata from 10.141.27.250:46044 conn188: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:20:11.614-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46050 #190 (20 connections now open)
2020-11-27T18:20:11.615-0800 I NETWORK  [conn190] received client metadata from 10.141.27.250:46050 conn190: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:20:11.637-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46052 #191 (21 connections now open)
2020-11-27T18:20:11.637-0800 I NETWORK  [conn191] received client metadata from 10.141.27.250:46052 conn191: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:20:19.898-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46092 #192 (22 connections now open)
2020-11-27T18:20:19.919-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46094 #193 (23 connections now open)
2020-11-27T18:20:19.919-0800 I NETWORK  [conn192] received client metadata from 10.141.27.250:46092 conn192: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:20:19.919-0800 I NETWORK  [conn193] received client metadata from 10.141.27.250:46094 conn193: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:20:19.920-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46096 #194 (24 connections now open)
2020-11-27T18:20:19.920-0800 I NETWORK  [conn194] received client metadata from 10.141.27.250:46096 conn194: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:20:27.846-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46102 #195 (25 connections now open)
2020-11-27T18:20:27.846-0800 I NETWORK  [conn195] received client metadata from 10.141.27.250:46102 conn195: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:20:27.862-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46104 #196 (26 connections now open)
2020-11-27T18:20:27.862-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46106 #197 (27 connections now open)
2020-11-27T18:20:27.863-0800 I NETWORK  [conn196] received client metadata from 10.141.27.250:46104 conn196: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:20:27.863-0800 I NETWORK  [conn197] received client metadata from 10.141.27.250:46106 conn197: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:20:27.867-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46108 #198 (28 connections now open)
2020-11-27T18:20:27.868-0800 I NETWORK  [conn198] received client metadata from 10.141.27.250:46108 conn198: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:20:27.878-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46110 #199 (29 connections now open)
2020-11-27T18:20:27.878-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46112 #200 (30 connections now open)
2020-11-27T18:20:27.878-0800 I NETWORK  [conn199] received client metadata from 10.141.27.250:46110 conn199: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:20:27.878-0800 I NETWORK  [conn200] received client metadata from 10.141.27.250:46112 conn200: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:20:38.697-0800 I -        [conn193] end connection 10.141.27.250:46094 (30 connections now open)
2020-11-27T18:20:38.777-0800 I -        [conn192] end connection 10.141.27.250:46092 (29 connections now open)
2020-11-27T18:20:38.849-0800 I -        [conn196] end connection 10.141.27.250:46104 (28 connections now open)
2020-11-27T18:20:38.850-0800 I -        [conn199] end connection 10.141.27.250:46110 (27 connections now open)
2020-11-27T18:20:38.851-0800 I -        [conn200] end connection 10.141.27.250:46112 (26 connections now open)
2020-11-27T18:20:38.851-0800 I -        [conn197] end connection 10.141.27.250:46106 (26 connections now open)
2020-11-27T18:20:39.107-0800 I -        [conn194] end connection 10.141.27.250:46096 (24 connections now open)
2020-11-27T18:20:39.715-0800 I -        [conn198] end connection 10.141.27.250:46108 (23 connections now open)
2020-11-27T18:20:39.715-0800 I -        [conn195] end connection 10.141.27.250:46102 (23 connections now open)
2020-11-27T18:20:40.132-0800 I -        [conn191] end connection 10.141.27.250:46052 (21 connections now open)
2020-11-27T18:20:40.132-0800 I -        [conn188] end connection 10.141.27.250:46044 (21 connections now open)
2020-11-27T18:20:40.709-0800 I -        [conn190] end connection 10.141.27.250:46050 (19 connections now open)
2020-11-27T18:20:40.709-0800 I -        [conn189] end connection 10.141.27.250:46046 (18 connections now open)
2020-11-27T18:21:06.713-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46148 #201 (18 connections now open)
2020-11-27T18:21:06.714-0800 I NETWORK  [conn201] received client metadata from 10.141.27.250:46148 conn201: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:21:06.718-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46152 #202 (19 connections now open)
2020-11-27T18:21:06.718-0800 I NETWORK  [conn202] received client metadata from 10.141.27.250:46152 conn202: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:21:06.726-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46154 #203 (20 connections now open)
2020-11-27T18:21:06.726-0800 I NETWORK  [conn203] received client metadata from 10.141.27.250:46154 conn203: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:21:06.745-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46156 #204 (21 connections now open)
2020-11-27T18:21:06.746-0800 I NETWORK  [conn204] received client metadata from 10.141.27.250:46156 conn204: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:21:16.054-0800 I WRITE    [conn13] update meteor.cache_files query: { status: { $in: [ "hit", "miss" ] }, hostname: "mamba" } planSummary: IXSCAN { status: 1 } update: { $set: { status: "miss" } } keysExamined:20704 docsExamined:20704 nMatched:10352 nModified:10352 keysInserted:10352 keysDeleted:10352 numYields:165 locks:{ Global: { acquireCount: { r: 10518, w: 10518 } }, Database: { acquireCount: { w: 10518 } }, Collection: { acquireCount: { w: 166 } }, Metadata: { acquireCount: { w: 10352 } }, oplog: { acquireCount: { w: 10352 } } } 378ms
2020-11-27T18:21:16.054-0800 I COMMAND  [conn13] command meteor.$cmd command: update { update: "cache_files", ordered: true, updates: [ { q: { status: { $in: [ "hit", "miss" ] }, hostname: "mamba" }, u: { $set: { status: "miss" } }, multi: true, upsert: false } ] } numYields:0 reslen:119 locks:{ Global: { acquireCount: { r: 10518, w: 10518 } }, Database: { acquireCount: { w: 10518 } }, Collection: { acquireCount: { w: 166 } }, Metadata: { acquireCount: { w: 10352 } }, oplog: { acquireCount: { w: 10352 } } } protocol:op_query 378ms
2020-11-27T18:25:16.691-0800 I -        [conn201] end connection 10.141.27.250:46148 (21 connections now open)
2020-11-27T18:25:16.691-0800 I -        [conn204] end connection 10.141.27.250:46156 (21 connections now open)
2020-11-27T18:25:17.583-0800 I -        [conn203] end connection 10.141.27.250:46154 (19 connections now open)
2020-11-27T18:25:17.583-0800 I -        [conn202] end connection 10.141.27.250:46152 (19 connections now open)
2020-11-27T18:25:44.207-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46298 #205 (18 connections now open)
2020-11-27T18:25:44.207-0800 I NETWORK  [conn205] received client metadata from 10.141.27.250:46298 conn205: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:25:44.213-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46302 #206 (19 connections now open)
2020-11-27T18:25:44.214-0800 I NETWORK  [conn206] received client metadata from 10.141.27.250:46302 conn206: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:25:44.221-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46304 #207 (20 connections now open)
2020-11-27T18:25:44.222-0800 I NETWORK  [conn207] received client metadata from 10.141.27.250:46304 conn207: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:25:44.237-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46306 #208 (21 connections now open)
2020-11-27T18:25:44.237-0800 I NETWORK  [conn208] received client metadata from 10.141.27.250:46306 conn208: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:26:11.911-0800 I -        [conn205] end connection 10.141.27.250:46298 (21 connections now open)
2020-11-27T18:26:11.911-0800 I -        [conn208] end connection 10.141.27.250:46306 (21 connections now open)
2020-11-27T18:26:12.303-0800 I -        [conn207] end connection 10.141.27.250:46304 (19 connections now open)
2020-11-27T18:26:12.303-0800 I -        [conn206] end connection 10.141.27.250:46302 (18 connections now open)
2020-11-27T18:28:33.473-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46402 #209 (18 connections now open)
2020-11-27T18:28:33.474-0800 I NETWORK  [conn209] received client metadata from 10.141.27.250:46402 conn209: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:28:33.474-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46404 #210 (19 connections now open)
2020-11-27T18:28:33.475-0800 I NETWORK  [conn210] received client metadata from 10.141.27.250:46404 conn210: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:28:33.483-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46408 #211 (20 connections now open)
2020-11-27T18:28:33.483-0800 I NETWORK  [conn211] received client metadata from 10.141.27.250:46408 conn211: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:28:33.505-0800 I NETWORK  [thread1] connection accepted from 10.141.27.250:46410 #212 (21 connections now open)
2020-11-27T18:28:33.505-0800 I NETWORK  [conn212] received client metadata from 10.141.27.250:46410 conn212: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.7.1908 Core", architecture: "x86_64", version: "3.10.0-1062.9.1.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2020-11-27T18:28:47.769-0800 I WRITE    [conn5] update meteor.cache_files query: { status: { $in: [ "hit", "miss" ] }, hostname: "mamba" } planSummary: IXSCAN { status: 1 } update: { $set: { status: "miss" } } keysExamined:20790 docsExamined:20790 nMatched:10395 nModified:10395 keysInserted:10395 keysDeleted:10395 numYields:165 locks:{ Global: { acquireCount: { r: 10561, w: 10561 } }, Database: { acquireCount: { w: 10561 } }, Collection: { acquireCount: { w: 166 } }, Metadata: { acquireCount: { w: 10395 } }, oplog: { acquireCount: { w: 10395 } } } 386ms
2020-11-27T18:28:47.769-0800 I COMMAND  [conn5] command meteor.$cmd command: update { update: "cache_files", ordered: true, updates: [ { q: { status: { $in: [ "hit", "miss" ] }, hostname: "mamba" }, u: { $set: { status: "miss" } }, multi: true, upsert: false } ] } numYields:0 reslen:119 locks:{ Global: { acquireCount: { r: 10561, w: 10561 } }, Database: { acquireCount: { w: 10561 } }, Collection: { acquireCount: { w: 166 } }, Metadata: { acquireCount: { w: 10395 } }, oplog: { acquireCount: { w: 10395 } } } protocol:op_query 386ms
2020-11-27T18:49:45.738-0800 I -        [conn209] end connection 10.141.27.250:46402 (21 connections now open)
2020-11-27T18:49:45.738-0800 I -        [conn212] end connection 10.141.27.250:46410 (21 connections now open)
2020-11-27T18:49:47.156-0800 I -        [conn210] end connection 10.141.27.250:46404 (19 connections now open)
2020-11-27T18:49:47.156-0800 I -        [conn211] end connection 10.141.27.250:46408 (19 connections now open)
2020-11-27T19:26:15.015-0800 E STORAGE  [thread2] WiredTiger error (17) [1606533975:15774][47286:0x2b7490bde700], file:WiredTiger.wt, WT_SESSION.checkpoint: /home/binshtem/software/cryosparc/cryosparc2_database/WiredTiger.turtle.set: handle-open: open: File exists
2020-11-27T19:26:15.017-0800 E STORAGE  [thread2] WiredTiger error (17) [1606533975:17282][47286:0x2b7490bde700], checkpoint-server: checkpoint server error: File exists
2020-11-27T19:26:15.017-0800 E STORAGE  [thread2] WiredTiger error (-31804) [1606533975:17312][47286:0x2b7490bde700], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2020-11-27T19:26:15.017-0800 I -        [thread2] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2020-11-27T19:26:15.017-0800 I -        [thread2] 

***aborting after fassert() failure


2020-11-27T19:26:15.021-0800 F -        [thread2] Got signal: 6 (Aborted).

 0x561091a5cd6b 0x561091a5bfe6 0x561091a5c4c3 0x2b748f3f45f0 0x2b748f637337 0x2b748f638a28 0x561090c7b887 0x561091753597 0x561090c85e75 0x561090c8608c 0x561090c862f5 0x561092406ae4 0x2b748f3ece65 0x2b748f6ff88d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"56109063C000","o":"1420D6B","s":"_ZN5mongo15printStackTraceERSo"},{"b":"56109063C000","o":"141FFE6"},{"b":"56109063C000","o":"14204C3"},{"b":"2B748F3E5000","o":"F5F0"},{"b":"2B748F601000","o":"36337","s":"gsignal"},{"b":"2B748F601000","o":"37A28","s":"abort"},{"b":"56109063C000","o":"63F887","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"56109063C000","o":"1117597"},{"b":"56109063C000","o":"649E75","s":"__wt_eventv"},{"b":"56109063C000","o":"64A08C","s":"__wt_err"},{"b":"56109063C000","o":"64A2F5","s":"__wt_panic"},{"b":"56109063C000","o":"1DCAAE4"},{"b":"2B748F3E5000","o":"7E65"},{"b":"2B748F601000","o":"FE88D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.10-4-g67ee356c6b", "gitVersion" : "67ee356c6be377cda547d16423daef5beb4e8377", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-1062.9.1.el7.x86_64", "version" : "#1 SMP Fri Dec 6 15:49:49 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "56109063C000", "elfType" : 3, "buildId" : "A14732EAAA95508462633A5030B501E08B1B106F" }, { "b" : "7FFC504F2000", "elfType" : 3, "buildId" : "B5A5458535A1397FA6BAAF5E8C13A6395426A1B2" }, { "b" : "2B748E298000", "path" : "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/lib/libpython2.7.so.1.0", "elfType" : 3 }, { "b" : "2B748E498000", "path" : "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/bin/../lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "2B748E70E000", "path" : "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/bin/../lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "2B748EB4F000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "4749697BF078337576C4629F0D30B296A0939779" }, { "b" : "2B748ED57000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "18113E6E83D8E981B8E8D808F7F3DBB23F950A1D" }, { "b" : "2B748EF5B000", "path" : "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/bin/../lib/libstdc++.so.6", "elfType" : 3 }, { "b" : "2B748F0CF000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5681C054FDABCF789F4DDA66E94F1F6ED1747327" }, { "b" : "2B748F3D1000", "path" : "/home/binshtem/software/cryosparc/cryosparc2_master/deps/anaconda/bin/../lib/libgcc_s.so.1", "elfType" : 3 }, { "b" : "2B748F3E5000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "8B33F7F8C86F8D544C63C5541A8E42B3DDFEF8B1" }, { "b" : "2B748F601000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "398944D32CF16A67AF51067A326E6C0CC14F90ED" }, { "b" : "2B748F9CF000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "E0D39E293DC99997E7B4C9B6203301E6CD904B50" }, { "b" : "2B748E274000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5CC1A53B747A7E4D21198723C2B633E54F3C06D9" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x3B) [0x561091a5cd6b]
 mongod(+0x141FFE6) [0x561091a5bfe6]
 mongod(+0x14204C3) [0x561091a5c4c3]
 libpthread.so.0(+0xF5F0) [0x2b748f3f45f0]
 libc.so.6(gsignal+0x37) [0x2b748f637337]
 libc.so.6(abort+0x148) [0x2b748f638a28]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x561090c7b887]
 mongod(+0x1117597) [0x561091753597]
 mongod(__wt_eventv+0x3E4) [0x561090c85e75]
 mongod(__wt_err+0xA0) [0x561090c8608c]
 mongod(__wt_panic+0x2F) [0x561090c862f5]
 mongod(+0x1DCAAE4) [0x561092406ae4]
 libpthread.so.0(+0x7E65) [0x2b748f3ece65]
 libc.so.6(clone+0x6D) [0x2b748f6ff88d]
-----  END BACKTRACE  -----

[donghuachen]

Hi All,

I got the same problem (database EXITED, others seemed normal based on the output of cryosparcm status) and the browser was not responsive after running normally for a couple of minutes. I can restart cryosparc v3.3.1+220315 again by running cryosparcm restart, sometimes cryosparc can restart successfully (and just run well for a few minutes), but sometimes I got a Connection Refused error like this post (https://discuss.cryosparc.com/t/pymongo-errors-serverselectiontimeouterror-localhost-errno-111-connection-refused/2830/4).
Any suggestion? Thanks in advance!