Database EXITED after accessing projects in the GUI

Troubleshooting
1

The issue is that cryosparcm start works and the GUI is accessible until the user accesses the projects. Then first the database goes to EXITED status and soon after app goes to EXITED as well.

cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/beagle3/eperozo/cryosparc_phaller/master
Current cryoSPARC version: v4.1.2
----------------------------------------------------------------------------

CryoSPARC process status:

app                              EXITED    Jul 27 12:28 PM
app_api                          RUNNING   pid 450224, uptime 20:12:55
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     RUNNING   pid 450041, uptime 20:13:21
command_rtp                      RUNNING   pid 450090, uptime 20:13:04
command_vis                      RUNNING   pid 450087, uptime 20:13:06
database                         EXITED    Jul 27 12:28 PM

----------------------------------------------------------------------------
License is valid
----------------------------------------------------------------------------

global config variables:
export CRYOSPARC_LICENSE_ID=XXX
export CRYOSPARC_MASTER_HOSTNAME=XXX
export CRYOSPARC_HOSTNAME_CHECK=XXX
export CRYOSPARC_DB_PATH="/beagle3/eperozo/cryosparc_phaller/db"
export CRYOSPARC_BASE_PORT=39120
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=true
export CRYOSPARC_CLICK_WRAP=true

I tried cryosparcm log database but did not find a clear error and solution.

How should we proceed?

Thanks!

2

What does /beagle3/eperozo/cryosparc_phaller/master/run/database.log show for this time?

3 
2023-07-27T12:28:06.553-0500 E STORAGE  [conn10] WiredTiger error (-31802) [1690478886:553870][449922:0x7fea949e9700], file:collection-81-2194768026313995945.wt, WT_SESSION.open_cursor: __posix_file_read, 449: /beagle3/eperozo/cryosparc_phaller/db/collection-81-2194768026313995945.wt: handle-read: pread: failed to read 172032 bytes at offset 41187008512: WT_ERROR: non-specific WiredTiger error Raw: [1690478886:553870][449922:0x7fea949e9700], file:collection-81-2194768026313995945.wt, WT_SESSION.open_cursor: __posix_file_read, 449: /beagle3/eperozo/cryosparc_phaller/db/collection-81-2194768026313995945.wt: handle-read: pread: failed to read 172032 bytes at offset 41187008512: WT_ERROR: non-specific WiredTiger error
2023-07-27T12:28:06.554-0500 F -        [conn10] Invariant failure: ret resulted in status UnknownError: -31802: WT_ERROR: non-specific WiredTiger error at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 130
2023-07-27T12:28:06.554-0500 F -        [conn10] \n\n***aborting after invariant() failure\n\n
2023-07-27T12:28:06.554-0500 E STORAGE  [conn25] WiredTiger error (-31802) [1690478886:554985][449922:0x7fea93ddd700], file:collection-81-2194768026313995945.wt, WT_SESSION.open_cursor: __posix_file_read, 449: /beagle3/eperozo/cryosparc_phaller/db/collection-81-2194768026313995945.wt: handle-read: pread: failed to read 172032 bytes at offset 41187008512: WT_ERROR: non-specific WiredTiger error Raw: [1690478886:554985][449922:0x7fea93ddd700], file:collection-81-2194768026313995945.wt, WT_SESSION.open_cursor: __posix_file_read, 449: /beagle3/eperozo/cryosparc_phaller/db/collection-81-2194768026313995945.wt: handle-read: pread: failed to read 172032 bytes at offset 41187008512: WT_ERROR: non-specific WiredTiger error
2023-07-27T12:28:06.555-0500 F -        [conn25] Invariant failure: ret resulted in status UnknownError: -31802: WT_ERROR: non-specific WiredTiger error at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 130
2023-07-27T12:28:06.555-0500 F -        [conn25] \n\n***aborting after invariant() failure\n\n
2023-07-27T12:28:06.573-0500 F -        [conn26] Got signal: 6 (Aborted).
4

You may want to confirm:

  • CryoSPARC is always started by the non-root Linux account that owns the CryoSPARC installation
  • there are/were no disruptions to the filesystem that holds /beagle3/eperozo/cryosparc_phaller/db
  • the database is reserved for use by this CryoSPARC instance only

If these checks are okay, the database may be corrupt. If the database is corrupt and

  • a recent database backup exists and no CryoSPARC jobs were run after the backup, you can restore the database from the backup.
  • a recent database backup does not exist, you can rebuild the database as described in this post.