Cryosparc does not start after backup

Install, Configure and Update
1

Hi Cryosparc,

I have recently run a backup, and now my cryosparc instance does not start correctly. It seems to get stuck trying to start the command core: see below

CryoSPARC process status:

app STOPPED Not started
app_api STOPPED Not started
app_api_dev STOPPED Not started
command_core STARTING
command_rtp STOPPED Not started
command_vis STOPPED Not started
database RUNNING pid 2427339, uptime 0:02:12

Attempt 1/3 to GET http://nodelogin02:41012/startup failed with exception: 500 Server Error: INTERNAL SERVER ERROR for url: http://nodelogin02:41012/startup
Retrying…
Attempt 2/3 to GET http://nodelogin02:41012/startup failed with exception: 500 Server Error: INTERNAL SERVER ERROR for url: http://nodelogin02:41012/startup
Retrying…
Attempt 3/3 to GET http://nodelogin02:41012/startup failed with exception: 500 Server Error: INTERNAL SERVER ERROR for url: http://nodelogin02:41012/startup
Failed to GET http://nodelogin02:41012/startup

Can anyone please help?

2

Welcome to the forum @Tijnvdvelden .
Please can you provide the following details

  1. Do you mean you created a backup copy or you tried to restore from a backup? Please describe the circumstances and commands involved.
  2. Are you aiming for a single workstation setup or planning to run job on separate worker nodes?
  3. With the database running, as shown in please run the following commands and post their outputs
    ps -eo user:12,pid,ppid,start,command | grep -e cryosparc_ -e mongo
ls -l /tmp/mongo*.sock /tmp/cryosparc*.sock
host nodelogin02
curl nodelogin02:41011
curl 127.0.0.1:41011
cryosparcm log command_core | tail -n 30
3

Dear Wtempel,

I tried to restore a backup, and run on seperate worker nodes. I got notified by our IT department that there was a ghost process still running that I hadn’t seen. Killing this process solved the issue and CryoSPARC starts again.

Unfortunately, I am running into a new issue. While the backup has restored all the jobs I have run, they do not seem to be occupied. The jobs do show, but I cant download the results. When looking into the specific job folder I only find the events.bson, an empty gridfs_data folder, and the job.json file. Has something gone wrong during the recovery of the backup? The image below is from a 2D classification job.

J67_Folder
J67_Folder1455×179 4.88 KB

4

Please can you post commands and other details (such as which directories were backed up) that were used during backup and restoration.

5

The instalation of CryoSPARC was done by our HPC support. The folowing commands were used for backup and restoration of the database, I did not specify a directory to back up.

cryosparcm backup

cryosparcm restore --file=/path/to/backup/cryosparc_backup_2024_11_15_13h57.archive

6

THe command

creates a backup of the CryoSPARC database, but not CryoSPARC project directories. For CryoSPARC to function, a database backup would need to be restored and paired with CryoSPARC project directories whose state must not have changed after the database backaup has been initiated.
The screenshot showing event.bson, gridfs_data/ and job.json corresponds to the database-derived components of a CryoSPARC job directory (a subdirectory of a project directory). I speculate that the job may have been cleared before that database backup has been initiated or *.cs and *.mrc files have been deleted after the database backup. Can you confirm what happened to the project directories after the database backup?