cryoSPARC update/re-installation

Install, Configure and Update
1

Dear cryoSPARC-Devs and Users,

I have a question regarding the upgrade of cryoSPARC.

Currently, we’re running cryoSPARC v3.3.2 in a main:worker (7 workers) configuration. The main interface is running on Centos 7, which hits EOL soon. So I not only want to upgrade cryoSPARC but also the OS.

And regarding this, I would like to have some feedback if my initial thoughts are correct:

  • backup the cryoSPARC database to an external server (the usual database dump). Also, backup the cryoSPARC config (for the license key). Also, simply copy the database folder
  • backup ssh-keys, host-key, etc., which is important for the interconnection between the machines.
  • change to OS from centos to Ubuntu
  • re-configure the OS (ssh keys, network mounts, etc.)
  • copy the database folder back into place

And now, I was thinking, I should go this way:

  • Install cryoSPARC v.3.3.2 and define the database folder during installation. By this, I should have all jobs back in cryoSPARC.
  • Upgrade to v4.x

I was thinking first going to v.3.3.2 again, as (based on my knowledge from the v4 release) the database from v4 is not compatible with v.3.3. So by taking the route via the old version, I let cryoSPARC take care of the database migration.

Is this the correct way to go?

Also going from v3.3.2 to v.4.x, is there anything additional, I need to take into consideration?

Best
Christian

2

Out of curiosity, based on your data storage arrangements, would it be easy to not wipe the CryoSPARC software installation and database during OS installation?

3

Not sure what exactly you mean. The cryosparc_master is on the OS’s root partition, so it gets lost during reinstallation.

Currently, cryosparc is located in the home directory:
$ which cryosparcm
~/cryosparc/cryosparc_master/bin/cryosparc

Can I just copy the entire cryosparc folder to a external storage, and after installation of ubuntu just move it back? And how is this different to reinstall with --dbpath <db_path> set to the database location, a folder I can easily copy/move/backup.

4

The OS installation may be an opportunity to separate the (new) CryoSPARC installation from the root filesystem. In my experience, this separation increases flexibility from a sys admin perspective.

I think it is a good idea to both dump the database and, after stopping CryoSPARC (please confirm no mongod process is running with
ps -eopid,ppid,cmd|grep mongo
), copy the database directory to storage that will not be affected by the OS re-installation.
A copy of config.sh that you also mentioned can serve as reference for relevant details like $CRYOSPARC_BASE_PORT/--port.
You may install CryoSPARC v4.2.1 without first installing v3.3.2. During installation, you should specify a temporary --dbpath where a blank database will be created.
After installation, you can

  1. stop CryoSPARC
  2. confirm that all CryoSPARC processes have indeed been terminated
    ps -eopid,ppid,cmd | grep -e cryosparc -e mongo
  3. edit the CRYOSPARC_DB_PATH= variable inside cryposparc_master/config.sh to point to the directory of the database copy (or a copy thereof) that you prepared before OS installation.
  4. after cryosparcm start, run
    cryosparcm listusers to confirm that the new CryoSPARC installation indeed accesses the information of the old database.
1 Like
5

Thank you for the detailed anwser.

It’s at pleasant surprise to see, that one can directly from v3.3.2 to v.4.x during reinstallation.

I guess, one must only reconnect all the workers afterwards? But this is pretty easy and straight forward.

I will also take into consideration to place the master on a drive, separated from the OS. Our systems are historically grown without a dedicated admin in the beginning, which is also the reason, that there is still centos running on those machines.

But for future operations, if the master is on a isolated disk, I can simply reinstall any OS, just re-add the bashrc entry, and can directly start cryosparc again?

I think this topic can be flagged solved/closed. The update is queued for mid august, I just wanted to clarify these things before running into issue. If there is any other issue than, I‘ll open a new, more specific issue.

Thanks,
Christian

6

Reconnection may not be necessary, as connection information is stored in the database. If the path of your worker installation changes, you may have to update the database records for workers with
the set_scheduler_target_property() cli command
Related: If your CryoSPARC instance is still running, you may want to record the output of the command

cryosparcm cli "get_scheduler_targets()"

for future reference.

… and ensure that CryoSPARC remains compatible with external dependencies. For example, if the path or version of the external CUDA toolkit has changed, you would run

cryosparcw newcuda /path/to/new/cuda

for each affected worker installation (details).
Alternatively, you could also separate the CUDA installation from the OS installation, in which case you would need to ensure that the toolkit installation remains compatible with the new OS installation. Note: The CUDA example is valid for v4.2.1 without 3DFlex dependencies.

7

Alright, thank you for these insights.

This is just for the master node, so there is no CUDA available.
For the workers, I am actually a fan of a clean re-installation, if anything changes.

So based on this, I can also do the following before the OS update:

  • Install a SSD drive in the machine
  • Stop cryosparc
  • (Optional, but highly recommended) make a backup of the database
  • Copy everything to the new SSD
  • (Optional) verify file integrity with hashes
  • Change paths in the master/config.sh
  • Change paths in the .bashrc

Restart cryoSPARC to see, if everything is fine. If so, I can simply change from CentOS to Ubuntu, update the .bashrc, and just start cryoSPARC. If everything runs fine at this stage, I can simply upgrade cryoSPARC to v.4.x from here.

Can you confirm this?

Thank you very much again,
Christian

8

One can change the CRYOSPARC_DB_PATH inside cryosparc_master/config.sh to point to a different path of the database directory. However, a change of /path/to/cryosparc_master will break the installation.

9

Hmm alright, so there is no way of moving the master directory, just the database.

So I’ll have to reinstall cryoSPARC at some point, to get it to a permanent storage drive, separated from the OS drive.

Thank for all this input, this will be helping me alot :slight_smile: