Upgrade OS change mount points etc

Data Management
1

Hi there,
We have a storage-server mounted on gpu-server (at e.g. /mnt/sto) on which cryoSPARC is running. I plan to upgrade both servers from CentOS7 to Ubuntu 22.04.

Thus, I plan to i) leave all data (~400 TB) on the storage-server ii) backup the cSPA database and iii) once the new cSPA instance is installed on the gpu-server the cSPA database will be restored (cryosparcm restore description in cryosparcm reference | CryoSPARC Guide ).

So far so good - but I would like to alter the mount point of the storage-server on the gpu-server (to e.g. /data/sto). Thus, all current data and symlink paths are not valid anymore.

There might be additional factors to consider in this whole upgrade process that I am currently not aware of. Could you tell me how to make this upgrade as smooth as possible so that users can seamlessly continue processing their data on the new, updated cSPA instance?

2

May I

  • emphasize the warning against restoring an outdated (w.r.t. the state of the project directory’s states at the time of database restoration) backup of the database
    and ask
  • whether the database directory is deleted or preserved during the OS upgrade?
  • whether it would be possible to maintain the /mnt/sto/ path after the upgrade as a symbolic link to the new mount point?
  • why Ubuntu-22.04 is preferred over Ubuntu-24.04?
3

Dear wtempel,

I am reaching out for advice, as I am not entirely sure what the best practice is in this situation. We have many unfinished projects (and a large amount of data) stored on our storage server, but the CentOS upgrade is necessary since we will no longer be able to run the latest version of CryoSPARC otherwise.

We currently have two GPU servers running CryoSPARC versions v4.6.2 and v4.5.1, both connected to a shared storage server. Since we already have a third GPU server and a second storage server running Ubuntu 22.04, we plan to upgrade these three remaining servers to the same OS version for consistency.

Our plan is to stop CryoSPARC, back up both databases, and then upgrade the operating systems on the three servers. The CryoSPARC backups and all project data are located on the storage server. To the best of my knowledge, the data on the storage server should not be affected by the OS upgrade.

Regarding mount points — symbolic links could be an option, but do you think it would generally be safer to keep all paths unchanged?
In your experience, what would be the safest and most reliable way to perform this upgrade?

4

Would the database directories (defined by CRYOSPARC_DB_PATH inside cryosparc_master/config.sh of each instance) be affected by the OS upgrade?

5

For one GPU server, it won’t be an issue if I mount the storage server in the same location it is mounted now. However, for the other server, the path will definitely change since its current location is not on the storage server.

6

To ensure I understand this correctly, please can you provide more details for the quoted comment?
Please can you also, for all servers you wish to upgrade, post the outputs of these commands

uname -a
hostname -f
ip -4 -br a
df -x tmpfs -Th
# if the the server is a cryosparc master
cd /path/to/cryosparc_master/ # replace actual path
pwd 
grep CRYOSPARC_DB_PATH config.sh
cat version
7

Dear wtempel,
I sent you the outputs info as a message.

8

Dear wtempel,

concerning “since its current location is not on the storage server”: This database is currently directed to an HDD which is also part of the host1 gpu-server (i.e. /data1). Thus, maybe this HDD will be wiped during the installation of Ubuntu.

please find outputs to commands below:
########################################################################

First GPU-server

$uname -a
Linux host1 3.10.0-1127.10.1.el7.x86_64 #1 SMP Wed Jun 3 14:28:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$hostname -f
host1

$ip -4 -br a
lo UNKNOWN 127.0.0.1/8
enp10s0f1 UP 140.XX.YY.ZZ/24
virbr0 DOWN 192.XX.XX.XX/24

$df -x tmpfs -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 63G 0 63G 0% /dev
/dev/md127 xfs 200G 102G 99G 51% /
/dev/md0 xfs 3.7T 3.2T 502G 87% /scr
/dev/md125 xfs 1020M 218M 803M 22% /boot
/dev/md124 vfat 201M 12M 190M 6% /boot/efi
/dev/md123 ext4 5.5T 3.7T 1.5T 71% /data1
140.XX.YY.ZZ:/storage1 nfs4 443T 399T 44T 90% /mnt/strand1
140.XX.YY.ZZ:/storage2 nfs4 217T 102T 116T 47% /mnt/strand2

master found in
/home/cryosparc/cryosparc2_master
$grep CRYOSPARC_DB_PATH config.sh
export CRYOSPARC_DB_PATH=“/data/cSPA_database”
$cat version
v4.6.2
########################################################################

Second GPU-server

$uname -a
Linux host2 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$hostname -f
host2

$ip -4 -br a
lo UNKNOWN 127.0.0.1/8
bond0 UP 140.XX.YY.ZZ/24

$df -x tmpfs -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 126G 0 126G 0% /dev
/dev/sda3 ext4 355G 298G 40G 89% /
/dev/sda1 vfat 1022M 12M 1011M 2% /boot/efi
/dev/md0 xfs 3.7T 864G 2.8T 24% /scr
140.XX.YY.ZZ:/storage1 nfs4 443T 399T 44T 90% /mnt/strand1
140.XX.YY.ZZ:/storage2 nfs4 217T 102T 116T 47% /mnt/strand2

master found in
/home/cryosparc/cryosparc_master
$grep CRYOSPARC_DB_PATH config.sh
export CRYOSPARC_DB_PATH=“/mnt/helix/cSPA_database/db_monster”
$cat version
v4.5.1

########################################################################

storage server

$uname -a
Linux host3 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$hostname -f
host3

$ip -4 -br a
Option “-br” is unknown, try “ip -help”. #also ip -4 as well as ip only prints the same as ip -help

$df -x tmpfs -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda2 xfs 210G 91G 120G 43% /
devtmpfs devtmpfs 16G 0 16G 0% /dev
/dev/sda1 ext4 488M 125M 328M 28% /boot
/dev/mapper/vg0-storage1 xfs 443T 399T 44T 90% /storage1
/dev/mapper/vg1-storage2 xfs 217T 102T 116T 47% /storage2

9

If the Linux user database, typically, but not always stored inside /etc/passwd, will be wiped out during the OS upgrade, make a note of the numeric user id of the Linux user that “owns” the CryoSPARC installation. Ensure that CryoSPARC commands and the database copy (see below) are run under the same numeric Linux user id before and after the OS update.

If changes to the CryoSPARC project directories’ absolute paths cannot not be avoided due to the update, apply the archive project action to all projects whose paths will change before shutting down CryoSPARC for the update. After the update and restarting CryoSPARC, one may unarchive projects at their changed absolute paths.

If I understand correctly, the CryoSPARC database for “host1” is stored on a system disk. If so, may I suggest:

  1. stopping (with confirmation) CryoSPARC on “host1”. Afterward, do not interact with the CryoSPARC UI until the OS update is complete (to avoid inconsistencies between the copied database and the state of project directories).
  2. copying the database directory (see precautions) to suitable (reliable, continuously monitored for capacity) alternative storage that will host the database after the OS update.
  3. for redundancy, also creating a backup at a suitable off-system location with
    cryosparcm backup with the --dir option. Because CryoSPARC was supposed to be stopped in an earlier step, the cryosparcm backup command will start the CryoSPARC database. After cryosparcm backup has completed, run again cryosparcm stop to shutdown the database.
  4. after the system update, but before starting CryoSPARC, editing cryosparc_master/config.sh to update the CRYOSPARC_DB_PATH definition to the new database path.

If the paths of imported movies or other imported data change during the update update symbolic links to imported data.

10

Dear wtempel,

thanks for your sugestions.

Concerning the archiving/unarchiving I read in the note under the section Unarchive “should only be used with the intention of keeping the project tied to the current instance of CryoSPARC. Users looking to transfer projects between CryoSPARC instance should refer to the Attach and Detach features instead.”

In addition, during the OS upgrade the CS instance will certainly change and also the licence will change since the previous “CS-admin” left the lab.

Will it therefore not be easier if I

  1. clear all intermediate results of all projects (optional to saving space on disc)
  2. detache all projects
  3. upgrade the OS
  4. install CS with my licence
  5. attache all projects back in the new CS instance.

To the best of my knowledge I would not have to deal with the database and the altered path issue would be solved as well, isn’t it?

Best,
JMB

11

This should (almost) work. If CryoSPARC is being reinstalled with a blank database, you will also have to

  • before attaching projects, create CryoSPARC logins. On CryoSPARC v4.7.1, for a single workstation installation with the
    cryosparc_master/install.sh --standalone option, the first (admin) login would be created when running the cryosparc_master/install.sh script. Otherwise, one would use the cryosparcm createuser command. Additional users can be created in the GUI.
  • if this is not a “single workstation” that was installed with the cryosparc_master/install.sh --standalone option, connect node- or cluster-type scheduler targets/lanes before running the first CryoSPARC job.