Multiple cryoSPARC Master Installations on one Computer

Hi,

We have a SLURM cluster where the login-node has been configured to use multiple cryoSPARC Master installations for each computer.

I configured it to use a different port range (incrementing by 20) and a different database directory (users home directory) for each cluster user.

However, upon a fresh installation in my home folder, it looks like I am pulling from other users databases. For instance, when I try to start a new project, it defaults to the $HOME directory of another user. But when I echo $HOME from my shell, it shows my own home directory. So it’s like the cryoSPARC Master process is running in a different environment, which makes no sense. (at least to me)

Furthermore, despite my database folder being empty, it says that I’ve run a certain number of jobs in the past week, etc.

Do I need to do anything to further isolate the environments?

@ccgauvin94 Please can you further explain what you mean with

Is my understanding of your description correct in:

  1. No Linux user “owns” (in the chown sense) more than one master installation. How would you enforce that?
  2. Each master installation is complete: database directory, program files, etc. independent from other master installations?

Questions:

  1. Does each Linux user who “owns” a master instance also “own” a complete and independent (from other Linux users) worker installation that is available on slurm execution hosts?
  2. Can you infer from the unexpected content you see (preexisting jobs, etc.) which particular Linux user might be owning the cryoSPARC instance whose contents you see? Could that instance unexpectedly be using “your” port range, perhaps because that instance has inadvertently been installed “outside” the regimen you mentioned:
1 Like

Hi,
I managed to have up to 5 independent installs running at the same time hooked up in our SLURM cluster, but I was increasing the port numbers by 100 each. Just in case you want to try that! :upside_down_face:

2 Likes

Yes, happy to.

Our university uses active directory for authentication, and permissions. No user has write permissions outside their home folder. I’ve written a script (see below) that each user has to run to get cryoSPARC setup. It asks for their license ID, and some other parameters, and runs the cryoSPARC installation with those settings. It also increments the ports so that we don’t have conflicting ports.

Yes, everything gets installed into home > username > cryosparc, and the directory structure there is

user
└── cryosparc
    ├── cryosparc_cache
    ├── cryosparc_db
    ├── cryosparc_master
    └── cryosparc_worker

Yes. (If I set it up right).

I know the user, and I checked my tracking file and the port ranges do not overlap.

But

This solved the issue. I went from incrementing by 20 to incrementing by 50, and suddenly things are working again. I’ve double, triple checked my code, and I don’t think I’m making an incrementing error, but even if I was, I think the most I’d be off by is 1, and as far as I’m aware, cryoSPARC doesn’t actually use the last few ports in the 20 it reserves.

So I’m not sure what is causing it, but it seems to be fixed. It’s very likely it was something in my configuration.

Below is the script I wrote for the installation (in case anyone is having a similar issue and finds this thread):

#!/bin/bash

export CRYOSPARC_VERSION="3.3.1"

#Check if already installed
if [ -d $HOME/cryosparc ]; then
        echo "You already seem to have cryoSPARC installed. If this is not the case, please contact a system administrator."
        exit
fi

#Setup cryosparc directories
mkdir $HOME/cryosparc

#Get license
echo "This script will guide you through the initial setup of cryoSPARC. You only need to run this script once. After that, you may start and stop cryosparc with <command>."
echo "Please enter your cryoSPARC license ID. This license is unique to you as a user - not your lab."
echo "License ID: "
read LICENSE_ID
echo $LICENSE_ID > $HOME/cryosparc/license

echo "Installing files..."
cp -rav /mnt/shared/moduleapps/cryoem/cryosparc/$CRYOSPARC_VERSION/. ~/cryosparc
echo "Extracting. This may take some time..."
#Modify line below to include progress bar by piping to pv
tar -xf ~/cryosparc/cryosparc_master.tar.gz --directory ~/cryosparc/

export COUNTER=$(cat /mnt/shared/moduleapps/cryoem/cryosparc/counter)
echo "There are $COUNTER cryoSPARC installations already. Yours will be $(( COUNTER + 1))."
export CRYOSPARC_PORT=$(( (COUNTER + 2) * 50))
export CRYOSPARC_PORT=$(( CRYOSPARC_PORT + 38980))
echo "Setting cryoSPARC port range to $CRYOSPARC_PORT"

bash ~/cryosparc/cryosparc_master/install.sh --license $LICENSE_ID \
        --hostname tempest-login.msu.montana.edu \
        --dbpath ~/cryosparc/cryosparc_db \
        --port $CRYOSPARC_PORT \
        --yes

echo CRYOSPARC_FORCE_HOSTNAME=true >> ~/cryosparc/cryosparc_master/config.sh

echo "Please enter the following details to create your user account: "
read -p "Email: " USER_EMAIL
read -s -p "Password: " USER_PASSWORD
read -p "Username: " USER_LOGIN
read -p "First name: " USER_FIRSTNAME
read -p "Last name: " USER_LASTNAME

~/cryosparc/cryosparc_master/bin/cryosparcm start
~/cryosparc/cryosparc_master/bin/cryosparcm createuser --email $USER_EMAIL \
        --password $USER_PASSWORD \
        --username $USER_LOGIN \
        --firstname $USER_FIRSTNAME \
        --lastname $USER_LASTNAME

echo "$CRYOSPARC_PORT, $HOME" >> /mnt/shared/moduleapps/cryoem/cryosparc/ports_in_use
sed -i "1s/.*/$((COUNTER + 1))/" /mnt/shared/moduleapps/cryoem/cryosparc/counter

mkdir ~/cryosparc/cache

echo "cryoSPARC Master has now finished installing. However, before you can submit jobs, the cluster configuration must be updated with your account details."

read -p "What Tempest account would you like to use? NOTE: This is probably NOT your NetID, but rather the Tempest account such as priority-<yourgroup> that will run the jobs on SLURM:" TEMPEST_ACCOUNT

sed -i "2s/.*/#SBATCH --account=$TEMPEST_ACCOUNT/" ~/cryosparc/cluster_script.sh

echo "cryoSPARC Worker will now install."
echo "Extracting files..."
tar -xf ~/cryosparc/cryosparc_worker.tar.gz --directory ~/cryosparc/
echo "Done extracting."
echo "Installing..."
~/cryosparc/cryosparc_worker/install.sh \
        --license $LICENSE_ID \
        --cudapath /mnt/shared/moduleapps/eb/CUDAcore/11.1.1 \
        --yes

echo "Connecting cluster..."
cd ~/cryosparc
~/cryosparc/cryosparc_master/bin/cryosparcm cluster connect
echo "Cleaning up..."
rm ~/cryosparc/cryosparc_master.tar.gz
rm ~/cryosparc/cryosparc_worker.tar.gz
echo ""
echo "##########"
echo ""
echo "The cryoSPARC installation script has run to completion. It is important you understand the following:"
echo "Your cryoSPARC installation can be accessed through the web browser by navigating to http://tempest-login.msu.montana.edu:$CRYOSPARC_PORT"
echo "You will only be able to access this GUI when on the campus network, or over the campus VPN."
echo "To start and stop the GUI, you can use the commands '~/cryosparc/cryosparc_master/bin/cryosparcm start' and '~/cryosparc/cryosparc_master/bin/cryosparcm stop'"
echo "The cryoSPARC installation folder is unique to you and is present in your home folder, at ~/cryosparc - That folder contains the cache directory, which is where files are cached on the SSD, as well as the cryosparc_database folder which is where your project properties are stored."