We have a SLURM cluster where the login-node has been configured to use multiple cryoSPARC Master installations for each computer.
I configured it to use a different port range (incrementing by 20) and a different database directory (users home directory) for each cluster user.
However, upon a fresh installation in my home folder, it looks like I am pulling from other users databases. For instance, when I try to start a new project, it defaults to the $HOME directory of another user. But when I echo $HOME from my shell, it shows my own home directory. So it’s like the cryoSPARC Master process is running in a different environment, which makes no sense. (at least to me)
Furthermore, despite my database folder being empty, it says that I’ve run a certain number of jobs in the past week, etc.
Do I need to do anything to further isolate the environments?
@ccgauvin94 Please can you further explain what you mean with
Is my understanding of your description correct in:
No Linux user “owns” (in the chown sense) more than one master installation. How would you enforce that?
Each master installation is complete: database directory, program files, etc. independent from other master installations?
Does each Linux user who “owns” a master instance also “own” a complete and independent (from other Linux users) worker installation that is available on slurm execution hosts?
Can you infer from the unexpected content you see (preexisting jobs, etc.) which particular Linux user might be owning the cryoSPARC instance whose contents you see? Could that instance unexpectedly be using “your” port range, perhaps because that instance has inadvertently been installed “outside” the regimen you mentioned:
Our university uses active directory for authentication, and permissions. No user has write permissions outside their home folder. I’ve written a script (see below) that each user has to run to get cryoSPARC setup. It asks for their license ID, and some other parameters, and runs the cryoSPARC installation with those settings. It also increments the ports so that we don’t have conflicting ports.
Yes, everything gets installed into home > username > cryosparc, and the directory structure there is
I know the user, and I checked my tracking file and the port ranges do not overlap.
This solved the issue. I went from incrementing by 20 to incrementing by 50, and suddenly things are working again. I’ve double, triple checked my code, and I don’t think I’m making an incrementing error, but even if I was, I think the most I’d be off by is 1, and as far as I’m aware, cryoSPARC doesn’t actually use the last few ports in the 20 it reserves.
So I’m not sure what is causing it, but it seems to be fixed. It’s very likely it was something in my configuration.
Below is the script I wrote for the installation (in case anyone is having a similar issue and finds this thread):
#Check if already installed
if [ -d $HOME/cryosparc ]; then
echo "You already seem to have cryoSPARC installed. If this is not the case, please contact a system administrator."
#Setup cryosparc directories
echo "This script will guide you through the initial setup of cryoSPARC. You only need to run this script once. After that, you may start and stop cryosparc with <command>."
echo "Please enter your cryoSPARC license ID. This license is unique to you as a user - not your lab."
echo "License ID: "
echo $LICENSE_ID > $HOME/cryosparc/license
echo "Installing files..."
cp -rav /mnt/shared/moduleapps/cryoem/cryosparc/$CRYOSPARC_VERSION/. ~/cryosparc
echo "Extracting. This may take some time..."
#Modify line below to include progress bar by piping to pv
tar -xf ~/cryosparc/cryosparc_master.tar.gz --directory ~/cryosparc/
export COUNTER=$(cat /mnt/shared/moduleapps/cryoem/cryosparc/counter)
echo "There are $COUNTER cryoSPARC installations already. Yours will be $(( COUNTER + 1))."
export CRYOSPARC_PORT=$(( (COUNTER + 2) * 50))
export CRYOSPARC_PORT=$(( CRYOSPARC_PORT + 38980))
echo "Setting cryoSPARC port range to $CRYOSPARC_PORT"
bash ~/cryosparc/cryosparc_master/install.sh --license $LICENSE_ID \
--hostname tempest-login.msu.montana.edu \
--dbpath ~/cryosparc/cryosparc_db \
--port $CRYOSPARC_PORT \
echo CRYOSPARC_FORCE_HOSTNAME=true >> ~/cryosparc/cryosparc_master/config.sh
echo "Please enter the following details to create your user account: "
read -p "Email: " USER_EMAIL
read -s -p "Password: " USER_PASSWORD
read -p "Username: " USER_LOGIN
read -p "First name: " USER_FIRSTNAME
read -p "Last name: " USER_LASTNAME
~/cryosparc/cryosparc_master/bin/cryosparcm createuser --email $USER_EMAIL \
--password $USER_PASSWORD \
--username $USER_LOGIN \
--firstname $USER_FIRSTNAME \
echo "$CRYOSPARC_PORT, $HOME" >> /mnt/shared/moduleapps/cryoem/cryosparc/ports_in_use
sed -i "1s/.*/$((COUNTER + 1))/" /mnt/shared/moduleapps/cryoem/cryosparc/counter
echo "cryoSPARC Master has now finished installing. However, before you can submit jobs, the cluster configuration must be updated with your account details."
read -p "What Tempest account would you like to use? NOTE: This is probably NOT your NetID, but rather the Tempest account such as priority-<yourgroup> that will run the jobs on SLURM:" TEMPEST_ACCOUNT
sed -i "2s/.*/#SBATCH --account=$TEMPEST_ACCOUNT/" ~/cryosparc/cluster_script.sh
echo "cryoSPARC Worker will now install."
echo "Extracting files..."
tar -xf ~/cryosparc/cryosparc_worker.tar.gz --directory ~/cryosparc/
echo "Done extracting."
--license $LICENSE_ID \
--cudapath /mnt/shared/moduleapps/eb/CUDAcore/11.1.1 \
echo "Connecting cluster..."
~/cryosparc/cryosparc_master/bin/cryosparcm cluster connect
echo "Cleaning up..."
echo "The cryoSPARC installation script has run to completion. It is important you understand the following:"
echo "Your cryoSPARC installation can be accessed through the web browser by navigating to http://tempest-login.msu.montana.edu:$CRYOSPARC_PORT"
echo "You will only be able to access this GUI when on the campus network, or over the campus VPN."
echo "To start and stop the GUI, you can use the commands '~/cryosparc/cryosparc_master/bin/cryosparcm start' and '~/cryosparc/cryosparc_master/bin/cryosparcm stop'"
echo "The cryoSPARC installation folder is unique to you and is present in your home folder, at ~/cryosparc - That folder contains the cache directory, which is where files are cached on the SSD, as well as the cryosparc_database folder which is where your project properties are stored."