Post-install scripts are failing

Install, Configure and Update
1

Hello,
We have been deploying CryoSparc on AWS since May and it was a success except for the user web interface. Strangely, we started having a rollback issue to our stack created by AWS ParallelCluster. We have been in contact with AWS in regards to the issue we started to face three weeks back, they come up with a syntax error within the post-install scripts. We are confused why we are facing this kind of issue now when we have been deploying it in different occasions with no issue using the same parallel-cluster-configration file and the same post-install.sh file provided by the CryoSPARC. We are also wondering why this is happening now since there are no changes made from our side.
The following is the error pulled from cfn-init-log if this is helping:

************ CRYOSPARC SYSTEM: MASTER INSTALLER **************\
\
 Installation Settings:\
   License ID              : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\
   Root Directory          : /shared/cryosparc/cryosparc_master\
   Master Hostname         : ip-xxxxxxxxxxxx\
   Database Directory      : /shared/cryosparc/cryosparc_db\
   Base HTTP Port Number   : 45000\
   Developer Install       : false\
   Operating System        : Linux\
   Insecure Install        : false\
   Standalone Install      : false\
   Version                 : v4.1.1\
\
******************************************************************\
\
The cryoSPARC System master installation will host a database,\
web application, and command layer for cryoSPARC, but will not\
perform processing jobs.\
\
Please read and confirm the above settings. The cryoSPARC database\
will be stored at the database directory which is relative to the\
root directory, unless provided as an absolute path.\
You can change the database directory now by passing the --dbapth\
parameter to this installer, or later via the command line.\
\
Are the above settings correct?\
\
******************************************************************\
\
 Setting up hard-coded config.sh environment variables\
\
******************************************************************\
\
 Installing all dependencies.\
\
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress\
ERROR: This command can only be called by the owner of this script: ssm-user. You are root!\
Alternatively, set CRYOSPARC_FORCE_USER=true in cryosparc_master/config.sh to suppress this error.\
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress\
ERROR: This command can only be called by the owner of this script: ssm-user. You are root!\
Alternatively, set CRYOSPARC_FORCE_USER=true in cryosparc_master/config.sh to suppress this error.\
******* CRYOSPARC SYSTEM: WORKER INSTALLER ***********************\
\
 Installation Settings:\
   License ID              : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\
   Root Directory          : /shared/cryosparc/cryosparc_worker\
   Standalone Installation : false\
   Version                 : v4.1.1\
\
******************************************************************\
\
 CUDA check..\
 Found nvidia-smi at /bin/nvidia-smi\
\
 CUDA Path was provided as /shared/cuda/11.3.1\
 Checking CUDA installation...\
 Found nvcc at /shared/cuda/11.3.1/bin/nvcc\
 The above cuda installation will be used but can be changed later.\
\
******************************************************************\
\
 Setting up hard-coded config.sh environment variables\
\
******************************************************************\
\
 Installing all dependencies.\
\
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress\
Checking dependencies...\
Dependencies for python have changed - reinstalling...\
  ------------------------------------------------------------------------\
  Installing anaconda python...\
  ------------------------------------------------------------------------\
PREFIX=/shared/cryosparc/cryosparc_worker/deps/anaconda\
Unpacking payload ...\
Collecting package metadata (current_repodata.json): ...working... done\
Solving environment: ...working... done\
\
## Package Plan ##\
\
  environment location: /shared/cryosparc/cryosparc_worker/deps/anaconda\
\
  added / updated specs:\
    - _libgcc_mutex==0.1=main\
    - _openmp_mutex==4.5=1_gnu\
    - brotlipy==0.7.0=py38h27cfd23_1003\
    - ca-certificates==2022.3.29=h06a4308_1\
    - certifi==2021.10.8=py38h06a4308_2\
    - cffi==1.15.0=py38hd667e15_1\
    - charset-normalizer==2.0.4=pyhd3eb1b0_0\
    - colorama==0.4.4=pyhd3eb1b0_0\
    - conda-content-trust==0.1.1=pyhd3eb1b0_0\
    - conda-package-handling==1.8.1=py38h7f8727e_0\
    - conda==4.12.0=py38h06a4308_0\
    - cryptography==36.0.0=py38h9ce1e76_0\
    - idna==3.3=pyhd3eb1b0_0\
    - ld_impl_linux-64==2.35.1=h7274673_9\
    - libffi==3.3=he6710b0_2\
    - libgcc-ng==9.3.0=h5101ec6_17\
    - libgomp==9.3.0=h5101ec6_17\
    - libstdcxx-ng==9.3.0=hd4cf53a_17\
    - ncurses==6.3=h7f8727e_2\
    - openssl==1.1.1n=h7f8727e_0\
    - pip==21.2.4=py38h06a4308_0\
    - pycosat==0.6.3=py38h7b6447c_1\
    - pycparser==2.21=pyhd3eb1b0_0\
    - pyopenssl==22.0.0=pyhd3eb1b0_0\
    - pysocks==1.7.1=py38h06a4308_0\
    - python==3.8.13=h12debd9_0\
    - readline==8.1.2=h7f8727e_1\
    - requests==2.27.1=pyhd3eb1b0_0\
    - ruamel_yaml==0.15.100=py38h27cfd23_0\
    - setuptools==61.2.0=py38h06a4308_0\
    - six==1.16.0=pyhd3eb1b0_1\
    - sqlite==3.38.2=hc218d9a_0\
    - tk==8.6.11=h1ccaba5_0\
    - tqdm==4.63.0=pyhd3eb1b0_0\
    - urllib3==1.26.8=pyhd3eb1b0_0\
    - wheel==0.37.1=pyhd3eb1b0_0\
    - xz==5.2.5=h7b6447c_0\
    - yaml==0.2.5=h7b6447c_0\
    - zlib==1.2.12=h7f8727e_1\
\
\
The following NEW packages will be INSTALLED:\
\
  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main\
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-4.5-1_gnu\
  brotlipy           pkgs/main/linux-64::brotlipy-0.7.0-py38h27cfd23_1003\
  ca-certificates    pkgs/main/linux-64::ca-certificates-2022.3.29-h06a4308_1\
  certifi            pkgs/main/linux-64::certifi-2021.10.8-py38h06a4308_2\
  cffi               pkgs/main/linux-64::cffi-1.15.0-py38hd667e15_1\
  charset-normalizer pkgs/main/noarch::charset-normalizer-2.0.4-pyhd3eb1b0_0\
  colorama           pkgs/main/noarch::colorama-0.4.4-pyhd3eb1b0_0\
  conda              pkgs/main/linux-64::conda-4.12.0-py38h06a4308_0\
  conda-content-tru~ pkgs/main/noarch::conda-content-trust-0.1.1-pyhd3eb1b0_0\
  conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.8.1-py38h7f8727e_0\
  cryptography       pkgs/main/linux-64::cryptography-36.0.0-py38h9ce1e76_0\
  idna               pkgs/main/noarch::idna-3.3-pyhd3eb1b0_0\
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.35.1-h7274673_9\
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2\
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.3.0-h5101ec6_17\
  libgomp            pkgs/main/linux-64::libgomp-9.3.0-h5101ec6_17\
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.3.0-hd4cf53a_17\
  ncurses            pkgs/main/linux-64::ncurses-6.3-h7f8727e_2\
  openssl            pkgs/main/linux-64::openssl-1.1.1n-h7f8727e_0\
  pip                pkgs/main/linux-64::pip-21.2.4-py38h06a4308_0\
  pycosat            pkgs/main/linux-64::pycosat-0.6.3-py38h7b6447c_1\
  pycparser          pkgs/main/noarch::pycparser-2.21-pyhd3eb1b0_0\
  pyopenssl          pkgs/main/noarch::pyopenssl-22.0.0-pyhd3eb1b0_0\
  pysocks            pkgs/main/linux-64::pysocks-1.7.1-py38h06a4308_0\
  python             pkgs/main/linux-64::python-3.8.13-h12debd9_0\
  readline           pkgs/main/linux-64::readline-8.1.2-h7f8727e_1\
  requests           pkgs/main/noarch::requests-2.27.1-pyhd3eb1b0_0\
  ruamel_yaml        pkgs/main/linux-64::ruamel_yaml-0.15.100-py38h27cfd23_0\
  setuptools         pkgs/main/linux-64::setuptools-61.2.0-py38h06a4308_0\
  six                pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1\
  sqlite             pkgs/main/linux-64::sqlite-3.38.2-hc218d9a_0\
  tk                 pkgs/main/linux-64::tk-8.6.11-h1ccaba5_0\
  tqdm               pkgs/main/noarch::tqdm-4.63.0-pyhd3eb1b0_0\
  urllib3            pkgs/main/noarch::urllib3-1.26.8-pyhd3eb1b0_0\
  wheel              pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0\
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0\
  yaml               pkgs/main/linux-64::yaml-0.2.5-h7b6447c_0\
  zlib               pkgs/main/linux-64::zlib-1.2.12-h7f8727e_1\
\
\
Preparing transaction: ...working... done\
Executing transaction: ...working... done\
installation finished.\
  ------------------------------------------------------------------------\
    Done.\
    anaconda python installation successful.\
  ------------------------------------------------------------------------\
  Extracting all conda packages...\
  ------------------------------------------------------------------------\
.....................................................................................................................................................................................\
  ------------------------------------------------------------------------\
    Done.\
    conda packages installation successful.\
  ------------------------------------------------------------------------\
  Preparing to install all pip packages...\
  ------------------------------------------------------------------------\
DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453\
Processing ./deps_bundle/python/python_packages/pip_packages/pycuda-2020.1.tar.gz\
  Preparing metadata (setup.py): started\
  Preparing metadata (setup.py): finished with status 'done'\
Installing collected packages: pycuda\
  DEPRECATION: pycuda is being installed using the legacy 'setup.py install' method, because the '--no-binary' option was enabled for it and this currentlydisables local wheel building for projects that don't have a 'pyproject.toml' file. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/11451\
  Running setup.py install for pycuda: started\
  Running setup.py install for pycuda: still running...\
  Running setup.py install for pycuda: finished with status 'done'\
Successfully installed pycuda-2020.1\
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\
  ------------------------------------------------------------------------\
    Done.\
    pip packages installation successful.\
  ------------------------------------------------------------------------\
  Main dependency installation completed. Continuing...\
  ------------------------------------------------------------------------\
Completed.\
Currently checking hash for ctffind\
Dependencies for ctffind have changed - reinstalling...\
  ------------------------------------------------------------------------\
  ctffind 4.1.10 installation successful.\
  ------------------------------------------------------------------------\
Completed.\
Currently checking hash for cudnn\
Dependencies for cudnn have changed - reinstalling...\
  ------------------------------------------------------------------------\
  cudnn 8.1.0.77 for CUDA 11 installation successful.\
  ------------------------------------------------------------------------\
Completed.\
Currently checking hash for gctf\
Dependencies for gctf have changed - reinstalling...\
  ------------------------------------------------------------------------\
  Gctf v1.06 installation successful.\
  ------------------------------------------------------------------------\
Completed.\
Completed dependency check.\
Generating '/shared/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/libtiff/tiff_h_4_4_0.py' from '/shared/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/../include/tiff.h'\
\
******* CRYOSPARC WORKER INSTALLATION COMPLETE *******************\
\
 In order to run processing jobs, you will need to connect this\
 worker to a cryoSPARC master.\
\
******************************************************************\
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress\
Traceback (most recent call last):\
  File "<string>", line 1, in <module>\
  File "cryosparc_compute/__init__.py", line 9, in <module>\
    from . import dataset  # noqa\
  File "cryosparc_compute/dataset.py", line 59\
    def to_dataframe(dset: Dataset[Row], exclude_uid=False):\
                         ^\
SyntaxError: invalid syntax\
Traceback (most recent call last):\
  File "<string>", line 1, in <module>\
  File "cryosparc_compute/__init__.py", line 9, in <module>\
    from . import dataset  # noqa\
  File "cryosparc_compute/dataset.py", line 59\
    def to_dataframe(dset: Dataset[Row], exclude_uid=False):\
                         ^\
SyntaxError: invalid syntax\
Starting cryoSPARC System master process..\
/shared/cryosparc/cryosparc_master/bin/cryosparcm: line 942: supervisorctl: command not found\
CryoSPARC is already running.\
If you would like to restart, use cryosparcm restart\
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress\
Traceback (most recent call last):\
  File "<string>", line 1, in <module>\
  File "cryosparc_compute/__init__.py", line 9, in <module>\
    from . import dataset  # noqa\
  File "cryosparc_compute/dataset.py", line 59\
    def to_dataframe(dset: Dataset[Row], exclude_uid=False):\
                         ^\
SyntaxError: invalid syntax\
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress\
Traceback (most recent call last):\
  File "<string>", line 1, in <module>\
  File "cryosparc_compute/__init__.py", line 9, in <module>\
    from . import dataset  # noqa\
  File "cryosparc_compute/dataset.py", line 59\
    def to_dataframe(dset: Dataset[Row], exclude_uid=False):\
                         ^\
SyntaxError: invalid syntax\
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress\
Traceback (most recent call last):\
  File "<string>", line 1, in <module>\
  File "cryosparc_compute/__init__.py", line 9, in <module>\
    from . import dataset  # noqa\
  File "cryosparc_compute/dataset.py", line 59\
    def to_dataframe(dset: Dataset[Row], exclude_uid=False):\
                         ^\
SyntaxError: invalid syntax\
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress\
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress\
/shared/cryosparc/cryosparc_master/bin/cryosparcm: line 172: supervisorctl: command not found\
CryoSPARC is running.\
Stopping cryoSPARC\
/shared/cryosparc/cryosparc_master/bin/cryosparcm: line 1048: supervisorctl: command not found\
parallelcluster: fetch_and_run - Failed to run postinstall, s3://fh-pi-testinclude-d-cryosparc-test-data-np/install-cryosparc_v1.sh failed with non 0 return code: 127\
\
2023-01-13 17:53:33,647 [ERROR] Error encountered during build of shellRunPostInstall: Command runpostinstall failed\
Traceback (most recent call last):\
  File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 576, in run_config\
    cmd_log.info("Config %s", config.name)\
  File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 276, in build\
    else:\
  File "/usr/lib/python3.7/site-packages/cfnbootstrap/command_tool.py", line 127, in apply\
    raise ToolError(u"Command %s failed" % name)\
cfnbootstrap.construction_errors.ToolError: Command runpostinstall failed\
2023-01-13 17:53:33,647 [ERROR] -----------------------BUILD FAILED!------------------------\
2023-01-13 17:53:33,647 [ERROR] Unhandled exception during build: Command runpostinstall failed\
Traceback (most recent call last):\
  File "/opt/aws/bin/cfn-init", line 176, in <module>\
    logging.error("Could not create \{\} to store metadata".format(data_dir))\
  File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 137, in build\
    Contractor(metadata, strict_mode).build(configSets, self)\
  File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 564, in build\
  File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 576, in run_config\
    cmd_log.info("Config %s", config.name)\
  File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 276, in build\
    else:\
  File "/usr/lib/python3.7/site-packages/cfnbootstrap/command_tool.py", line 127, in apply\
    raise ToolError(u"Command %s failed" % name)\
cfnbootstrap.construction_errors.ToolError: Command runpostinstall failed}

Looking forward for your assistance. Thanks
B

2

Your post install script needs to be adjusted for compatibility with the current version of CryoSPARC, likely such that:

  • cryosparcm is not run by the root user, but under the Linux account that will eventually control CryoSPARC processes
  • the need for setting CRYOSPARC_FORCE_USER=true is removed
3

Hi,
I will check with the updated version and deploy it again and see what the outcome will be.
Thanks!

4

Hello, Good Day.
Where can i get the updated post install script that is compatible with the newer version of CryoSparc, i tried to remove CRYOSPARC_FORCE_USER=true but i still have the issue. Or i am missing something here.
Since my cluster creation is rolling back, I had to re-create the cluster using the --rollback-on-failure false flag, so that I can login to the HeadNode to troubleshoot but the logs are showing so many errors.
is there a way or a command to add in the post script itself to do the update so that the stack won’t be rolling back destroying all the resources created.

Regards,
Brikti

5

I am not aware of an updated script having been released, but you may modify your script. Without having seen your script, I cannot make specific suggestions. I suspect, however, that it needs to be ensured that cryosparcm is only called by ssm-user (based on the error message), maybe by running the command with a
/bin/su ssm-user prefix.

6

This is the script i have been using;

#!/bin/bash

# Log script output to a file to reference later
set +e
exec &> >(tee -a "/tmp/post_install.log")

. "/etc/parallelcluster/cfnconfig"

# Get the local commands to run yum and apt
YUM_CMD=$(which yum)
APT_GET_CMD=$(which apt-get)

# If we have yum installed, use it to install prerequisites. If not, use apt
if [[ -n $YUM_CMD ]]; then
    wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -P /tmp
    yum install -y /tmp/epel-release-latest-7.noarch.rpm

    yum install -y perl-Switch python3 python3-pip links
    user_test=$(getent passwd ec2-user)
    if [[ -n "${user_test}" ]]; then
        OSUSER=ec2-user
        OSGROUP=ec2-user
    else
        OSUSER=centos
        OSGROUP=centos
    fi
elif [[ -n $APT_GET_CMD ]]; then
    apt-get update
    apt-get install -y libswitch-perl python3 python3-pip links
    OSUSER=ubuntu
    OSGROUP=ubuntu
else
    # If we don't have yum or apt, we couldn't install the prerequisites, so exit
    echo "error can't install package $PACKAGE"
    exit 1;
fi

# Get the cryoSPARC license ID, optional path, and optional versions from the script arguments
CRYOSPARC_LICENSE_ID=$2
CRYOSPARC_INSTALL_PATH=${3:-/shared/cryosparc}
CUDA_INSTALL_PATH=${4:-/shared/cuda}
CUDA_VERSION=${5:-11.3.1}
CUDA_LONG_VERSION=${6:-11.3.1_465.19.01}

# Install the AWS CLI
pip3 install --upgrade awscli boto3

yum -y update

# Configure AWS
AWS_DEFAULT_REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | rev | cut -c 2- | rev)
aws configure set default.region "${AWS_DEFAULT_REGION}"
aws configure set default.output json


# Install CUDA Toolkit
mkdir -p "${CUDA_INSTALL_PATH}"
cd "${CUDA_INSTALL_PATH}" || return
wget "https://developer.download.nvidia.com/compute/cuda/${CUDA_VERSION}/local_installers/cuda_${CUDA_LONG_VERSION}_linux.run"
sh cuda_"${CUDA_LONG_VERSION}"_linux.run --defaultroot="${CUDA_INSTALL_PATH}" --toolkit --toolkitpath="${CUDA_INSTALL_PATH}"/"${CUDA_VERSION}" --samples --silent
rm cuda_"${CUDA_LONG_VERSION}"_linux.run

# Add CUDA to the path
cat > /etc/profile.d/cuda.sh << 'EOF'
PATH=$PATH:@CUDA_INSTALL_PATH@/CUDA_VERSION/bin
EOF
sed -i "s|@CUDA_INSTALL_PATH@|${CUDA_INSTALL_PATH}|g" /etc/profile.d/cuda.sh
sed -i "s|@CUDA_VERSION@|${CUDA_VERSION}|g" /etc/profile.d/cuda.sh
. /etc/profile.d/cuda.sh

# Download cryoSPARC
mkdir -p "${CRYOSPARC_INSTALL_PATH}"
cd "${CRYOSPARC_INSTALL_PATH}" || return
curl -L https://get.cryosparc.com/download/master-latest/${CRYOSPARC_LICENSE_ID} -o cryosparc_master.tar.gz
curl -L https://get.cryosparc.com/download/worker-latest/${CRYOSPARC_LICENSE_ID} -o cryosparc_worker.tar.gz

# Install master process
tar -xf cryosparc_master.tar.gz
cd cryosparc_master || return
./install.sh --license "${CRYOSPARC_LICENSE_ID}" \
      --hostname "${HOSTNAME}" \
      --dbpath "${CRYOSPARC_INSTALL_PATH}"/cryosparc_db \
      --port 45000 \
      --allowroot \
      --yes

# Start cryoSPARC master package
"${CRYOSPARC_INSTALL_PATH}"/cryosparc_master/bin/cryosparcm start

# Add CryoSPARC to the path
cat > /etc/profile.d/cryosparc.sh << 'EOF'
PATH=$PATH:@CRYOSPARC_INSTALL_PATH@/cryosparc_master/bin
EOF
sed -i "s|@CRYOSPARC_INSTALL_PATH@|${CRYOSPARC_INSTALL_PATH}|g" /etc/profile.d/cryosparc.sh
. /etc/profile.d/cryosparc.sh

echo "export CRYOSPARC_FORCE_USER=true" >> "${CRYOSPARC_INSTALL_PATH}"/cryosparc_master/config.sh
echo "export CRYOSPARC_FORCE_HOSTNAME=true" >> "${CRYOSPARC_INSTALL_PATH}"/cryosparc_master/config.sh
echo "export CRYOSPARC_DISABLE_IMPORT_ON_MASTER=true" >> "${CRYOSPARC_INSTALL_PATH}"/cryosparc_master/config.sh

# Install cryoSPARC work package
cd "${CRYOSPARC_INSTALL_PATH}" || return
tar -xf cryosparc_worker.tar.gz
cd cryosparc_worker || return
./install.sh --license "${CRYOSPARC_LICENSE_ID}" \
      --cudapath "${CUDA_INSTALL_PATH}/${CUDA_VERSION}" \
      --yes

rm ${CRYOSPARC_INSTALL_PATH}/*.tar.gz
chown -R ${OSUSER}:${OSGROUP} /shared/cryosparc

# Start cluster
/bin/su -c "${CRYOSPARC_INSTALL_PATH}/cryosparc_master/bin/cryosparcm start" - ${OSUSER}

# Create cluster config files for each GPU partition in SLURM
for PARTITION in gpu-large gpu-med gpu-small 
do

   cat > ${CRYOSPARC_INSTALL_PATH}/cluster_info.json << 'EOF'
{
"qdel_cmd_tpl": "scancel {{ cluster_job_id }}",
"worker_bin_path": "$CRYOSPARC_INSTALL_PATH/cryosparc_worker/bin/cryosparcw",
"title": "cryosparc-cluster",
"cache_path": "",
"qinfo_cmd_tpl": "sinfo",
"qsub_cmd_tpl": "sbatch {{ script_path_abs }}",
"qstat_cmd_tpl": "squeue -j {{ cluster_job_id }}",
"send_cmd_tpl": "{{ command }}",
"name": "@PARTITION@"
}
EOF

sed -i "s|@CRYOSPARC_INSTALL_PATH@|${CRYOSPARC_INSTALL_PATH}|g" "${CRYOSPARC_INSTALL_PATH}"/cluster_info.json
sed -i "s|@PARTITION@|${PARTITION}|g" "${CRYOSPARC_INSTALL_PATH}"/cluster_info.json

cat > ${CRYOSPARC_INSTALL_PATH}/cluster_script.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=cryosparc_{{ project_uid }}_{{ job_uid }}
#SBATCH --output={{ job_log_path_abs }}
#SBATCH --error={{ job_log_path_abs }}
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task={{ num_cpu }}
#SBATCH --gres=gpu:{{ num_gpu }}
#SBATCH --partition=@PARTITION@
{{ run_cmd }}
EOF

sed -i "s|@PARTITION@|${PARTITION}|g" "${CRYOSPARC_INSTALL_PATH}"/cluster_script.sh

# Connect CryoSPARC worker nodes to cluster
/bin/su -c "cd ${CRYOSPARC_INSTALL_PATH} && ${CRYOSPARC_INSTALL_PATH}/cryosparc_master/bin/cryosparcm cluster connect" - ${OSUSER}

done

# Restart CryoSPARC master
/bin/su -c "cd ${CRYOSPARC_INSTALL_PATH} && ${CRYOSPARC_INSTALL_PATH}/cryosparc_master/bin/cryosparcm restart" - ${OSUSER}

Really appreciate your help.

7

The
cryosparcm start command at the bottom of this snippet may be the problem. I am not sure right now why this command is even needed at this point of the script. What errors do you get with this line omitted from the script?

8

The stack is still rolling back and failing to create the stack so with the rollback i was able to login to the node so the cfn-init.log doesn’t show me any errors but when trying to start cryosparc, i have this:

ryosparcm start
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
ERROR: This command can only be called by the owner of this script: ec2-user. You are root!
Alternatively, set CRYOSPARC_FORCE_USER=true in cryosparc_master/config.sh to suppress this error.

I tried to edit the config.sh as suggested to see what will happen, this is what shows:

ryosparcm start
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
Traceback (most recent call last):
File “”, line 1, in
File “cryosparc_compute/init.py”, line 9, in
from . import dataset # noqa
File “cryosparc_compute/dataset.py”, line 59
def to_dataframe(dset: Dataset[Row], exclude_uid=False):
^
SyntaxError: invalid syntax
Traceback (most recent call last):
File “”, line 1, in
File “cryosparc_compute/init.py”, line 9, in
from . import dataset # noqa
File “cryosparc_compute/dataset.py”, line 59
def to_dataframe(dset: Dataset[Row], exclude_uid=False):
^
SyntaxError: invalid syntax
Starting cryoSPARC System master process…
/shared/cryosparc/cryosparc_master/bin/cryosparcm: line 942: supervisorctl: command not found
CryoSPARC is already running.
If you would like to restart, use cryosparcm restart

And restarting;

arning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
/shared/cryosparc/cryosparc_master/bin/cryosparcm: line 172: supervisorctl: command not found
CryoSPARC is running.
Stopping cryoSPARC
/shared/cryosparc/cryosparc_master/bin/cryosparcm: line 1048: supervisorctl: command not found

9

Please can you post:

  • the current postinstall script, as downloaded from your S3 bucket
  • output of
    ls -l /shared/cryosparc/cryosparc_master/ and
    ls -l /shared/cryosparc/
  • were you in fact root when you executed cryosparcm start?
  • was config.sh already present or did you have to create it?
  • contents of /tmp/post_install.log

As a troubleshooting strategy, I suggest (as ec2-user or whatever chosen non-root user on this instance):

  • ps xww | grep -e cryosparc -e mongo
  • kill CryoSPARC related processes
  • remove directories cryosparc_master, cryosparc_worker, cryosparc_db.
  • install the CryoSPARC master package manually to the point where you can successfully run
    cryosparc start
    • to familiarize yourself with the installation process
    • to create a blueprint for a functional post install script
10

Current postinstall script:

#!/bin/bash

# Log script output to a file to reference later
set +e
exec &> >(tee -a "/tmp/post_install.log")

. "/etc/parallelcluster/cfnconfig"

# Get the local commands to run yum and apt
YUM_CMD=$(which yum)
APT_GET_CMD=$(which apt-get)

# If we have yum installed, use it to install prerequisites. If not, use apt
if [[ -n $YUM_CMD ]]; then
    wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -P /tmp
    yum install -y /tmp/epel-release-latest-7.noarch.rpm

    yum install -y perl-Switch python3 python3-pip links
    user_test=$(getent passwd ec2-user)
    if [[ -n "${user_test}" ]]; then
        OSUSER=ec2-user
        OSGROUP=ec2-user
    else
        OSUSER=centos
        OSGROUP=centos
    fi
elif [[ -n $APT_GET_CMD ]]; then
    apt-get update
    apt-get install -y libswitch-perl python3 python3-pip links
    OSUSER=ubuntu
    OSGROUP=ubuntu
else
    # If we don't have yum or apt, we couldn't install the prerequisites, so exit
    echo "error can't install package $PACKAGE"
    exit 1;
fi

# Get the cryoSPARC license ID, optional path, and optional versions from the script arguments
CRYOSPARC_LICENSE_ID=$2
CRYOSPARC_INSTALL_PATH=${3:-/shared/cryosparc}
CUDA_INSTALL_PATH=${4:-/shared/cuda}
CUDA_VERSION=${5:-11.3.1}
CUDA_LONG_VERSION=${6:-11.3.1_465.19.01}

# Install the AWS CLI
pip3 install --upgrade awscli boto3

yum -y update

# Configure AWS
AWS_DEFAULT_REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | rev | cut -c 2- | rev)
aws configure set default.region "${AWS_DEFAULT_REGION}"
aws configure set default.output json


# Install CUDA Toolkit
mkdir -p "${CUDA_INSTALL_PATH}"
cd "${CUDA_INSTALL_PATH}" || return
wget "https://developer.download.nvidia.com/compute/cuda/${CUDA_VERSION}/local_installers/cuda_${CUDA_LONG_VERSION}_linux.run"
sh cuda_"${CUDA_LONG_VERSION}"_linux.run --defaultroot="${CUDA_INSTALL_PATH}" --toolkit --toolkitpath="${CUDA_INSTALL_PATH}"/"${CUDA_VERSION}" --samples --silent
rm cuda_"${CUDA_LONG_VERSION}"_linux.run

# Add CUDA to the path
cat > /etc/profile.d/cuda.sh << 'EOF'
PATH=$PATH:@CUDA_INSTALL_PATH@/CUDA_VERSION/bin
EOF
sed -i "s|@CUDA_INSTALL_PATH@|${CUDA_INSTALL_PATH}|g" /etc/profile.d/cuda.sh
sed -i "s|@CUDA_VERSION@|${CUDA_VERSION}|g" /etc/profile.d/cuda.sh
. /etc/profile.d/cuda.sh

# Download cryoSPARC
mkdir -p "${CRYOSPARC_INSTALL_PATH}"
cd "${CRYOSPARC_INSTALL_PATH}" || return
curl -L https://get.cryosparc.com/download/master-latest/${CRYOSPARC_LICENSE_ID} -o cryosparc_master.tar.gz
curl -L https://get.cryosparc.com/download/worker-latest/${CRYOSPARC_LICENSE_ID} -o cryosparc_worker.tar.gz

# Install master process
tar -xf cryosparc_master.tar.gz
cd cryosparc_master || return
./install.sh --license "${CRYOSPARC_LICENSE_ID}" \
      --hostname "${HOSTNAME}" \
      --dbpath "${CRYOSPARC_INSTALL_PATH}"/cryosparc_db \
      --port 45000 \
      --allowroot \
      --yes

# Start cryoSPARC master package
#"${CRYOSPARC_INSTALL_PATH}"/cryosparc_master/bin/cryosparcm start

# Add CryoSPARC to the path
cat > /etc/profile.d/cryosparc.sh << 'EOF'
PATH=$PATH:@CRYOSPARC_INSTALL_PATH@/cryosparc_master/bin
EOF
sed -i "s|@CRYOSPARC_INSTALL_PATH@|${CRYOSPARC_INSTALL_PATH}|g" /etc/profile.d/cryosparc.sh
. /etc/profile.d/cryosparc.sh

#echo "export CRYOSPARC_FORCE_USER=true" >> "${CRYOSPARC_INSTALL_PATH}"/cryosparc_master/config.sh
echo "export CRYOSPARC_FORCE_HOSTNAME=true" >> "${CRYOSPARC_INSTALL_PATH}"/cryosparc_master/config.sh
echo "export CRYOSPARC_DISABLE_IMPORT_ON_MASTER=true" >> "${CRYOSPARC_INSTALL_PATH}"/cryosparc_master/config.sh

# Install cryoSPARC work package
cd "${CRYOSPARC_INSTALL_PATH}" || return
tar -xf cryosparc_worker.tar.gz
cd cryosparc_worker || return
./install.sh --license "${CRYOSPARC_LICENSE_ID}" \
      --cudapath "${CUDA_INSTALL_PATH}/${CUDA_VERSION}" \
      --yes

rm ${CRYOSPARC_INSTALL_PATH}/*.tar.gz
chown -R ${OSUSER}:${OSGROUP} /shared/cryosparc

# Start cluster
/bin/su -c "${CRYOSPARC_INSTALL_PATH}/cryosparc_master/bin/cryosparcm start" - ${OSUSER}

# Create cluster config files for each GPU partition in SLURM
for PARTITION in gpu-large gpu-med gpu-small 
do

   cat > ${CRYOSPARC_INSTALL_PATH}/cluster_info.json << 'EOF'
{
"qdel_cmd_tpl": "scancel {{ cluster_job_id }}",
"worker_bin_path": "$CRYOSPARC_INSTALL_PATH/cryosparc_worker/bin/cryosparcw",
"title": "cryosparc-cluster",
"cache_path": "",
"qinfo_cmd_tpl": "sinfo",
"qsub_cmd_tpl": "sbatch {{ script_path_abs }}",
"qstat_cmd_tpl": "squeue -j {{ cluster_job_id }}",
"send_cmd_tpl": "{{ command }}",
"name": "@PARTITION@"
}
EOF

sed -i "s|@CRYOSPARC_INSTALL_PATH@|${CRYOSPARC_INSTALL_PATH}|g" "${CRYOSPARC_INSTALL_PATH}"/cluster_info.json
sed -i "s|@PARTITION@|${PARTITION}|g" "${CRYOSPARC_INSTALL_PATH}"/cluster_info.json

cat > ${CRYOSPARC_INSTALL_PATH}/cluster_script.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=cryosparc_{{ project_uid }}_{{ job_uid }}
#SBATCH --output={{ job_log_path_abs }}
#SBATCH --error={{ job_log_path_abs }}
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task={{ num_cpu }}
#SBATCH --gres=gpu:{{ num_gpu }}
#SBATCH --partition=@PARTITION@
{{ run_cmd }}
EOF

sed -i "s|@PARTITION@|${PARTITION}|g" "${CRYOSPARC_INSTALL_PATH}"/cluster_script.sh

# Connect CryoSPARC worker nodes to cluster
/bin/su -c "cd ${CRYOSPARC_INSTALL_PATH} && ${CRYOSPARC_INSTALL_PATH}/cryosparc_master/bin/cryosparcm cluster connect" - ${OSUSER}

done

# Restart CryoSPARC master
/bin/su -c "cd ${CRYOSPARC_INSTALL_PATH} && ${CRYOSPARC_INSTALL_PATH}/cryosparc_master/bin/cryosparcm restart" - ${OSUSER}

ls -l /shared/cryosparc/cryosparc_master
total 116
drwxrwxr-x  2 ec2-user ec2-user  4096 Jan 23 15:55 bin
drwxrwxr-x  3 ec2-user ec2-user  4096 Jan 23 15:55 build
-rwxrwxr-x  1 ec2-user ec2-user  5458 Jan 23 15:55 check_install_deps.sh
-rw-r--r--  1 ec2-user ec2-user   729 Jan 26 23:43 config.sh
drwxrwxr-x  5 ec2-user ec2-user  4096 Jan 23 15:55 cryosparc_app
drwxrwxr-x  9 ec2-user ec2-user  4096 Jan 23 15:55 cryosparc_command
drwxrwxr-x 11 ec2-user ec2-user  4096 Jan 26 23:45 cryosparc_compute
drwxrwxr-x  4 ec2-user ec2-user  4096 Jan 23 15:55 cryosparc_tools
drwxrwxr-x  4 ec2-user ec2-user  4096 Jan 23 15:55 cryosparc_webapp
drwxrwxr-x  4 ec2-user ec2-user  4096 Jan 23 15:55 deps_bundle
drwxrwxr-x  4 ec2-user ec2-user  4096 Jan 23 15:55 deps_bundle_hashes
-rw-rw-r--  1 ec2-user ec2-user  6170 Jan 23 15:55 environment.yml
-rw-rw-r--  1 ec2-user ec2-user  3056 Jan 23 15:55 install.py
-rwxrwxr-x  1 ec2-user ec2-user 13320 Jan 23 15:55 install.sh
-rw-rw-r--  1 ec2-user ec2-user 15088 Jan 23 15:55 patch.py
drwxrwxr-x  2 ec2-user ec2-user  4096 Jan 26 23:45 run
-rw-rw-r--  1 ec2-user ec2-user  4072 Jan 23 15:55 setup_client_ipython.py
-rw-rw-r--  1 ec2-user ec2-user  5217 Jan 23 15:55 supervisord.conf
drwxrwxr-x  2 ec2-user ec2-user  4096 Jan 23 15:55 systemd
-rw-rw-r--  1 ec2-user ec2-user     7 Jan 23 15:55 version

ls -l /shared/cryosparc/
total 24
-rw-r--r--  1 root     root      356 Jan 26 23:45 cluster_info.json
-rw-r--r--  1 root     root      313 Jan 26 23:45 cluster_script.sh
drwxrwxr-x  2 ec2-user ec2-user 4096 Jan 23 15:55 cryosparc2_worker
drwxrwxr-x  2 ec2-user ec2-user 4096 Jan 26 23:45 cryosparc_db
drwxrwxr-x 13 ec2-user ec2-user 4096 Jan 26 23:45 cryosparc_master
drwxrwxr-x  9 ec2-user ec2-user 4096 Jan 26 23:44 cryosparc_worker

When running cryosparcm start as

sh-4.2$ whoami
ssm-user

this is what i get

sh: cryosparcm: command not found

I didn’t create config.sh file, it’s already created

h-4.2$ ls
bin                    config.sh          cryosparc_compute  deps_bundle         install.py  run                      systemd
build                  cryosparc_app      cryosparc_tools    deps_bundle_hashes  install.sh  setup_client_ipython.py  version
check_install_deps.sh  cryosparc_command  cryosparc_webapp   environment.yml     patch.py    supervisord.conf

contents of `/tmp/post_install.log`
3084350K .......... .......... .......... .......... .......... 99%  315M 0s
3084400K .......... .......... .......... .......... .......... 99%  410M 0s
3084450K .......... ......                                     100%  441M=10s

2023-01-26 23:40:59 (299 MB/s) - ‘cuda_11.3.1_465.19.01_linux.run’ saved [3158494112/3158494112]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
100  771M  100  771M    0     0  37.8M      0  0:00:20  0:00:20 --:--:-- 45.0M
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 1797M  100 1797M    0     0  39.6M      0  0:00:45  0:00:45 --:--:-- 46.4M

************ CRYOSPARC SYSTEM: MASTER INSTALLER **************

 Installation Settings:
   License ID              : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
   Root Directory          : /shared/cryosparc/cryosparc_master
   Master Hostname         : ip-xxxxxxxxxxxxxxxxxxxx
   Database Directory      : /shared/cryosparc/cryosparc_db
   Base HTTP Port Number   : 45000
   Developer Install       : false
   Operating System        : Linux
   Insecure Install        : false
   Standalone Install      : false
   Version                 : v4.1.2

******************************************************************

The cryoSPARC System master installation will host a database,
web application, and command layer for cryoSPARC, but will not
perform processing jobs.

Please read and confirm the above settings. The cryoSPARC database
will be stored at the database directory which is relative to the
root directory, unless provided as an absolute path.
You can change the database directory now by passing the --dbapth
parameter to this installer, or later via the command line.

Are the above settings correct?

******************************************************************

 Setting up hard-coded config.sh environment variables

******************************************************************

 Installing all dependencies.

Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
ERROR: This command can only be called by the owner of this script: UNKNOWN. You are root!
Alternatively, set CRYOSPARC_FORCE_USER=true in cryosparc_master/config.sh to suppress this error.
******* CRYOSPARC SYSTEM: WORKER INSTALLER ***********************

 Installation Settings:
   License ID              : xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
   Root Directory          : /shared/cryosparc/cryosparc_worker
   Standalone Installation : false
   Version                 : v4.1.2

******************************************************************

 CUDA check..
 Found nvidia-smi at /bin/nvidia-smi

 CUDA Path was provided as /shared/cuda/11.3.1
 Checking CUDA installation...
 Found nvcc at /shared/cuda/11.3.1/bin/nvcc
 The above cuda installation will be used but can be changed later.

******************************************************************

 Setting up hard-coded config.sh environment variables

******************************************************************

 Installing all dependencies.

Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
Checking dependencies...
Dependencies for python have changed - reinstalling...
  ------------------------------------------------------------------------
  Installing anaconda python...
  ------------------------------------------------------------------------
PREFIX=/shared/cryosparc/cryosparc_worker/deps/anaconda
Unpacking payload ...
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /shared/cryosparc/cryosparc_worker/deps/anaconda

  added / updated specs:
    - _libgcc_mutex==0.1=main
    - _openmp_mutex==4.5=1_gnu
    - brotlipy==0.7.0=py38h27cfd23_1003
    - ca-certificates==2022.3.29=h06a4308_1
    - certifi==2021.10.8=py38h06a4308_2
    - cffi==1.15.0=py38hd667e15_1
    - charset-normalizer==2.0.4=pyhd3eb1b0_0
    - colorama==0.4.4=pyhd3eb1b0_0
    - conda-content-trust==0.1.1=pyhd3eb1b0_0
    - conda-package-handling==1.8.1=py38h7f8727e_0
    - conda==4.12.0=py38h06a4308_0
    - cryptography==36.0.0=py38h9ce1e76_0
    - idna==3.3=pyhd3eb1b0_0
    - ld_impl_linux-64==2.35.1=h7274673_9
    - libffi==3.3=he6710b0_2
    - libgcc-ng==9.3.0=h5101ec6_17
    - libgomp==9.3.0=h5101ec6_17
    - libstdcxx-ng==9.3.0=hd4cf53a_17
    - ncurses==6.3=h7f8727e_2
    - openssl==1.1.1n=h7f8727e_0
    - pip==21.2.4=py38h06a4308_0
    - pycosat==0.6.3=py38h7b6447c_1
    - pycparser==2.21=pyhd3eb1b0_0
    - pyopenssl==22.0.0=pyhd3eb1b0_0
    - pysocks==1.7.1=py38h06a4308_0
    - python==3.8.13=h12debd9_0
    - readline==8.1.2=h7f8727e_1
    - requests==2.27.1=pyhd3eb1b0_0
    - ruamel_yaml==0.15.100=py38h27cfd23_0
    - setuptools==61.2.0=py38h06a4308_0
    - six==1.16.0=pyhd3eb1b0_1
    - sqlite==3.38.2=hc218d9a_0
    - tk==8.6.11=h1ccaba5_0
    - tqdm==4.63.0=pyhd3eb1b0_0
    - urllib3==1.26.8=pyhd3eb1b0_0
    - wheel==0.37.1=pyhd3eb1b0_0
    - xz==5.2.5=h7b6447c_0
    - yaml==0.2.5=h7b6447c_0
    - zlib==1.2.12=h7f8727e_1


The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-4.5-1_gnu
  brotlipy           pkgs/main/linux-64::brotlipy-0.7.0-py38h27cfd23_1003
  ca-certificates    pkgs/main/linux-64::ca-certificates-2022.3.29-h06a4308_1
  certifi            pkgs/main/linux-64::certifi-2021.10.8-py38h06a4308_2
  cffi               pkgs/main/linux-64::cffi-1.15.0-py38hd667e15_1
  charset-normalizer pkgs/main/noarch::charset-normalizer-2.0.4-pyhd3eb1b0_0
  colorama           pkgs/main/noarch::colorama-0.4.4-pyhd3eb1b0_0
  conda              pkgs/main/linux-64::conda-4.12.0-py38h06a4308_0
  conda-content-tru~ pkgs/main/noarch::conda-content-trust-0.1.1-pyhd3eb1b0_0
  conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.8.1-py38h7f8727e_0
  cryptography       pkgs/main/linux-64::cryptography-36.0.0-py38h9ce1e76_0
  idna               pkgs/main/noarch::idna-3.3-pyhd3eb1b0_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.35.1-h7274673_9
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.3.0-h5101ec6_17
  libgomp            pkgs/main/linux-64::libgomp-9.3.0-h5101ec6_17
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.3.0-hd4cf53a_17
  ncurses            pkgs/main/linux-64::ncurses-6.3-h7f8727e_2
  openssl            pkgs/main/linux-64::openssl-1.1.1n-h7f8727e_0
  pip                pkgs/main/linux-64::pip-21.2.4-py38h06a4308_0
  pycosat            pkgs/main/linux-64::pycosat-0.6.3-py38h7b6447c_1
  pycparser          pkgs/main/noarch::pycparser-2.21-pyhd3eb1b0_0
  pyopenssl          pkgs/main/noarch::pyopenssl-22.0.0-pyhd3eb1b0_0
  pysocks            pkgs/main/linux-64::pysocks-1.7.1-py38h06a4308_0
  python             pkgs/main/linux-64::python-3.8.13-h12debd9_0
  readline           pkgs/main/linux-64::readline-8.1.2-h7f8727e_1
  requests           pkgs/main/noarch::requests-2.27.1-pyhd3eb1b0_0
  ruamel_yaml        pkgs/main/linux-64::ruamel_yaml-0.15.100-py38h27cfd23_0
  setuptools         pkgs/main/linux-64::setuptools-61.2.0-py38h06a4308_0
  six                pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1
  sqlite             pkgs/main/linux-64::sqlite-3.38.2-hc218d9a_0
  tk                 pkgs/main/linux-64::tk-8.6.11-h1ccaba5_0
  tqdm               pkgs/main/noarch::tqdm-4.63.0-pyhd3eb1b0_0
  urllib3            pkgs/main/noarch::urllib3-1.26.8-pyhd3eb1b0_0
  wheel              pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  yaml               pkgs/main/linux-64::yaml-0.2.5-h7b6447c_0
  zlib               pkgs/main/linux-64::zlib-1.2.12-h7f8727e_1


Preparing transaction: ...working... done
Executing transaction: ...working... done
installation finished.
  ------------------------------------------------------------------------
    Done.
    anaconda python installation successful.
  ------------------------------------------------------------------------
  Extracting all conda packages...
  ------------------------------------------------------------------------
.....................................................................................................................................................................................
  ------------------------------------------------------------------------
    Done.
    conda packages installation successful.
  ------------------------------------------------------------------------
  Preparing to install all pip packages...
  ------------------------------------------------------------------------
DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453
Processing ./deps_bundle/python/python_packages/pip_packages/pycuda-2020.1.tar.gz
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Installing collected packages: pycuda
  DEPRECATION: pycuda is being installed using the legacy 'setup.py install' method, because the '--no-binary' option was enabled for it and this currently disables local wheel building for projects that don't have a 'pyproject.toml' file. pip 23.1 will enforce this behaviour change. A possible replacement is toenable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/11451
  Running setup.py install for pycuda: started
  Running setup.py install for pycuda: still running...
  Running setup.py install for pycuda: finished with status 'done'
Successfully installed pycuda-2020.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to usea virtual environment instead: https://pip.pypa.io/warnings/venv
  ------------------------------------------------------------------------
    Done.
    pip packages installation successful.
  ------------------------------------------------------------------------
  Main dependency installation completed. Continuing...
  ------------------------------------------------------------------------
Completed.
Currently checking hash for ctffind
Dependencies for ctffind have changed - reinstalling...
  ------------------------------------------------------------------------
  ctffind 4.1.10 installation successful.
  ------------------------------------------------------------------------
Completed.
Currently checking hash for cudnn
Dependencies for cudnn have changed - reinstalling...
  ------------------------------------------------------------------------
  cudnn 8.1.0.77 for CUDA 11 installation successful.
  ------------------------------------------------------------------------
Completed.
Currently checking hash for gctf
Dependencies for gctf have changed - reinstalling...
  ------------------------------------------------------------------------
  Gctf v1.06 installation successful.
  ------------------------------------------------------------------------
Completed.
Completed dependency check.
Generating '/shared/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/libtiff/tiff_h_4_4_0.py' from '/shared/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/../include/tiff.h'

******* CRYOSPARC WORKER INSTALLATION COMPLETE *******************

 In order to run processing jobs, you will need to connect this
 worker to a cryoSPARC master.

******************************************************************
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "cryosparc_compute/__init__.py", line 9, in <module>
    from . import dataset  # noqa
  File "cryosparc_compute/dataset.py", line 59
    def to_dataframe(dset: Dataset[Row], exclude_uid=False):
                         ^
SyntaxError: invalid syntax
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "cryosparc_compute/__init__.py", line 9, in <module>
    from . import dataset  # noqa
  File "cryosparc_compute/dataset.py", line 59
    def to_dataframe(dset: Dataset[Row], exclude_uid=False):
                         ^
SyntaxError: invalid syntax
Starting cryoSPARC System master process..
/shared/cryosparc/cryosparc_master/bin/cryosparcm: line 942: supervisorctl: command not found
CryoSPARC is already running.
If you would like to restart, use cryosparcm restart
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "cryosparc_compute/__init__.py", line 9, in <module>
    from . import dataset  # noqa
  File "cryosparc_compute/dataset.py", line 59
    def to_dataframe(dset: Dataset[Row], exclude_uid=False):
                         ^
SyntaxError: invalid syntax
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "cryosparc_compute/__init__.py", line 9, in <module>
    from . import dataset  # noqa
  File "cryosparc_compute/dataset.py", line 59
    def to_dataframe(dset: Dataset[Row], exclude_uid=False):
                         ^
SyntaxError: invalid syntax
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "cryosparc_compute/__init__.py", line 9, in <module>
    from . import dataset  # noqa
  File "cryosparc_compute/dataset.py", line 59
    def to_dataframe(dset: Dataset[Row], exclude_uid=False):
                         ^
SyntaxError: invalid syntax
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
/shared/cryosparc/cryosparc_master/bin/cryosparcm: line 172: supervisorctl: command not found
CryoSPARC is running.
Stopping cryoSPARC
/shared/cryosparc/cryosparc_master/bin/cryosparcm: line 1048: supervisorctl: command not found

As a root user this is what i am getting
[root@ip-xxxxxxxxx cryosparc_master]# whoami
root
[root@ip-xxxxxxxxx cryosparc_master]# cryosparcm start
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
ERROR: This command can only be called by the owner of this script: ec2-user. You are root!
Alternatively, set CRYOSPARC_FORCE_USER=true in cryosparc_master/config.sh to suppress this error.
[root@ip-xxxxxxxxx cryosparc_master]# cryosparcm restart
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
ERROR: This command can only be called by the owner of this script: ec2-user. You are root!
Alternatively, set CRYOSPARC_FORCE_USER=true in cryosparc_master/config.sh to suppress this error.

cryosparc_master: ls
bin                    config.sh          cryosparc_compute  deps_bundle         install.py  run                      systemd
build                  cryosparc_app      cryosparc_tools    deps_bundle_hashes  install.sh  setup_client_ipython.py  version
check_install_deps.sh  cryosparc_command  cryosparc_webapp   environment.yml     patch.py    supervisord.conf

I hope this will be helpful and answered all your questions, sorry if i missed anything.
Always appreciating your support.

11

Based on directory ownership, this CryoSPARC instance should be managed by the ec2-user, but neither the root nor the ssm-user account.

12

My question is, i have been using the same script to deploy the CryoSparc using AWS ParallelCluster, i never had a rollback issue, ParallelCluster was creating all the resources without any issue and i have been using AWS Session Manager to access the Head Node and i used to start, restart the cryosparcm as a root user. The deployment was successful, for instance this is the screenshot taken sometime in August:

image
image698×588 53.2 KB

I still don’t get the answer why this is happening now after 7 months and if there is a way i can do or add something on the script to fix the new issue. The stack is failing because of AWS::CloudFormation::WaitCondition which is happening at the HeadNode.
AWS ParallelCluster experts were trying to help me with the issue but they said they don’t own the scripts and can’t fix it.
This was the guide i have been using Deploying CryoSPARC on AWS - CryoSPARC Guide but i had to make some changes based on our requirements.
I had to change the cryosparc-pcluster.config.template file to fit with ParallelCluster Version 3 plus i had to use an existing VPC other than that i used the install-cryosparc_v1.sh file provided by your company.
I wish if there is a way to fix the issue.
Really Appreciate your time.
Regards,

13

Don’t worry about the information in the screenshot, that EC2 doesn’t exist any more.

14

@brikti – you could change the tarball versions and install a previous version. E.g.,

curl -L "https://get.cryosparc.com/download/master-v4.0.3/${CRYOSPARC_LICENSE_ID}" -o cryosparc_master.tar.gz
curl -L "https://get.cryosparc.com/download/worker-v4.0.3/${CRYOSPARC_LICENSE_ID}" -o cryosparc_worker.tar.gz

Not exactly the right solution for the problem, but it will get CryoSPARC running for productivity. I’m unsure whether upgrading the dist to latest from the admin panel will work, or if it will require you to once again consider revising the post-install process.

15

Hi Bollig, after editing the script with your suggestion the Cryosparc is up and running and the stack is not rolling back any more. I know you mentioned that this is not the right solution but we will running some tests and will let you know.
Thank you so much for help.

16

Hello all,

I ran into the same issue described here but have a fix. The post-install script described in this thread is the example provided by AWS in this repo and I’ve submitted a PR to make it work with 4.0 and 4.1.

Long story short, it looks like trying to install everything as root and then changing the user afterwords doesn’t work well with the conda installation/configuration/install/etc done in 4.1. This isn’t too surprising, and the general recommendation to not install as root should be heeded, so the change as seen in the PR is basically “installing as as normal user”. Hopefully this (or something similar) gets merged here before others run into this, but if not, this comment should help others.

1 Like