cryoSPARC v2 → v3 Update Issues

Some people have been experiencing problems with the new v3 update. Most cases seem to be that the update only partially completes and does not fully update the worker module.

Here are some things to try before you attempt a full re-install:

  1. Save the failed installation log somewhere for future reference

  2. Make sure any existing conda environments are disabled

  3. Run these commands:

    cryosparcm stop
    cryosparcm start
    

    If you see any errors in the output, save the full output

  4. Check whether you can access the web interface

  5. In the cryosparc2_master folder, look for a cryosparc2_worker.tar.gz folder with a modification date that corresponds to when you tried to update to v3

  6. Copy or upload this cryosparc2_worker.tar.gz into the cryosparc2_worker installation folder

  7. Log into the machine with the cryosparc2_worker folder is hosted (if required) and navigate to it via command line

  8. Run this command to install the worker update:

    bin/cryosparcw update --override
    

    Save the log if you see any errors

  9. If you see any permission errors, try running these commands (substituting $LICENSE_ID and $CUDA_PATH with your cryoSPARC license key and the CUDA installation path on your system, respectively):

    chmod -R 755 deps
    ./install.sh --license $LICENSE_ID --cudapath $CUDA_PATH --yes
    
  10. Try running a job

If you get stuck at any point, post any error logs you got in this thread for further troubleshooting.

3 Likes

We have permissions issues when updating from v2.15.0 to v3.0:

cryosparcm update 
CryoSPARC current version v2.15.0
          update starting on Thu Dec 10 16:53:17 EST 2020

No version specified - updating to latest version.

=============================
Updating to version v3.0.0.
=============================
CryoSPARC is running.
Stopping cryosparc 
command_proxy: stopped
command_vis: stopped
webapp: stopped
command_core: stopped
database: stopped
Shut down
  Downloading master update...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  769M  100  769M    0     0  35.1M      0  0:00:21  0:00:21 --:--:-- 57.3M
  Downloading worker update...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 2903M  100 2903M    0     0  15.6M      0  0:03:05  0:03:05 --:--:-- 12.8M
  Done.

 Update will now be applied to the master installation, 
 followed by worker installations on other nodes.

  Deleting old files...
  Extracting...
  Done.
  Updating dependencies...
Could not find conda environment: cryosparc_master_env
You can list all discoverable environments with `conda info --envs`.

Warning: Could not activate conda environment; this indicates that a cryoSPARC installation is either incomplete or in progress
===================================================
Warning: cryoSPARC v2 dependencies and cryosparc2_worker.tar.gz detected.
Performing full v3 worker update in 10 seconds (to cancel, press Ctrl + C)
...
 
===================================================
Installing latest master dependencies.
===================================================
 
  Checking dependencies... 
  Dependencies for python have changed - reinstalling...
  ------------------------------------------------------------------------
  Installing anaconda python...
  ------------------------------------------------------------------------
PREFIX=/mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps/anaconda
Unpacking payload ...
Collecting package metadata (current_repodata.json): done                                                                                                                                                                               
Solving environment: done

## Package Plan ##

  environment location: /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps/anaconda

  added / updated specs:
    - _libgcc_mutex==0.1=main
    - ca-certificates==2020.1.1=0
    - certifi==2020.4.5.1=py37_0
    - cffi==1.14.0=py37he30daa8_1
    - chardet==3.0.4=py37_1003
    - conda-package-handling==1.6.1=py37h7b6447c_0
    - conda==4.8.3=py37_0
    - cryptography==2.9.2=py37h1ba5d50_0
    - idna==2.9=py_1
    - ld_impl_linux-64==2.33.1=h53a641e_7
    - libedit==3.1.20181209=hc058e9b_0
    - libffi==3.3=he6710b0_1
    - libgcc-ng==9.1.0=hdf63c60_0
    - libstdcxx-ng==9.1.0=hdf63c60_0
    - ncurses==6.2=he6710b0_1
    - openssl==1.1.1g=h7b6447c_0
    - pip==20.0.2=py37_3
    - pycosat==0.6.3=py37h7b6447c_0
    - pycparser==2.20=py_0
    - pyopenssl==19.1.0=py37_0
    - pysocks==1.7.1=py37_0
    - python==3.7.7=hcff3b4d_5
    - readline==8.0=h7b6447c_0
    - requests==2.23.0=py37_0
    - ruamel_yaml==0.15.87=py37h7b6447c_0
    - setuptools==46.4.0=py37_0
    - six==1.14.0=py37_0
    - sqlite==3.31.1=h62c20be_1
    - tk==8.6.8=hbc83047_0
    - tqdm==4.46.0=py_0
    - urllib3==1.25.8=py37_0
    - wheel==0.34.2=py37_0
    - xz==5.2.5=h7b6447c_0
    - yaml==0.1.7=had09818_2
    - zlib==1.2.11=h7b6447c_3


The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.1.1-0
  certifi            pkgs/main/linux-64::certifi-2020.4.5.1-py37_0
  cffi               pkgs/main/linux-64::cffi-1.14.0-py37he30daa8_1
  chardet            pkgs/main/linux-64::chardet-3.0.4-py37_1003
  conda              pkgs/main/linux-64::conda-4.8.3-py37_0
  conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.6.1-py37h7b6447c_0
  cryptography       pkgs/main/linux-64::cryptography-2.9.2-py37h1ba5d50_0
  idna               pkgs/main/noarch::idna-2.9-py_1
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_1
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1g-h7b6447c_0
  pip                pkgs/main/linux-64::pip-20.0.2-py37_3
  pycosat            pkgs/main/linux-64::pycosat-0.6.3-py37h7b6447c_0
  pycparser          pkgs/main/noarch::pycparser-2.20-py_0
  pyopenssl          pkgs/main/linux-64::pyopenssl-19.1.0-py37_0
  pysocks            pkgs/main/linux-64::pysocks-1.7.1-py37_0
  python             pkgs/main/linux-64::python-3.7.7-hcff3b4d_5
  readline           pkgs/main/linux-64::readline-8.0-h7b6447c_0
  requests           pkgs/main/linux-64::requests-2.23.0-py37_0
  ruamel_yaml        pkgs/main/linux-64::ruamel_yaml-0.15.87-py37h7b6447c_0
  setuptools         pkgs/main/linux-64::setuptools-46.4.0-py37_0
  six                pkgs/main/linux-64::six-1.14.0-py37_0
  sqlite             pkgs/main/linux-64::sqlite-3.31.1-h62c20be_1
  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
  tqdm               pkgs/main/noarch::tqdm-4.46.0-py_0
  urllib3            pkgs/main/linux-64::urllib3-1.25.8-py37_0
  wheel              pkgs/main/linux-64::wheel-0.34.2-py37_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  yaml               pkgs/main/linux-64::yaml-0.1.7-had09818_2
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3


Preparing transaction: done
Executing transaction: done
/mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps_bundle/python/anaconda.sh: line 449: /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps/anaconda/bin/python: Permission denied
  ------------------------------------------------------------------------
    Done.
    anaconda python installation successful.
  ------------------------------------------------------------------------
  Extracting all conda packages...
  ------------------------------------------------------------------------
........................................................................../mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps/anaconda/etc/profile.d/conda.sh: line 100: /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps/anaconda/bin/conda: Permission denied
/mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps/anaconda/etc/profile.d/conda.sh: line 100: /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps/anaconda/bin/conda: Permission denied
/mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps/anaconda/etc/profile.d/conda.sh: line 100: /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps/anaconda/bin/conda: Permission denied

  ------------------------------------------------------------------------
    Done.
    conda packages installation successful.
  ------------------------------------------------------------------------
  Main dependency installation completed. Continuing...
  ------------------------------------------------------------------------
  Completed.
  Currently checking hash for mongodb
  Dependencies for mongodb have changed - reinstalling...
  ------------------------------------------------------------------------
  mongodb 3.4.10 installation successful.
  ------------------------------------------------------------------------
  Completed.
  Completed dependency check. 
/mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps/anaconda/etc/profile.d/conda.sh: line 55: /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps/anaconda/bin/conda: Permission denied
Warning: Could not activate conda environment; this indicates that a cryoSPARC installation is either incomplete or in progress
 
===================================================
Successfully updated master to version v3.0.0.
===================================================
 
/mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps/anaconda/etc/profile.d/conda.sh: line 55: /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_master/deps/anaconda/bin/conda: Permission denied
Warning: Could not activate conda environment; this indicates that a cryoSPARC installation is either incomplete or in progress
Starting cryoSPARC System master process..
Fatal Python error: _PyMainInterpreterConfig_Read: memory allocation failed
ValueError: character U+6900632d is not in range [U+0000; U+10ffff]

Current thread 0x00007f7147b4d740 (most recent call first):
============================

We’ve also tried chmod 755 * at this path cryosparc2_master/deps/anaconda/bin. This allows the installation to complete when installing on top of the install, but jobs fail due to pycuda issues (permissions issues arise right before pycuda installation).
We’ve tried a fresh install and we have the same problems.
After issues, we reinstalled v2.15.0 in which there were no permission issues and installation proceeded fine.

Any ideas why we are having these permission issues?

Thanks,
Matt

1 Like

We have not seen these permission issues before, your workaround of calling chmod to the contents definitely seems like the right way to fix it.

To finish the installation after you’ve changed your permissions, don’t redo the install on top of the initial install. Instead, follow my instructions above from step 5 to complete the worker module update.

I should have mentioned that we’ve tried installing the worker and also have the same permissions in the worker folder. But I’ve followed your instructions to regenerate the error:

europa-cryosparc2_worker: bin/cryosparcw update --override

Forcing update...
=============================
Updating worker...
=============================
  Deleting old files...
  Extracting...
  Done.
  Updating dependencies...
Checking dependencies... 
Forcing dependencies to be reinstalled...
  ------------------------------------------------------------------------
  Installing anaconda python...
  ------------------------------------------------------------------------
PREFIX=/mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_worker/deps/anaconda
Unpacking payload ...
Collecting package metadata (current_repodata.json): done                                                                   
Solving environment: done

## Package Plan ##

  environment location: /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_worker/deps/anaconda

  added / updated specs:
    - _libgcc_mutex==0.1=main
    - ca-certificates==2020.1.1=0
    - certifi==2020.4.5.1=py37_0
    - cffi==1.14.0=py37he30daa8_1
    - chardet==3.0.4=py37_1003
    - conda-package-handling==1.6.1=py37h7b6447c_0
    - conda==4.8.3=py37_0
    - cryptography==2.9.2=py37h1ba5d50_0
    - idna==2.9=py_1
    - ld_impl_linux-64==2.33.1=h53a641e_7
    - libedit==3.1.20181209=hc058e9b_0
    - libffi==3.3=he6710b0_1
    - libgcc-ng==9.1.0=hdf63c60_0
    - libstdcxx-ng==9.1.0=hdf63c60_0
    - ncurses==6.2=he6710b0_1
    - openssl==1.1.1g=h7b6447c_0
    - pip==20.0.2=py37_3
    - pycosat==0.6.3=py37h7b6447c_0
    - pycparser==2.20=py_0
    - pyopenssl==19.1.0=py37_0
    - pysocks==1.7.1=py37_0
    - python==3.7.7=hcff3b4d_5
    - readline==8.0=h7b6447c_0
    - requests==2.23.0=py37_0
    - ruamel_yaml==0.15.87=py37h7b6447c_0
    - setuptools==46.4.0=py37_0
    - six==1.14.0=py37_0
    - sqlite==3.31.1=h62c20be_1
    - tk==8.6.8=hbc83047_0
    - tqdm==4.46.0=py_0
    - urllib3==1.25.8=py37_0
    - wheel==0.34.2=py37_0
    - xz==5.2.5=h7b6447c_0
    - yaml==0.1.7=had09818_2
    - zlib==1.2.11=h7b6447c_3


The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.1.1-0
  certifi            pkgs/main/linux-64::certifi-2020.4.5.1-py37_0
  cffi               pkgs/main/linux-64::cffi-1.14.0-py37he30daa8_1
  chardet            pkgs/main/linux-64::chardet-3.0.4-py37_1003
  conda              pkgs/main/linux-64::conda-4.8.3-py37_0
  conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.6.1-py37h7b6447c_0
  cryptography       pkgs/main/linux-64::cryptography-2.9.2-py37h1ba5d50_0
  idna               pkgs/main/noarch::idna-2.9-py_1
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_1
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1g-h7b6447c_0
  pip                pkgs/main/linux-64::pip-20.0.2-py37_3
  pycosat            pkgs/main/linux-64::pycosat-0.6.3-py37h7b6447c_0
  pycparser          pkgs/main/noarch::pycparser-2.20-py_0
  pyopenssl          pkgs/main/linux-64::pyopenssl-19.1.0-py37_0
  pysocks            pkgs/main/linux-64::pysocks-1.7.1-py37_0
  python             pkgs/main/linux-64::python-3.7.7-hcff3b4d_5
  readline           pkgs/main/linux-64::readline-8.0-h7b6447c_0
  requests           pkgs/main/linux-64::requests-2.23.0-py37_0
  ruamel_yaml        pkgs/main/linux-64::ruamel_yaml-0.15.87-py37h7b6447c_0
  setuptools         pkgs/main/linux-64::setuptools-46.4.0-py37_0
  six                pkgs/main/linux-64::six-1.14.0-py37_0
  sqlite             pkgs/main/linux-64::sqlite-3.31.1-h62c20be_1
  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
  tqdm               pkgs/main/noarch::tqdm-4.46.0-py_0
  urllib3            pkgs/main/linux-64::urllib3-1.25.8-py37_0
  wheel              pkgs/main/linux-64::wheel-0.34.2-py37_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  yaml               pkgs/main/linux-64::yaml-0.1.7-had09818_2
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3


Preparing transaction: done
Executing transaction: done
/mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_worker/deps_bundle/python/anaconda.sh: line 449: /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_worker/deps/anaconda/bin/python: Permission denied
  ------------------------------------------------------------------------
    Done.
    anaconda python installation successful.
  ------------------------------------------------------------------------
  Extracting all conda packages...
  ------------------------------------------------------------------------
................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................../mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_worker/deps/anaconda/etc/profile.d/conda.sh: line 100: /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_worker/deps/anaconda/bin/conda: Permission denied
/mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_worker/deps/anaconda/etc/profile.d/conda.sh: line 100: /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_worker/deps/anaconda/bin/conda: Permission denied
/mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_worker/deps/anaconda/etc/profile.d/conda.sh: line 100: /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_worker/deps/anaconda/bin/conda: Permission denied

  ------------------------------------------------------------------------
    Done.
    conda packages installation successful.
  ------------------------------------------------------------------------
  Preparing to install all pip packages...
  ------------------------------------------------------------------------
Processing ./deps_bundle/python/python_packages/pip_packages/pycuda-2020.1.tar.gz
Processing ./deps_bundle/python/python_packages/pip_packages/scikit_cuda-0.5.3-py2.py3-none-any.whl
Skipping wheel build for pycuda, due to binaries being disabled for it.
Installing collected packages: scikit-cuda, pycuda
    Running setup.py install for pycuda ... error
    ERROR: Command errored out with exit status 1:
     command: /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-ma1amt6j/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-ma1amt6j/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-h31t32h9/install-record.txt --single-version-externally-managed --compile --install-headers /mnt/grid/joshua-tor/hpc/home/data/jaremko/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m/pycuda
         cwd: /tmp/pip-req-build-ma1amt6j/
    Complete output (7000 lines):
   XXXXXXdeleted 7000 lines here because of character limitXXXXX
  ------------------------------------------------------------------------
    Done.
    pip packages installation successful.
  ------------------------------------------------------------------------
  Main dependency installation completed. Continuing...
  ------------------------------------------------------------------------
Completed.
Currently checking hash for ctffind
Forcing reinstall for dependency ctffind...
  ------------------------------------------------------------------------
  ctffind 4.1.10 installation successful.
  ------------------------------------------------------------------------
Completed.
Currently checking hash for gctf
Forcing reinstall for dependency gctf...
  ------------------------------------------------------------------------
  Gctf v1.06 installation successful.
  ------------------------------------------------------------------------
Completed.
Completed dependency check. 
 
Successfully updated.

============

Can you send me the long directory listing for the cryosparc2_worker/deps/anaconda/bin folder? i.e., the output of

ls -la cryosparc2_worker/deps/anaconda/bin

This will help me further troubleshoot why this is happening.

Then I think you should be able to get it working like this:

  1. Change the permissions to each file in cryosparc2_worker/deps/anaconda/bin to 0755, as you did before
  2. On a machine with GPUs, navigate into the worker directory
    cd cryosparc2_worker
    
  3. Enable cryoSPARC’s worker python environment in your current shell:
    eval $(bin/cryosparcw env)
    
  4. Run these commands to finish installing pip dependencies, including pycuda
    pip install deps_bundle/python/python_packages/pip_packages/* --no-dependencies --no-binary :all: --force-reinstall
    

Save any errors you get in this process and send them to me.

1 Like

I will give this steps a shot. Here is the list:

ls -la cryosparc2_worker/deps/anaconda/bin
total 24160
drwxr-sr-x.  2 jaremko joshuatonew   32768 Dec 11 13:54 .
drwxr-sr-x. 15 jaremko joshuatonew    4096 Dec 11 13:54 ..
-rwxrwxr-x.  1 jaremko joshuatonew     184 Dec 11 13:52 2to3
-rwxrwxr-x.  1 jaremko joshuatonew    2394 Dec 11 13:52 activate
-rwxr-xr-x.  2 jaremko joshuatonew 1040032 Jun 14  2016 assistant-qt4
-rwxrwxr-x.  1 jaremko joshuatonew     305 Dec 11 13:52 bokeh
-rwxrwxr-x.  1 jaremko joshuatonew    3089 Dec 11 13:52 cairo-trace
lrwxrwxrwx.  1 jaremko joshuatonew       3 Dec 11 13:53 captoinfo -> tic
-rwxrwxr-x.  1 jaremko joshuatonew     313 Dec 11 13:52 chardetect
-rwxrwxr-x.  2 jaremko joshuatonew   39552 Nov  8  2017 cjpeg
-rwxrwxr-x.  2 jaremko joshuatonew   14296 Nov 28  2018 clear
-rwxrwxr-x.  1 jaremko joshuatonew     300 Dec 11 13:52 conda
-rwxrwxr-x.  1 jaremko joshuatonew     309 Dec 11 13:52 conda-env
-rwxrwxr-x.  1 jaremko joshuatonew    5314 Dec 11 13:52 c_rehash
-rwxr-xr-x.  1 jaremko joshuatonew     226 Dec 11 13:52 cygdb
-rwxr-xr-x.  1 jaremko joshuatonew     235 Dec 11 13:52 cython
-rwxr-xr-x.  1 jaremko joshuatonew     228 Dec 11 13:52 cythonize
-rwxrwxr-x.  1 jaremko joshuatonew     311 Dec 11 13:52 dask-mpi
-rwxrwxr-x.  1 jaremko joshuatonew     314 Dec 11 13:52 dask-remote
-rwxrwxr-x.  1 jaremko joshuatonew     317 Dec 11 13:52 dask-scheduler
-rwxrwxr-x.  1 jaremko joshuatonew     311 Dec 11 13:52 dask-ssh
-rwxrwxr-x.  1 jaremko joshuatonew     314 Dec 11 13:52 dask-submit
-rwxrwxr-x.  1 jaremko joshuatonew     314 Dec 11 13:52 dask-worker
-rwxrwxr-x.  1 jaremko joshuatonew    2381 Dec 11 13:52 deactivate
-rwxr-xr-x.  2 jaremko joshuatonew  492648 Jun 14  2016 designer-qt4
-rwxrwxr-x.  2 jaremko joshuatonew   34936 Nov  8  2017 djpeg
-rwxrwxr-x.  1 jaremko joshuatonew     322 Dec 11 13:52 easy_install
-rwxrwxr-x.  1 jaremko joshuatonew     846 Dec 11 13:52 f2py
-rwxrwxr-x.  2 jaremko joshuatonew   25856 Nov 21  2017 fax2ps
-rwxrwxr-x.  2 jaremko joshuatonew   25624 Nov 21  2017 fax2tiff
-rwxrwxr-x.  2 jaremko joshuatonew   29488 Feb  4  2017 fc-cache
-rwxrwxr-x.  2 jaremko joshuatonew   32216 Feb  4  2017 fc-cat
-rwxrwxr-x.  2 jaremko joshuatonew   19960 Feb  4  2017 fc-list
-rwxrwxr-x.  2 jaremko joshuatonew   21432 Feb  4  2017 fc-match
-rwxrwxr-x.  2 jaremko joshuatonew   19744 Feb  4  2017 fc-pattern
-rwxrwxr-x.  1 jaremko joshuatonew   20408 Dec 11 13:52 fc-query
-rwxrwxr-x.  1 jaremko joshuatonew   20320 Dec 11 13:52 fc-scan
-rwxrwxr-x.  1 jaremko joshuatonew   28808 Dec 11 13:52 fc-validate
-rwxrwxr-x.  2 jaremko joshuatonew   57083 Jul  5  2018 fftwf-wisdom
-rwxrwxr-x.  2 jaremko joshuatonew   57083 Jul  5  2018 fftwl-wisdom
-rwxrwxr-x.  2 jaremko joshuatonew   57035 Jul  5  2018 fftw-wisdom
-rwxrwxr-x.  2 jaremko joshuatonew    2279 Jul  5  2018 fftw-wisdom-to-conf
-rwxrwxr-x.  1 jaremko joshuatonew     300 Dec 11 13:52 flask
-rwxrwxr-x.  1 jaremko joshuatonew    4122 Dec 11 13:52 freetype-config
-rwxrwxr-x.  1 jaremko joshuatonew   34778 Dec 11 13:52 iconv
-rwxrwxr-x.  1 jaremko joshuatonew     182 Dec 11 13:52 idle
-rwxrwxr-x.  1 jaremko joshuatonew     333 Dec 11 13:52 imageio_download_bin
-rwxrwxr-x.  1 jaremko joshuatonew     329 Dec 11 13:52 imageio_remove_bin
-rwxrwxr-x.  2 jaremko joshuatonew   63520 Nov 28  2018 infocmp
lrwxrwxrwx.  1 jaremko joshuatonew       3 Dec 11 13:53 infotocap -> tic
-rwxrwxr-x.  1 jaremko joshuatonew     253 Dec 11 13:52 iptest
-rwxrwxr-x.  1 jaremko joshuatonew     253 Dec 11 13:52 iptest2
-rwxr-xr-x.  1 jaremko joshuatonew     207 Dec 11 13:52 ipython
-rwxr-xr-x.  1 jaremko joshuatonew     207 Dec 11 13:52 ipython2
-rwxrwxr-x.  2 jaremko joshuatonew   43120 Nov  8  2017 jpegtran
-rwxr-xr-x.  2 jaremko joshuatonew  292216 Jun 14  2016 lconvert-qt4
-rwxrwxr-x.  1 jaremko joshuatonew    2410 Dec 11 13:52 libpng16-config
lrwxrwxrwx.  1 jaremko joshuatonew      15 Dec 11 13:53 libpng-config -> libpng16-config
-rwxr-xr-x.  2 jaremko joshuatonew 1610096 Jun 14  2016 linguist-qt4
-rwxr-xr-x.  2 jaremko joshuatonew 1755608 Jun 14  2016 lrelease-qt4
-rwxr-xr-x.  2 jaremko joshuatonew  942904 Jun 14  2016 lupdate-qt4
lrwxrwxrwx.  1 jaremko joshuatonew       2 Dec 11 13:54 lzcat -> xz
lrwxrwxrwx.  1 jaremko joshuatonew       6 Dec 11 13:54 lzcmp -> xzdiff
lrwxrwxrwx.  1 jaremko joshuatonew       6 Dec 11 13:54 lzdiff -> xzdiff
lrwxrwxrwx.  1 jaremko joshuatonew       6 Dec 11 13:54 lzegrep -> xzgrep
lrwxrwxrwx.  1 jaremko joshuatonew       6 Dec 11 13:54 lzfgrep -> xzgrep
lrwxrwxrwx.  1 jaremko joshuatonew       6 Dec 11 13:54 lzgrep -> xzgrep
lrwxrwxrwx.  1 jaremko joshuatonew       6 Dec 11 13:54 lzless -> xzless
lrwxrwxrwx.  1 jaremko joshuatonew       2 Dec 11 13:54 lzma -> xz
-rwxrwxr-x.  2 jaremko joshuatonew   20848 May 16  2018 lzmadec
-rwxrwxr-x.  1 jaremko joshuatonew   16656 Dec 11 13:52 lzmainfo
lrwxrwxrwx.  1 jaremko joshuatonew       6 Dec 11 13:54 lzmore -> xzmore
-rwxrwxr-x.  1 jaremko joshuatonew     305 Dec 11 13:52 mako-render
-rwxr-xr-x.  2 jaremko joshuatonew 1229688 Jun 14  2016 moc-qt4
-rwxrwxr-x.  1 jaremko joshuatonew    6435 Dec 11 13:52 ncursesw6-config
-rwxrwxr-x.  1 jaremko joshuatonew     308 Dec 11 13:52 numba
-rwxrwxr-x.  2 jaremko joshuatonew  667312 Aug 14  2018 openssl
-rwxrwxr-x.  2 jaremko joshuatonew   21104 Nov 21  2017 pal2rgb
lrwxrwxrwx.  1 jaremko joshuatonew       6 Dec 11 13:53 pbunzip2 -> pbzip2
lrwxrwxrwx.  1 jaremko joshuatonew       6 Dec 11 13:53 pbzcat -> pbzip2
-rwxr-xr-x.  2 jaremko joshuatonew  301108 Aug  8  2016 pbzip2
-rwxrwxr-x.  1 jaremko joshuatonew     294 Dec 11 13:52 pip
-rwxr-xr-x.  2 jaremko joshuatonew   49432 Jun 14  2016 pixeltool-qt4
-rwxrwxr-x.  2 jaremko joshuatonew   60440 Apr 23  2019 pngfix
-rwxrwxr-x.  2 jaremko joshuatonew   19800 Apr 23  2019 png-fix-itxt
-rwxrwxr-x.  2 jaremko joshuatonew   20968 Nov 21  2017 ppm2tiff
-rwxrwxr-x.  1 jaremko joshuatonew     301 Dec 11 13:52 pycc
-rwxrwxr-x.  1 jaremko joshuatonew     167 Dec 11 13:52 pydoc
-rwxr-xr-x.  1 jaremko joshuatonew     309 Dec 11 13:53 pygmentize
-rwxrwxr-x.  2 jaremko joshuatonew  220474 Jul 14  2016 pylupdate4
-rwxrwxr-x.  2 jaremko joshuatonew   67245 Jul 14  2016 pyrcc4
-rwxrwxr-x.  1 jaremko joshuatonew     298 Dec 11 13:52 pytest
-rwxrwxr-x.  1 jaremko joshuatonew     298 Dec 11 13:52 py.test
lrwxrwxrwx.  1 jaremko joshuatonew       9 Dec 11 13:53 python -> python2.7
lrwxrwxrwx.  1 jaremko joshuatonew       9 Dec 11 13:53 python2 -> python2.7
-rwxrwxr-x.  1 jaremko joshuatonew   19760 Oct 10  2018 python2.7
-rwxrwxr-x.  1 jaremko joshuatonew    1770 Dec 11 13:52 python2.7-config
lrwxrwxrwx.  1 jaremko joshuatonew      16 Dec 11 13:53 python2-config -> python2.7-config
lrwxrwxrwx.  1 jaremko joshuatonew      16 Dec 11 13:53 python-config -> python2.7-config
-rwxrwxr-x.  1 jaremko joshuatonew     271 Dec 11 13:52 pyuic4
-rwxr-xr-x.  2 jaremko joshuatonew   70848 Jun 14  2016 qcollectiongenerator-qt4
-rwxr-xr-x.  2 jaremko joshuatonew   28528 Jun 14  2016 qdbuscpp2xml-qt4
-rwxr-xr-x.  2 jaremko joshuatonew   65616 Jun 14  2016 qdbus-qt4
-rwxr-xr-x.  2 jaremko joshuatonew  120568 Jun 14  2016 qdbusviewer-qt4
-rwxr-xr-x.  2 jaremko joshuatonew   69328 Jun 14  2016 qdbusxml2cpp-qt4
-rwxr-xr-x.  2 jaremko joshuatonew 1815864 Jun 14  2016 qdoc3-qt4
-rwxr-xr-x.  2 jaremko joshuatonew  173432 Jun 14  2016 qhelpconverter-qt4
-rwxr-xr-x.  2 jaremko joshuatonew   24032 Jun 14  2016 qhelpgenerator-qt4
-rwxr-xr-x.  2 jaremko joshuatonew 3550784 Jun 14  2016 qmake-qt4
-rwxr-xr-x.  2 jaremko joshuatonew   94512 Jun 14  2016 qmlplugindump-qt4
-rwxr-xr-x.  2 jaremko joshuatonew  339904 Jun 14  2016 qmlviewer-qt4
-rwxr-xr-x.  2 jaremko joshuatonew 1006072 Jun 14  2016 qt3to4-qt4
-rw-rw-r--.  1 jaremko joshuatonew     454 Dec 11 13:52 qt.conf
-rwxr-xr-x.  2 jaremko joshuatonew  221880 Jun 14  2016 qtconfig-qt4
-rwxr-xr-x.  2 jaremko joshuatonew   37464 Jun 14  2016 qttracereplay-qt4
-rwxrwxr-x.  2 jaremko joshuatonew   29160 Nov 21  2017 raw2tiff
-rwxr-xr-x.  2 jaremko joshuatonew 1326848 Jun 14  2016 rcc-qt4
-rwxrwxr-x.  2 jaremko joshuatonew   16264 Nov  8  2017 rdjpgcom
lrwxrwxrwx.  1 jaremko joshuatonew       4 Dec 11 13:53 reset -> tset
-rwxrwxr-x.  2 jaremko joshuatonew  400796 Jul 14  2016 sip
-rwxrwxr-x.  1 jaremko joshuatonew     312 Dec 11 13:52 skivi
-rwxrwxr-x.  1 jaremko joshuatonew   18630 Dec 11 13:52 smtpd.py
-rwxrwxr-x.  2 jaremko joshuatonew 1440768 Oct 25  2019 sqlite3
-rwxrwxr-x.  2 jaremko joshuatonew   29841 Aug 28  2018 sqlite3_analyzer
-rwxrwxr-x.  2 jaremko joshuatonew   14288 Nov 28  2018 tabs
lrwxrwxrwx.  1 jaremko joshuatonew       8 Dec 11 13:54 tclsh -> tclsh8.6
-rwxrwxr-x.  2 jaremko joshuatonew   20616 Aug 28  2018 tclsh8.6
-rwxrwxr-x.  2 jaremko joshuatonew   88136 Nov 28  2018 tic
-rwxrwxr-x.  2 jaremko joshuatonew   21152 Nov 21  2017 tiff2bw
-rwxrwxr-x.  2 jaremko joshuatonew   83496 Nov 21  2017 tiff2pdf
-rwxrwxr-x.  2 jaremko joshuatonew   66432 Nov 21  2017 tiff2ps
-rwxrwxr-x.  2 jaremko joshuatonew   20928 Nov 21  2017 tiff2rgba
-rwxrwxr-x.  2 jaremko joshuatonew   25152 Nov 21  2017 tiffcmp
-rwxrwxr-x.  2 jaremko joshuatonew   49104 Nov 21  2017 tiffcp
-rwxrwxr-x.  2 jaremko joshuatonew  117096 Nov 21  2017 tiffcrop
-rwxrwxr-x.  2 jaremko joshuatonew   20816 Nov 21  2017 tiffdither
-rwxrwxr-x.  2 jaremko joshuatonew   29928 Nov 21  2017 tiffdump
-rwxrwxr-x.  2 jaremko joshuatonew   25960 Nov 21  2017 tiffinfo
-rwxrwxr-x.  2 jaremko joshuatonew   25168 Nov 21  2017 tiffmedian
-rwxrwxr-x.  2 jaremko joshuatonew   20560 Nov 21  2017 tiffset
-rwxrwxr-x.  2 jaremko joshuatonew   20640 Nov 21  2017 tiffsplit
-rwxrwxr-x.  2 jaremko joshuatonew   22480 Nov 28  2018 toe
-rwxrwxr-x.  2 jaremko joshuatonew   22512 Nov 28  2018 tput
-rwxrwxr-x.  2 jaremko joshuatonew   30680 Nov 28  2018 tset
-rwxr-xr-x.  2 jaremko joshuatonew 1853288 Jun 14  2016 uic-qt4
lrwxrwxrwx.  1 jaremko joshuatonew       2 Dec 11 13:54 unlzma -> xz
lrwxrwxrwx.  1 jaremko joshuatonew       2 Dec 11 13:54 unxz -> xz
-rwxrwxr-x.  1 jaremko joshuatonew     300 Dec 11 13:52 wheel
lrwxrwxrwx.  1 jaremko joshuatonew       7 Dec 11 13:54 wish -> wish8.6
-rwxrwxr-x.  2 jaremko joshuatonew   20768 Aug 28  2018 wish8.6
-rwxrwxr-x.  2 jaremko joshuatonew   16368 Nov  8  2017 wrjpgcom
-rwxrwxr-x.  1 jaremko joshuatonew    2777 Dec 11 13:52 xml2-config
-rwxrwxr-x.  2 jaremko joshuatonew   30272 Jun 24  2019 xmlcatalog
-rwxrwxr-x.  2 jaremko joshuatonew   93152 Jun 24  2019 xmllint
-rwxr-xr-x.  2 jaremko joshuatonew   69480 Jun 14  2016 xmlpatterns-qt4
-rwxr-xr-x.  2 jaremko joshuatonew   12160 Jun 14  2016 xmlpatternsvalidator-qt4
-rwxrwxr-x.  1 jaremko joshuatonew   99208 Dec 11 13:52 xz
lrwxrwxrwx.  1 jaremko joshuatonew       2 Dec 11 13:54 xzcat -> xz
lrwxrwxrwx.  1 jaremko joshuatonew       6 Dec 11 13:54 xzcmp -> xzdiff
-rwxrwxr-x.  2 jaremko joshuatonew   20848 May 16  2018 xzdec
-rwxrwxr-x.  2 jaremko joshuatonew    6632 May 16  2018 xzdiff
lrwxrwxrwx.  1 jaremko joshuatonew       6 Dec 11 13:54 xzegrep -> xzgrep
lrwxrwxrwx.  1 jaremko joshuatonew       6 Dec 11 13:54 xzfgrep -> xzgrep
-rwxrwxr-x.  2 jaremko joshuatonew    5628 May 16  2018 xzgrep
-rwxrwxr-x.  2 jaremko joshuatonew    1802 May 16  2018 xzless
-rwxrwxr-x.  2 jaremko joshuatonew    2161 May 16  2018 xzmore

I get the following error when trying to update worker node
(Which I have to rename as cryosparc_worker.tar.gz manually)

ERROR: Command errored out with exit status 1:
command: /home/stavros/apps/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin/python3.7 -u -c ‘import sys, setuptools, tokenize; sys.argv[0] = ‘"’"’/tmp/pip-req-build-0bu3ybi3/setup.py’"’"’; file=’"’"’/tmp/pip-req-build-0bu3ybi3/setup.py’"’"’;f=getattr(tokenize, ‘"’"‘open’"’"’, open)(file);code=f.read().replace(’"’"’\r\n’"’"’, ‘"’"’\n’"’"’);f.close();exec(compile(code, file, ‘"’"‘exec’"’"’))’ install --record /tmp/pip-record-7jmp7_9q/install-record.txt --single-version-externally-managed --compile --install-headers /home/stavros/apps/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m/pycuda
cwd: /tmp/pip-req-build-0bu3ybi3/
Complete output (571 lines):
*************************************************************
*** I have detected that you have not run configure.py.
*************************************************************
*** Additionally, no global config files were found.
*** I will go ahead with the default configuration.
*** In all likelihood, this will not work out.
***
*** See README_SETUP.txt for more information.
***
*** If the build does fail, just re-run configure.py with the
*** correct arguments, and then retry. Good luck!
*************************************************************
*** HIT Ctrl-C NOW IF THIS IS NOT WHAT YOU WANT
*************************************************************
Continuing in 1 seconds…
/home/stavros/apps/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: ‘test_requires’
warnings.warn(msg)
ERROR: Command errored out with exit status 1: /home/stavros/apps/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin/python3.7 -u -c ‘import sys, setuptools, tokenize; sys.argv[0] = ‘"’"’/tmp/pip-req-build-0bu3ybi3/setup.py’"’"’; file=’"’"’/tmp/pip-req-build-0bu3ybi3/setup.py’"’"’;f=getattr(tokenize, ‘"’"‘open’"’"’, open)(file);code=f.read().replace(’"’"’\r\n’"’"’, ‘"’"’\n’"’"’);f.close();exec(compile(code, file, ‘"’"‘exec’"’"’))’ install --record /tmp/pip-record-7jmp7_9q/install-record.txt --single-version-externally-managed --compile --install-headers /home/stavros/apps/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m/pycuda Check the logs for full command output.

@stravos, can you post the output of nvidia-smi?

Then try the worker update steps I noted above, starting with step 2:

Do you still get the same error?

1 Like
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.40.04    Driver Version: 418.40.04    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 680     On   | 00000000:02:00.0 N/A |                  N/A |
| 30%   31C    P8    N/A /  N/A |     36MiB /  4037MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 680     On   | 00000000:03:00.0 N/A |                  N/A |
| 30%   32C    P8    N/A /  N/A |     41MiB /  4037MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 208...  On   | 00000000:83:00.0 Off |                  N/A |
| 31%   31C    P8     6W / 250W |     41MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 208...  On   | 00000000:84:00.0 Off |                  N/A |
| 33%   32C    P8    17W / 250W |     41MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0                    Not Supported                                       |
|    1                    Not Supported                                       |
|    2      5477      G   /usr/bin/Xorg                                 40MiB |
|    3      5477      G   /usr/bin/Xorg                                 40MiB |
+-----------------------------------------------------------------------------+

@nfrasser When I performed step 4 as above, I get the following:

pycuda requires Python ‘~=3.6’ but the running Python is 2.7.6

Blockquote

Performed an upgrade from 2.15 to 3. For good measure, restarted cryosparc to make sure no errors show up. I get this
/mnt/HDD1/Software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libtinfo.so.6: no version information available (required by bash)

Webapp loads alright but when I try to import a project, I get
Unable to import project from /mnt/12T_HDD2/P8: 'ascii' codec can't decode byte 0x89 in position 0: ordinal not in range(128)
This happens with any project I want to import. I’m not sure what is going on anymore.

I am able to create projects and run jobs (to an extent). In the Patch Motion, I am not able to pick the GPU I can run my jobs on (see attached). The job runs on the default GPU but the selection is definitely necessary.
image

I’m also wondering if this is due to the mix and match of GPUs I have in the system (2080Ti and 780Ti). I use the 780Ti as the display and 2080Ti for processing.

Hi @nfrasser, we still receive an error after going through these steps (see below, couldn’t paste all of the errors). In addition, we are installing on a GPFS, which is showing permission issues with v3.0, but not v2.15.0. We installed v3.0 on a local node with no problem, but we would like to continue to have the installation on the GPFS for our nodes to access. So it seems something in the v3.0 installation doesn’t like the GPFS.


bpl-subset/bpl_subset/boost/lexical_cast.hpp:1737:17: note: in expansion of macro ‘BOOST_STATIC_ASSERT_MSG’
                     BOOST_STATIC_ASSERT_MSG(( sizeof(CharT) == sizeof(T) ),
                     ^
    bpl-subset/bpl_subset/boost/python/detail/defaults_def.hpp: In function ‘void pycudaboost::python::detail::define_with_defaults(const char*, const OverloadsT&, NameSpaceT&, const SigT&)’:
    bpl-subset/bpl_subset/boost/static_assert.hpp:125:21: warning: typedef ‘boost_static_assert_typedef_247’ locally defined but not used [-Wunused-local-typedefs]
              BOOST_JOIN(boost_static_assert_typedef_, __LINE__)
                         ^
    bpl-subset/bpl_subset/boost/python/detail/defaults_def.hpp:246:7: note: in expansion of macro ‘BOOST_STATIC_ASSERT’
           BOOST_STATIC_ASSERT(
           ^
    bpl-subset/bpl_subset/boost/python/pure_virtual.hpp: In member function ‘void pycudaboost::python::detail::pure_virtual_visitor<PointerToMemberFunction>::visit(C_&, const char*, Options&) const’:
    bpl-subset/bpl_subset/boost/static_assert.hpp:125:21: warning: typedef ‘boost_static_assert_typedef_80’ locally defined but not used [-Wunused-local-typedefs]
              BOOST_JOIN(boost_static_assert_typedef_, __LINE__)
                         ^
    bpl-subset/bpl_subset/boost/python/pure_virtual.hpp:80:11: note: in expansion of macro ‘BOOST_STATIC_ASSERT’
               BOOST_STATIC_ASSERT(!Options::has_default_implementation);
               ^
    bpl-subset/bpl_subset/boost/python/return_arg.hpp: In static member function ‘static PyObject* pycudaboost::python::return_arg<arg_pos, Base>::postcall(const ArgumentPackage&, PyObject*)’:
    bpl-subset/bpl_subset/boost/static_assert.hpp:125:21: warning: typedef ‘boost_static_assert_typedef_85’ locally defined but not used [-Wunused-local-typedefs]
              BOOST_JOIN(boost_static_assert_typedef_, __LINE__)
                         ^
    bpl-subset/bpl_subset/boost/python/return_arg.hpp:85:9: note: in expansion of macro ‘BOOST_STATIC_ASSERT’
             BOOST_STATIC_ASSERT(arg_pos > 0);
             ^
    bpl-subset/bpl_subset/boost/optional/optional.hpp: In member function ‘void pycudaboost::optional_detail::optional_base<T>::construct(const Expr&, const pycudaboost::in_place_factory_base*)’:
    bpl-subset/bpl_subset/boost/static_assert.hpp:125:21: warning: typedef ‘boost_static_assert_typedef_355’ locally defined but not used [-Wunused-local-typedefs]
              BOOST_JOIN(boost_static_assert_typedef_, __LINE__)
                         ^
    bpl-subset/bpl_subset/boost/optional/optional.hpp:355:8: note: in expansion of macro ‘BOOST_STATIC_ASSERT’
            BOOST_STATIC_ASSERT ( ::boost::mpl::not_<is_reference_predicate>::value ) ;
            ^
    bpl-subset/bpl_subset/boost/optional/optional.hpp: In member function ‘void pycudaboost::optional_detail::optional_base<T>::construct(const Expr&, const pycudaboost::typed_in_place_factory_base*)’:
    bpl-subset/bpl_subset/boost/static_assert.hpp:125:21: warning: typedef ‘boost_static_assert_typedef_364’ locally defined but not used [-Wunused-local-typedefs]
              BOOST_JOIN(boost_static_assert_typedef_, __LINE__)
                         ^
    bpl-subset/bpl_subset/boost/optional/optional.hpp:364:8: note: in expansion of macro ‘BOOST_STATIC_ASSERT’
            BOOST_STATIC_ASSERT ( ::boost::mpl::not_<is_reference_predicate>::value ) ;
            ^
    In file included from src/wrapper/wrap_curand.cpp:1:0:
    src/cpp/cuda.hpp: In member function ‘pycudaboost::python::tuple pycuda::device::compute_capability()’:
    src/cpp/cuda.hpp:448:29: warning: ‘CUresult cuDeviceComputeCapability(int*, int*, CUdevice)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:2984) [-Wdeprecated-declarations]
             CUDAPP_CALL_GUARDED(cuDeviceComputeCapability, (&major, &minor, m_device));
                                 ^
    src/cpp/cuda.hpp:143:22: note: in definition of macro ‘CUDAPP_CALL_GUARDED’
         cu_status_code = NAME ARGLIST; \
                          ^
    src/cpp/cuda.hpp:448:81: warning: ‘CUresult cuDeviceComputeCapability(int*, int*, CUdevice)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:2984) [-Wdeprecated-declarations]
             CUDAPP_CALL_GUARDED(cuDeviceComputeCapability, (&major, &minor, m_device));
                                                                                     ^
    src/cpp/cuda.hpp:143:27: note: in definition of macro ‘CUDAPP_CALL_GUARDED’
         cu_status_code = NAME ARGLIST; \
                               ^
    src/cpp/cuda.hpp: In static member function ‘static pycudaboost::shared_ptr<pycuda::context> pycuda::context::attach(unsigned int)’:
    src/cpp/cuda.hpp:645:29: warning: ‘CUresult cuCtxAttach(CUctx_st**, unsigned int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:4026) [-Wdeprecated-declarations]
             CUDAPP_CALL_GUARDED(cuCtxAttach, (&current, flags));
                                 ^
    src/cpp/cuda.hpp:143:22: note: in definition of macro ‘CUDAPP_CALL_GUARDED’
         cu_status_code = NAME ARGLIST; \
                          ^
    src/cpp/cuda.hpp:645:58: warning: ‘CUresult cuCtxAttach(CUctx_st**, unsigned int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:4026) [-Wdeprecated-declarations]
             CUDAPP_CALL_GUARDED(cuCtxAttach, (&current, flags));
                                                              ^
    src/cpp/cuda.hpp:143:27: note: in definition of macro ‘CUDAPP_CALL_GUARDED’
         cu_status_code = NAME ARGLIST; \
                               ^
    src/cpp/cuda.hpp: In member function ‘virtual void pycuda::context::detach_internal()’:
    src/cpp/cuda.hpp:654:37: warning: ‘CUresult cuCtxDetach(CUcontext)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:4062) [-Wdeprecated-declarations]
             CUDAPP_CALL_GUARDED_CLEANUP(cuCtxDetach, (m_context));
                                         ^
    src/cpp/cuda.hpp:152:22: note: in definition of macro ‘CUDAPP_CALL_GUARDED_CLEANUP’
         cu_status_code = NAME ARGLIST; \
                          ^
    src/cpp/cuda.hpp:654:60: warning: ‘CUresult cuCtxDetach(CUcontext)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:4062) [-Wdeprecated-declarations]
             CUDAPP_CALL_GUARDED_CLEANUP(cuCtxDetach, (m_context));
                                                                ^
    src/cpp/cuda.hpp:152:27: note: in definition of macro ‘CUDAPP_CALL_GUARDED_CLEANUP’
         cu_status_code = NAME ARGLIST; \
                               ^
    src/cpp/cuda.hpp: In constructor ‘pycuda::texture_reference::texture_reference()’:
    src/cpp/cuda.hpp:1127:29: warning: ‘CUresult cuTexRefCreate(CUtexref_st**)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:13235) [-Wdeprecated-declarations]
           { CUDAPP_CALL_GUARDED(cuTexRefCreate, (&m_texref)); }
                                 ^
    src/cpp/cuda.hpp:143:22: note: in definition of macro ‘CUDAPP_CALL_GUARDED’
         cu_status_code = NAME ARGLIST; \
                          ^
    src/cpp/cuda.hpp:1127:55: warning: ‘CUresult cuTexRefCreate(CUtexref_st**)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:13235) [-Wdeprecated-declarations]
           { CUDAPP_CALL_GUARDED(cuTexRefCreate, (&m_texref)); }
                                                           ^
    src/cpp/cuda.hpp:143:27: note: in definition of macro ‘CUDAPP_CALL_GUARDED’
         cu_status_code = NAME ARGLIST; \
                               ^
    src/cpp/cuda.hpp: In destructor ‘pycuda::texture_reference::~texture_reference()’:
    src/cpp/cuda.hpp:1137:39: warning: ‘CUresult cuTexRefDestroy(CUtexref)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:13255) [-Wdeprecated-declarations]
               CUDAPP_CALL_GUARDED_CLEANUP(cuTexRefDestroy, (m_texref));
                                           ^
    src/cpp/cuda.hpp:152:22: note: in definition of macro ‘CUDAPP_CALL_GUARDED_CLEANUP’
         cu_status_code = NAME ARGLIST; \
                          ^
    src/cpp/cuda.hpp:1137:65: warning: ‘CUresult cuTexRefDestroy(CUtexref)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:13255) [-Wdeprecated-declarations]
               CUDAPP_CALL_GUARDED_CLEANUP(cuTexRefDestroy, (m_texref));
                                                                     ^
    src/cpp/cuda.hpp:152:27: note: in definition of macro ‘CUDAPP_CALL_GUARDED_CLEANUP’
         cu_status_code = NAME ARGLIST; \
                               ^
    src/cpp/cuda.hpp: In member function ‘void pycuda::function::set_block_shape(int, int, int)’:
    src/cpp/cuda.hpp:1360:13: warning: ‘CUresult cuFuncSetBlockShape(CUfunction, int, int, int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:10932) [-Wdeprecated-declarations]
                 cuFuncSetBlockShape, (m_function, x, y, z), m_symbol);
                 ^
    src/cpp/cuda.hpp:121:22: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                          ^
    src/cpp/cuda.hpp:1360:54: warning: ‘CUresult cuFuncSetBlockShape(CUfunction, int, int, int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:10932) [-Wdeprecated-declarations]
                 cuFuncSetBlockShape, (m_function, x, y, z), m_symbol);
                                                          ^
    src/cpp/cuda.hpp:121:27: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                               ^
    src/cpp/cuda.hpp: In member function ‘void pycuda::function::set_shared_size(unsigned int)’:
    src/cpp/cuda.hpp:1365:13: warning: ‘CUresult cuFuncSetSharedSize(CUfunction, unsigned int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:10966) [-Wdeprecated-declarations]
                 cuFuncSetSharedSize, (m_function, bytes), m_symbol);
                 ^
    src/cpp/cuda.hpp:121:22: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                          ^
    src/cpp/cuda.hpp:1365:52: warning: ‘CUresult cuFuncSetSharedSize(CUfunction, unsigned int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:10966) [-Wdeprecated-declarations]
                 cuFuncSetSharedSize, (m_function, bytes), m_symbol);
                                                        ^
    src/cpp/cuda.hpp:121:27: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                               ^
    src/cpp/cuda.hpp: In member function ‘void pycuda::function::param_set_size(unsigned int)’:
    src/cpp/cuda.hpp:1371:13: warning: ‘CUresult cuParamSetSize(CUfunction, unsigned int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:10998) [-Wdeprecated-declarations]
                 cuParamSetSize, (m_function, bytes), m_symbol);
                 ^
    src/cpp/cuda.hpp:121:22: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                          ^
    src/cpp/cuda.hpp:1371:47: warning: ‘CUresult cuParamSetSize(CUfunction, unsigned int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:10998) [-Wdeprecated-declarations]
                 cuParamSetSize, (m_function, bytes), m_symbol);
                                                   ^
    src/cpp/cuda.hpp:121:27: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                               ^
    src/cpp/cuda.hpp: In member function ‘void pycuda::function::param_set(int, unsigned int)’:
    src/cpp/cuda.hpp:1376:13: warning: ‘CUresult cuParamSeti(CUfunction, int, unsigned int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11031) [-Wdeprecated-declarations]
                 cuParamSeti, (m_function, offset, value), m_symbol);
                 ^
    src/cpp/cuda.hpp:121:22: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                          ^
    src/cpp/cuda.hpp:1376:52: warning: ‘CUresult cuParamSeti(CUfunction, int, unsigned int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11031) [-Wdeprecated-declarations]
                 cuParamSeti, (m_function, offset, value), m_symbol);
                                                        ^
    src/cpp/cuda.hpp:121:27: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                               ^
    src/cpp/cuda.hpp: In member function ‘void pycuda::function::param_set(int, float)’:
    src/cpp/cuda.hpp:1381:11: warning: ‘CUresult cuParamSetf(CUfunction, int, float)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11064) [-Wdeprecated-declarations]
               cuParamSetf, (m_function, offset, value), m_symbol);
               ^
    src/cpp/cuda.hpp:121:22: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                          ^
    src/cpp/cuda.hpp:1381:50: warning: ‘CUresult cuParamSetf(CUfunction, int, float)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11064) [-Wdeprecated-declarations]
               cuParamSetf, (m_function, offset, value), m_symbol);
                                                      ^
    src/cpp/cuda.hpp:121:27: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                               ^
    src/cpp/cuda.hpp: In member function ‘void pycuda::function::param_setv(int, void*, size_t)’:
    src/cpp/cuda.hpp:1387:11: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11099) [-Wdeprecated-declarations]
               cuParamSetv, (m_function, offset, buf, (unsigned int) len), m_symbol);
               ^
    src/cpp/cuda.hpp:121:22: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                          ^
    src/cpp/cuda.hpp:1387:68: warning: ‘CUresult cuParamSetv(CUfunction, int, void*, unsigned int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11099) [-Wdeprecated-declarations]
               cuParamSetv, (m_function, offset, buf, (unsigned int) len), m_symbol);
                                                                        ^
    src/cpp/cuda.hpp:121:27: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                               ^
    src/cpp/cuda.hpp: In member function ‘void pycuda::function::param_set_texref(const pycuda::texture_reference&)’:
    src/cpp/cuda.hpp:1391:45: warning: ‘CUresult cuParamSetTexRef(CUfunction, int, CUtexref)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11247) [-Wdeprecated-declarations]
             CUDAPP_CALL_GUARDED_WITH_TRACE_INFO(cuParamSetTexRef, (m_function,
                                                 ^
    src/cpp/cuda.hpp:121:22: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                          ^
    src/cpp/cuda.hpp:1392:45: warning: ‘CUresult cuParamSetTexRef(CUfunction, int, CUtexref)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11247) [-Wdeprecated-declarations]
                 CU_PARAM_TR_DEFAULT, tr.handle()), m_symbol);
                                                 ^
    src/cpp/cuda.hpp:121:27: note: in definition of macro ‘CUDAPP_CALL_GUARDED_WITH_TRACE_INFO’
         cu_status_code = NAME ARGLIST; \
                               ^
    src/cpp/cuda.hpp: In member function ‘void pycuda::function::launch()’:
    src/cpp/cuda.hpp:1398:13: warning: ‘CUresult cuLaunch(CUfunction)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11136) [-Wdeprecated-declarations]
                 cuLaunch, (m_function), m_symbol);
                 ^
    src/cpp/cuda.hpp:111:24: note: in definition of macro ‘CUDAPP_CALL_GUARDED_THREADED_WITH_TRACE_INFO’
           cu_status_code = NAME ARGLIST; \
                            ^
    src/cpp/cuda.hpp:1398:34: warning: ‘CUresult cuLaunch(CUfunction)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11136) [-Wdeprecated-declarations]
                 cuLaunch, (m_function), m_symbol);
                                      ^
    src/cpp/cuda.hpp:111:29: note: in definition of macro ‘CUDAPP_CALL_GUARDED_THREADED_WITH_TRACE_INFO’
           cu_status_code = NAME ARGLIST; \
                                 ^
    src/cpp/cuda.hpp: In member function ‘void pycuda::function::launch_grid(int, int)’:
    src/cpp/cuda.hpp:1403:11: warning: ‘CUresult cuLaunchGrid(CUfunction, int, int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11175) [-Wdeprecated-declarations]
               cuLaunchGrid, (m_function, grid_width, grid_height), m_symbol);
               ^
    src/cpp/cuda.hpp:111:24: note: in definition of macro ‘CUDAPP_CALL_GUARDED_THREADED_WITH_TRACE_INFO’
           cu_status_code = NAME ARGLIST; \
                            ^
    src/cpp/cuda.hpp:1403:61: warning: ‘CUresult cuLaunchGrid(CUfunction, int, int)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11175) [-Wdeprecated-declarations]
               cuLaunchGrid, (m_function, grid_width, grid_height), m_symbol);
                                                                 ^
    src/cpp/cuda.hpp:111:29: note: in definition of macro ‘CUDAPP_CALL_GUARDED_THREADED_WITH_TRACE_INFO’
           cu_status_code = NAME ARGLIST; \
                                 ^
    src/cpp/cuda.hpp: In member function ‘void pycuda::function::launch_grid_async(int, int, const pycuda::stream&)’:
    src/cpp/cuda.hpp:1408:13: warning: ‘CUresult cuLaunchGridAsync(CUfunction, int, int, CUstream)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11222) [-Wdeprecated-declarations]
                 cuLaunchGridAsync, (m_function, grid_width, grid_height, s.handle()),
                 ^
    src/cpp/cuda.hpp:111:24: note: in definition of macro ‘CUDAPP_CALL_GUARDED_THREADED_WITH_TRACE_INFO’
           cu_status_code = NAME ARGLIST; \
                            ^
    src/cpp/cuda.hpp:1408:80: warning: ‘CUresult cuLaunchGridAsync(CUfunction, int, int, CUstream)’ is deprecated (declared at /usr/local/cuda/include/cuda.h:11222) [-Wdeprecated-declarations]
                 cuLaunchGridAsync, (m_function, grid_width, grid_height, s.handle()),
                                                                                    ^
    src/cpp/cuda.hpp:111:29: note: in definition of macro ‘CUDAPP_CALL_GUARDED_THREADED_WITH_TRACE_INFO’
           cu_status_code = NAME ARGLIST; \
                                 ^
    In file included from src/wrapper/wrap_curand.cpp:2:0:
    src/cpp/curand.hpp: In function ‘void pycuda::curandom::py_curand_get_direction_vectors(curandDirectionVectorSet_t, pycudaboost::python::api::object, int)’:
    src/cpp/curand.hpp:55:26: warning: unused variable ‘len’ [-Wunused-variable]
         PYCUDA_BUFFER_SIZE_T len = buf_wrapper.m_buf.len;
                              ^
    In file included from src/wrapper/wrap_curand.cpp:2:0:
    src/cpp/curand.hpp: In function ‘void pycuda::curandom::py_curand_get_scramble_constants32(pycudaboost::python::api::object, int)’:
    src/cpp/curand.hpp:96:26: warning: unused variable ‘len’ [-Wunused-variable]
         PYCUDA_BUFFER_SIZE_T len = buf_wrapper.m_buf.len;
                              ^
    src/cpp/curand.hpp: In function ‘void pycuda::curandom::py_curand_get_scramble_constants64(pycudaboost::python::api::object, int)’:
    src/cpp/curand.hpp:118:26: warning: unused variable ‘len’ [-Wunused-variable]
         PYCUDA_BUFFER_SIZE_T len = buf_wrapper.m_buf.len;
                              ^
    In file included from /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:21:0,
                     from /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                     from src/wrapper/tools.hpp:10,
                     from src/wrapper/wrap_curand.cpp:4:
    /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/numpy/core/include/numpy/__multiarray_api.h: At global scope:
    /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1463:1: warning: ‘int _import_array()’ defined but not used [-Wunused-function]
     _import_array(void)
     ^
    g++ -pthread -shared -B /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/compiler_compat -L/grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib -Wl,-rpath=/grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/src/cpp/cuda.o build/temp.linux-x86_64-3.7/src/cpp/bitlog.o build/temp.linux-x86_64-3.7/src/wrapper/wrap_cudadrv.o build/temp.linux-x86_64-3.7/src/wrapper/mempool.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/converter/arg_to_python_base.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/converter/builtin_converters.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/converter/from_python.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/converter/registry.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/converter/type_id.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/object/class.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/object/enum.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/object/function.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/object/function_doc_signature.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/object/inheritance.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/object/iterator.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/object/life_support.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/object/pickle_support.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/object/stl_iterator.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/dict.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/errors.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/exec.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/import.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/list.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/long.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/module.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/numeric.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/object_operators.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/object_protocol.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/slice.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/str.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/tuple.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/wrapper.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/smart_ptr/src/sp_collector.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/smart_ptr/src/sp_debug_hooks.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/system/src/error_code.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/thread/src/pthread/once.o build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/thread/src/pthread/thread.o build/temp.linux-x86_64-3.7/src/wrapper/wrap_curand.o -L/usr/local/cuda/lib -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib/stubs -L/usr/local/cuda/lib64/stubs -L/usr/local/cuda/lib -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib/stubs -L/usr/local/cuda/lib64/stubs -lcuda -lcurand -o build/lib.linux-x86_64-3.7/pycuda/_driver.cpython-37m-x86_64-linux-gnu.so
    building '_pvt_struct' extension
    gcc -pthread -B /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -fwrapv -Wall -O3 -DNDEBUG -fPIC -I/grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/numpy/core/include -I/grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m -c src/wrapper/_pvt_struct_v3.cpp -o build/temp.linux-x86_64-3.7/src/wrapper/_pvt_struct_v3.o
    In file included from /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822:0,
                     from /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                     from /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                     from src/wrapper/_pvt_struct_v3.cpp:11:
    /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
     #warning "Using deprecated NumPy API, disable it with " \
      ^
    src/wrapper/_pvt_struct_v3.cpp: In function ‘int prepare_s(PyStructObject*)’:
    src/wrapper/_pvt_struct_v3.cpp:959:57: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
         if ((len + 1) > (PY_SSIZE_T_MAX / sizeof(formatcode))) {
                                                             ^
    src/wrapper/_pvt_struct_v3.cpp: In function ‘int s_init(PyObject*, PyObject*, PyObject*)’:
    src/wrapper/_pvt_struct_v3.cpp:1049:41: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
         static char *kwlist[] = {"format", 0};
                                             ^
    src/wrapper/_pvt_struct_v3.cpp: In function ‘PyObject* s_unpack_from(PyObject*, PyObject*, PyObject*)’:
    src/wrapper/_pvt_struct_v3.cpp:1168:51: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
         static char *kwlist[] = {"buffer", "offset", 0};
                                                       ^
    src/wrapper/_pvt_struct_v3.cpp:1168:51: warning: deprecated conversion from string constant to ‘char*’ [-Wwrite-strings]
    src/wrapper/_pvt_struct_v3.cpp: In function ‘int s_pack_internal(PyStructObject*, PyObject*, int, char*)’:
    src/wrapper/_pvt_struct_v3.cpp:1226:17: warning: unused variable ‘isstring’ [-Wunused-variable]
                 int isstring;
                     ^
    src/wrapper/_pvt_struct_v3.cpp: In function ‘PyObject* s_pack_into(PyObject*, PyObject*)’:
    src/wrapper/_pvt_struct_v3.cpp:1368:10: warning: ‘int PyObject_AsWriteBuffer(PyObject*, void**, Py_ssize_t*)’ is deprecated (declared at /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m/abstract.h:500) [-Wdeprecated-declarations]
         if ( PyObject_AsWriteBuffer(PyTuple_GET_ITEM(args, 0),
              ^
    src/wrapper/_pvt_struct_v3.cpp:1369:89: warning: ‘int PyObject_AsWriteBuffer(PyObject*, void**, Py_ssize_t*)’ is deprecated (declared at /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m/abstract.h:500) [-Wdeprecated-declarations]
                                                                 (void**)&buffer, &buffer_len) == -1 ) {
                                                                                             ^
    g++ -pthread -shared -B /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/compiler_compat -L/grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib -Wl,-rpath=/grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/src/wrapper/_pvt_struct_v3.o -o build/lib.linux-x86_64-3.7/pycuda/_pvt_struct.cpython-37m-x86_64-linux-gnu.so
    running install_lib
    copying build/lib.linux-x86_64-3.7/pycuda/_driver.cpython-37m-x86_64-linux-gnu.so -> /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda
    copying build/lib.linux-x86_64-3.7/pycuda/_pvt_struct.cpython-37m-x86_64-linux-gnu.so -> /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda
    running install_egg_info
    Copying pycuda.egg-info to /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda-2020.1-py3.7.egg-info
    error: [Errno 13] Permission denied: '/grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda-2020.1-py3.7.egg-info/PKG-INFO'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-fp9_fqws/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-fp9_fqws/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-4zyc9pgg/install-record.txt --single-version-externally-managed --compile --install-headers /grid/joshua-tor/home/wangm/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m/pycuda Check the logs for full command output.
1 Like

Hi @jaremko,

Is it possible if you can run the command: umask 002 in your shell, and run the installation process again? Doing so sets the base permissions of the files created by the installer. With this mask, the default directory permissions are 775 and default file permissions are 664.

I’m also having permissions issues when upgrading from v2.15 to v3.0, causing the update to fail. When I try to repeat the update, I’m told that I’m already up to date with v3.0.0. However, cryoSPARC will not start. Should I try to reinstall? Is there a way to reinstall without losing the data from previous jobs?

I’ve copied the top and the bottom of what is a very long error list:

Update will now be applied to the master installation,
followed by worker installations on other nodes.

Deleting old files…
Extracting…
tar: cryosparc_app/bundle/programs/server/app/app.js: Cannot open: Permission denied
tar: cryosparc_app/bundle/programs/server/app/app.js.map: Cannot open: Permission denied
.
.
.
tar: cryosparc_webapp/bundle/server/.bundle_version.txt: Cannot open: Permission denied
tar: Exiting with failure status due to previous errors

@stravos I suggest you update your Nvidia driver to a newer version such as 455, you can get it here. Then try the dependency installation again from the cryosparc2_worker directory with command bin/cryosparcw forcedeps

If that still doesn’t work, can you send me just the output this env command? (Important: please remove the line that starts with CRYOSPARC_LICENSE_ID= before you post)

bin/cryosparcw env

@vamsee if cryosparc status reports that everything is running correctly, then you can safely ignore the no version information available message. We’ll address this in a future release.

We’ll be addressing the Unable to import project error in a patch coming tomorrow.

1 Like

One thing I noticed when doing some deep vs topaz tests in new CS3 (also love how NU-refine NEW is ~2x faster! :smiley: ).

Topaz no longer worked.

I reinstalled topaz using python 3.8.5 (previous was 2.7.18) and cuda tool kit 11. This seemed to resolve the <topaz_path>/topaz -v error in topaz_utils.py. Maybe helpful for others with similar issue.

cheers- J

3 Likes

Hi @stephan and @nfrasser,

I have an update.

During are troubleshooting we had two issues:

  1. Permission issues in /anaconda/bin during installation
  2. Older drivers of nvidia and cuda

Both these issues caused a problem with v3.0.0 and v3.0.1, but not v2.15.0. Updating the nvidia and cuda drivers fixed gpu errors. To get around permissions, we updated/installed from v2.15.0 to v3.0.0 like this:

cryoparcm update #leads to /anaconda/bin permission issues, which fails worker installation

cp /cryosparc2_master/cryosparc2_worker.tar.gz /cryosparc2_worker/

./cryosparc2_worker/bin/cryosparcw update –override #will lead to /anaconda/bin permission errors

chmod -R 755 /cryosparc2_worker

./cryosparc2_worker/install.sh --license $LICENSE_ID
–cudapath /usr/local/cuda-10.0/
–yes

Not a very intuitive way to update, but it works. Have you found any reason why these permissions arise in v3 but not v2?

Best,
Matt

1 Like

Hi,
There is a similar kind of error i am facing when i tried reinstallation of Cryosparc 3.1 . Any suggestions?

Also shows " tar: Exiting with failure status due to previous errors" error message.

With regards