No module named 'torch' after 4.1 upgrade

nfrasser · June 27, 2023, 8:31pm

@andreym you should able to do this by adding pypi.org as a trusted host in pip and disabling SSL verification in conda. Should be something like this:

cryosparcw call pip config set global.trusted-host pypi.org
SSL_NO_VERIFY=1 cryosparcw install-3dflex

Let me know how that goes

andreym · June 27, 2023, 11:05pm

@nfrasser Thank you for the suggestion. I have tried it, unfortunately I am still running into the error:
" Installing 3D Flex Refine dependencies…

Collecting package metadata (current_repodata.json): failed

CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://conda.anaconda.org/nvidia/label/cuda-11.8.0/linux-64/current_repodata.json

Elapsed: -

An HTTP error occurred when trying to retrieve this URL.

HTTP errors are often intermittent, and a simple retry will get you on your way.

‘Linux 64 :: Anaconda.org’"

Tru · June 28, 2023, 7:11am

It is not your proxy, as https://conda.anaconda.org/nvidia/label/cuda-11.8.0/linux-64/current_repodata.json lands at " The page you are looking for does not exist."

nfrasser · June 28, 2023, 3:47pm

@andreym could you post the full output of the following command?

SSL_NO_VERIFY=1 cryosparcw call conda install -y cuda-nvcc=11.8 cuda-toolkit=11.8 -c nvidia/label/cuda-11.8.0 -v

andreym · June 28, 2023, 4:33pm

Hello. The error output is a bit extensive, please see below:

~]$ SSL_NO_VERIFY=1 /ssd/CryoSparc/cryosparc_worker/bin/cryosparcw call conda install -y cuda-nvcc=11.8 cuda-toolkit=11.8 -c nvidia/label/cuda-11.8.0 -v
Collecting package metadata (current_repodata.json): ...working... Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))': /nvidia/label/cuda-11.8.0/linux-64/current_repodata.json

Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))': /nvidia/label/cuda-11.8.0/noarch/current_repodata.json

Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))': /nvidia/label/cuda-11.8.0/linux-64/current_repodata.json

Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))': /nvidia/label/cuda-11.8.0/noarch/current_repodata.json

Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))': /nvidia/label/cuda-11.8.0/noarch/current_repodata.json

Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))': /nvidia/label/cuda-11.8.0/linux-64/current_repodata.json

failed
Traceback (most recent call last):
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connection.py", line 416, in connect
    self.sock = ssl_wrap_socket(
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
    return self.urlopen(
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
    return self.urlopen(
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
    return self.urlopen(
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='conda.anaconda.org', port=443): Max retries exceeded with url: /nvidia/label/cuda-11.8.0/linux-64/current_repodata.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 701, in fetch_repodata_remote_request
    resp = session.get(join_url(url, filename), headers=headers, proxies=session.proxies,
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/requests/sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='conda.anaconda.org', port=443): Max retries exceeded with url: /nvidia/label/cuda-11.8.0/linux-64/current_repodata.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/exceptions.py", line 1082, in __call__
    return func(*args, **kwargs)
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/cli/main.py", line 87, in _main
    exit_code = do_call(args, p)
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/cli/conda_argparse.py", line 84, in do_call
    return getattr(module, func_name)(args, parser)
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/cli/main_install.py", line 20, in execute
    install(args, parser, 'install')
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/cli/install.py", line 260, in install
    unlink_link_transaction = solver.solve_for_transaction(
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/solve.py", line 152, in solve_for_transaction
    unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/solve.py", line 195, in solve_for_diff
    final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/solve.py", line 300, in solve_final_state
    ssc = self._collect_all_metadata(ssc)
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
    return f(*args, **kwds)
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/solve.py", line 463, in _collect_all_metadata
    index, r = self._prepare(prepared_specs)
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/solve.py", line 1058, in _prepare
    reduced_index = get_reduced_index(self.prefix, self.channels,
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/index.py", line 288, in get_reduced_index
    new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs,
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 140, in query_all
    result = tuple(concat(executor.map(subdir_query, channel_urls)))
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 132, in <lambda>
    subdir_query = lambda url: tuple(SubdirData(Channel(url), repodata_fn=repodata_fn).query(
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 145, in query
    self.load()
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 210, in load
    _internal_state = self._load()
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 375, in _load
    raw_repodata_str = fetch_repodata_remote_request(
  File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 806, in fetch_repodata_remote_request
    raise CondaHTTPError(help_message,
conda.exceptions.CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/nvidia/label/cuda-11.8.0/linux-64/current_repodata.json>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
'https://conda.anaconda.org/nvidia/label/cuda-11.8.0/linux-64'

nfrasser · June 29, 2023, 5:11pm

@andreym it looks like SSL_NO_VERIFY=1 is not having the intended effect. I’d like to see what else you have in your environment: Could you you please post the output of the following commands? Make sure to obfuscate any sensitive information you see.

cryosparcw call /usr/bin/env | grep -v LICENSE
cryosparcw call conda config -v --show

andreym · June 29, 2023, 7:49pm

Hi @nfrasser. Thanks for your help. Please see below:

]$ ./cryosparcw call conda config -v --show
add_anaconda_token: True
add_pip_as_python_dependency: True
aggressive_update_packages:
  - ca-certificates
  - certifi
  - openssl
allow_conda_downgrades: False
allow_cycles: True
allow_non_channel_urls: False
allow_softlinks: False
always_copy: False
always_softlink: False
always_yes: None
anaconda_upload: None
auto_activate_base: True
auto_stack: 0
auto_update_conda: False
bld_path: 
changeps1: True
channel_alias: https://conda.anaconda.org
channel_priority: flexible
channels:
  - defaults
client_ssl_cert: None
client_ssl_cert_key: None
clobber: False
conda_build: {}
create_default_packages: []
croot: /ssd/CryoSparc/cryosparc_worker/deps/anaconda/conda-bld
custom_channels:
  pkgs/main: https://repo.anaconda.com
  pkgs/r: https://repo.anaconda.com
  pkgs/pro: https://repo.anaconda.com
custom_multichannels:
  defaults: 
    - https://repo.anaconda.com/pkgs/main
    - https://repo.anaconda.com/pkgs/r
  local: 
debug: False
default_channels:
  - https://repo.anaconda.com/pkgs/main
  - https://repo.anaconda.com/pkgs/r
default_python: 3.8
default_threads: None
deps_modifier: not_set
dev: False
disallowed_packages: []
download_only: False
dry_run: False
enable_private_envs: False
env_prompt: ({default_env}) 
envs_dirs:
  - /ssd/CryoSparc/cryosparc_worker/deps/anaconda/envs
  - /home/cryosparc/.conda/envs
error_upload_url: https://conda.io/conda-post/unexpected-error
execute_threads: 1
experimental_solver: classic
extra_safety_checks: False
force: False
force_32bit: False
force_reinstall: False
force_remove: False
ignore_pinned: False
json: False
local_repodata_ttl: 1
migrated_channel_aliases: []
migrated_custom_channels: {}
non_admin_enabled: True
notify_outdated_conda: True
offline: False
override_channels_enabled: True
path_conflict: clobber
pinned_packages: []
pip_interop_enabled: True
pkgs_dirs:
  - /ssd/CryoSparc/cryosparc_worker/deps/anaconda/pkgs
  - /home/cryosparc/.conda/pkgs
proxy_servers: {}
quiet: False
remote_backoff_factor: 1
remote_connect_timeout_secs: 9.15
remote_max_retries: 3
remote_read_timeout_secs: 60.0
repodata_fns:
  - current_repodata.json
  - repodata.json
repodata_threads: None
report_errors: None
restore_free_channel: False
rollback_enabled: True
root_prefix: /ssd/CryoSparc/cryosparc_worker/deps/anaconda
safety_checks: warn
sat_solver: pycosat
separate_format_cache: False
shortcuts: True
show_channel_urls: None
signing_metadata_url_base: None
solver_ignore_timestamps: False
ssl_verify: True
subdir: linux-64
subdirs:
  - linux-64
  - noarch
target_prefix_override: 
track_features: []
unsatisfiable_hints: True
unsatisfiable_hints_check_depth: 2
update_modifier: update_specs
use_index_cache: False
use_local: False
use_only_tar_bz2: False
verbosity: 1
verify_threads: 1
whitelist_channels: []

And:

]$ ./cryosparcw call /usr/bin/env | grep -v LICENSE
CRYOSPARC_USE_GPU=true
RELION_ALLOW_CHANGE_MINIMUM_DEDICATED=1
CUDA_PATH=/cm/shared/apps/cuda11.4/toolkit/11.4.1
MANPATH=/cm/shared/apps/IMOD/man:/cm/local/apps/cm-setup/man:/cm/local/apps/cluster-tools/man:/cm/local/apps/cmd/man:/cm/local/apps/cuda/libs/current/share/man:/cm/shared/apps/slurm/current/man:/cm/local/apps/environment-modules/4.5.3//share/man:/cm/shared/apps/IMOD/man:/usr/local/share/man:/usr/share/man/overrides:/usr/share/man:/opt/ibutils/share/man:/cm/local/apps/environment-modules/current/share/man:/cm/local/apps/environment-modules/current/share/man
CUDA_INC_PATH=/cm/shared/apps/cuda11.4/toolkit/11.4.1:/cm/shared/apps/cuda10.1/toolkit/10.1.243
XDG_SESSION_ID=119328
IMOD_DIR=/cm/shared/apps/IMOD
HOSTNAME=headnode
_LMFILES__modshare=/cm/shared/modulefiles/openmpi/4.0.5:1:/cm/shared/modulefiles/cuda10.1/toolkit/10.1.243:1:/cm/shared/modulefiles/cuda11.4/toolkit/11.4.1:1:/cm/local/modulefiles/cmd:1:/cm/shared/modulefiles/relion/4.0.1:1:/cm/local/modulefiles/cm-setup/9.1:1:/cm/local/modulefiles/slurm/slurm/20.02.6:1:/cm/local/modulefiles/gcc/10.2.0:1:/cm/local/modulefiles/shared:1:/cm/local/modulefiles/cmsh:1:/cm/local/modulefiles/cluster-tools/9.1:1
MODULEPATH_modshare=/cm/shared/modulefiles:1:/cm/local/modulefiles:1
LD_RUN_PATH_modshare=/cm/local/apps/cuda/libs/current/lib64:2
MODULES_CMD=/cm/local/apps/environment-modules/4.5.3/libexec/modulecmd.tcl
TERM=xterm-256color
SHELL=/bin/bash
PYTHONNOUSERSITE=true
IMOD_QTLIBDIR=/cm/shared/apps/IMOD/qtlib
HISTSIZE=1000
CUDA_SDK=/cm/shared/apps/cuda11.4/sdk/11.4.1
SSH_CLIENT=Redacted
CONDA_SHLVL=1
RELION_WARNING_LOCAL_MPI=12
LIBRARY_PATH_modshare=/cm/shared/apps/slurm/current/lib64:1:/cm/shared/apps/cuda11.4/toolkit/11.4.1/targets/x86_64-linux/lib:1:/cm/shared/apps/cuda10.1/toolkit/10.1.243/targets/x86_64-linux/lib:1:/cm/shared/apps/slurm/current/lib64/slurm:1:/cm/local/apps/cuda/libs/current/lib64:2
LIBRARY_PATH=/cm/shared/apps/cuda11.4/toolkit/11.4.1/targets/x86_64-linux/lib:/cm/local/apps/cuda/libs/current/lib64:/cm/shared/apps/cuda10.1/toolkit/10.1.243/targets/x86_64-linux/lib:/cm/shared/apps/slurm/current/lib64/slurm:/cm/shared/apps/slurm/current/lib64
PERL5LIB=/home/cryosparc/perl5/lib/perl5:/root/perl5/lib/perl5:
LD_PRELOAD=/ssd/CryoSparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/libpython3.8.so
CONDA_PROMPT_MODIFIER=(cryosparc_worker_env) 
CRYOSPARC_CONDA_ENV=cryosparc_worker_env
RELION_MOTIONCOR2_EXECUTABLE=/cm/shared/apps/bin/MotionCor2
QTDIR=/usr/lib64/qt-3.3
OLDPWD=/ssd/CryoSparc/cryosparc_worker
QTINC=/usr/lib64/qt-3.3/include
PERL_MB_OPT=--install_base /home/cryosparc/perl5
SSH_TTY=/dev/pts/0
CRYOSPARC_DEVELOP=false
QT_GRAPHICSSYSTEM_CHECKED=1
CUDA_HOME=/cm/shared/apps/cuda11.4/toolkit/11.4.1
USER=cryosparc
LD_LIBRARY_PATH=/cm/shared/apps/cuda11.2/toolkit/11.2.0/lib64:/ssd/CryoSparc/cryosparc_worker/deps/external/cudnn/lib:/cm/local/apps/gcc/10.2.0/lib:/cm/local/apps/gcc/10.2.0/lib64:/cm/local/apps/gcc/10.2.0/lib32:/cm/shared/apps/relion/4.0.1/lib:/cm/shared/apps/cuda11.4/toolkit/11.4.1/targets/x86_64-linux/lib:/cm/shared/apps/cuda10.1/toolkit/10.1.243/extras/CUPTI/lib64:/cm/local/apps/cuda/libs/current/lib64:/cm/shared/apps/cuda10.1/toolkit/10.1.243/targets/x86_64-linux/lib:/cm/shared/apps/openmpi/4.0.5/lib64:/cm/shared/apps/slurm/current/lib64/slurm:/cm/shared/apps/slurm/current/lib64

CONDA_EXE=/ssd/CryoSparc/cryosparc_worker/deps/anaconda/bin/conda
RELION_UNBLUR_EXECUTABLE=/cm/shared/apps/bin/unblur_openmp_7_17_15.exe
ENV=/cm/local/apps/environment-modules/4.5.3//init/profile.sh
RELION_MINIMUM_DEDICATED=64
CPATH=/cm/shared/apps/cuda11.4/sdk/11.4.1/common/inc:/cm/shared/apps/cuda11.4/toolkit/11.4.1/targets/x86_64-linux/include:/cm/shared/apps/cuda10.1/sdk/10.1.243/common/inc:/cm/shared/apps/cuda10.1/toolkit/10.1.243/targets/x86_64-linux/include:/cm/shared/apps/slurm/current/include
CUDA_CACHE_DISABLE=1
_CE_CONDA=
CUDA_INSTALL_PATH=/cm/shared/apps/cuda11.4/toolkit/11.4.1
MAIL=/var/spool/mail/root
PATH=/cm/shared/apps/cuda11.2/toolkit/11.2.0/bin:/ssd/CryoSparc/cryosparc_worker/bin:/ssd/CryoSparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/ssd/CryoSparc/cryosparc_worker/deps/anaconda/condabin:/ssd/CryoSparc/cryosparc_master/bin:/cm/local/apps/gcc/10.2.0/bin:/home/cryosparc/perl5/bin:/cm/local/apps/environment-modules/4.5.3/bin:/cm/shared/apps/conda/bin:/cm/shared/apps/conda/condabin:/cm/local/apps/cm-setup/bin:/cm/local/apps/cluster-tools/bin:/cm/local/apps/cmd/sbin:/cm/local/apps/cmd/bin:/cm/shared/apps/relion/4.0.1/bin:/cm/shared/apps/cuda11.4/sdk/11.4.1/bin/x86_64/linux/release:/cm/shared/apps/cuda11.4/toolkit/11.4.1/bin:/cm/local/apps/cuda/libs/current/bin:/cm/shared/apps/cuda10.1/sdk/10.1.243/bin/x86_64/linux/release:/cm/shared/apps/cuda10.1/toolkit/10.1.243/bin:/cm/shared/apps/openmpi/4.0.5/bin:/cm/shared/apps/slurm/current/sbin:/cm/shared/apps/slurm/current/bin:/usr/lib64/qt-3.3/bin:/root/perl5/bin:/cm/local/apps/environment-modules/4.5.3/bin:/cm/shared/apps/IMOD/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/ibutils/bin:/cm/local/apps/environment-modules/4.5.3/bin:/bin:/sbin:/root/bin:/sbin:/usr/sbin:/cm/local/apps/environment-modules/4.5.3/bin
SLURM_CONF=/cm/shared/apps/slurm/var/etc/slurm/slurm.conf
MKL_NUM_THREADS=1
XML_CATALOG_FILES=file:///cm/shared/apps/conda/etc/xml/catalog file:///etc/xml/catalog
IMOD_PLUGIN_DIR=/cm/shared/apps/IMOD/lib/imodplug
CONDA_PREFIX=/ssd/CryoSparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env
LD_RUN_PATH=/cm/local/apps/cuda/libs/current/lib64
CRYOSPARC_ROOT_DIR=/ssd/CryoSparc/cryosparc_worker
INCLUDEPATH_modshare=/cm/shared/apps/cuda11.4/toolkit/11.4.1/extras/Debugger/include:1:/cm/shared/apps/cuda11.4/toolkit/11.4.1/targets/x86_64-linux/include/CL:1:/cm/shared/apps/cuda11.4/sdk/11.4.1/common/inc:1:/cm/shared/apps/cuda10.1/toolkit/10.1.243/extras/Debugger/include:1:/cm/shared/apps/cuda10.1/toolkit/10.1.243/extras/CUPTI/include:1:/cm/shared/apps/cuda10.1/toolkit/10.1.243/targets/x86_64-linux/include/CL:1:/cm/shared/apps/cuda10.1/sdk/10.1.243/common/inc:1:/cm/shared/apps/cuda11.4/toolkit/11.4.1/targets/x86_64-linux/include:1:/cm/shared/apps/cuda10.1/toolkit/10.1.243/targets/x86_64-linux/include:1
PWD=/ssd/CryoSparc/cryosparc_worker
NUMEXPR_NUM_THREADS=1
_LMFILES_=/cm/local/modulefiles/shared:/cm/local/modulefiles/slurm/slurm/20.02.6:/cm/shared/modulefiles/openmpi/4.0.5:/cm/shared/modulefiles/cuda10.1/toolkit/10.1.243:/cm/shared/modulefiles/cuda11.4/toolkit/11.4.1:/cm/shared/modulefiles/relion/4.0.1:/cm/local/modulefiles/cmsh:/cm/local/modulefiles/cmd:/cm/local/modulefiles/cluster-tools/9.1:/cm/local/modulefiles/cm-setup/9.1:/cm/local/modulefiles/gcc/10.2.0
RELION_RESMAP_EXECUTABLE=/cm/shared/apps/bin/ResMap
IMOD_JAVADIR=/usr/local/java
CUDA_INC_PATH_modshare=/cm/shared/apps/cuda11.4/toolkit/11.4.1:1:/cm/shared/apps/cuda10.1/toolkit/10.1.243:1
LANG=en_US.UTF-8
MODULEPATH=/cm/local/modulefiles:/cm/shared/modulefiles
LOADEDMODULES=shared:slurm/slurm/20.02.6:openmpi/4.0.5:cuda10.1/toolkit/10.1.243:cuda11.4/toolkit/11.4.1:relion/4.0.1:cmsh:cmd:cluster-tools/9.1:cm-setup/9.1:gcc/10.2.0
KDEDIRS=/usr
FOR_DISABLE_STACK_TRACE=1
CPATH_modshare=/cm/shared/apps/cuda11.4/sdk/11.4.1/common/inc:1:/cm/shared/apps/cuda10.1/sdk/10.1.243/common/inc:1:/cm/shared/apps/cuda11.4/toolkit/11.4.1/targets/x86_64-linux/include:1:/cm/shared/apps/cuda10.1/toolkit/10.1.243/targets/x86_64-linux/include:1:/cm/shared/apps/slurm/current/include:1
ENABLE_LMOD=0
_CE_M=
RELION_CTFFIND_EXECUTABLE=/cm/shared/apps/bin/ctffind
HISTCONTROL=ignoredups
PATH_modshare=/cm/shared/apps/IMOD/bin:1:/cm/shared/apps/cuda10.1/sdk/10.1.243/bin/x86_64/linux/release:1:/cm/shared/apps/relion/4.0.1/bin:1:/cm/shared/apps/cuda11.4/sdk/11.4.1/bin/x86_64/linux/release:1:/usr/bin:1:/cm/local/apps/cm-setup/bin:1:/cm/local/apps/cmd/sbin:1:/root/bin:1:/usr/local/bin:1:/cm/shared/apps/openmpi/4.0.5/bin:1:/cm/local/apps/cuda/libs/current/bin:2:/cm/local/apps/environment-modules/4.5.3//bin:1:/home/cryosparc/perl5/bin:1:/cm/shared/apps/slurm/current/bin:1:/opt/ibutils/bin:1:/cm/local/apps/gcc/10.2.0/bin:1:/cm/shared/apps/cuda11.4/toolkit/11.4.1/bin:1:/cm/local/apps/cmd/bin:1:/cm/shared/apps/cuda10.1/toolkit/10.1.243/bin:1:/cm/shared/apps/slurm/current/sbin:1:/bin:1:/cm/shared/apps/conda/condabin:1:/root/perl5/bin:1:/cm/local/apps/cluster-tools/bin:1:/sbin:1:/cm/shared/apps/conda/bin:1:/usr/sbin:1:/cm/local/apps/environment-modules/4.5.3/bin:1:/usr/lib64/qt-3.3/bin:1:/usr/local/sbin:1
SHLVL=2
HOME=/home/cryosparc
PYTHONPATH_modshare=/cm/local/apps/cuda/libs/current/pynvml:1
CRYOSPARC_PATH=/ssd/CryoSparc/cryosparc_worker/bin
INCLUDEPATH=/cm/shared/apps/cuda11.4/toolkit/11.4.1/extras/Debugger/include:/cm/shared/apps/cuda11.4/toolkit/11.4.1/targets/x86_64-linux/include/CL:/cm/shared/apps/cuda11.4/sdk/11.4.1/common/inc:/cm/shared/apps/cuda11.4/toolkit/11.4.1/targets/x86_64-linux/include:/cm/shared/apps/cuda10.1/toolkit/10.1.243/extras/Debugger/include:/cm/shared/apps/cuda10.1/toolkit/10.1.243/extras/CUPTI/include:/cm/shared/apps/cuda10.1/toolkit/10.1.243/targets/x86_64-linux/include/CL:/cm/shared/apps/cuda10.1/sdk/10.1.243/common/inc:/cm/shared/apps/cuda10.1/toolkit/10.1.243/targets/x86_64-linux/include
MANPATH_modshare=/usr/local/share/man:1:/cm/local/apps/cuda/libs/current/share/man:2:/cm/local/apps/environment-modules/4.5.3//share/man:1:/opt/ibutils/share/man:1:/cm/shared/apps/IMOD/man:1:/cm/local/apps/cm-setup/man:1:/cm/shared/apps/slurm/current/man:1:/usr/share/man/overrides:1:/cm/local/apps/cmd/man:2:/cm/local/apps/environment-modules/current/share/man:1:/usr/share/man:1:/cm/local/apps/cluster-tools/man:1
RELION_SUMMOVIE_EXECUTABLE=/cm/shared/apps/bin/sum_movie_openmp_7_17_15.exe
PERL_LOCAL_LIB_ROOT=:/root/perl5:/home/cryosparc/perl5
BASH_ENV=/cm/local/apps/environment-modules/4.5.3//init/bash
PYTHONPATH=/ssd/CryoSparc/cryosparc_worker
CONDA_PYTHON_EXE=/ssd/CryoSparc/cryosparc_worker/deps/anaconda/bin/python
LOGNAME=cryosparc
LD_LIBRARY_PATH_modshare=/cm/local/apps/gcc/10.2.0/lib:1:/cm/shared/apps/slurm/current/lib64:1:/cm/local/apps/gcc/10.2.0/lib32:1:/cm/shared/apps/cuda11.4/toolkit/11.4.1/targets/x86_64-linux/lib:1:/cm/shared/apps/cuda10.1/toolkit/10.1.243/extras/CUPTI/lib64:1:/cm/shared/apps/cuda10.1/toolkit/10.1.243/targets/x86_64-linux/lib:1:/cm/local/apps/gcc/10.2.0/lib64:1:/cm/shared/apps/relion/4.0.1/lib:1:/cm/shared/apps/slurm/current/lib64/slurm:1:/cm/local/apps/cuda/libs/current/lib64:2:/cm/shared/apps/openmpi/4.0.5/lib64:1
QTLIB=/usr/lib64/qt-3.3/lib
XDG_DATA_DIRS=/root/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
SSH_CONNECTION=Redacted
MODULESHOME=/cm/local/apps/environment-modules/4.5.3/
OMP_NUM_THREADS=1
CONDA_DEFAULT_ENV=cryosparc_worker_env
RELION_GCTF_EXECUTABLE=/cm/shared/apps/bin/Gctf
LESSOPEN=||/usr/bin/lesspipe.sh %s
IMOD_CALIB_DIR=/usr/local/ImodCalib
RELION_PDFVIEWER_EXECUTABLEe=evince
LOADEDMODULES_modshare=openmpi/4.0.5:1:cuda10.1/toolkit/10.1.243:1:cuda11.4/toolkit/11.4.1:1:cmd:1:relion/4.0.1:1:cm-setup/9.1:1:slurm/slurm/20.02.6:1:gcc/10.2.0:1:shared:1:cmsh:1:cluster-tools/9.1:1
XDG_RUNTIME_DIR=/run/user/0
DISPLAY=localhost:10.0
QT_PLUGIN_PATH=/usr/lib64/kde4/plugins:/usr/lib/kde4/plugins
CUDA_ROOT=/cm/shared/apps/cuda11.4/toolkit/11.4.1
CUDA_CMLOCAL_ROOT=/cm/local/apps/cuda/libs/current
CRYOSPARC_CUDA_PATH=/cm/shared/apps/cuda11.2/toolkit/11.2.0
CMD_WLM_CLUSTER_NAME=slurm
PERL_MM_OPT=INSTALL_BASE=/home/cryosparc/perl5

nfrasser · June 30, 2023, 3:28pm

@andreym, one more command to try:

cryosparcw call conda config --set ssl_verify False

And then try installing 3D Flex dependencies as per the standard instructions.

andreym · July 5, 2023, 3:46pm

Hello, @nfrasser. Thank you for the suggestion. There is some progress, per your suggestion. Please see below. Update is able to download (temporarily) some of the libraries, but gets interrupted with others. Rerunning the command again or with
cryosparcw call pip config set global.trusted-host pypi.org SSL_NO_VERIFY=1 cryosparcw install-3dflex
the process pretty much repeats from beginning.

Thank you,
Andrey

]$ ./cryosparcw install-3dflex
Installing 3D Flex Refine dependencies...
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.12.0
  latest version: 23.5.0

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /ssd/CryoSparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env

  added / updated specs:
    - cuda-nvcc=11.8
    - cuda-toolkit=11.8


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cuda-nvvp-11.8.87          |                0       114.4 MB  nvidia/label/cuda-11.8.0
    libcublas-11.11.3.6        |                0       364.0 MB  nvidia/label/cuda-11.8.0
    libcublas-dev-11.11.3.6    |                0       394.1 MB  nvidia/label/cuda-11.8.0
    libcufft-10.9.0.58         |                0       142.8 MB  nvidia/label/cuda-11.8.0
    libcufft-dev-10.9.0.58     |                0       275.8 MB  nvidia/label/cuda-11.8.0
    libcusolver-11.4.1.48      |                0        96.5 MB  nvidia/label/cuda-11.8.0
    libcusolver-dev-11.4.1.48  |                0        66.3 MB  nvidia/label/cuda-11.8.0
    libcusparse-11.7.5.86      |                0       176.3 MB  nvidia/label/cuda-11.8.0
    libcusparse-dev-11.7.5.86  |                0       359.7 MB  nvidia/label/cuda-11.8.0
    libnpp-11.8.0.86           |                0       147.8 MB  nvidia/label/cuda-11.8.0
    libnpp-dev-11.8.0.86       |                0       144.5 MB  nvidia/label/cuda-11.8.0
    ------------------------------------------------------------
                                           Total:        2.23 GB

The following NEW packages will be INSTALLED:

  cuda-cccl          nvidia/label/cuda-11.8.0/linux-64::cuda-cccl-11.8.89-0
  cuda-command-line~ nvidia/label/cuda-11.8.0/linux-64::cuda-command-line-tools-11.8.0-0
  cuda-compiler      nvidia/label/cuda-11.8.0/linux-64::cuda-compiler-11.8.0-0
  cuda-cudart        nvidia/label/cuda-11.8.0/linux-64::cuda-cudart-11.8.89-0
  cuda-cudart-dev    nvidia/label/cuda-11.8.0/linux-64::cuda-cudart-dev-11.8.89-0
  cuda-cuobjdump     nvidia/label/cuda-11.8.0/linux-64::cuda-cuobjdump-11.8.86-0
  cuda-cupti         nvidia/label/cuda-11.8.0/linux-64::cuda-cupti-11.8.87-0
  cuda-cuxxfilt      nvidia/label/cuda-11.8.0/linux-64::cuda-cuxxfilt-11.8.86-0
  cuda-documentation nvidia/label/cuda-11.8.0/linux-64::cuda-documentation-11.8.86-0
  cuda-driver-dev    nvidia/label/cuda-11.8.0/linux-64::cuda-driver-dev-11.8.89-0
  cuda-gdb           nvidia/label/cuda-11.8.0/linux-64::cuda-gdb-11.8.86-0
  cuda-libraries     nvidia/label/cuda-11.8.0/linux-64::cuda-libraries-11.8.0-0
  cuda-libraries-dev nvidia/label/cuda-11.8.0/linux-64::cuda-libraries-dev-11.8.0-0
  cuda-memcheck      nvidia/label/cuda-11.8.0/linux-64::cuda-memcheck-11.8.86-0
  cuda-nsight        nvidia/label/cuda-11.8.0/linux-64::cuda-nsight-11.8.86-0
  cuda-nsight-compu~ nvidia/label/cuda-11.8.0/linux-64::cuda-nsight-compute-11.8.0-0
  cuda-nvcc          nvidia/label/cuda-11.8.0/linux-64::cuda-nvcc-11.8.89-0
  cuda-nvdisasm      nvidia/label/cuda-11.8.0/linux-64::cuda-nvdisasm-11.8.86-0
  cuda-nvml-dev      nvidia/label/cuda-11.8.0/linux-64::cuda-nvml-dev-11.8.86-0
  cuda-nvprof        nvidia/label/cuda-11.8.0/linux-64::cuda-nvprof-11.8.87-0
  cuda-nvprune       nvidia/label/cuda-11.8.0/linux-64::cuda-nvprune-11.8.86-0
  cuda-nvrtc         nvidia/label/cuda-11.8.0/linux-64::cuda-nvrtc-11.8.89-0
  cuda-nvrtc-dev     nvidia/label/cuda-11.8.0/linux-64::cuda-nvrtc-dev-11.8.89-0
  cuda-nvtx          nvidia/label/cuda-11.8.0/linux-64::cuda-nvtx-11.8.86-0
  cuda-nvvp          nvidia/label/cuda-11.8.0/linux-64::cuda-nvvp-11.8.87-0
  cuda-profiler-api  nvidia/label/cuda-11.8.0/linux-64::cuda-profiler-api-11.8.86-0
  cuda-sanitizer-api nvidia/label/cuda-11.8.0/linux-64::cuda-sanitizer-api-11.8.86-0
  cuda-toolkit       nvidia/label/cuda-11.8.0/linux-64::cuda-toolkit-11.8.0-0
  cuda-tools         nvidia/label/cuda-11.8.0/linux-64::cuda-tools-11.8.0-0
  cuda-visual-tools  nvidia/label/cuda-11.8.0/linux-64::cuda-visual-tools-11.8.0-0
  gds-tools          nvidia/label/cuda-11.8.0/linux-64::gds-tools-1.4.0.31-0
  libcublas          nvidia/label/cuda-11.8.0/linux-64::libcublas-11.11.3.6-0
  libcublas-dev      nvidia/label/cuda-11.8.0/linux-64::libcublas-dev-11.11.3.6-0
  libcufft           nvidia/label/cuda-11.8.0/linux-64::libcufft-10.9.0.58-0
  libcufft-dev       nvidia/label/cuda-11.8.0/linux-64::libcufft-dev-10.9.0.58-0
  libcufile          nvidia/label/cuda-11.8.0/linux-64::libcufile-1.4.0.31-0
  libcufile-dev      nvidia/label/cuda-11.8.0/linux-64::libcufile-dev-1.4.0.31-0
  libcurand          nvidia/label/cuda-11.8.0/linux-64::libcurand-10.3.0.86-0
  libcurand-dev      nvidia/label/cuda-11.8.0/linux-64::libcurand-dev-10.3.0.86-0
  libcusolver        nvidia/label/cuda-11.8.0/linux-64::libcusolver-11.4.1.48-0
  libcusolver-dev    nvidia/label/cuda-11.8.0/linux-64::libcusolver-dev-11.4.1.48-0
  libcusparse        nvidia/label/cuda-11.8.0/linux-64::libcusparse-11.7.5.86-0
  libcusparse-dev    nvidia/label/cuda-11.8.0/linux-64::libcusparse-dev-11.7.5.86-0
  libnpp             nvidia/label/cuda-11.8.0/linux-64::libnpp-11.8.0.86-0
  libnpp-dev         nvidia/label/cuda-11.8.0/linux-64::libnpp-dev-11.8.0.86-0
  libnvjpeg          nvidia/label/cuda-11.8.0/linux-64::libnvjpeg-11.9.0.86-0
  libnvjpeg-dev      nvidia/label/cuda-11.8.0/linux-64::libnvjpeg-dev-11.9.0.86-0
  nsight-compute     nvidia/label/cuda-11.8.0/linux-64::nsight-compute-2022.3.0.22-0

The following packages will be UPDATED:

  ca-certificates    conda-forge::ca-certificates-2022.12.~ --> pkgs/main::ca-certificates-2023.05.30-h06a4308_0
  certifi            conda-forge/noarch::certifi-2022.12.7~ --> pkgs/main/linux-64::certifi-2023.5.7-py38h06a4308_0
  openssl            conda-forge::openssl-1.1.1s-h0b41bf4_1 --> pkgs/main::openssl-1.1.1u-h7f8727e_0



Downloading and Extracting Packages
libnpp-11.8.0.86     | 147.8 MB  | ####################################################################################9 | 100% 
libcublas-11.11.3.6  | 364.0 MB  | ####################################################################################9 | 100% 
libnpp-dev-11.8.0.86 | 144.5 MB  | ####################################################################################9 | 100% 
libcusolver-dev-11.4 | 66.3 MB   | ####################################################################################9 | 100% 
libcusparse-11.7.5.8 | 176.3 MB  | ##########################9                                                           |  32% 
libcusolver-11.4.1.4 | 96.5 MB   | #############################################################################6        |  91% 
cuda-nvvp-11.8.87    | 114.4 MB  | #######################################6                                              |  47% 
libcublas-dev-11.11. | 394.1 MB  | ####################################################################################9 | 100% 
libcufft-dev-10.9.0. | 275.8 MB  | ##############################################################################9       |  93% 
libcusparse-dev-11.7 | 359.7 MB  | ###################################################################################9  |  99% 
libcufft-10.9.0.58   | 142.8 MB  | #####################################################5                                |  63% 

("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/nvidia/label/cuda-11.8.0/linux-64/cuda-nvvp-11.8.87-0.tar.bz2>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

[cryosparc@headnode bin]$

andreym · July 20, 2023, 2:38pm

I am still working with our IT to identify the source of interruption. But I was able to install dependencies manually.

Run > cryosparcw install-3dflex as normal to identify the libraries you might need. For me, these were

libcusparse-dev-11.7 
libnpp-11.8.0.86   
libcublas-dev-11.11. 
libcublas-11.11.3.6  
libcusolver-dev-11.4 
libnpp-dev-11.8.0.86 
libcufft-10.9.0.58  
libcusparse-11.7.5.8 
libcufft-dev-10.9.0. 
libcusolver-11.4.1.4

Download the needed *tar.bz2 files on any other system from :: Anaconda.org On my system, behind our network, it still took several (3+) attempts at downloading most of the packages before the download completed successfully.
Transfer the packages to the server with you CS instal
Run > cryosparcw call conda install /path/*tar.bz2
Re-run > cryosparcw install-3dflex. This last step was able to download and install torch successfully and validate 3D Flex dependencies installation.
Run a 3D Flex job to make sure full installation was successful.

Hopefully this will be of help to anyone in the same boat.