Hey everyone,
Thanks for reporting.
If you’d like to run 3DFlex jobs, you will need to install the dependencies required via the install-3dflex command as mentioned here:
There seems to be an issue with the installation on some systems, we’re working on an update to fix this.
Thanks @stephan …
We were able to install and start a run, but eventually get this error which seems related to GPU memory:
cryosparc_compute.jobs.flex_refine.flexmod.TetraSVFunction.forward torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.15 GiB (GPU 0; 10.76 GiB total capacity; 7.15 GiB already allocated; 940.94 MiB free; 9.57 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Is there a place we should change max_split_size_mb ?
Also, will you post back here when you have a fix for the 3D flex dips?
Many thanks!
Is it expected that the worker environment is using the system gcc/g++ instead of the conda version?
I am running into /usr/bin/gcc and cuda/nvcc from conda potential conflicts on ubuntu 20.04. I have remove the possible conflicting packages from ubuntu (apt remove nvidia-cuda-toolkit…) but ./bin/cryosparcw install-3dflex keeps failing.
Then trying to revert with:
cryoem@myrdal:~/cryosparc2/cryosparc_worker$ ./bin/cryosparcw forcedeps
yields:
...
gcc -pthread -B /home/cryoem/cryosparc2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -fwrapv -Wall -O3 -DNDEBUG -fPIC -DBOOST_ALL_NO_LIB=1 -DBOOST_THREAD_BUILD_DLL=1 -DBOOST_MULTI_INDEX_DISABLE_SERIALIZATION=1 -DBOOST_PYTHON_SOURCE=1 -Dboost=pycudaboost -DBOOST_THREAD_DONT_USE_CHRONO=1 -DPYGPU_PACKAGE=pycuda -DPYGPU_PYCUDA=1 -DHAVE_CURAND=1 -Isrc/cpp -Ibpl-subset/bpl_subset -I/home/cryoem/cryosparc2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/numpy/core/include -I/home/cryoem/cryosparc2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m -c src/cpp/cuda.cpp -o build/temp.linux-x86_64-3.7/src/cpp/cuda.o
In file included from src/cpp/cuda.cpp:4:
src/cpp/cuda.hpp:14:10: fatal error: cuda.h: No such file or directory
14 | #include <cuda.h>
| ^~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /home/cryoem/cryosparc2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin/python3.7 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-3334l4mw/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-3334l4mw/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-8lgu7bn9/install-record.txt --single-version-externally-managed --compile --install-headers /home/cryoem/cryosparc2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m/pycuda Check the logs for full command output.
check_install_deps.sh: 59: ERROR: installing python failed.
I had to re-add nvidia-cuda-toolkit and the system provided cuda10.1
maybe relates to this thread 3DFlex Dependencies; Building pycuda - #10 by qitsweauca
tru@myrdal:~$ dpkg -l nvidia-cuda-toolkit
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===================-============-============-=================================
ii nvidia-cuda-toolkit 10.1.243-3 amd64 NVIDIA CUDA development toolkit
tru@myrdal:~$ dpkg -l gcc g++
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-================-============-=================================
ii g++ 4:9.3.0-1ubuntu2 amd64 GNU C++ compiler
ii gcc 4:9.3.0-1ubuntu2 amd64 GNU C compiler
now I can revert to previous setup:
cryoem@myrdal:~/cryosparc2/cryosparc_worker$ ./bin/cryosparcw forcedeps
Checking dependencies...
Forcing dependencies to be reinstalled...
------------------------------------------------------------------------
Installing anaconda python...
------------------------------------------------------------------------
PREFIX=/home/cryoem/cryosparc2/cryosparc_worker/deps/anaconda
Unpacking payload ...
...
Extracting all conda packages...
------------------------------------------------------------------------
...................................................................................................................................................................................
------------------------------------------------------------------------
Done.
conda packages installation successful.
------------------------------------------------------------------------
Preparing to install all pip packages...
------------------------------------------------------------------------
Processing ./deps_bundle/python/python_packages/pip_packages/pycuda-2020.1.tar.gz
Preparing metadata (setup.py) ... done
Skipping wheel build for pycuda, due to binaries being disabled for it.
Installing collected packages: pycuda
Running setup.py install for pycuda ... done
Successfully installed pycuda-2020.1
------------------------------------------------------------------------
Done.
pip packages installation successful.
------------------------------------------------------------------------
Main dependency installation completed. Continuing...
------------------------------------------------------------------------
Completed.
Currently checking hash for ctffind
Forcing reinstall for dependency ctffind...
------------------------------------------------------------------------
ctffind 4.1.10 installation successful.
------------------------------------------------------------------------
Completed.
Currently checking hash for cudnn
Forcing reinstall for dependency cudnn...
------------------------------------------------------------------------
cudnn 8.1.0.77 for CUDA 11 installation successful.
------------------------------------------------------------------------
Completed.
Currently checking hash for gctf
Forcing reinstall for dependency gctf...
------------------------------------------------------------------------
Gctf v1.06 installation successful.
------------------------------------------------------------------------
Completed.
Completed dependency check.
Generating '/home/cryoem/cryosparc2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/libtiff/tiff_h_4_4_0.py' from '/home/cryoem/cryosparc2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/../include/tiff.h'
reverted system yields non function cryosparc with errors such as:
I use the fixed provided at 3DFlex Dependencies; Building pycuda - #15 by scaiola
replacing line 457 of cryosparc_worker/bin/cryosparcw
conda install -y cuda-nvcc=11.7 cuda-toolkit=11.7 -c nvidia
by:
conda install -y cuda-nvcc=11.7 cuda-toolkit=11.7 -c nvidia/label/cuda-11.7.0
and running cryosparc_worker/bin/cryosparcw install-3dflex after, seems to have fixed everything and activate the 3dflex functionnality
This fix appears to work for us as well on a CentOS system.
Reference:
I did have to first revert the system:
./bin/cryosparcw forcedeps
Then edit cryosparcw and change from:
conda install -y cuda-nvcc=11.7 cuda-toolkit=11.7 -c nvidia
To:
conda install -y cuda-nvcc=11.7 cuda-toolkit=11.7 -c nvidia/label/cuda-11.7.0
Then finally run the 3dflex installer:
./bin/cryosparcw install-3dflex
No more lengthy errors, and users report that jobs appear ok so far.
Hi
No idea if this is related to the previous fix, here is an error reported by our users when using topaz:
UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors.
This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor.
You may want to copy the array to protect its data or make it writeable before converting it to a tensor.
This type of warning will be suppressed for the rest of this program.
(Triggered internally at /opt/conda/conda-bld/pytorch_1607370156314/work/torch/csrc/utils/tensor_numpy.cpp:141.)
and another one during extraction:
Thanks for the update!
Hey everyone,
I see that v4.1.1 addressed this, but I have run into this same ‘no torch’ issue in CryoSPARC v4.1.2 on Centos 7 when starting a 3DFlex Train job (Data Prep and Mesh Prep ran fine).
" ModuleNotFoundError: No module named ‘torch’ " is the precise wording in the log.
Before I attempt any of the above solutions, is there one that is currently recommended for v4.1.2?
Thanks!
Please try, without editing cryosparcw,
/path/to/cryosparc_worker/bin/cryosparcw install-3dflex 2>&1 | tee install_3dflex.log
Does this work?
The command runs with the follow output:
Preparing transaction: …working… done
Verifying transaction: …working… done
Executing transaction: …working… done==> WARNING: A newer version of conda exists. <==
current version: 4.12.0
latest version: 23.1.0Please update conda by running
$ conda update -n base -c defaults condaFound existing installation: pycuda 2020.1
Uninstalling pycuda-2020.1:
Successfully uninstalled pycuda-2020.1
Collecting torch
Downloading torch-1.13.1-cp38-cp38-manylinux1_x86_64.whl (887.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 887.4/887.4 MB 3.3 MB/s eta 0:00:00
Installing collected packages: torch
Successfully installed torch-1.13.1
Processing ./deps_bundle/python/python_packages/pip_packages/pycuda-2020.1-cp38-cp38-linux_x86_64.whl
Installing collected packages: pycuda
Successfully installed pycuda-2020.1
PyTorch not installed correctly, or NVIDIA GPU not detected.
Running 3DFlex Train fails upon initializing torch
This is using a Quadro P6000 with CUDA Version 10.1.243 (not sure what “found version 10020” maps to from the above error). Seems like the workstation needs an updated CUDA driver to properly install PyTorch?
EDIT: @wtempel seems like all jobs fail now on this machine, not just 3DFlex Train. See below for a Non-Uniform Refinement, and similar error seen for Ab Initio. Suggestion?
I had also run the command on a second workstation set up as a worker for the above workstation. The command ran with the following output:
Preparing transaction: …working… done
Verifying transaction: …working… done
Executing transaction: …working… done==> WARNING: A newer version of conda exists. <==
current version: 4.12.0
latest version: 23.1.0Please update conda by running
$ conda update -n base -c defaults condaFound existing installation: pycuda 2020.1
Uninstalling pycuda-2020.1:
Successfully uninstalled pycuda-2020.1
Collecting torch
Downloading torch-1.13.1-cp38-cp38-manylinux1_x86_64.whl (887.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 887.4/887.4 MB 4.6 MB/s eta 0:00:00
Installing collected packages: torch
Successfully installed torch-1.13.1
Processing ./deps_bundle/python/python_packages/pip_packages/pycuda-2020.1-cp38-cp38-linux_x86_64.whl
Installing collected packages: pycuda
Successfully installed pycuda-2020.1
PyTorch not installed correctly, or NVIDIA GPU not detected.
I am able to run 3D Flex Train and other jobs without crashing (at least, thus far). The worker is Centos 7, GeForce RTX 2080 Ti with CUDA Version 10.2.89.
Hello, @wtempel. What commands are run during “install-3dflex” ? Currently I am getting Conda HTTP error due to the company firewall settings. Usually, when trying to create Conda environments, I need to add “–insecure” to bypass this issue. Is it possible to do so in this case?
@andreym you should able to do this by adding pypi.org as a trusted host in pip and disabling SSL verification in conda. Should be something like this:
cryosparcw call pip config set global.trusted-host pypi.org
SSL_NO_VERIFY=1 cryosparcw install-3dflex
Let me know how that goes
@nfrasser Thank you for the suggestion. I have tried it, unfortunately I am still running into the error:
" Installing 3D Flex Refine dependencies…
Collecting package metadata (current_repodata.json): failed
CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://conda.anaconda.org/nvidia/label/cuda-11.8.0/linux-64/current_repodata.json
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
It is not your proxy, as https://conda.anaconda.org/nvidia/label/cuda-11.8.0/linux-64/current_repodata.json lands at " The page you are looking for does not exist."
@andreym could you post the full output of the following command?
SSL_NO_VERIFY=1 cryosparcw call conda install -y cuda-nvcc=11.8 cuda-toolkit=11.8 -c nvidia/label/cuda-11.8.0 -v
Hello. The error output is a bit extensive, please see below:
~]$ SSL_NO_VERIFY=1 /ssd/CryoSparc/cryosparc_worker/bin/cryosparcw call conda install -y cuda-nvcc=11.8 cuda-toolkit=11.8 -c nvidia/label/cuda-11.8.0 -v
Collecting package metadata (current_repodata.json): ...working... Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))': /nvidia/label/cuda-11.8.0/linux-64/current_repodata.json
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))': /nvidia/label/cuda-11.8.0/noarch/current_repodata.json
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))': /nvidia/label/cuda-11.8.0/linux-64/current_repodata.json
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))': /nvidia/label/cuda-11.8.0/noarch/current_repodata.json
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))': /nvidia/label/cuda-11.8.0/noarch/current_repodata.json
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))': /nvidia/label/cuda-11.8.0/linux-64/current_repodata.json
failed
Traceback (most recent call last):
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
conn.connect()
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connection.py", line 416, in connect
self.sock = ssl_wrap_socket(
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
resp = conn.urlopen(
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='conda.anaconda.org', port=443): Max retries exceeded with url: /nvidia/label/cuda-11.8.0/linux-64/current_repodata.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 701, in fetch_repodata_remote_request
resp = session.get(join_url(url, filename), headers=headers, proxies=session.proxies,
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/requests/sessions.py", line 542, in get
return self.request('GET', url, **kwargs)
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/requests/adapters.py", line 517, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='conda.anaconda.org', port=443): Max retries exceeded with url: /nvidia/label/cuda-11.8.0/linux-64/current_repodata.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/exceptions.py", line 1082, in __call__
return func(*args, **kwargs)
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/cli/main.py", line 87, in _main
exit_code = do_call(args, p)
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/cli/conda_argparse.py", line 84, in do_call
return getattr(module, func_name)(args, parser)
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/cli/main_install.py", line 20, in execute
install(args, parser, 'install')
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/cli/install.py", line 260, in install
unlink_link_transaction = solver.solve_for_transaction(
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/solve.py", line 152, in solve_for_transaction
unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/solve.py", line 195, in solve_for_diff
final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/solve.py", line 300, in solve_final_state
ssc = self._collect_all_metadata(ssc)
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
return f(*args, **kwds)
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/solve.py", line 463, in _collect_all_metadata
index, r = self._prepare(prepared_specs)
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/solve.py", line 1058, in _prepare
reduced_index = get_reduced_index(self.prefix, self.channels,
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/index.py", line 288, in get_reduced_index
new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs,
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 140, in query_all
result = tuple(concat(executor.map(subdir_query, channel_urls)))
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
yield fs.pop().result()
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.__get_result()
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 132, in <lambda>
subdir_query = lambda url: tuple(SubdirData(Channel(url), repodata_fn=repodata_fn).query(
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 145, in query
self.load()
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 210, in load
_internal_state = self._load()
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 375, in _load
raw_repodata_str = fetch_repodata_remote_request(
File "/ssd/CryoSparc/cryosparc_worker/deps/anaconda/lib/python3.8/site-packages/conda/core/subdir_data.py", line 806, in fetch_repodata_remote_request
raise CondaHTTPError(help_message,
conda.exceptions.CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/nvidia/label/cuda-11.8.0/linux-64/current_repodata.json>
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
'https://conda.anaconda.org/nvidia/label/cuda-11.8.0/linux-64'