When update to cryosparc 4.2.1 pycuda installed failed

gaorongchao · June 16, 2023, 1:17pm

gcc -pthread -B /home/someone/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/compiler_compat -Wno-unused-result -Wsign-compare -fwrapv -Wall -O3 -DNDEBUG -fPIC -DBOOST_ALL_NO_LIB=1 -DBOOST_THREAD_BUILD_DLL=1 -DBOOST_MULTI_INDEX_DISABLE_SERIALIZATION=1 -DBOOST_PYTHON_SOURCE=1 -Dboost=pycudaboost -DBOOST_THREAD_DONT_USE_CHRONO=1 -DPYGPU_PACKAGE=pycuda -DPYGPU_PYCUDA=1 -DHAVE_CURAND=1 -Isrc/cpp -Ibpl-subset/bpl_subset -I/usr/local/cuda-11.7/include -I/home/someone/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numpy/core/include -I/home/someone/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.8 -c bpl-subset/bpl_subset/libs/python/src/converter/arg_to_python_base.cpp -o build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs/python/src/converter/arg_to_python_base.o
      In file included from /usr/include/limits.h:191:0,
                       from /opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7/include/limits.h:194,
                       from /opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7/include/syslimits.h:7,
                       from /opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7/include/limits.h:34,
                       from bpl-subset/bpl_subset/boost/python/detail/wrap_python.hpp:61,
                       from bpl-subset/bpl_subset/boost/python/detail/prefix.hpp:13,
                       from bpl-subset/bpl_subset/boost/python/handle.hpp:8,
                       from bpl-subset/bpl_subset/boost/python/converter/arg_to_python_base.hpp:7,
                       from bpl-subset/bpl_subset/libs/python/src/converter/arg_to_python_base.cpp:6:
      /usr/include/bits/xopen_lim.h:64:10: fatal error: bits/uio_lim.h: No such file or directory
       #include <bits/uio_lim.h>
                ^~~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/opt/rh/devtoolset-7/root/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> pycuda

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
check_install_deps.sh: 66: ERROR: installing python failed.

when use /cryosparcw install-3dflex to success install pycuda, get new error:

"nvcc fatal   : Value 'sm_86' is not defined for option 'gpu-architecture'\n"

wtempel · June 16, 2023, 1:30pm

Welcome to the forum @gaorongchao.
Please can you post the output of these commands:

uname -a
free -g
nvidia-smi

gaorongchao · June 16, 2023, 1:48pm

Thank you for your replay, the output of these commands:
uname -a

Linux gpu7 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

free -g

              total        used        free      shared  buff/cache   available
Mem:           1006          12         103           0         890         992
Swap:            33           0          33

nvidia-smi

Fri Jun 16 21:46:09 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3090    Off  | 00000000:1A:00.0 Off |                  N/A |
| 30%   33C    P0   109W / 350W |      0MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 3090    Off  | 00000000:1B:00.0 Off |                  N/A |
| 30%   33C    P0   104W / 350W |      0MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 3090    Off  | 00000000:3D:00.0 Off |                  N/A |
| 30%   30C    P0   106W / 350W |      0MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 3090    Off  | 00000000:3E:00.0 Off |                  N/A |
| 30%   33C    P0   105W / 350W |      0MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  GeForce RTX 3090    Off  | 00000000:88:00.0 Off |                  N/A |
| 30%   32C    P0    99W / 350W |      0MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  GeForce RTX 3090    Off  | 00000000:89:00.0 Off |                  N/A |
| 30%   31C    P0   102W / 350W |      0MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  GeForce RTX 3090    Off  | 00000000:B1:00.0 Off |                  N/A |
| 30%   31C    P0    94W / 350W |      0MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  GeForce RTX 3090    Off  | 00000000:B2:00.0 Off |                  N/A |
| 30%   31C    P0    57W / 350W |      0MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

wtempel · June 16, 2023, 6:51pm

Do you know how /usr/include/bits/xopen_lim.h was installed?

rpm -qf /usr/include/bits/xopen_lim.h
ls -l /usr/include/bits/xopen_lim.h

gaorongchao · June 19, 2023, 1:45am

(base) [root@gpu7 bits]# rpm -qf xopen_lim.h 
glibc-headers-2.17-326.el7_9.x86_64
(base) [root@gpu7 bits]# ls -l xopen_lim.h 
-rw-r--r-- 1 root root 3857 Feb 28 12:40 xopen_lim.h
(base) [root@gpu7 bits]#

gaorongchao · June 19, 2023, 1:49am

Errors when run the Ab-Initio Reconstruction:

[CPU:  502.3 MB  Avail:1015.21 GB]
Traceback (most recent call last):
  File "/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/tools.py", line 429, in context_dependent_memoize
    return ctx_dict[cur_ctx][args]
KeyError: <pycuda._driver.Context object at 0x7fad24720dd0>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 96, in cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/abinit/run.py", line 165, in cryosparc_compute.jobs.abinit.run.run_homo_abinit
  File "cryosparc_master/cryosparc_compute/jobs/abinit/run.py", line 556, in cryosparc_compute.jobs.abinit.run.generate_initial_density_from_projections
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 964, in cryosparc_compute.engine.engine.process
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 974, in cryosparc_compute.engine.engine.process
  File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 156, in cryosparc_compute.engine.cuda_core.allocate_gpu
  File "/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/gpuarray.py", line 549, in fill
    func = elementwise.get_fill_kernel(self.dtype)
  File "<decorator-gen-13>", line 2, in get_fill_kernel
  File "/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/tools.py", line 433, in context_dependent_memoize
    result = func(*args)
  File "/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/elementwise.py", line 493, in get_fill_kernel
    return get_elwise_kernel(
  File "/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/elementwise.py", line 162, in get_elwise_kernel
    mod, func, arguments = get_elwise_kernel_and_types(
  File "/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/elementwise.py", line 148, in get_elwise_kernel_and_types
    mod = module_builder(arguments, operation, name,
  File "/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/elementwise.py", line 45, in get_elwise_module
    return SourceModule("""
  File "/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/compiler.py", line 290, in __init__
    cubin = compile(source, nvcc, options, keep, no_extern_c,
  File "/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/compiler.py", line 254, in compile
    return compile_plain(source, options, keep, nvcc, cache_dir, target)
  File "/homecryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/compiler.py", line 78, in compile_plain
    checksum.update(preprocess_source(source, options, nvcc).encode("utf-8"))
  File "/homecryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/compiler.py", line 54, in preprocess_source
    raise CompileError("nvcc preprocessing of %s failed" % source_path,
pycuda.driver.CompileError: nvcc preprocessing of /tmp/tmpy0eku7j3.cu failed
[command: nvcc --preprocess -arch sm_86 -I/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/cuda /tmp/tmpy0eku7j3.cu --compiler-options -P]
[stderr:
b'In file included from /usr/include/limits.h:191:0,\n                
 from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/limits.h:168,\n                 
from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/syslimits.h:7,\n                
 from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/limits.h:34,\n                 
from /usr/local/cuda-11.8/bin/../targets/x86_64-linux/include/driver_types.h:81,\n                
 from /usr/local/cuda-11.8/bin/../targets/x86_64-linux/include/builtin_types.h:59,\n                
 from /usr/local/cuda-11.8/bin/../targets/x86_64-linux/include/cuda_runtime.h:91,\n                
 from <command-line>:0:\n/usr/include/bits/xopen_lim.h:64:26: fatal error: bits/uio_lim.h: No such file or directory\n #include <bits/uio_lim.h>\n                          
^\ncompilation terminated.\n']

gaorongchao · June 19, 2023, 2:02am

When I run the command：

nvcc --preprocess -arch sm_86 -I/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/cuda /tmp/tmp35xs09t1.cu --compiler-options -P

The Error is

{
    unsigned int x, y, z;
    __attribute__((host)) __attribute__((device)) dim3(unsigned int vx = 1, unsigned int vy = 1, unsigned int vz = 1) : x(vx), y(vy), z(vz) {}
    __attribute__((host)) __attribute__((device)) dim3(uint3 v) : x(v.x), y(v.y), z(v.z) {}
    __attribute__((host)) __attribute__((device)) operator uint3(void) const { uint3 t; t.x = x; t.y = y; t.z = z; return t; }
};
In file included from /usr/include/limits.h:191:0,
                 from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/limits.h:168,
                 from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/syslimits.h:7,
                 from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/limits.h:34,
                 from /usr/local/cuda-11.2/bin/../targets/x86_64-linux/include/driver_types.h:77,
                 from /usr/local/cuda-11.2/bin/../targets/x86_64-linux/include/builtin_types.h:59,
                 from /usr/local/cuda-11.2/bin/../targets/x86_64-linux/include/cuda_runtime.h:91,
                 from <command-line>:0:
/usr/include/bits/xopen_lim.h:64:26: fatal error: bits/uio_lim.h: No such file or directory
 #include <bits/uio_lim.h>
                          ^
compilation terminated.

wtempel · June 19, 2023, 3:36pm

Is it possible that your header file installation on that computer is broken in that the file /usr/include/bits/xopen_lim.h originally included with that package has been overwritten with a newer version of the file, resulting in a broken header file installation?
The EL7 version of the package seems not to contain bits/uio_lim.h (glibc-headers-2.17 for CentOS-7), whereas the EL8 version does contain the package (glibc-headers-2.28 for CentOS-8-stream).
An inconsistent header file collection may have lead to a failed CryoSPARC installation.

gaorongchao · June 19, 2023, 10:34pm

So the solution of the problem is update the version to glibc-headers-2.28? The OS is CentOS-7, is there any risk for the glibc-header update?

wtempel · June 20, 2023, 12:45pm

Probably not. To clarify, I checked on an AWS instance with CentOS-7 and CryoSPARC installed and found that /usr/include/bits/xopen_lim.h does not include a line
#include <bits/uio_lim.h>. This suggests to me that /usr/include/bits/xopen_lim.h has in some way been modified/overwritten on your machine. I do not know enough about the specific requirements/circumstances of your OS installation to propose a specific solution, but you may want to consider a clean installation of the latest “long-term-support” version of a popular Linux distribution (RHEL-9, Ubuntu-22.04 or similar) with the corresponding “original” glibc-headers and compiler packages.

gaorongchao · June 21, 2023, 5:02am

Thank you very much! Under your guidance,

I found all files which modification time is different from most of files in /usr/include.
backup all the files
replace the different files with files in another machine (which is also CentOS7)
reinstall cryosparc

Final, it success!
Thanks again.

wtempel · June 21, 2023, 1:53pm