v3.1 crashes on CentOS 7.8 with the following error:
[stderr:
cicc: /lib64/libc.so.6: version `GLIBC_2.18’ not found (required by /data1/progs/cryosparc2_worker/cryosparc_compute/libs/libstdc++.so.6)
]
1 Like
I’m also getting the same error while trying to run refinement jobs. I just updated to v3.1 from v2 and same CentOS 7.8.
There’s also a pycuda error:
[CPU: 681.5 MB] Traceback (most recent call last):
File “/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py”, line 429, in context_dependent_memoize
return ctx_dict[cur_ctx][args]
KeyError: <pycuda._driver.Context object at 0x7fe945484990>
Thanks for reporting this. I can confirm that this is a bug on our end, and specific to very old systems like centos/rhel 7. A fix is in progress. (Edit: I’m referring to the GLIBC error, please see my colleague’s question below regarding the pycuda error)
-Harris
Hi @nschnick,
Is it possible if you can send more of the text (the came before this error message) from the overview tab of this job?
Hi @stephan,
Here’s the full text for the errors -
[CPU: 634.5 MB] Traceback (most recent call last):
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 429, in context_dependent_memoize
return ctx_dict[cur_ctx][args]
KeyError: <pycuda._driver.Context object at 0x7fb3c95db210>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py", line 1726, in run_with_except_hook
run_old(*args, **kw)
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 130, in cryosparc_compute.engine.cuda_core.GPUThread.run
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 1027, in cryosparc_compute.engine.engine.process.work
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 84, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py", line 549, in fill
func = elementwise.get_fill_kernel(self.dtype)
File "<decorator-gen-13>", line 2, in get_fill_kernel
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 433, in context_dependent_memoize
result = func(*args)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 498, in get_fill_kernel
"fill")
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 163, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 149, in get_elwise_kernel_and_types
keep, options, **kwargs)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 76, in get_elwise_module
options=options, keep=keep, no_extern_c=True)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 291, in __init__
arch, code, cache_dir, include_dirs)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 137, in compile_plain
stderr=stderr.decode("utf-8", "replace"))
pycuda.driver.CompileError: nvcc compilation of /tmp/tmp7k55kfgh/kernel.cu failed
[command: nvcc --cubin -arch sm_75 -I/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda kernel.cu]
[stderr:
cicc: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /home/cryosparc_user/software/cryosparc/cryosparc2_worker/cryosparc_compute/libs/libstdc++.so.6)
]
[CPU: 634.5 MB] Traceback (most recent call last):
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 429, in context_dependent_memoize
return ctx_dict[cur_ctx][args]
KeyError: (dtype('complex64'),)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py", line 1726, in run_with_except_hook
run_old(*args, **kw)
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 130, in cryosparc_compute.engine.cuda_core.GPUThread.run
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 1027, in cryosparc_compute.engine.engine.process.work
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 84, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py", line 549, in fill
func = elementwise.get_fill_kernel(self.dtype)
File "<decorator-gen-13>", line 2, in get_fill_kernel
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 433, in context_dependent_memoize
result = func(*args)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 498, in get_fill_kernel
"fill")
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 163, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 149, in get_elwise_kernel_and_types
keep, options, **kwargs)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 76, in get_elwise_module
options=options, keep=keep, no_extern_c=True)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 291, in __init__
arch, code, cache_dir, include_dirs)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 137, in compile_plain
stderr=stderr.decode("utf-8", "replace"))
pycuda.driver.CompileError: nvcc compilation of /tmp/tmp9_c3rzqr/kernel.cu failed
[command: nvcc --cubin -arch sm_75 -I/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda kernel.cu]
[stderr:
cicc: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /home/cryosparc_user/software/cryosparc/cryosparc2_worker/cryosparc_compute/libs/libstdc++.so.6)
]Ah I see, so the pycuda error is caused by the GLIBC error. A fix will be available in a few hours.
-Harris
2 Likes
Thanks for getting on this quickly. RHEL 7 derivatives are still on GCC 4.8, but we can install GCC 8.3 easily via the software collections. If it’s easier to fix by relying on source scl_source enable devtoolset-8, I would have no problem with that.
I am also getting similar error when doing new local refinement.
Traceback (most recent call last):
File "/usr/local/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py", line 1726, in run_with_except_hook
run_old(*args, **kw)
File "/usr/local/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "cryosparc_worker/cryosparc_compute/engine/newengine.py", line 2109, in cryosparc_compute.engine.newengine.process.work
File "cryosparc_worker/cryosparc_compute/engine/newengine.py", line 2155, in cryosparc_compute.engine.newengine.process.work
File "cryosparc_worker/cryosparc_compute/engine/newengine.py", line 1049, in cryosparc_compute.engine.newengine.EngineThread.compute_resid_pow
File "cryosparc_worker/cryosparc_compute/engine/newcuda_kernels.py", line 4688, in cryosparc_compute.engine.newcuda_kernels.compute_resid_pow
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 392, in cryosparc_compute.engine.cuda_core.context_dependent_memoize.wrapper
File "cryosparc_worker/cryosparc_compute/engine/newcuda_kernels.py", line 4629, in cryosparc_compute.engine.newcuda_kernels.get_compute_resid_pow_kernel
File "/usr/local/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 291, in __init__
arch, code, cache_dir, include_dirs)
File "/usr/local/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/usr/local/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 137, in compile_plain
stderr=stderr.decode("utf-8", "replace"))
pycuda.driver.CompileError: nvcc compilation of /tmp/tmp7jqad3ch/kernel.cu failed
[command: nvcc --cubin -arch sm_61 -I/usr/local/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda kernel.cu]
[stderr:
cicc: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /usr/local/cryosparc2/cryosparc2_worker/cryosparc_compute/libs/libstdc++.so.6)
]
HI @zhenyu_tan, @DanielAsarnow, @nschnick @bcanax,
Can you try running a Patch Motion Correction job and see if you get the same error?
Hi @stephan,
Yes, I’m getting the same error - copying below for completeness. Thank you for trying to fix this so quickly!
[CPU: 222.7 MB] Error occurred while processing J67/imported/FoilHole_11533752_Data_11532319_11532321_20210113_014633_Fractions.mrc
Traceback (most recent call last):
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 429, in context_dependent_memoize
return ctx_dict[cur_ctx][args]
KeyError: <pycuda._driver.Context object at 0x7f70f435e090>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/pipeline.py", line 59, in exec
return self.process(item)
File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 190, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 193, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 195, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 251, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 260, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 117, in cryosparc_compute.jobs.motioncorrection.patchmotion.prepare_movie_for_processing
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py", line 549, in fill
func = elementwise.get_fill_kernel(self.dtype)
File "<decorator-gen-120>", line 2, in get_fill_kernel
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 433, in context_dependent_memoize
result = func(*args)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 498, in get_fill_kernel
"fill")
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 163, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 149, in get_elwise_kernel_and_types
keep, options, **kwargs)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 76, in get_elwise_module
options=options, keep=keep, no_extern_c=True)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 291, in __init__
arch, code, cache_dir, include_dirs)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 137, in compile_plain
stderr=stderr.decode("utf-8", "replace"))
pycuda.driver.CompileError: nvcc compilation of /tmp/tmp2fqwfgiz/kernel.cu failed
[command: nvcc --cubin -arch sm_75 -I/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda kernel.cu]
[stderr:
cicc: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /home/cryosparc_user/software/cryosparc/cryosparc2_worker/cryosparc_compute/libs/libstdc++.so.6)
]
Marking J67/imported/FoilHole_11533752_Data_11532319_11532321_20210113_014633_Fractions.mrc as incomplete and continuing...Thank you! This is very helpful. The re-release should be out within the next few minutes.
1 Like
Hi everyone,
The fixed 3.1 release is available now. You can install it as follows:
cryosparcm update --override
Sorry for the inconvenience this caused.
-Harris
2 Likes
Works fine. Thanks for quickly resolving the issue.
2 Likes
Glad to hear it. Thank you for letting us know!
2 Likes
Seeing this error of several successful helical runs.
Traceback (most recent call last):
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 429, in context_dependent_memoize
return ctx_dict[cur_ctx][args]
KeyError: <pycuda._driver.Context object at 0x7fd4a3d95e10>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pytools/prefork.py", line 50, in call_capture_output
stderr=PIPE)
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'nvcc': 'nvcc'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "cryosparc_worker/cryosparc_compute/run.py", line 84, in cryosparc_compute.run.main
File "cryosparc_worker/cryosparc_compute/jobs/helix/run_refine.py", line 606, in cryosparc_compute.jobs.helix.run_refine.run
File "cryosparc_worker/cryosparc_compute/jobs/helix/run_refine.py", line 140, in cryosparc_compute.jobs.helix.run_refine.generate_cylindrical_density_from_projections
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 963, in cryosparc_compute.engine.engine.process
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 973, in cryosparc_compute.engine.engine.process
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 155, in cryosparc_compute.engine.cuda_core.allocate_gpu
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py", line 549, in fill
func = elementwise.get_fill_kernel(self.dtype)
File "<decorator-gen-13>", line 2, in get_fill_kernel
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 433, in context_dependent_memoize
result = func(*args)
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 498, in get_fill_kernel
"fill")
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 163, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs)
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 149, in get_elwise_kernel_and_types
keep, options, **kwargs)
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 76, in get_elwise_module
options=options, keep=keep, no_extern_c=True)
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 291, in __init__
arch, code, cache_dir, include_dirs)
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 78, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode("utf-8"))
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 50, in preprocess_source
result, stdout, stderr = call_capture_output(cmdline, error_on_nonzero=False)
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pytools/prefork.py", line 227, in call_capture_output
return forker.call_capture_output(cmdline, cwd, error_on_nonzero)
File "/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pytools/prefork.py", line 61, in call_capture_output
% (" ".join(cmdline), e))
pytools.prefork.ExecError: error invoking 'nvcc --preprocess -arch sm_61 -I/home/cryosparc_user/software/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda /tmp/tmpzs84z517.cu --compiler-options -P': [Errno 2] No such file or directory: 'nvcc': 'nvcc'
Tracked this to a CUDA version error. CUDA 11.1 crashes but with CUDA 10.2 all is ok.