Hi, I am getting the following error when launching an homogeneous refinement in a fresh updated v4.0.1 instance. Previous jobs up to this point worked without problem (from importing raw frames).
[CPU: 1.21 GB]
====== Starting Refinement Iterations ======
[CPU: 1.21 GB]
----------------------------- Start Iteration 0
[CPU: 1.21 GB]
Using Max Alignment Radius 18.057 (30.000A)
[CPU: 1.21 GB]
Auto batchsize: 12300 in each split
[CPU: 1.60 GB]
-- THR 1 BATCH 500 NUM 6000 TOTAL 4.7805840 ELAPSED 17.061522 --
[CPU: 1.67 GB]
Processed 24600.000 images in 17.592s.
[CPU: 1.87 GB]
Computing FSCs...
[CPU: 1.87 GB]
Using full box size 256, downsampled box size 128, with low memory mode disabled.
[CPU: 1.87 GB]
Computing FFTs on GPU.
[CPU: 977.2 MB]
Traceback (most recent call last):
File "cryosparc_worker/cryosparc_compute/run.py", line 93, in cryosparc_compute.run.main
File "cryosparc_worker/cryosparc_compute/jobs/refine/newrun.py", line 638, in cryosparc_compute.jobs.refine.newrun.run_homo_refine
File "cryosparc_worker/cryosparc_compute/engine/gsigproc.py", line 463, in cryosparc_compute.engine.gsigproc.compute_all_fscs
File "cryosparc_worker/cryosparc_compute/engine/gsigproc.py", line 506, in cryosparc_compute.engine.gsigproc.compute_all_fscs
File "cryosparc_worker/cryosparc_compute/engine/gsigproc.py", line 316, in cryosparc_compute.engine.gsigproc.GPUFSC.edt
File "cryosparc_worker/cryosparc_compute/engine/newcuda_kernels.py", line 6056, in cryosparc_compute.engine.newcuda_kernels.euclidian_distance_transform
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 416, in cryosparc_compute.engine.cuda_core.context_dependent_memoize.wrapper
File "cryosparc_worker/cryosparc_compute/engine/newcuda_kernels.py", line 6036, in cryosparc_compute.engine.newcuda_kernels.get_edt_kernels
File "/scicore/home/engel0006/GROUP/pool-engel/soft/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 291, in __init__
arch, code, cache_dir, include_dirs)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/scicore/home/engel0006/GROUP/pool-engel/soft/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 137, in compile_plain
stderr=stderr.decode("utf-8", "replace"))
pycuda.driver.CompileError: nvcc compilation of /scratch/tmp0x9q70ry/kernel.cu failed
[command: nvcc --cubin -arch sm_86 -I/scicore/home/engel0006/GROUP/pool-engel/soft/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda kernel.cu]
[stderr:
cicc: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /scicore/home/engel0006/GROUP/pool-engel/soft/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/./libLerc.so)
cicc: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /scicore/home/engel0006/GROUP/pool-engel/soft/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/./libLerc.so)
]
Everything used to work fine before the update. Is there any fix for this?
Thank you!
Info about master node:
Linux worker08.cluster.bc2.ch 3.10.0-1160.66.1.el7.x86_64 #1 SMP Wed May 18 16:02:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
total used free shared buff/cache available
Mem: 2015 76 56 0 1882 1936
Swap: 7 0 7
Info about worker node (same as master):
diogori@worker08:cryosparc_worker$ eval $(bin/cryosparcw env)
diogori@worker08:cryosparc_worker$ echo $CRYOSPARC_CUDA_PATH
/scicore/soft/apps/CUDA/11.3.1
diogori@worker08:cryosparc_worker$ ${CRYOSPARC_CUDA_PATH}/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:15:13_PDT_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0
diogori@worker08:cryosparc_worker$ python -c "import pycuda.driver; print(pycuda.driver.get_version())"
(10, 1, 0)
diogori@worker08:cryosparc_worker$ uname -a && free -g && nvidia-smi
Linux worker08.cluster.bc2.ch 3.10.0-1160.66.1.el7.x86_64 #1 SMP Wed May 18 16:02:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
total used free shared buff/cache available
Mem: 2015 76 56 0 1882 1936
Swap: 7 0 7
Sat Oct 8 13:16:20 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.43.04 Driver Version: 515.43.04 CUDA Version: 11.7 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A40 On | 00000000:23:00.0 Off | 0 |
| 0% 34C P0 35W / 300W | 743MiB / 46068MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA A40 On | 00000000:41:00.0 Off | 0 |
| 0% 34C P0 32W / 300W | 22MiB / 46068MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 NVIDIA A40 On | 00000000:A1:00.0 Off | 0 |
| 0% 35C P8 33W / 300W | 22MiB / 46068MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 NVIDIA A40 On | 00000000:C1:00.0 Off | 0 |
| 0% 33C P8 31W / 300W | 22MiB / 46068MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 4321 G /usr/bin/X 23MiB |
| 0 N/A N/A 29801 G .../napari/napari/bin/python 4MiB |
| 0 N/A N/A 31419 G ...himera-1.16/bin/python2.7 70MiB |
| 0 N/A N/A 36338 G ...himera-1.16/bin/python2.7 122MiB |
| 0 N/A N/A 92685 G ...himera-1.16/bin/python2.7 520MiB |
| 1 N/A N/A 4321 G /usr/bin/X 22MiB |
| 2 N/A N/A 4321 G /usr/bin/X 22MiB |
| 3 N/A N/A 4321 G /usr/bin/X 22MiB |
+-----------------------------------------------------------------------------+