In short
A user ran a 3D classification job on CryoSPARC v4.4.1, which claims to have succeeded after 1157 iteration, but all output volumes are 0 (every value in the mrc is 0) and all 2,3 million particles are assigned to class 0 (whose volume is also all 0) .
CryoSPARC instance information
- Type: single workstation
- Software version from
cryosparcm status:
Current cryoSPARC version: v4.4.1 - Output of
uname -a && free -gon master node
Linux [REDACTED] 3.10.0-1160.83.1.el7.x86_64 #1 SMP Wed Jan 25 16:41:43 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
total used free shared buff/cache available
Mem: 62 11 0 0 50 50
Swap: 63 15 48
CryoSPARC worker environment
[sander@REDACTED ~]$ bash
[sander@REDACTED ~]$ eval $(/home/sander/cryosparc_test/cryosparc_worker/bin/cryosparcw env)
[sander@REDACTED ~]$ env | grep PATH
NUMBA_CUDA_INCLUDE_PATH=/home/sander/cryosparc_test/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include
LD_LIBRARY_PATH=
PATH=/home/sander/cryosparc_test/cryosparc_worker/bin:/home/sander/cryosparc_test/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/home/sander/cryosparc_test/cryosparc_worker/deps/anaconda/condabin:/home/sander/cryosparc_test/cryosparc_master/bin:/home/sander/cryosparc_test/cryosparc_master/bin:/home/sander/cryosparc_test/cryosparc_master/bin:/home/sander/cryosparc_test/cryosparc_master/bin:/home/sander/cryosparc_test/cryosparc_master/bin:/home/sander/cryosparc_test/cryosparc_master/bin:/home/sander/cryosparc_test/cryosparc_master/bin:/home/sander/cryosparc_test/cryosparc_master/bin:/home/sander/cryosparc_test/cryosparc_master/bin:/home/sander/cryosparc_test/cryosparc_master/bin:/opt/apps/imod/IMOD/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/apps/imod/IMOD/pythonLink:/var/lib/snapd/snap/bin:/home/sander/.local/bin:/home/sander/bin
MODULEPATH=/etc/modulefiles
CRYOSPARC_PATH=/home/sander/cryosparc_test/cryosparc_worker/bin
PYTHONPATH=/home/sander/cryosparc_test/cryosparc_worker
CRYOSPARC_CUDA_PATH=/usr/local/cuda-11.8
[sander@REDACTED ~]$ /sbin/ldconfig -p | grep -i cuda
libpcsamplingutil.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libpcsamplingutil.so
libnvrtc.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvrtc.so.12
libnvrtc.so.11.2 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvrtc.so.11.2
libnvrtc.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvrtc.so
libnvrtc.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvrtc.so
libnvrtc-builtins.so.12.0 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvrtc-builtins.so.12.0
libnvrtc-builtins.so.11.8 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvrtc-builtins.so.11.8
libnvrtc-builtins.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvrtc-builtins.so
libnvrtc-builtins.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvrtc-builtins.so
libnvperf_target.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvperf_target.so
libnvperf_host.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvperf_host.so
libnvjpeg.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvjpeg.so.12
libnvjpeg.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvjpeg.so.11
libnvjpeg.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvjpeg.so
libnvjpeg.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvjpeg.so
libnvblas.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvblas.so.12
libnvblas.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvblas.so.11
libnvblas.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvblas.so
libnvblas.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvblas.so
libnvToolsExt.so.1 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvToolsExt.so.1
libnvToolsExt.so.1 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvToolsExt.so.1
libnvToolsExt.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvToolsExt.so
libnvToolsExt.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvToolsExt.so
libnvJitLink.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvJitLink.so.12
libnvJitLink.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvJitLink.so
libnpps.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnpps.so.12
libnpps.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnpps.so.11
libnpps.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnpps.so
libnpps.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnpps.so
libnppitc.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppitc.so.12
libnppitc.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppitc.so.11
libnppitc.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppitc.so
libnppitc.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppitc.so
libnppisu.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppisu.so.12
libnppisu.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppisu.so.11
libnppisu.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppisu.so
libnppisu.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppisu.so
libnppist.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppist.so.12
libnppist.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppist.so.11
libnppist.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppist.so
libnppist.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppist.so
libnppim.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppim.so.12
libnppim.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppim.so.11
libnppim.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppim.so
libnppim.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppim.so
libnppig.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppig.so.12
libnppig.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppig.so.11
libnppig.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppig.so
libnppig.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppig.so
libnppif.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppif.so.12
libnppif.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppif.so.11
libnppif.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppif.so
libnppif.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppif.so
libnppidei.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppidei.so.12
libnppidei.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppidei.so.11
libnppidei.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppidei.so
libnppidei.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppidei.so
libnppicc.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppicc.so.12
libnppicc.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppicc.so.11
libnppicc.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppicc.so
libnppicc.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppicc.so
libnppial.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppial.so.12
libnppial.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppial.so.11
libnppial.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppial.so
libnppial.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppial.so
libnppc.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppc.so.12
libnppc.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppc.so.11
libnppc.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppc.so
libnppc.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppc.so
libicudata.so.50 (libc6,x86-64) => /lib64/libicudata.so.50
libicudata.so (libc6,x86-64) => /lib64/libicudata.so
libcusparse.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcusparse.so.12
libcusparse.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcusparse.so.11
libcusparse.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcusparse.so
libcusparse.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcusparse.so
libcusolverMg.so.11 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcusolverMg.so.11
libcusolverMg.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcusolverMg.so.11
libcusolverMg.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcusolverMg.so
libcusolverMg.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcusolverMg.so
libcusolver.so.11 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcusolver.so.11
libcusolver.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcusolver.so.11
libcusolver.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcusolver.so
libcusolver.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcusolver.so
libcurand.so.10 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcurand.so.10
libcurand.so.10 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcurand.so.10
libcurand.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcurand.so
libcurand.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcurand.so
libcupti.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcupti.so.12
libcupti.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcupti.so
libcuinj64.so.12.0 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcuinj64.so.12.0
libcuinj64.so.11.8 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcuinj64.so.11.8
libcuinj64.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcuinj64.so
libcuinj64.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcuinj64.so
libcufile_rdma.so.1 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufile_rdma.so.1
libcufile_rdma.so.1 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufile_rdma.so.1
libcufile_rdma.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufile_rdma.so
libcufile_rdma.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufile_rdma.so
libcufile.so.0 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufile.so.0
libcufile.so.0 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufile.so.0
libcufile.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufile.so
libcufile.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufile.so
libcufftw.so.11 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufftw.so.11
libcufftw.so.10 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufftw.so.10
libcufftw.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufftw.so
libcufftw.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufftw.so
libcufft.so.11 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufft.so.11
libcufft.so.10 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufft.so.10
libcufft.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufft.so
libcufft.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufft.so
libcudart.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12
libcudart.so.11.0 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudart.so.11.0
libcudart.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so
libcudart.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudart.so
libcudadebugger.so.1 (libc6,x86-64) => /lib64/libcudadebugger.so.1
libcuda.so.1 (libc6,x86-64) => /lib64/libcuda.so.1
libcuda.so (libc6,x86-64) => /lib64/libcuda.so
libcublasLt.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcublasLt.so.12
libcublasLt.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublasLt.so.11
libcublasLt.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcublasLt.so
libcublasLt.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublasLt.so
libcublas.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so.12
libcublas.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublas.so.11
libcublas.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so
libcublas.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublas.so
libcheckpoint.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcheckpoint.so
libaccinj64.so.12.0 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libaccinj64.so.12.0
libaccinj64.so.11.8 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libaccinj64.so.11.8
libaccinj64.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libaccinj64.so
libaccinj64.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libaccinj64.so
libOpenCL.so.1 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libOpenCL.so.1
libOpenCL.so.1 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libOpenCL.so.1
libOpenCL.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libOpenCL.so
libOpenCL.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libOpenCL.so
[sander@REDACTED ~]$ uname -a
Linux REDACTED 3.10.0-1160.83.1.el7.x86_64 #1 SMP Wed Jan 25 16:41:43 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[sander@REDACTED ~]$ free -g
total used free shared buff/cache available
Mem: 62 11 0 0 50 50
Swap: 63 15 48
[sander@REDACTED ~]$ nvidia-smi
Fri Apr 19 13:25:46 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12 Driver Version: 525.85.12 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:01:00.0 Off | N/A |
| 27% 31C P8 10W / 180W | 370MiB / 8192MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce ... On | 00000000:02:00.0 Off | N/A |
| 27% 27C P8 6W / 180W | 2MiB / 8192MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1227 G /usr/bin/X 87MiB |
| 0 N/A N/A 2611 G /usr/bin/gnome-shell 63MiB |
| 0 N/A N/A 4344 G /usr/bin/X 32MiB |
| 0 N/A N/A 24507 G /usr/bin/X 28MiB |
| 0 N/A N/A 25187 G /usr/bin/gnome-shell 91MiB |
| 0 N/A N/A 26637 G /usr/bin/gnome-shell 61MiB |
+-----------------------------------------------------------------------------+
Issue
A user ran a 3D classification job that claims to have succeeded after 1157 but all output volumes are 0 and all particles are assigned to class 0. I would have expected this job to fail (with an informative error) or at least give a warning.
Running header on any of the output MRC gives back
header ~/Downloads/cryosparc_P18_J208_class_00_01157_volume.mrc
RO image file on unit 1 : /home/sander/Downloads/cryosparc_P18_J208_class_00_01157_volume.mrc Size= 290 K
Number of columns, rows, sections ..... 42 42 42
Map mode .............................. 2 (32-bit real)
Start cols, rows, sects, grid x,y,z ... 0 0 0 42 42 42
Pixel spacing (Angstroms).............. 8.599 8.599 8.599
Cell angles ........................... 90.000 90.000 90.000
Fast, medium, slow axes ............... X Y Z
Origin on x,y,z ....................... 0.000 0.000 0.000
Minimum density ....................... 0.0000
Maximum density ....................... 0.0000
Mean density .......................... 0.0000
tilt angles (original,current) ........ 0.0 0.0 0.0 0.0 0.0 0.0
Space group,# extra bytes,idtype,lens . 0 0 0 0
0 Titles :
Job info
- Name: 3D classification
- Inputs for job:
- 1 paricle stack with 2,310,789 particles
- 4 different initial volumes coming from a heterogeneous refinement job
- No solvent maks
- 1 Focus mask (quite small and offcenter if this makes a difference)
- 10 non-default parameters:
class3D_N_K: 4class3D_target_res: 20class3D_rms_conv_thresh: 0.1class3D_init_res: 10class3D_init_mode: inputclass3D_filter_hp_res: 6class3D_mask_thresh_factor: 0.1class3D_mask_near_ang: 20class3D_mask_far_ang: 31class3D_force_hard_class: true
Joblog
can’t be added due to character limit for creating a topic and no text format that is allowed for uploading files. Does contain the following (unique) warnings:
home/sander/cryosparc_test/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/dispatcher.py:538: NumbaPerformanceWarning: Grid size 1 will likely result in GPU under-utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
/home/sander/cryosparc_test/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py:2919: UserWarning: NVRTC log messages whilst compiling kernel:
kernel(963): warning #177-D: variable "Nb2p1" was declared but never referenced
\00
warnings.warn(msg)
/home/sander/cryosparc_test/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/dispatcher.py:538: NumbaPerformanceWarning: Grid size 3 will likely result in GPU under-utilization due to low occupancy.
warn(NumbaPerformanceWarning(msg))
The next one occurs a lot (I assume this might be the main issue)
/home/sander/cryosparc_test/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/process.py:108: RuntimeWarning: divide by zero encountered in log
self._target(*self._args, **self._kwargs)
/home/sander/cryosparc_test/cryosparc_worker/cryosparc_compute/plotutil.py:1301: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). Consider using `matplotlib.pyplot.close()`.
fig = plt.figure(figsize=figsize)
/home/sander/cryosparc_test/cryosparc_worker/cryosparc_compute/plotutil.py:1320: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). Consider using `matplotlib.pyplot.close()`.
fig = plt.figure(figsize=figsize)
/home/sander/cryosparc_test/cryosparc_worker/cryosparc_compute/plotutil.py:177: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). Consider using `matplotlib.pyplot.close()`.
fig = plt.figure(figsize=figsize)
/home/sander/cryosparc_test/cryosparc_worker/cryosparc_compute/plotutil.py:1169: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). Consider using `matplotlib.pyplot.close()`.
fig = plt.figure(figsize=figsize)
/home/sander/cryosparc_test/cryosparc_worker/cryosparc_compute/plotutil.py:775: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). Consider using `matplotlib.pyplot.close()`.
fig = plt.figure(figsize=figsize)
/home/sander/cryosparc_test/cryosparc_worker/cryosparc_compute/plotutil.py:1441: UserWarning:
The connectionstyle keyword argument is not applicable when drawing edges
with LineCollection.
To make this warning go away, either specify `arrows=True` to
force FancyArrowPatches or use the default value for connectionstyle.
Note that using FancyArrowPatches may be slow for large graphs.
nx.draw_networkx_edges(G, pos, edgelist=cross_edges, ax=ax, width=widths[cross_ids], edge_color=colors[cross_ids], alpha=alphas[cross_ids], connectionstyle="arc3,rad=1")
/home/sander/cryosparc_test/cryosparc_worker/cryosparc_compute/plotutil.py:1442: UserWarning:
The connectionstyle keyword argument is not applicable when drawing edges
with LineCollection.
To make this warning go away, either specify `arrows=True` to
force FancyArrowPatches or use the default value for connectionstyle.
Note that using FancyArrowPatches may be slow for large graphs.
nx.draw_networkx_edges(G, pos, edgelist=self_edges, ax=ax, width=widths[self_ids], edge_color=colors[self_ids], alpha=alphas[self_ids], connectionstyle="arc3,rad=1")
Similar issues
looks similar, except that all 4 import volumes are different (came from a Heterogeneous refinement job)
Please let me know if you need any more info or data from me to help track down this issue!
(It would have also been nice to be able to upload a txt file to attach the joblog to this topic instead of running into the max character count limit for posting topics)
)