Hello,
I am brand new to Cryosparc and still a little too new to Linux and I am having a problem running the T20S tutorial, specifically I am having issues running the Patch motion correction portion of the job.
Environment:
Current cryoSPARC version: v2.15.0
Cuda version export CRYOSPARC_CUDA_PATH="/usr/local/cuda"
CentOS 7
Standalone install
Issue:
Whenever I try and run patch motion correction on the imported movies from the tutorial, I get an error. I think I might have tracked down the problem, as I looked on the Cryosparc troubleshooting guide and I followed the directions for my error, namely the section titled “Job runs but ends unexpectedly with status ‘Failed.’”
Originally, I believed my error was due to the fact our workstations were using CUDA version 11, but I did install CUDA 10.2, which should be compatible. However, when I input the command bin/cryosparcw from the cyrosparc2_worker folder, I get the following message output:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "connect.py", line 22, in print_gpu_list
import pycuda.driver as cudrv
This error led me to a following discussion here in your website (the limit on new users linking discussions has prevented me from copying it here for some reason)
I found there the following set of command to try and reinstall pycuda
cd cryosparc2_worker
cat config.sh
#ensure the `CRYOSPARC_CUDA_PATH` var is set to the parent directory of `/bin/nvcc`
#i.e. `/usr/local/cuda-10.1`
eval $(./bin/cryosparcw env)
which pip
#should be inside the cryosparc2_worker dir,
#which means it's part of cryoSPARC's python environment.
pip uninstall pycuda
pip install "./deps_bundle/python/python_packages/pip_packages/pycuda-2018.1.1.tar.gz" --no-cache-dir
Everything seemed to work fine until I attempted the pip install command, and when I issued it gave me the following error
*** WARNING: nvcc not in path.
*** May need to set CUDA_INC_DIR for installation to succeed.
***************************************************************
*************************************************************
*** I have detected that you have not run configure.py.
*************************************************************
*** Additionally, no global config files were found.
*** I will go ahead with the default configuration.
*** In all likelihood, this will not work out.
***
*** See README_SETUP.txt for more information.
***
*** If the build does fail, just re-run configure.py with the
*** correct arguments, and then retry. Good luck!
*************************************************************
*** HIT Ctrl-C NOW IF THIS IS NOT WHAT YOU WANT
*************************************************************
src/cpp/cuda.hpp:14:18: fatal error: cuda.h: No such file or directory
#include <cuda.h>
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Below is the full log of the actual error I am receiving from the cryosparc joblog Pxx Jyy
================= CRYOSPARCW ======= 2020-10-08 10:32:01.510737 =========
Project P1 Job J12
===========================================================================
========= monitor process now starting main process
MAINPROCESS PID 49491
========= monitor process now waiting for main process
MAIN PID 49491
motioncorrection.run_patch cryosparc2_compute.jobs.jobregister
***************************************************************
Process Process-1:1:
Traceback (most recent call last):
File "/home/cryosparc/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
self.run()
File "/home/cryosparc/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "cryosparc2_compute/jobs/pipeline.py", line 155, in process_work_simple
process_setup(proc_idx) # do any setup you want on a per-process basis
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 80, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.process_setup
File "cryosparc2_compute/engine/__init__.py", line 8, in <module>
from engine import *
File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 4, in init cryosparc2_compute.engine.engine
ImportError: No module named pycuda.driver
Traceback (most recent call last):
File "/home/cryosparc/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/home/cryosparc/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/home/cryosparc/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/home/cryosparc/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/home/cryosparc/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/home/cryosparc/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
send(obj)
IOError: [Errno 32] Broken pipe
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
**** handle exception rc
set status to failed
Traceback (most recent call last):
File "cryosparc2_worker/cryosparc2_compute/run.py", line 85, in cryosparc2_compute.run.main
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 359, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi
AssertionError: Child process with PID 49526 has terminated unexpectedly!
========= main process now complete.
========= monitor process now complete.
I have a feeling there is something very basic I am missing here, I am just not sure what it might be. I hope this information is helpful, please let me know if anymore would be as well. Thank you all ahead of time for your help!
Joe