I’m on centos7, trying to run cryosparc3 with cuda 10.
I had a lot of trouble trying to update my standalone 2.15 installation to 3.0. The worker didn’t update, and when I tried to update worker manually, cuda stopped working. Eventually just moved the database archive (as well as directly copy pasted the unarchived database) elsewhere and deleted everything in the cryosparc folder. I reinstalled everything inside the same cryosparc folder again and then replaced the contents of cryosparc_database with those from what I had before (previously cryosparc2_database). Now when I try to do things (I’ve tried particle extraction and 2d classification so far), I get a brief popup bar at the top of the screen that reads:
[Errno2] No such file or directory: '/home/user/MySoftwares/cryosparc/cryosparc2_worker/bin/cryosparcw': '/home/user/MySoftwares/cryosparc/cryosparc2_worker/bin/cryosparcw'
My worker directory no longer has a 2 in it since I reinstalled. Is there a way to get the program to try to find cryosparcw inside of cryosparc_worker instead of non-existent cryosparc2_worker?
Another thing I notice is there is no config.sh inside the worker directory. The installation did seem to potentially end prematurely when it had trouble creating my cryosparc user account (that seems to happen while trying to reinstall inside this cryosparc directory, but replacing the database content with my old database makes the cryosparc user account available to log in with again).
Hi @afrangaj, this indicates that you need to reconnect your worker so that the master module registers the new path in the database, you can do this from the cryosparc_worker directory with the bin/cryosparcw connect command.
I encountered the exact same issues as @afrangaj above (and had a similar situation). I was updating from version 2.15 to 3, and the installation was unsuccessful through cryosparcm update, so I ended up doing a fresh reinstall of cryoSPARC v3.0 after backing up my database.
I had to perform two steps to fix the worker node issue:
Go into the cryosparc_worker directory and run ./install.sh --license <license_id> --cudapath <cuda_path>, where <license_id> and <cuda_path> are your license and path to cuda (e.g. /usr/local/cuda). This step will generate the missing config.sh file.
Go into the cryosparc_worker/bin/ directory and run ./cryosparcw connect --worker localhost --master localhost --port <port_number> --ssdpath <ssd_path> --update
@nfrasser@mchakra Thank you both for your advice. I could generate the config.sh in worker as mchakra said and reconnecting the worker seemed to work as well. However now an earlier problem is popping up, where jobs fail with:
File "/u/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 1711, in run_with_except_hook
run_old(*args, **kw)
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 129, in cryosparc_compute.engine.cuda_core.GPUThread.run
File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 130, in cryosparc_compute.engine.cuda_core.GPUThread.run
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 997, in cryosparc_compute.engine.engine.process.work
File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 106, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File "cryosparc_worker/cryosparc_compute/engine/gfourier.py", line 33, in cryosparc_compute.engine.gfourier.fft2_on_gpu_inplace
File "/u/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/fft.py", line 102, in __init__
capability = misc.get_compute_capability(misc.get_current_device())
File "/u/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/misc.py", line 254, in get_current_device
return drv.Device(cuda.cudaGetDevice())
File "/u/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/cudart.py", line 767, in cudaGetDevice
cudaCheckStatus(status)
File "/u/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/cudart.py", line 565, in cudaCheckStatus
raise e
skcuda.cudart.cudaErrorInsufficientDriver
I’m on driver 410.79, and nvidia-smi seems to work and still asserts I’m on cuda 10. Is there a new higher driver minimum for c3?
After installing driver 455.45.01 and cuda 10.1, I started cryosparc again and thinks look like they are running so far. Thanks for all the help. Looking forward to using the new features.