The error messages include the following:
UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 10020). Please update your GPU driver by downloading and installing a new version from the URL: Official Drivers | NVIDIA Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at /opt/conda/conda-bld/pytorch_1659484683044/work/c10/cuda/CUDAFunctions.cpp:109.)
[CPU: 226.2 MB] CudaWarning: The NVIDIA driver on your system is too old (found version 10020). Please update your GPU driver by downloading and installing a new version from the URL: Official Drivers | NVIDIA Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver.
[CPU: 226.2 MB] Falling back to CPU.
[CPU: 226.2 MB] # Using device=0 with cuda=False
> FileNotFoundError: [Errno 2] No such file or directory: ‘/mnt/disks/data-1/20240222_cryosparc_Screening/P20/J54/image_list_train.txt’
[CPU: 226.2 MB] Traceback (most recent call last):
File “cryosparc_worker/cryosparc_compute/run.py”, line 85, in cryosparc_compute.run.main
File “/home/xxxxxlab/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py”, line 357, in run_topaz_wrapper_train
assert len(glob.glob(os.path.join(model_dir, ‘*’))) > 0, “Training failed, no models were created.”
AssertionError: Training failed, no models were created.
I am using cryosparc v3.1.1 and I think topaz v0.2.5. Python is version 2.7.17, and nvidida driver is version 440.33.01. CUDA is version 10.2.
I understand that more recent versions of these exist and are recommended, but these versions work with all our other software at the moment and nothing has been updated since topaz was last used successfully on this workstation. I’m hesitant to update the drivers since we are also using this workstation for other projects currently, and if something else stops working as a result, I don’t know that we’ll be able to address it quickly.
Is there something else that might be wrong that is causing the image list train file not to be found? I don’t understand how it could be the version if nothing has changed since it was last used.
Thank you.