I recently ran updates on our Ubuntu 18.04.2 cryosparc worker nodes, which updated the Nvidia drivers from version 430 to version 440. Now cryosparc doesn’t work any more (with either the Cuda 9.1 or the Cuda 9.2 libraries). Jobs which ran prior to the update now fail with this error message:
[CPU: 244.37 GB] Traceback (most recent call last): File "cryosparc2_master/cryosparc2_compute/run.py", line 78, in cryosparc2_compute.run.main File "cryosparc2_compute/jobs/select2D/run.py", line 262, in run _, particles_to_plot = mrc.read_mrc(os.path.join(proj_dir_abs, particles_dset_exclude.data['blob/path'])) File "cryosparc2_compute/blobio/mrc.py", line 114, in read_mrc data = read_mrc_data(file_obj, header, start_page, end_page, out) File "cryosparc2_compute/blobio/mrc.py", line 77, in read_mrc_data data = n.fromfile(file_obj, dtype=dtype, count= num_pages * ny * nx).reshape(num_pages, ny, nx) MemoryError
I’m going to try and revert the nvidia drivers, but am a loss for why an nvidia kernel driver updated would cause cryosparc to stop working.
In general, CryoSPARC seems to be very brittle with respect to Nvidia drivers/Cuda libraries. Can someone tell me which versions of the Nvidia drivers / Cuda libraries have been tested to work with CryoSPARC?