The Metadata Log:
================= CRYOSPARCW ======= 2024-05-07 16:23:43.531641 =========
Project P83 Job J3
Master syrah.qb3.berkeley.edu Port 39002
===========================================================================
MAIN PROCESS PID 8006
========= now starting main process at 2024-05-07 16:23:43.532445
motioncorrection.run_patch cryosparc_compute.jobs.jobregister
MONITOR PROCESS PID 8008
========= monitor process now waiting for main process
========= sending heartbeat at 2024-05-07 16:23:45.432702
/home/cryosparc2/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/cryosparc2/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
return self._float_to_str(self.smallest_subnormal)
/home/cryosparc2/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/cryosparc2/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
return self._float_to_str(self.smallest_subnormal)
***************************************************************
Running job on hostname %s syrah.qb3.berkeley.edu
Allocated Resources : {'fixed': {'SSD': False}, 'hostname': 'syrah.qb3.berkeley.edu', 'lane': 'default', 'lane_type': 'node', 'license': True, 'licenses_acquired': 1, 'slots': {'CPU': [2, 3, 4, 5, 6, 7], 'GPU': [1], 'RAM': [1, 2]}, 'target': {'cache_path': '/mount/ssd/cryosparc2_cache', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 11554717696, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 1, 'mem': 11554717696, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 2, 'mem': 11554717696, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 3, 'mem': 11554717696, 'name': 'NVIDIA GeForce RTX 2080 Ti'}], 'hostname': 'syrah.qb3.berkeley.edu', 'lane': 'default', 'name': 'syrah.qb3.berkeley.edu', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], 'GPU': [0, 1, 2, 3], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}, 'ssh_str': 'cryosparc2', 'title': 'Worker node syrah.qb3.berkeley.edu', 'type': 'node', 'worker_bin_path': '/home/cryosparc2/cryosparc2/cryosparc2_worker/bin/cryosparcw'}}
Process Process-1:
Traceback (most recent call last):
File "/home/cryosparc2/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/cryosparc2/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/cryosparc2/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/pipeline.py", line 199, in process_work_simple
process_setup(proc_idx) # do any setup you want on a per-process basis
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 115, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.process_setup
File "cryosparc_master/cryosparc_compute/gpu/gpucore.py", line 47, in cryosparc_master.cryosparc_compute.gpu.gpucore.initialize
File "/home/cryosparc2/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 3216, in get_version
return driver.get_version()
File "/home/cryosparc2/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 461, in get_version
version = driver.cuDriverGetVersion()
File "/home/cryosparc2/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 292, in __getattr__
self.ensure_initialized()
File "/home/cryosparc2/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 254, in ensure_initialized
self.cuInit(0)
File "/home/cryosparc2/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 348, in safe_cuda_api_call
return self._check_cuda_python_error(fname, libfn(*args))
File "cuda/cuda.pyx", line 11325, in cuda.cuda.cuInit
File "cuda/ccuda.pyx", line 17, in cuda.ccuda.cuInit
File "cuda/_cuda/ccuda.pyx", line 2353, in cuda._cuda.ccuda._cuInit
RuntimeError: Function "cuInit" not found
========= sending heartbeat at 2024-05-07 16:23:55.449315
========= sending heartbeat at 2024-05-07 16:24:05.469318
========= sending heartbeat at 2024-05-07 16:24:15.493143
/home/cryosparc2/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/home/cryosparc2/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
***************************************************************
========= main process now complete at 2024-05-07 16:24:24.817011
========= sending heartbeat at 2024-05-07 16:24:25.516081
========= heartbeat failed at 2024-05-07 16:24:25.529758:
========= main process now complete at 2024-05-07 16:24:35.538837.
========= monitor process now complete at 2024-05-07 16:24:35.547084.