Traceback (most recent call last):
File “/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 2306, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 136, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 137, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
File “cryosparc_master/cryosparc_compute/jobs/class2D/newrun.py”, line 640, in cryosparc_master.cryosparc_compute.jobs.class2D.newrun.class2D_engine_run.work
File “cryosparc_master/cryosparc_compute/engine/newengine.py”, line 1399, in cryosparc_master.cryosparc_compute.engine.newengine.EngineThread.compute_resid_pow
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 382, in cryosparc_master.cryosparc_compute.gpu.gpucore.EngineBaseThread.ensure_allocated
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/devices.py”, line 232, in _require_cuda_context
return fn(*args, **kws)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/api.py”, line 189, in pinned_array
buffer = current_context().memhostalloc(bytesize)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py”, line 1378, in memhostalloc
return self.memory_manager.memhostalloc(bytesize, mapped, portable, wc)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py”, line 889, in memhostalloc
pointer = allocator()
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py”, line 884, in allocator
return driver.cuMemHostAlloc(size, flags)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py”, line 348, in safe_cuda_api_call
return self._check_cuda_python_error(fname, libfn(*args))
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py”, line 408, in _check_cuda_python_error
raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [CUresult.CUDA_ERROR_INVALID_VALUE] Call to cuMemHostAlloc results in CUDA_ERROR_INVALID_VALUE
@yang1 what check point did this happen / how long was the job running ?
before the zero iteration,
License is valid.
Launching job on lane default target spgpu …
Running job on master node hostname spgpu
[CPU: 91.1 MB Avail: 31.65 GB]
Job J35 Started
[CPU: 91.1 MB Avail: 31.65 GB]
Master running v4.7.0, worker running v4.7.0
[CPU: 91.4 MB Avail: 31.65 GB]
Working in directory: /opt/cryosparc_containers/CS-exocyst-snare-shuimu/J35
[CPU: 91.4 MB Avail: 31.65 GB]
Running on lane default
[CPU: 91.4 MB Avail: 31.65 GB]
Resources allocated:
[CPU: 91.4 MB Avail: 31.64 GB]
Worker: spgpu
[CPU: 91.4 MB Avail: 31.64 GB]
CPU : [0, 1, 2, 3]
[CPU: 91.4 MB Avail: 31.64 GB]
GPU : [0]
[CPU: 91.4 MB Avail: 31.64 GB]
RAM : [0, 1, 2]
[CPU: 91.4 MB Avail: 31.64 GB]
SSD : False
[CPU: 91.4 MB Avail: 31.64 GB]
[CPU: 91.4 MB Avail: 31.64 GB]
Importing job module for job type class_2D_new…
[CPU: 273.5 MB Avail: 31.55 GB]
Job ready to run
[CPU: 273.5 MB Avail: 31.55 GB]
[CPU: 469.3 MB Avail: 31.17 GB]
Using random seed of 464557121
[CPU: 469.3 MB Avail: 31.17 GB]
Using pixel size from location/micrograph_psize_A for duplicate removal
[CPU: 469.3 MB Avail: 31.17 GB]
Loading a ParticleStack with 263780 items…
[CPU: 580.0 MB Avail: 28.81 GB]
Done.
[CPU: 580.0 MB Avail: 28.81 GB]
Windowing particles
[CPU: 580.0 MB Avail: 28.81 GB]
Done.
[CPU: 598.6 MB Avail: 28.79 GB]
Using 80 classes.
[CPU: 598.8 MB Avail: 28.79 GB]
Computing 2D class averages:
[CPU: 598.8 MB Avail: 28.79 GB]
Volume Size: 64 (voxel size 8.24A)
[CPU: 598.8 MB Avail: 28.79 GB]
Zeropadded Volume Size: 128
[CPU: 598.8 MB Avail: 28.79 GB]
Data Size: 64 (pixel size 8.24A)
[CPU: 598.8 MB Avail: 28.79 GB]
Using Reconstruction Resolution: 6.00A (31.0 radius)
[CPU: 598.8 MB Avail: 28.79 GB]
Using Alignment Resolution: 6.00A (31.0 radius)
[CPU: 598.8 MB Avail: 28.79 GB]
Windowing only corners of 2D classes at each iteration.
[CPU: 598.8 MB Avail: 28.79 GB]
Using random seed for initialization of 464557121
[CPU: 601.9 MB Avail: 28.79 GB]
Done in 0.229s.
[CPU: 643.0 MB Avail: 28.25 GB]
Start of Iteration 0
[CPU: 643.0 MB Avail: 28.25 GB]
[CPU: 918.7 MB Avail: 27.43 GB]
Traceback (most recent call last):