2d classification-cuda error

Traceback (most recent call last):
File “/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 2306, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 136, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 137, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
File “cryosparc_master/cryosparc_compute/jobs/class2D/newrun.py”, line 640, in cryosparc_master.cryosparc_compute.jobs.class2D.newrun.class2D_engine_run.work
File “cryosparc_master/cryosparc_compute/engine/newengine.py”, line 1399, in cryosparc_master.cryosparc_compute.engine.newengine.EngineThread.compute_resid_pow
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 382, in cryosparc_master.cryosparc_compute.gpu.gpucore.EngineBaseThread.ensure_allocated
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/devices.py”, line 232, in _require_cuda_context
return fn(*args, **kws)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/api.py”, line 189, in pinned_array
buffer = current_context().memhostalloc(bytesize)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py”, line 1378, in memhostalloc
return self.memory_manager.memhostalloc(bytesize, mapped, portable, wc)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py”, line 889, in memhostalloc
pointer = allocator()
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py”, line 884, in allocator
return driver.cuMemHostAlloc(size, flags)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py”, line 348, in safe_cuda_api_call
return self._check_cuda_python_error(fname, libfn(*args))
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py”, line 408, in _check_cuda_python_error
raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [CUresult.CUDA_ERROR_INVALID_VALUE] Call to cuMemHostAlloc results in CUDA_ERROR_INVALID_VALUE

@yang1 what check point did this happen / how long was the job running ?

before the zero iteration,

License is valid.

Launching job on lane default target spgpu …

Running job on master node hostname spgpu
[CPU: 91.1 MB Avail: 31.65 GB]

Job J35 Started
[CPU: 91.1 MB Avail: 31.65 GB]

Master running v4.7.0, worker running v4.7.0
[CPU: 91.4 MB Avail: 31.65 GB]

Working in directory: /opt/cryosparc_containers/CS-exocyst-snare-shuimu/J35
[CPU: 91.4 MB Avail: 31.65 GB]

Running on lane default
[CPU: 91.4 MB Avail: 31.65 GB]

Resources allocated:
[CPU: 91.4 MB Avail: 31.64 GB]

Worker: spgpu
[CPU: 91.4 MB Avail: 31.64 GB]

CPU : [0, 1, 2, 3]
[CPU: 91.4 MB Avail: 31.64 GB]

GPU : [0]
[CPU: 91.4 MB Avail: 31.64 GB]

RAM : [0, 1, 2]
[CPU: 91.4 MB Avail: 31.64 GB]

SSD : False
[CPU: 91.4 MB Avail: 31.64 GB]


[CPU: 91.4 MB Avail: 31.64 GB]

Importing job module for job type class_2D_new…
[CPU: 273.5 MB Avail: 31.55 GB]

Job ready to run
[CPU: 273.5 MB Avail: 31.55 GB]


[CPU: 469.3 MB Avail: 31.17 GB]

Using random seed of 464557121
[CPU: 469.3 MB Avail: 31.17 GB]

Using pixel size from location/micrograph_psize_A for duplicate removal
[CPU: 469.3 MB Avail: 31.17 GB]

Loading a ParticleStack with 263780 items…
[CPU: 580.0 MB Avail: 28.81 GB]

Done.
[CPU: 580.0 MB Avail: 28.81 GB]

Windowing particles
[CPU: 580.0 MB Avail: 28.81 GB]

Done.
[CPU: 598.6 MB Avail: 28.79 GB]

Using 80 classes.
[CPU: 598.8 MB Avail: 28.79 GB]

Computing 2D class averages:
[CPU: 598.8 MB Avail: 28.79 GB]

Volume Size: 64 (voxel size 8.24A)
[CPU: 598.8 MB Avail: 28.79 GB]

Zeropadded Volume Size: 128
[CPU: 598.8 MB Avail: 28.79 GB]

Data Size: 64 (pixel size 8.24A)
[CPU: 598.8 MB Avail: 28.79 GB]

Using Reconstruction Resolution: 6.00A (31.0 radius)
[CPU: 598.8 MB Avail: 28.79 GB]

Using Alignment Resolution: 6.00A (31.0 radius)
[CPU: 598.8 MB Avail: 28.79 GB]

Windowing only corners of 2D classes at each iteration.
[CPU: 598.8 MB Avail: 28.79 GB]

Using random seed for initialization of 464557121
[CPU: 601.9 MB Avail: 28.79 GB]

Done in 0.229s.
[CPU: 643.0 MB Avail: 28.25 GB]

Start of Iteration 0
[CPU: 643.0 MB Avail: 28.25 GB]

[CPU: 918.7 MB Avail: 27.43 GB]

Traceback (most recent call last):

@yang1 Does the error persist if you append the line

export CRYOSPARC_NO_PAGELOCK="true"

to the file

/opt/cryosparc/cryosparc_worker/config.sh

(guide)?