NU Refinement in v4.4 AssertionError: "refine_relax_symmetry" parameter not available

3D Refinement
1

Hi All,

I just updated cryosparc to v4.4 and tried to run NU refinement job but got an error. I did not see any parameter for refine_relax_symmetry in NU refinement. I also restarted cryosparc by ‘cryosparcm restart’ but got the same error again. Did I miss something? Thanks!

Master running v4.4.0, worker running v4.4.0
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/refine/newrun.py”, line 237, in cryosparc_master.cryosparc_compute.jobs.refine.newrun.run_homo_refine
File “/data/cryosparc/cryosparc_worker/cryosparc_compute/util/init.py”, line 59, in getitem
assert key in self, “”""%s" parameter not available. This job may be stale.
AssertionError: “refine_relax_symmetry” parameter not available. This job may be stale.
Please try the following steps in order until the issue is resolved:
1. Build a new job and try again.
2. Run the following via command line: cryosparcm cli “refresh_job_types()”.
3. Ensure no orphaned cryoSPARC processes are running in the background.

2

Did you try this?

Run the following via command line: cryosparcm cli “refresh_job_types()”.

3

Was the failing job originally built in a CryoSPARC version older than 4.4?

4

Yes, I restarted from an older job. Maybe I need to try a completely new job? When I tried the following but got another error:

cryosparcm cli “refresh_job_types()”
-bash: syntax error near unexpected token `(’

5

Please try that and let us know if it resolves the problem.

6

Did you type the command or copy paste it? Try typing it if you pasted it in.

7

Unfortunately got a new error when I started a completely new NU Refinement job:

Traceback (most recent call last):
  File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 3007, in add_ptx
    driver.cuLinkAddData(self.handle, input_ptx, ptx, len(ptx),
  File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 352, in safe_cuda_api_call
    return self._check_cuda_python_error(fname, libfn(*args))
  File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 412, in _check_cuda_python_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [CUresult.CUDA_ERROR_UNSUPPORTED_PTX_VERSION] Call to cuLinkAddData results in CUDA_ERROR_UNSUPPORTED_PTX_VERSION

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 2192, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc_master/cryosparc_compute/gpu/gpucore.py", line 117, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
  File "cryosparc_master/cryosparc_compute/gpu/gpucore.py", line 118, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 1123, in cryosparc_master.cryosparc_compute.engine.engine.process.work
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 350, in cryosparc_master.cryosparc_compute.engine.engine.EngineThread.compute_resid_pow
  File "/data/cryosparc/cryosparc_worker/cryosparc_compute/gpu/gpuarray.py", line 110, in fill
    from .elementwise.fill import fill
  File "/data/cryosparc/cryosparc_worker/cryosparc_compute/gpu/elementwise/fill.py", line 24, in <module>
    def fill(arr, x, out):
  File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/np/ufunc/decorators.py", line 203, in wrap
    guvec.add(fty)
  File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/np/ufunc/deviceufunc.py", line 475, in add
    kernel = self._compile_kernel(fnobj, sig=tuple(outertys))
  File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/vectorizers.py", line 241, in _compile_kernel
    return cuda.jit(sig)(fnobj)
  File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/decorators.py", line 133, in _jit
    disp.compile(argtypes)
  File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/dispatcher.py", line 928, in compile
    kernel.bind()
  File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/dispatcher.py", line 207, in bind
    self._codelibrary.get_cufunc()
  File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/codegen.py", line 184, in get_cufunc
    cubin = self.get_cubin(cc=device.compute_capability)
  File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/codegen.py", line 159, in get_cubin
    linker.add_ptx(ptx.encode())
  File "/data/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numba/cuda/cudadrv/driver.py", line 3010, in add_ptx
    raise LinkerError("%s\n%s" % (e, self.error_log))
numba.cuda.cudadrv.driver.LinkerError: [CUresult.CUDA_ERROR_UNSUPPORTED_PTX_VERSION] Call to cuLinkAddData results in CUDA_ERROR_UNSUPPORTED_PTX_VERSION
ptxas application ptx input, line 9; fatal   : Unsupported .version 7.8; current version is '7.5'
8

I would suggest trying the refresh job types command, typing the command directly at the terminal prompt, not pasting it

1 Like
9

Is the nvidia driver version ≥ 520?

10

NVIDIA-SMI 495.29.05 Driver Version: 495.29.05 CUDA Version: 11.5

11

I typed the command and the screen output a lot of text. I attached a couple of lines:

cryosparcm cli “refresh_job_types()”
[{‘contains’: [{‘description’: ‘Import one or more movies in MRC, TIFF or EER format.’, ‘develop_only’: False, ‘hidden’: False, ‘import’: True, ‘input_slot_groups’: , ‘is_interactive’: False, ‘is_lightweight’: True, ‘name’: ‘import_movies’,…

I restarted my new NU refinement job and still had the above new error.

12

This driver version is not compatible with CryoSPARC v4.4. Please update the driver to v520 (or higher).

13

After updating the GPU driver version (535.129.03), the problem of CUDA_ERROR_UNSUPPORTED_PTX_VERSION was solved! Again the NU refinement has to be newly built in v4.4.
Thank all for the help!

3 Likes