Patch motion correction multi error

[gabor_oroszlan]

Hi All,

My motion correction failed with all of my movies, the same error message shows during run for each movies. I paste below. Thank you for your help.
"
Error occurred while processing J1/imported/hcmvpol_100_001_X-1Y-1-0.tif
Traceback (most recent call last):
File “/nfs/userdocs/lab-jabraham/ja-cryosparc/sw/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py”, line 429, in context_dependent_memoize
return ctx_dict[cur_ctx][args]
KeyError: <pycuda._driver.Context object at 0x7f53e2212cf0>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/nfs/userdocs/lab-jabraham/ja-cryosparc/sw/cryosparc_worker/cryosparc_compute/jobs/pipeline.py”, line 59, in exec
return self.process(item)
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 190, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 193, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 195, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 255, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 264, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 121, in cryosparc_compute.jobs.motioncorrection.patchmotion.prepare_movie_for_processing
File “/nfs/userdocs/lab-jabraham/ja-cryosparc/sw/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py”, line 549, in fill
func = elementwise.get_fill_kernel(self.dtype)
File “”, line 2, in get_fill_kernel
File “/nfs/userdocs/lab-jabraham/ja-cryosparc/sw/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py”, line 433, in context_dependent_memoize
result = func(*args)
File “/nfs/userdocs/lab-jabraham/ja-cryosparc/sw/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 498, in get_fill_kernel
“fill”)
File “/nfs/userdocs/lab-jabraham/ja-cryosparc/sw/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 163, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs)
File “/nfs/userdocs/lab-jabraham/ja-cryosparc/sw/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 149, in get_elwise_kernel_and_types
keep, options, **kwargs)
File “/nfs/userdocs/lab-jabraham/ja-cryosparc/sw/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 76, in get_elwise_module
options=options, keep=keep, no_extern_c=True)
File “/nfs/userdocs/lab-jabraham/ja-cryosparc/sw/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 291, in init
arch, code, cache_dir, include_dirs)
File “/nfs/userdocs/lab-jabraham/ja-cryosparc/sw/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File “/nfs/userdocs/lab-jabraham/ja-cryosparc/sw/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 78, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode(“utf-8”))
File “/nfs/userdocs/lab-jabraham/ja-cryosparc/sw/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 55, in preprocess_source
cmdline, stderr=stderr)
pycuda.driver.CompileError: nvcc preprocessing of /tmp/tmp5brevv9b.cu failed
[command: nvcc --preprocess -arch sm_75 -I/nfs/userdocs/lab-jabraham/ja-cryosparc/sw/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda /tmp/tmp5brevv9b.cu --compiler-options -P]
[stderr:
b"gcc: error trying to exec ‘cc1plus’: execvp: No such file or directory\nnvcc fatal : Failed to preprocess host compiler properties.\n"]

Marking J1/imported/hcmvpol_100_001_X-1Y-1-0.tif as incomplete and continuing…
"

[wtempel]

Welcome to the forum @gabor_oroszlan.

@gabor_oroszlan

b"gcc: error trying to exec ‘cc1plus’: execvp: No such file or directory\nnvcc fatal : Failed to preprocess host compiler properties.\n"]

The CUDA installation that cryoSPARC is supposed to use on the worker node may be broken. You may want to ask the linux user that runs the cryoSPARC instance (ja-cryosparc?) to collect (and share) some information running commands on the cryoSPARC worker node where the error occurred.

@stephan

cryoSPARC worker environment

• CUDA toolkit and pycuda driver information
  eval $(<path-to-cryosparc_worker>/bin/cryosparcw env)
${CRYOSPARC_CUDA_PATH}/bin/nvcc --version
python -c "import pycuda.driver; print(pycuda.driver.get_version())"
• Output of
  uname -a && free -g && nvidia-smi
  on relevant worker node(s)

The cryoSPARC worker node is identified in the job’s Overview tab (“Show from top”). Do you know version and patch level of this cryoSPARC installation? This information can also be found near the top of the job’s Overview tab.

[gabor_oroszlan]

Thank you for getting back to me. Here are the first lines from the overview tab:License is valid.
Launching job on lane ja-gpu3 target ja-gpu3.in.hwlab …
Running job on remote worker node hostname ja-gpu3.in.hwlab
[CPU: 68.0 MB] Project P26 Job J2 Started
[CPU: 68.0 MB] Master running v3.2.0, worker running v3.2.0
[CPU: 68.2 MB] Running on lane ja-gpu3
[CPU: 68.2 MB] Resources allocated:
[CPU: 68.2 MB] Worker: ja-gpu3.in.hwlab
[CPU: 68.2 MB] CPU : [0, 1, 2, 3, 4, 5]
[CPU: 68.2 MB] GPU : [0]
[CPU: 68.2 MB] RAM : [0, 1]
[CPU: 68.2 MB] SSD : False
[CPU: 68.2 MB] --------------------------------------------------------------
[CPU: 68.2 MB] Importing job module for job type patch_motion_correction_multi…
[CPU: 207.3 MB] Job ready to run
[CPU: 207.4 MB] ***************************************************************
[CPU: 209.2 MB] Job will process this many movies: 3445
[CPU: 209.3 MB] parent process is 153333
[CPU: 172.5 MB] Calling CUDA init from 153613
[CPU: 312.8 MB] – 0.0: processing 0 of 3445: J1/imported/hcmvpol_100_001_X-1Y-1-0.tif
loading /nfs/userdocs/lab-jabraham/ja-cryosparc/P26/J1/imported/hcmvpol_100_001_X-1Y-1-0.tif
Loading raw movie data from J1/imported/hcmvpol_100_001_X-1Y-1-0.tif …
Done in 1.60s
Loading gain data from J1/imported/gain.mrc …
Done in 0.23s
Processing …

[wtempel]

@gabor_oroszlan Were you able to resolve this problem? In case the problem persists, please post

@stephan

CryoSPARC worker environment

• Load the cryoSPARC worker environment (for execution of the following commands):
  eval $(<path-to-cryosparc_worker>/bin/cryosparcw env)
• CUDA toolkit path:
  echo $CRYOSPARC_CUDA_PATH
• CUDA toolkit version:
  ${CRYOSPARC_CUDA_PATH}/bin/nvcc --version
• pycuda information (run command in same shell):
  python -c "import pycuda.driver; print(pycuda.driver.get_version())"
• Output of
  uname -a && free -g && nvidia-smi