Motion Correction Not Working with T20S Proteasome Tutorial Data

Hello,

I am running the cryosparc tutorial and cannot get past the motion correction step. All of the import jobs seem to work, however, motion correction does not. I have tried resetting the path to direct to both cryosparc and cuda, however, it does not seem to change the error message.

[CPU: 221.8 MB]  Error occurred while processing J1/imported/14sep05c_00024sq_00003hl_00002es.frames.tif
Traceback (most recent call last):
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 429, in context_dependent_memoize
    return ctx_dict[cur_ctx][args]
KeyError: <pycuda._driver.Context object at 0x2aaad73bb1b0>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pytools/prefork.py", line 50, in call_capture_output
    stderr=PIPE)
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'nvcc': 'nvcc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data1/cryosparc_user/v3/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 59, in exec
    return self.process(item)
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 190, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 193, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 195, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 251, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 260, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 117, in cryosparc_compute.jobs.motioncorrection.patchmotion.prepare_movie_for_processing
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py", line 549, in fill
    func = elementwise.get_fill_kernel(self.dtype)
  File "<decorator-gen-120>", line 2, in get_fill_kernel
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 433, in context_dependent_memoize
    result = func(*args)
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 498, in get_fill_kernel
    "fill")
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 163, in get_elwise_kernel
    arguments, operation, name, keep, options, **kwargs)
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 149, in get_elwise_kernel_and_types
    keep, options, **kwargs)
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 76, in get_elwise_module
    options=options, keep=keep, no_extern_c=True)
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 291, in __init__
    arch, code, cache_dir, include_dirs)
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 254, in compile
    return compile_plain(source, options, keep, nvcc, cache_dir, target)
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 78, in compile_plain
    checksum.update(preprocess_source(source, options, nvcc).encode("utf-8"))
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 50, in preprocess_source
    result, stdout, stderr = call_capture_output(cmdline, error_on_nonzero=False)
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pytools/prefork.py", line 227, in call_capture_output
    return forker.call_capture_output(cmdline, cwd, error_on_nonzero)
  File "/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pytools/prefork.py", line 61, in call_capture_output
    % (" ".join(cmdline), e))
pytools.prefork.ExecError: error invoking 'nvcc --preprocess -arch sm_75 -I/data1/cryosparc_user/v3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda /tmp/tmp0i5ikmg6.cu --compiler-options -P': [Errno 2] No such file or directory: 'nvcc': 'nvcc'

Marking J1/imported/14sep05c_00024sq_00003hl_00002es.frames.tif as incomplete and continuing...

Any insight into what the problem is or the solution would help! Thank you so much!

Hi @linaria17,

What version of cryoSPARC are you running? Could you please reply with the output of nvidia-smi on the machine you tried to run this motion correction job?

Thanks,
Suhail

Hi @sdawood,

We recently upgraded our cuda and the error has now changed on the patch motion correction job to the below:

[CPU: 261.9 MB]  Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 84, in cryosparc_compute.run.main
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 402, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi
AssertionError: Child process with PID 21308 has terminated unexpectedly!

This is what our nvidia-smi outputs:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  On   | 00000000:3B:00.0 Off |                  N/A |
| 27%   24C    P8     3W / 250W |      0MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  On   | 00000000:5E:00.0 Off |                  N/A |
| 27%   24C    P8     1W / 250W |      0MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 208...  On   | 00000000:AF:00.0 Off |                  N/A |
| 27%   25C    P8     1W / 250W |      0MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 208...  On   | 00000000:D8:00.0 Off |                  N/A |
| 27%   23C    P8    14W / 250W |      0MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+ 

Thank you so much for helping us troubleshoot!

@linaria17 could you also share the internal job log for that latest job? You can get it with this command (replace PX and JY with the Project ID and Job ID, for example P1 J2:

cryosparcm joblog PX JY | cat