Error during patch motion correction

Motion Correction
1

Hi,
I have imported my movies and was trying to do the patch motion correction, where I am experiencing an a error.
“Child process with PID 1745820 terminated unexpectedly with exit code 1”

2

Welcome to the forum @Wathsara .
Please can you post the final ≈ 20 lines from the job log (available under Metadata|Log.

3

Thank you for replying!.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/exx/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 199, in process_work_simple
    process_setup(proc_idx) # do any setup you want on a per-process basis
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 118, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.process_setup
  File "cryosparc_master/cryosparc_compute/gpu/gpucore.py", line 49, in cryosparc_master.cryosparc_compute.gpu.gpucore.initialize
  File "/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 3216, in get_version
    return driver.get_version()
  File "/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 461, in get_version
    version = driver.cuDriverGetVersion()
  File "/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 292, in __getattr__
    self.ensure_initialized()
  File "/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 258, in ensure_initialized
    raise CudaSupportError(f"Error at driver init: {description}")
numba.cuda.cudadrv.error.CudaSupportError: Error at driver init: Call to cuInit results in CUDA_ERROR_NO_DEVICE (100)
/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
4

Hello… please help me with above error

5

@Wathsara Please can you post the outputs of these commands

  1. On the CryoSPARC master computer:
    hostname -f
cryosparcm cli "get_scheduler_targets()"
# replace P99, J199 with project and job IDs of failed patch motion job
cryosparcm cli "get_job('P99', 'J199', 'job_type', 'version', 'status',  'params_spec', 'errors_run', 'instance_information')"
  2. On the CryoSPARC worker where the job ran and failed
    uname -a
nvidia-smi
1 Like
6

Hi,
I have run above commands and outputs are as follows.

  1. [{‘cache_path’: ‘/tmp’, ‘cache_quota_mb’: None, ‘cache_reserve_mb’: 10000, ‘desc’: None, ‘gpus’: [{‘id’: 0, ‘mem’: 25298141184, ‘name’: ‘NVIDIA GeForce RTX 3090’}], ‘hostname’: ‘localhost’, ‘lane’: ‘default’, ‘monitor_port’: None, ‘name’: ‘localhost’, ‘resource_fixed’: {‘SSD’: True}, ‘resource_slots’: {‘CPU’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31], ‘GPU’: [0], ‘RAM’: [0, 1, 2, 3]}, ‘ssh_str’: ‘exx@localhost’, ‘title’: ‘Worker node localhost’, ‘type’: ‘node’, ‘worker_bin_path’: ‘/home/exx/cryosparc/cryosparc_worker/bin/cryosparcw’}]
    *** (http://sn4622126433:61002, code 400) Encountered ServerError from JSONRPC function “get_job” with params (‘P99’, ‘J199’, ‘job_type’, ‘version’, ‘status’, ‘params_spec’, ‘errors_run’, ‘instance_information’):
    ServerError: P99 J199 does not exist.
    Traceback (most recent call last):
    File “/home/exx/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py”, line 196, in wrapper
    res = func(*args, **kwargs)
    File “/home/exx/cryosparc/cryosparc_master/cryosparc_command/command_core/init.py”, line 6118, in get_job
    raise ValueError(f"{project_uid} {job_uid} does not exist.")
    ValueError: P99 J199 does not exist.

  2. Linux sn4622126433 5.15.0-144-generic #157-Ubuntu SMP Mon Jun 16 07:33:10 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
    NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

7

This failure might explain

You may try if a reboot of the GPU computer restores the nvidia-smi function.

1 Like
8

Thank you ! It actually helped and running right now.