Child process with PID terminated unexpectedly

Motion Correction
1

Hi,
I’m trying to run a Patch Motion Correction but I’m facing the bellow error:
Calling CUDA init from 14468
Child process with PID 14468 terminated unexpectedly with exit code 1.

I checked the output of cryosparcm cli "get_scheduler_targets()" and the things look like normal
[{‘cache_path’: ‘/home/impmc’, ‘cache_quota_mb’: None, ‘cache_reserve_mb’: 10000, ‘desc’: None, ‘gpus’: [{‘id’: 0, ‘mem’: 11546394624, ‘name’: ‘NVIDIA GeForce RTX 2080 Ti’}, {‘id’: 1, ‘mem’: 11546394624, ‘name’: ‘NVIDIA GeForce RTX 2080 Ti’}, {‘id’: 2, ‘mem’: 11546394624, ‘name’: ‘NVIDIA GeForce RTX 2080 Ti’}, {‘id’: 3, ‘mem’: 11543379968, ‘name’: ‘NVIDIA GeForce RTX 2080 Ti’}], ‘hostname’: ‘titane2’, ‘lane’: ‘default’, ‘monitor_port’: None, ‘name’: ‘titane2’, ‘resource_fixed’: {‘SSD’: True}, ‘resource_slots’: {‘CPU’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], ‘GPU’: [0, 1, 2, 3], ‘RAM’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]}, ‘ssh_str’: ‘impmc@titane2’, ‘title’: ‘Worker node titane2’, ‘type’: ‘node’, ‘worker_bin_path’: ‘/home/impmc/cryosparc_worker/bin/cryosparcw’}]

However, when I look to the job.log file, I see this error:
ImportError: libcufft.so.10: cannot open shared object file: No such file or directory

Any ideas what could be the issue?
Thanks in advance,
Carlos.

2

Welcome to the forum @CarlosFernandes.
Has motion correction worked previously on this installation of CryoSPARC?
Please can you post the the outputs of these commands:

/home/impmc/cryosparc_worker/bin/cryosparcw call nvcc --version
/home/impmc/cryosparc_worker/bin/cryosparcw call which nvcc
/home/impmc/cryosparc_worker/bin/cryosparcw call python -c "import pycuda.driver; print(pycuda.driver.get_version())"
3

Hi! Thanks for the response @wtempel !
It is the first time that I’m trying motion correction in this installation. Actually, I just installed CryoSPARC in this machine.
The outputs that you asked:

For /home/impmc/cryosparc_worker/bin/cryosparcw call nvcc --version:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85

For /home/impmc/cryosparc_worker/bin/cryosparcw call which nvcc:
/usr/lib/nvidia-cuda-toolkit/bin/nvcc

For /home/impmc/cryosparc_worker/bin/cryosparcw call python -c “import pycuda.driver; print(pycuda.driver.get_version())”:
(9, 1, 0)

Thanks for the help!
Best regards,
Carlos

4

This version of the toolkit may be too old to work with CryoSPARC.
You may want to install CUDA-11.8 independently of the existing CUDA-9.1 toolkit as a non-root user (with the --toolkit, --toolkitpath= and --defaultroot= parameters).
You would need to

  • ensure the nvidia driver is compatible with v11.8 of the toolkit (≥v450.80.02)
  • run
    /home/impmc/cryosparc_worker/bin/cryosparcw newcuda /home/impmc/cuda-11.8.0
    (assuming you previously installed the toolkit with (no sudo)
    bash cuda_11.8.0_520.61.05_linux.run --silent --toolkit --toolkitpath=/home/impmc/cuda-11.8.0 --defaultroot=/home/impmc/cuda-11.8.0
    )
    [Edited to correct a command]
5

Its working now!
There is only one thing, this command
/home/impmc/cryosparc_worker/bin/cryosparcw /home/impmc/cuda-11.8.0
shoud include newcuda, the command should be
/home/impmc/cryosparc_worker/bin/cryosparcw newcuda /home/impmc/cuda-11.8.0
Thank you so much for your help!
Best regards,
Carlos.

1 Like
6

You are correct. I will correct my post immediately.