Extract From Micrographs Job can't complete

[CleoShen]

Here is the latest status of my Extract from Micrographs job, it has been like this for a night, seems to be frozen, and can’t complete.

[CPU: 3.00 GB]   (11400 of 11406) Finished processing micrograph 11401.

[CPU: 3.00 GB]   (11401 of 11406) Finished processing micrograph 11399.

[CPU: 3.00 GB]   (11402 of 11406) Finished processing micrograph 11403.

[CPU: 3.00 GB]   (11403 of 11406) Finished processing micrograph 11405.

[CPU: 3.00 GB]   (11404 of 11406) Finished processing micrograph 11404.

[CPU: 3.00 GB]   (11405 of 11406) Finished processing micrograph 11402.

[CPU: 3.00 GB]   (11406 of 11406) Finished processing micrograph 11406.type or paste code here

[wtempel]

Please can you post the “job.log” for this job?
You can find it in the job directory, or print it to the screen with
cryosparcm joblog <project_id> <job_id>

[CleoShen]

Thank you for your replying. The joblog is really big, I was trying to paste it here, but the character is over. How can I share it with you?

[wtempel]

The job log has the same error as

@CleoShen

pycuda._driver.LogicError: cuDeviceGet failed: invalid device ordinal

Was this the same worker node?
Please can you post the outputs of:

@stephan

cryoSPARC worker environment

• Load the cryoSPARC worker environment (for execution of the following commands):
  eval $(<path-to-cryosparc_worker>/bin/cryosparcw env)
• CUDA toolkit path:
  echo $CRYOSPARC_CUDA_PATH
• CUDA toolkit version:
  ${CRYOSPARC_CUDA_PATH}/bin/nvcc --version
• pycuda information (run command in same shell):
  python -c "import pycuda.driver; print(pycuda.driver.get_version())"
• Output of
  uname -a && free -g && nvidia-smi

on relevant worker node(s)