Issues in running Denoise Micrograph job

aayushi · July 3, 2026, 12:32pm

Hi All,

I have been having some troubles in running the denoise micrograph job. Given very limited experience in the field of cryoEM I assumed it may be a memeory issue, as in few interactive jobs like inspect pick and select 2D jobs ran smoothly with smaller batches of dataset. So, I divided the data using Exposure Set job to split my 9620 exposures set into 40 batches and rerun the denoising job on just 262 exposures. To my surprise it seem to give the same error. Since I am new to the field of CryoEM and have not changes any default parameters for any job so far. My question is - 1) What could have lead to this issue and how can I communicate to our IT team about the potential source of problem that leads to this error?

Event log Screenshot of the error message (in case of full data set).

Error message when rerunning the same job on 262 micrographs

wtempel · July 9, 2026, 6:12pm

Welcome to the forum @aayushi . Please can you post the outputs of these commands on the GPU computer where the job ran:

uname -a 
free -h
nvidia-smi
cat /cm/shared/apps/cryosparc_worker/config.sh
/cm/shared/apps/cryosparc_worker/bin/cryosparcw gpulist

A friendly reminder: Please post text outputs as text to improve search results for other interested forum users.
The error message can be printed to a terminal window by running the following commands on the CryoSPARC master host:

project_uid="P99" # replace with actual ID
job_uid="J999" # replace with actual ID 
cryosparcm eventlog $project_uid $job_uid | tail -n 40

aayushi · July 13, 2026, 4:46pm

Dear CryoSPARC Team,

Thank you for the reply. To add some context: we initially suspected a memory issue and increased the master node’s RAM to over 720 GB, but the job still fails. With some research based on a few similar errors on the CryoSPARC discussion forum and on GitHub (e.g. whisperX issue #1304), we tried forcing PyTORCH compatibility by adding export TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 and export CRYOSPARC_CUDA_FORCE_PTX_JIT=1 to cryosparc_worker/config.sh and restarting it. Unfortunately, It did not change the final output as I still am unable to run this job. Was it the right fix for v4.6.2, or if there’s other recommended approach? Do we need to update the cryoSPARC version to fix this issue?

Here is the outputs as requested:

±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 2 N/A N/A 378935 C python 3544MiB |
| 3 N/A N/A 377757 C python 884MiB |
| 5 N/A N/A 357198 C python 932MiB |
±----------------------------------------------------------------------------------------+

[root@gpu103 cryosparc_worker-frueh]# cat config.sh

export CRYOSPARC_LICENSE_ID=“redacted”
export CRYOSPARC_USE_GPU=true
export TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1

[root@gpu103 cryosparc_worker-frueh]# ./bin/cryosparcw gpulist
Detected 6 CUDA devices.

id pci-bus name

   0                24  Tesla T4                                                                
   1                94  Tesla T4                                                                
   2               175  Tesla T4                                                                
   3               176  Tesla T4                                                                
   4               216  Tesla T4                                                                
   5               217  Tesla T4

[root@gpu103 cryosparc_worker-frueh]#

The text for the new error message after forcing the PyTORCH compatibility -

[cryosparcuser@cryosparc-server-frueh ~]$ cryosparcm eventlog P1 J154 | tail -n 40
###########
EOF

sbatch /tmp/asing176_P1_J154.sbatch

==========================================================================

[Fri, 10 Jul 2026 19:18:05 GMT] -------- Submission command:
bash /projects/frueh_lab/CS-hmwp2-apo-asingh-processing/J154/queue_sub_script.sh
[Fri, 10 Jul 2026 19:18:05 GMT] -------- Cluster Job ID:
4041290
[Fri, 10 Jul 2026 19:18:05 GMT] -------- Queued on cluster at 2026-07-10 15:18:05.998857
[Fri, 10 Jul 2026 19:18:06 GMT] -------- Cluster job status at 2026-07-10 15:18:16.309822 (1 retries)
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
4041290 gpu cryospar cryospar R 0:05 1 gpu110
[Fri, 10 Jul 2026 19:18:16 GMT] [CPU RAM used: 90 MB] Job J154 Started
[Fri, 10 Jul 2026 19:18:16 GMT] [CPU RAM used: 90 MB] Master running v4.6.2, worker running v4.6.2
[Fri, 10 Jul 2026 19:18:16 GMT] [CPU RAM used: 90 MB] Working in directory: /projects/frueh_lab/CS-hmwp2-apo-asingh-processing/J154
[Fri, 10 Jul 2026 19:18:16 GMT] [CPU RAM used: 90 MB] Running on lane rithpc
[Fri, 10 Jul 2026 19:18:16 GMT] [CPU RAM used: 90 MB] Resources allocated:
[Fri, 10 Jul 2026 19:18:16 GMT] [CPU RAM used: 90 MB] Worker: rithpc

Fri, 10 Jul 2026 19:18:16 GMT\] \[CPU RAM used: 90 MB\] CPU : \[0, 1, 2, 3, 4, 5, 6, 7

Fri, 10 Jul 2026 19:18:16 GMT\] \[CPU RAM used: 90 MB\] GPU : \[0

Fri, 10 Jul 2026 19:18:16 GMT\] \[CPU RAM used: 90 MB\] RAM : \[0, 1

[Fri, 10 Jul 2026 19:18:16 GMT] [CPU RAM used: 90 MB] SSD : False
[Fri, 10 Jul 2026 19:18:16 GMT] [CPU RAM used: 90 MB] --------------------------------------------------------------
[Fri, 10 Jul 2026 19:18:16 GMT] [CPU RAM used: 90 MB] Importing job module for job type denoise_train…
[Fri, 10 Jul 2026 19:18:34 GMT] [CPU RAM used: 705 MB] Job ready to run
[Fri, 10 Jul 2026 19:18:34 GMT] [CPU RAM used: 705 MB] ***************************************************************
[Fri, 10 Jul 2026 19:18:34 GMT] [CPU RAM used: 705 MB] Transparent hugepages are enabled. You may encounter stalls or performance problems with CryoSPARC jobs.
[Fri, 10 Jul 2026 19:18:34 GMT] [CPU RAM used: 709 MB] Loaded info for 600 micrographs
[Fri, 10 Jul 2026 19:18:34 GMT] [CPU RAM used: 712 MB] Frame shape in : [4096 4096], psize_in : 0.9300000071525574
[Fri, 10 Jul 2026 19:18:34 GMT] [CPU RAM used: 712 MB] Frame shape out : [1280 1280], psize_out: 2.9760000228881838
[Fri, 10 Jul 2026 19:18:34 GMT] [CPU RAM used: 712 MB] Frame shape crop: [1256 1256] (only for training)
[Fri, 10 Jul 2026 19:18:34 GMT] [CPU RAM used: 719 MB] Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 129, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/micrograph_analysis/run_denoise.py”, line 180, in cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.run_denoise.run_train
File “/cm/shared/apps/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/torch/cuda/init.py”, line 491, in _lazy_init
torch._C._cuda_init()
RuntimeError: The NVIDIA driver on your system is too old (found version 12080). Please update your GPU driver by downloading and installing a new version from the URL: Download The Latest Official NVIDIA Drivers Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver.
[cryosparcuser@cryosparc-server-frueh ~]$

wtempel · July 13, 2026, 8:25pm

We recommend

updating CryoSPARC to version 5, provided your infrastructure is compatible
and removing the TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD and CRYOSPARC_CUDA_FORCE_PTX_JIT settings from cryosparc_worker/config.sh