Patch Motion Correction gain_ref_blob/path error

Motion Correction
1

When trying to run the patch motion correction job, after a couple of micrographs, it begins showing

KeyError: ‘gain_ref_blob/path’

and then

Marking [X].eer as incomplete and continuing…

and then proceeds to do so for every micrograph. If allowed to run, the job will eventually finish, but with all micrographs being marked as incomplete.

The import movies job has a seemingly valid .gain file included and this same cryosparc instance has run this same exact workflow many times. We’ve also restarted cryosparc and the same issue continues to appear.

2

@ajm Please can you let us know:

  • the CryoSPARC version
  • the outputs of how many Import Movies job were connected to this motion correction job?
3

This is on version 4.6.2 (but the version has not changed since other successful job runs).

16,453 movies were imported. I also tried re-running the job to only use 100 micrographs and the same error was present.

4

With a single Import Movies job?

5

Yes. Previous successful patch motion correction jobs have typically used more in the range of 4,000-9,000 imported movies, but this import movies job was successful with that number, and we’ve had some successful motion correction jobs with around 15,000 movies.

6

Thanks @ajm for these details. Please can you post the outputs of these commands (which apply to CryoSPARC v4, not v5):

project_id=P99 # edit actual project ID
import_job=J1 # edit actual movie import job ID
motion_job=J2 # edit actual patch motion job ID
cryosparcm cli "get_job('$project_id', '$import_job', 'job_type', 'version', 'created_at', 'started_at', 'output_result_groups', 'params_spec')"
cryosparcm cli "get_job('$project_id', '$motion_job', 'job_type', 'version', 'created_at', 'started_at', 'input_slot_groups', 'params_spec')"
7

Should I have the motion correction job running as we run these commands? We’ve cleared it since I posted the original question.

8

Ideally, the job would be in the same state as you described

but you may try the command with the motion correction job in its current state.

9

After restarting the job, it’s now running without errors. We had already restarted several times and gotten the error again, and even after restarting cryosparc had gotten the same error.

The only two things that might be different would be the seed, which has now changed as we created a new job (it looks like someone deleted the original during troubleshooting attempts), or the exact GPUs utilized. We set for a 3 GPU run and perhaps there is a bad GPU that was getting included before, though earlier today we had been utilizing all 4 GPUs across multiple jobs without issue.

10

Good to know. In case you observe the issue again, please preserve the failed job(s) in their failed state and post the outputs of the commands in Patch Motion Correction gain_ref_blob/path error - #6 by wtempel.

11

Although the original job is gone, I actually have a saved pdf of the log file from before while running. Just one other interesting note- the original job never marked as failed, it did complete running, but marked every micrograph as incomplete.

The job started like this:

> [2026-02-24 15:28:09.35] License is valid.

> [2026-02-24 15:28:09.35] Launching job on lane default target DHVI-R124852.dhe.duke.edu …

> [2026-02-24 15:28:09.37] Running job on master node hostname DHVI-R124852.dhe.duke.edu

> [2026-02-24 15:28:10.90] [CPU: 89.8 MB] [Avail: 156.71 GB] Job J29 Started

> [2026-02-24 15:28:10.92] [CPU: 89.8 MB] [Avail: 156.71 GB] Master running v4.6.2, worker running v4.6.2

> [2026-02-24 15:28:10.94] [CPU: 90.3 MB] [Avail: 156.71 GB] Working in directory:

/home/cryosparc_user/cryosparc/MainDataPool/MainDrive2/Coco/CS-heva1-

2-f-9a9-029kjl-176rda/J29

> [2026-02-24 15:28:10.94] [CPU: 90.3 MB] [Avail: 156.71 GB] Running on lane default

> [2026-02-24 15:28:10.95] [CPU: 90.3 MB] [Avail: 156.71 GB] Resources allocated:

> [2026-02-24 15:28:10.95] [CPU: 90.3 MB] [Avail: 156.71 GB] Worker: DHVI-R124852.dhe.duke.edu

> [2026-02-24 15:28:10.96] [CPU: 90.3 MB] [Avail: 156.71 GB] CPU : [0, 1, 2, 3, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,

19, 20, 21]

> [2026-02-24 15:28:10.96] [CPU: 90.3 MB] [Avail: 156.71 GB] GPU : [0, 2, 3]

> [2026-02-24 15:28:10.96] [CPU: 90.3 MB] [Avail: 156.71 GB] RAM : [0, 1, 5, 6, 7, 8]

> [2026-02-24 15:28:10.97] [CPU: 90.3 MB] [Avail: 156.71 GB] SSD : False

> [2026-02-24 15:28:10.97] [CPU: 90.3 MB] [Avail: 156.71 GB] --------------------------------------------------------------

> [2026-02-24 15:28:10.98] [CPU: 90.3 MB] [Avail: 156.71 GB] Importing job module for job type patch_motion_correction_multi…

> [2026-02-24 15:28:16.74] [CPU: 247.1 MB] [Avail: 156.67 GB] Job ready to run

> [2026-02-24 15:28:16.75] [CPU: 247.1 MB] [Avail: 156.67 GB] ***************************************************************

> [2026-02-24 15:28:16.89] [CPU: 258.6 MB] [Avail: 156.65 GB] Job will process this many movies: 16453

> [2026-02-24 15:28:16.90] [CPU: 258.6 MB] [Avail: 156.65 GB] Job will output denoiser training data for this many movies: 200

> [2026-02-24 15:28:16.90] [CPU: 258.6 MB] [Avail: 156.65 GB] Random seed: 710278332

> [2026-02-24 15:28:16.91] [CPU: 258.6 MB] [Avail: 156.65 GB] parent process is 3639271

> [2026-02-24 15:28:17.78] [CPU: 252.3 MB] [Avail: 156.62 GB] Calling CUDA init from 3639327

> [2026-02-24 15:28:17.78] [CPU: 251.9 MB] [Avail: 156.62 GB] Calling CUDA init from 3639328

> [2026-02-24 15:28:17.79] [CPU: 252.9 MB] [Avail: 156.62 GB] Calling CUDA init from 3639329

> [2026-02-24 15:28:18.57] [CPU: 396.8 MB] [Avail: 156.32 GB] – 0.0: processing 1 of 16453:

J28/imported/012671193859860078723_FoilHole_20277293_Data_20230888_

19_20260222_101256_EER.eer

loading

/home/cryosparc_user/cryosparc/MainDataPool/MainDrive2/Coco/CS-heva1-

2-f-9a9-029kjl-176rda/J28/imported/012671193859860078723_FoilHol

e_20277293_Data_20230888_19_20260222_101256_EER.eer

Loading raw movie data from

J28/imported/012671193859860078723_FoilHole_20277293_Data_20230888_

19_20260222_101256_EER.eer …

Done in 22.60s

> [2026-02-24 15:28:18.58] [CPU: 396.4 MB] [Avail: 156.31 GB] – 2.0: processing 2 of 16453:

J28/imported/012033094416284745072_FoilHole_20277299_Data_20230888_

7_20260222_101305_EER.eer

loading

/home/cryosparc_user/cryosparc/MainDataPool/MainDrive2/Coco/CS-heva1-

2-f-9a9-029kjl-176rda/J28/imported/012033094416284745072_FoilHol

e_20277299_Data_20230888_7_20260222_101305_EER.eer

Loading raw movie data from

J28/imported/012033094416284745072_FoilHole_20277299_Data_20230888_

7_20260222_101305_EER.eer …

Done in 22.59s

> [2026-02-24 15:28:18.58] [CPU: 395.9 MB] [Avail: 156.31 GB] – 1.0: processing 3 of 16453:

J28/imported/012651017804146027142_FoilHole_20277300_Data_20230888_

4_20260222_101314_EER.eer

loading

/home/cryosparc_user/cryosparc/MainDataPool/MainDrive2/Coco/CS-heva1-

2-f-9a9-029kjl-176rda/J28/imported/012651017804146027142_FoilHol

e_20277300_Data_20230888_4_20260222_101314_EER.eer

Loading raw movie data from

J28/imported/012651017804146027142_FoilHole_20277300_Data_20230888_

4_20260222_101314_EER.eer …

Done in 22.59s

> [2026-02-24 15:28:41.20] [CPU: 646.9 MB] [Avail: 155.06 GB] – 2.0: processing 4 of 16453:

J28/imported/001908414285768319843_FoilHole_20277301_Data_20230888_

11_20260222_101323_EER.eer

loading

/home/cryosparc_user/cryosparc/MainDataPool/MainDrive2/Coco/CS-heva1-

2-f-9a9-029kjl-176rda/J28/imported/001908414285768319843_FoilHol

e_20277301_Data_20230888_11_20260222_101323_EER.eer

Loading raw movie data from

J28/imported/001908414285768319843_FoilHole_20277301_Data_20230888_

11_20260222_101323_EER.eer …

Done in 21.37s

> [2026-02-24 15:28:41.20] [CPU: 338.1 MB] [Avail: 155.09 GB] Error occurred while processing

J28/imported/012033094416284745072_FoilHole_20277299_Data_20230888_

7_20260222_101305_EER.eer

Traceback (most recent call last):

File

"/home/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/

jobs/pipeline.py", line 59, in exec

return self.process(item)

File

"cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch

.py", line 152, in cryosparc_master.cryosparc_compute.jobs.motionco

rrection.run_patch.run_patch_motion_correction_multi.motionworker.p

rocess

File

"cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch

.py", line 174, in cryosparc_master.cryosparc_compute.jobs.motionco

rrection.run_patch.run_patch_motion_correction_multi.motionworker.p

rocess

KeyError: ‘gain_ref_blob/path’

Marking

J28/imported/012033094416284745072_FoilHole_20277299_Data_20230888_

7_20260222_101305_EER.eer as incomplete and continuing…

> [2026-02-24 15:28:41.20] [CPU: 530.3 MB] [Avail: 155.11 GB] – 0.0: processing 5 of 16453:

J28/imported/001491181146136650929_FoilHole_20277305_Data_20230888_

10_20260222_101332_EER.eer

loading

/home/cryosparc_user/cryosparc/MainDataPool/MainDrive2/Coco/CS-heva1-

2-f-9a9-029kjl-176rda/J28/imported/001491181146136650929_FoilHol

e_20277305_Data_20230888_10_20260222_101332_EER.eer

Loading raw movie data from

J28/imported/001491181146136650929_FoilHole_20277305_Data_20230888_

10_20260222_101332_EER.eer …

Done in 21.25s

> [2026-02-24 15:28:41.21] [CPU: 525.6 MB] [Avail: 155.11 GB] – 1.0: processing 6 of 16453:

J28/imported/015234369737136985561_FoilHole_20277306_Data_20230888_

2_20260222_101341_EER.eer

loading

/home/cryosparc_user/cryosparc/MainDataPool/MainDrive2/Coco/CS-heva1-

2-f-9a9-029kjl-176rda/J28/imported/015234369737136985561_FoilHol

e_20277306_Data_20230888_2_20260222_101341_EER.eer

Loading raw movie data from

J28/imported/015234369737136985561_FoilHole_20277306_Data_20230888_

2_20260222_101341_EER.eer …

Done in 21.23s

> [2026-02-24 15:28:41.21] [CPU: 338.1 MB] [Avail: 155.11 GB] Error occurred while processing

J28/imported/012671193859860078723_FoilHole_20277293_Data_20230888_

19_20260222_101256_EER.eer

Traceback (most recent call last):

File

"/home/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/

jobs/pipeline.py", line 59, in exec

return self.process(item)

File

"cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch

.py", line 152, in cryosparc_master.cryosparc_compute.jobs.motionco

rrection.run_patch.run_patch_motion_correction_multi.motionworker.p

rocess

File

"cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch

.py", line 174, in cryosparc_master.cryosparc_compute.jobs.motionco

rrection.run_patch.run_patch_motion_correction_multi.motionworker.p

rocess

KeyError: ‘gain_ref_blob/path’

Marking

J28/imported/012671193859860078723_FoilHole_20277293_Data_20230888_

19_20260222_101256_EER.eer as incomplete and continuing…

> [2026-02-24 15:28:41.22] [CPU: 338.1 MB] [Avail: 155.11 GB] Error occurred while processing

J28/imported/012651017804146027142_FoilHole_20277300_Data_20230888_

4_20260222_101314_EER.eer

Traceback (most recent call last):

File

"/home/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/

jobs/pipeline.py", line 59, in exec

return self.process(item)

File

"cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch

.py", line 152, in cryosparc_master.cryosparc_compute.jobs.motionco

rrection.run_patch.run_patch_motion_correction_multi.motionworker.p

rocess

File

"cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch

.py", line 174, in cryosparc_master.cryosparc_compute.jobs.motionco

rrection.run_patch.run_patch_motion_correction_multi.motionworker.p

rocess

KeyError: ‘gain_ref_blob/path’

Marking

J28/imported/012651017804146027142_FoilHole_20277300_Data_20230888_

4_20260222_101314_EER.eer as incomplete and continuing…

> [2026-02-24 15:29:02.46] [CPU: 725.8 MB] [Avail: 154.63 GB] – 1.0: processing 7 of 16453:

J28/imported/005647219310244941381_FoilHole_20277307_Data_20230888_

0_20260222_101244_EER.eer

loading

/home/cryosparc_user/cryosparc/MainDataPool/MainDrive2/Coco/CS-heva1-

2-f-9a9-029kjl-176rda/J28/imported/005647219310244941381_FoilHol

e_20277307_Data_20230888_0_20260222_101244_EER.eer

Loading raw movie data from

J28/imported/005647219310244941381_FoilHole_20277307_Data_20230888_

0_20260222_101244_EER.eer …

Done in 49.70s

> [2026-02-24 15:29:02.46] [CPU: 338.1 MB] [Avail: 154.63 GB] Error occurred while processing

J28/imported/015234369737136985561_FoilHole_20277306_Data_20230888_

2_20260222_101341_EER.eer

Traceback (most recent call last):

File

"/home/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/

jobs/pipeline.py", line 59, in exec

return self.process(item)

File

"cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch

.py", line 152, in cryosparc_master.cryosparc_compute.jobs.motionco

rrection.run_patch.run_patch_motion_correction_multi.motionworker.p

rocess

File

"cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch

.py", line 174, in cryosparc_master.cryosparc_compute.jobs.motionco

rrection.run_patch.run_patch_motion_correction_multi.motionworker.p

rocess

KeyError: ‘gain_ref_blob/path’

Marking

J28/imported/015234369737136985561_FoilHole_20277306_Data_20230888_

2_20260222_101341_EER.eer as incomplete and continuing…

> [2026-02-24 15:29:02.49] [CPU: 674.3 MB] [Avail: 154.79 GB] – 0.0: processing 8 of 16453:

J28/imported/001105090438727037532_FoilHole_20277308_Data_20230888_

3_20260222_101350_EER.eer

loading

/home/cryosparc_user/cryosparc/MainDataPool/MainDrive2/Coco/CS-heva1-

2-f-9a9-029kjl-176rda/J28/imported/001105090438727037532_FoilHol

e_20277308_Data_20230888_3_20260222_101350_EER.eer

Loading raw movie data from

J28/imported/001105090438727037532_FoilHole_20277308_Data_20230888_

3_20260222_101350_EER.eer …

Done in 20.19s

> [2026-02-24 15:29:02.49] [CPU: 338.1 MB] [Avail: 154.79 GB] Error occurred while processing

J28/imported/001491181146136650929_FoilHole_20277305_Data_20230888_

10_20260222_101332_EER.eer

Traceback (most recent call last):

File

"/home/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/

jobs/pipeline.py", line 59, in exec

return self.process(item)

File

"cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch

.py", line 152, in cryosparc_master.cryosparc_compute.jobs.motionco

rrection.run_patch.run_patch_motion_correction_multi.motionworker.p

rocess

File

"cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch

.py", line 174, in cryosparc_master.cryosparc_compute.jobs.motionco

rrection.run_patch.run_patch_motion_correction_multi.motionworker.p

rocess

KeyError: ‘gain_ref_blob/path’

Marking

J28/imported/001491181146136650929_FoilHole_20277305_Data_20230888_

10_20260222_101332_EER.eer as incomplete and continuing…

> [2026-02-24 15:29:02.59] [CPU: 839.4 MB] [Avail: 155.25 GB] – 2.0: processing 9 of 16453:

J28/imported/010610266103133146596_FoilHole_20277309_Data_20230888_

8_20260222_101359_EER.eer

loading

/home/cryosparc_user/cryosparc/MainDataPool/MainDrive2/Coco/CS-heva1-

2-f-9a9-029kjl-176rda/J28/imported/010610266103133146596_FoilHol

e_20277309_Data_20230888_8_20260222_101359_EER.eer

Loading raw movie data from

J28/imported/010610266103133146596_FoilHole_20277309_Data_20230888_

8_20260222_101359_EER.eer …

Done in 20.12s

> [2026-02-24 15:29:02.59] [CPU: 338.1 MB] [Avail: 155.25 GB] Error occurred while processing

J28/imported/001908414285768319843_FoilHole_20277301_Data_20230888_

11_20260222_101323_EER.eer

Traceback (most recent call last):

File

"/home/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/

jobs/pipeline.py", line 59, in exec

return self.process(item)

File

"cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch

.py", line 152, in cryosparc_master.cryosparc_compute.jobs.motionco

rrection.run_patch.run_patch_motion_correction_multi.motionworker.p

rocess

File

"cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch

.py", line 174, in cryosparc_master.cryosparc_compute.jobs.motionco

rrection.run_patch.run_patch_motion_correction_multi.motionworker.p

rocess

KeyError: ‘gain_ref_blob/path’

Marking

J28/imported/001908414285768319843_FoilHole_20277301_Data_20230888_

11_20260222_101323_EER.eer as incomplete

and proceeded like that until finishing with all marked incomplete.