Motioncor For K3

zjr · December 15, 2018, 2:20am

Hi

I got err when I run motion correction for data collected on k3, The error read like:

Traceback (most recent call last):
File “cryosparc2_compute/jobs/runcommon.py”, line 738, in run_with_except_hook
run_old(*args, **kw)
File “/data/cryoSPARC_v2/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py”, line 754, in run
self.__target(*self.__args, **self.__kwargs)
File “cryosparc2_compute/jobs/pipeline.py”, line 53, in stage_target
work = processor.process(item)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 138, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction_multi.motionworker.process
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 141, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction_multi.motionworker.process
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 142, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction_multi.motionworker.process
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 272, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 282, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/cuda_kernels.py”, line 149, in cryosparc2_compute.jobs.motioncorrection.cuda_kernels.do_extract_and_weight_gpu
File “/data/cryoSPARC_v2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/driver.py”, line 418, in function_call
func._launch_kernel(grid, block, arg_buf, shared, stream)
LogicError: cuLaunchKernel failed: invalid argument

Any suggestion? Thanks!

Jinru

stephan · December 21, 2018, 5:30pm

Hi @zjr,

Did you use the multi-gpu rigid motion correction job?
Is it possible if you can try this on the single gpu rigid motion correction job?

zjr · December 21, 2018, 6:47pm

Hi sarulthasan,

Thanks for the suggestion. I tried the single gpu but has a err too read like:
Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 387, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 388, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 272, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 282, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/cuda_kernels.py”, line 149, in cryosparc2_compute.jobs.motioncorrection.cuda_kernels.do_extract_and_weight_gpu
File “/data/cryoSPARC_v2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/driver.py”, line 418, in function_call
func._launch_kernel(grid, block, arg_buf, shared, stream)
LogicError: cuLaunchKernel failed: invalid argument

Jinru

stephan · December 21, 2018, 6:49pm

Hi @zjr,

Can you post the details about your movie, including resolution and pixel size? Is there a gain correction file?
Did you alter any parameters in the job?
Also, can you provide information about the GPU you are using (model, memory)?

zjr · December 21, 2018, 7:08pm

Hi @stephan,

This is single particle movies, pixels size 1.096, resolution should be around 1.9 A something, I cannot remember now. There is a gain ref file, I did not change any parameters in this job and the import job work fine.

Here is my GPU info:
04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
1a:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)

I have 8 gpu installed with memory 11178MiB/per.

Jinru

stephan · December 21, 2018, 7:12pm

What are the x & y resolutions of the frames in your movie?
Is it 5,760 x 4,092?

zjr · December 21, 2018, 7:20pm

Here is the info when I import the movie:

movie_blob/psize_A : set([1.096])

movie_blob/shape : [ 40 8184 11520]

movie_blob/is_gain_corrected : set([0])

Can you read the x & y resolutions from here?

Jinru

stephan · December 21, 2018, 7:27pm

Thats great! I’m going to take this back and try to reproduce the error you’re getting.

zjr · December 21, 2018, 7:31pm

Thanks a lot @stephan

Jinru

MHB · February 17, 2019, 11:59pm

Any update on this issue? Still observed with K3 data an latest version.

Feng10 · June 18, 2019, 2:39pm

yes, I also came across the same problem

stephan · June 18, 2019, 3:27pm

Hi @MHB, @Feng10, @zjr,

Did you experience this error while motion correcting K3 movies in v2.8.3 with Patch Motion Correction?

Feng10 · June 18, 2019, 3:53pm

Hi @sarulthasan,

I am using v2.8.0. Patch Motion Correction works for my dataset. Since patch motion works, I don’t have to run full-frame correction, right?

FYI, my dataset:
mscope_params/accel_kv : set([200.0])
mscope_params/cs_mm : set([2.7])
mscope_params/total_dose_e_per_A2 : set([67.720001])
movie_blob/psize_A : set([0.41999999])
movie_blob/shape : [ 50 8184 11520]

I just wonder, full-frame does not work because my movies and @zjr movies are too big? T20S tutorial movies are smaller than ours so full-frame works?

Cheers

stephan · June 18, 2019, 3:58pm

Hi @Feng10,

To address your first question about having to run Full-Frame Motion Correction or not, please see this blog post that has details on an updated and optimized cryoSPARC workflow using the new jobs.
https://cryosparc.com/docs/tutorials/patch-motion-ctf/

To address your second question, yes, Patch Motion Correction is more optimized than Full-Frame motion correction, so it uses less GPU memory for larger movies.

MHB · June 18, 2019, 4:01pm

No problem with patch motion-corr and K3 data so far.

MHB · July 3, 2019, 11:05pm

With 2.9 am getting anew error with patch motion corr…see below.
This used to happen with GPU 0 that which was assigned the system CPU. Now with 2.9 it happens for all GPUs.

Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 830, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/cryosparc_user/V2.X/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 53, in stage_target
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 142, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:5032)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 145, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:4927)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 146, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:4834)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:18557)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 222, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:12760)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 312, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:9953)
  File "/home/cryosparc_user/V2.X/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py", line 210, in __init__
    self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory

stephan · July 4, 2019, 4:27pm

Hi @MHB,

Can you tell us the GPU Models, Movie frame size (x, y & z), and any non-default parameters you used with the Patch Motion Correction Job?

MHB · July 5, 2019, 7:15pm

This is running GeForce GTX 1080, 8GB RAM, 2560 CUDA cores. Default parameters.
Blobshape [ 50 8184 11520] (K3 images)

stephan · July 24, 2019, 7:31pm

Hi @MHB,

Unfortunately 8GB is not enough to correct the motion of a 12K x 12K image. I’ve seen GPU Memory Utilization go up to 9.6GB when processing K3 superres data.

MHB · July 24, 2019, 11:30pm

Thanks. well understood.