I am a bit confused about the difference between the local motion correction and local motion correction (multi-gpu) job types. They both seem to accept number of GPUs as an input. It seems like the local motion correction (not-multi) job type will correctly reserve multiple GPUs if the number of GPUs is set to be >1, but it will only use one of them (at least only one shows usage by nvidia-smi) but the multi job type will request and actually use multiple GPUs. Despite this, they both seem to take approx the same amount of time if the same number of GPUs is requested. What is the actual difference under the hood between these job types? Why does it seem like the non-multi isn’t using multiple GPUs by resource monitoring but is taking an amount of time consistent with using multiple GPUs?
@kpahil The “non-multi” job type should not ask you to specify the number of GPUs. Thank you for reporting this bug.
The multi-GPU job may fail to improve performance if there’s a non-GPU bottleneck. “Promising” candidates for a bottleneck would be reading a large amount of data over limited storage or network bandwidth or decompressing certain formats like LZW-TIFF or BZ2.
Thank you! That makes sense. I’m certainly not saturating GPU usage, but am saturating CPU usage in the allocated cores. Is there anything I can do to better make use of my GPU (ex. allocate more CPU cores)?
Also, I cannot seem to find the “Output F-crop factor” option for either local motion correction job type (I want to apply a 2x bin, preferably within the job to avoid using a ton of storage). Screenshot:
no you’re absolutely right, it’s not there, I just checked on my system, I agree that’s weird - would be useful especially when working with super res data
funnily enough that’s exactly what I’m doing. I had binned the data during the patch motion correction that I’m using as an input for the local motion correction, but the local motion is going back to the raw data and extracting with the small pixel size (all the other jobs that I’ve run have been using the binned micrographs, so this behavior was a bit surprising to me tbh). The doc definitely says that local motion should have both the option to F-crop…
@kpahil
The job type should use additional CPU cores. Do you observe that one CPU is running near 100% load while all others are idling, or could there be a bottleneck other than the CPU?
The guide is is listing Output F-crop factor as a parameter for this job type in error. If down-sampling is desired, a separate job is required.
