PatchMotion failure 2.13.2

stephan · March 4, 2020, 7:34pm

To recap,

Make sure you’re running as the user that is running cryoSPARC. If you’re NOT running cryoSPARC on a single workstation (i.e. standalone instance), make sure you run this script on a worker node.
Locate cryosparcw- it should be in cryosparc2_worker/bin/
Run the script

cd cryosparc2_worker
eval $(bin/cryosparcw env)
python -c "import pycuda.driver as pycu; pycu.init(); print [(pycu.Device(i).name(), pycu.Device(i).compute_capability(), pycu.Device(i).total_memory(), pycu.Device(i).get_attributes()) for i in range(pycu.Device.count())]"

I hope this helps!

MHB · March 5, 2020, 11:32pm

Got it thanks. Here is the output

[('GeForce GTX 1080 Ti', 
(6, 1), 
11721113600, 
{pycuda._driver.device_attribute.MAX_THREADS_PER_BLOCK: 1024, 
pycuda._driver.device_attribute.MAX_BLOCK_DIM_X: 1024, 
pycuda._driver.device_attribute.MAX_BLOCK_DIM_Y: 1024, 
pycuda._driver.device_attribute.MAX_BLOCK_DIM_Z: 64, 
pycuda._driver.device_attribute.MAX_GRID_DIM_X: 2147483647, 
pycuda._driver.device_attribute.MAX_GRID_DIM_Y: 65535, 
pycuda._driver.device_attribute.MAX_GRID_DIM_Z: 65535, 
pycuda._driver.device_attribute.MAX_SHARED_MEMORY_PER_BLOCK: 49152, 
pycuda._driver.device_attribute.TOTAL_CONSTANT_MEMORY: 65536, 
pycuda._driver.device_attribute.WARP_SIZE: 32, pycuda._driver.device_attribute.MAX_PITCH: 
2147483647, pycuda._driver.device_attribute.MAX_REGISTERS_PER_BLOCK: 65536, 
pycuda._driver.device_attribute.CLOCK_RATE: 1620000, 
pycuda._driver.device_attribute.TEXTURE_ALIGNMENT: 512, 
pycuda._driver.device_attribute.GPU_OVERLAP: 1, 
pycuda._driver.device_attribute.MULTIPROCESSOR_COUNT: 28, 
pycuda._driver.device_attribute.KERNEL_EXEC_TIMEOUT: 1, 
pycuda._driver.device_attribute.INTEGRATED: 0, 
pycuda._driver.device_attribute.CAN_MAP_HOST_MEMORY: 1, 
pycuda._driver.device_attribute.COMPUTE_MODE: pycuda._driver.compute_mode.DEFAULT, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_WIDTH: 131072, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_WIDTH: 131072, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_HEIGHT: 65536, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_WIDTH: 16384, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_HEIGHT: 16384, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_DEPTH: 16384, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_ARRAY_WIDTH: 32768, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_ARRAY_HEIGHT: 32768, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_ARRAY_NUMSLICES: 2048, 
pycuda._driver.device_attribute.SURFACE_ALIGNMENT: 512, 
pycuda._driver.device_attribute.CONCURRENT_KERNELS: 1, 
pycuda._driver.device_attribute.ECC_ENABLED: 0, pycuda._driver.device_attribute.PCI_BUS_ID: 5, 
pycuda._driver.device_attribute.PCI_DEVICE_ID: 0, pycuda._driver.device_attribute.TCC_DRIVER: 0, 
pycuda._driver.device_attribute.MEMORY_CLOCK_RATE: 5505000, 
pycuda._driver.device_attribute.GLOBAL_MEMORY_BUS_WIDTH: 352, 
pycuda._driver.device_attribute.L2_CACHE_SIZE: 2883584, 
pycuda._driver.device_attribute.MAX_THREADS_PER_MULTIPROCESSOR: 2048, 
pycuda._driver.device_attribute.ASYNC_ENGINE_COUNT: 2, 
pycuda._driver.device_attribute.UNIFIED_ADDRESSING: 1, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_LAYERED_WIDTH: 32768, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_LAYERED_LAYERS: 2048, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_GATHER_WIDTH: 32768, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_GATHER_HEIGHT: 32768, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_WIDTH_ALTERNATE: 8192, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_HEIGHT_ALTERNATE: 8192, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_DEPTH_ALTERNATE: 32768, 
pycuda._driver.device_attribute.PCI_DOMAIN_ID: 0, 
pycuda._driver.device_attribute.TEXTURE_PITCH_ALIGNMENT: 32, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURECUBEMAP_WIDTH: 32768, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURECUBEMAP_LAYERED_WIDTH: 32768, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURECUBEMAP_LAYERED_LAYERS: 2046, 
pycuda._driver.device_attribute.MAXIMUM_SURFACE1D_WIDTH: 32768, 
pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_WIDTH: 131072, 
pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_HEIGHT: 65536, 
pycuda._driver.device_attribute.MAXIMUM_SURFACE3D_WIDTH: 16384, 
pycuda._driver.device_attribute.MAXIMUM_SURFACE3D_HEIGHT: 16384, 
pycuda._driver.device_attribute.MAXIMUM_SURFACE3D_DEPTH: 16384, 
pycuda._driver.device_attribute.MAXIMUM_SURFACE1D_LAYERED_WIDTH: 32768, 
pycuda._driver.device_attribute.MAXIMUM_SURFACE1D_LAYERED_LAYERS: 2048, 
pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_LAYERED_WIDTH: 32768, 
pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_LAYERED_HEIGHT: 32768, 
pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_LAYERED_LAYERS: 2048, 
pycuda._driver.device_attribute.MAXIMUM_SURFACECUBEMAP_WIDTH: 32768, 
pycuda._driver.device_attribute.MAXIMUM_SURFACECUBEMAP_LAYERED_WIDTH: 32768, 
pycuda._driver.device_attribute.MAXIMUM_SURFACECUBEMAP_LAYERED_LAYERS: 2046, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_LINEAR_WIDTH: 268435456, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_LINEAR_WIDTH: 131072, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_LINEAR_HEIGHT: 65000, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_LINEAR_PITCH: 2097120, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_MIPMAPPED_WIDTH: 32768, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_MIPMAPPED_HEIGHT: 32768, 
pycuda._driver.device_attribute.COMPUTE_CAPABILITY_MAJOR: 6, 
pycuda._driver.device_attribute.COMPUTE_CAPABILITY_MINOR: 1, 
pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_MIPMAPPED_WIDTH: 16384, pycuda._driver.device_attribute.STREAM_PRIORITIES_SUPPORTED: 1, 
pycuda._driver.device_attribute.GLOBAL_L1_CACHE_SUPPORTED: 1, 
pycuda._driver.device_attribute.LOCAL_L1_CACHE_SUPPORTED: 1, 
pycuda._driver.device_attribute.MAX_SHARED_MEMORY_PER_MULTIPROCESSOR: 98304, 
pycuda._driver.device_attribute.MAX_REGISTERS_PER_MULTIPROCESSOR: 65536, 
pycuda._driver.device_attribute.MANAGED_MEMORY: 1, 
pycuda._driver.device_attribute.MULTI_GPU_BOARD: 0, 
pycuda._driver.device_attribute.MULTI_GPU_BOARD_GROUP_ID: 0}), 

('GeForce GTX 1080 Ti', 
(6, 1), 
11721506816,
 {pycuda._driver.device_attribute.MAX_THREADS_PER_BLOCK: 1024, 
pycuda._driver.device_attribute.MAX_BLOCK_DIM_X: 1024, 
pycuda._driver.device_attribute.MAX_BLOCK_DIM_Y: 1024, pycuda._driver.device_attribute.MAX_BLOCK_DIM_Z: 64, pycuda._driver.device_attribute.MAX_GRID_DIM_X: 2147483647, pycuda._driver.device_attribute.MAX_GRID_DIM_Y: 65535, pycuda._driver.device_attribute.MAX_GRID_DIM_Z: 65535, pycuda._driver.device_attribute.MAX_SHARED_MEMORY_PER_BLOCK: 49152, pycuda._driver.device_attribute.TOTAL_CONSTANT_MEMORY: 65536, pycuda._driver.device_attribute.WARP_SIZE: 32, pycuda._driver.device_attribute.MAX_PITCH: 2147483647, pycuda._driver.device_attribute.MAX_REGISTERS_PER_BLOCK: 65536, pycuda._driver.device_attribute.CLOCK_RATE: 1620000, pycuda._driver.device_attribute.TEXTURE_ALIGNMENT: 512, pycuda._driver.device_attribute.GPU_OVERLAP: 1, pycuda._driver.device_attribute.MULTIPROCESSOR_COUNT: 28, pycuda._driver.device_attribute.KERNEL_EXEC_TIMEOUT: 0, pycuda._driver.device_attribute.INTEGRATED: 0, pycuda._driver.device_attribute.CAN_MAP_HOST_MEMORY: 1, pycuda._driver.device_attribute.COMPUTE_MODE: pycuda._driver.compute_mode.DEFAULT, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_WIDTH: 131072, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_WIDTH: 131072, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_HEIGHT: 65536, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_WIDTH: 16384, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_HEIGHT: 16384, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_DEPTH: 16384, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_ARRAY_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_ARRAY_HEIGHT: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_ARRAY_NUMSLICES: 2048, pycuda._driver.device_attribute.SURFACE_ALIGNMENT: 512, pycuda._driver.device_attribute.CONCURRENT_KERNELS: 1, pycuda._driver.device_attribute.ECC_ENABLED: 0, pycuda._driver.device_attribute.PCI_BUS_ID: 6, pycuda._driver.device_attribute.PCI_DEVICE_ID: 0, pycuda._driver.device_attribute.TCC_DRIVER: 0, pycuda._driver.device_attribute.MEMORY_CLOCK_RATE: 5505000, pycuda._driver.device_attribute.GLOBAL_MEMORY_BUS_WIDTH: 352, pycuda._driver.device_attribute.L2_CACHE_SIZE: 2883584, pycuda._driver.device_attribute.MAX_THREADS_PER_MULTIPROCESSOR: 2048, pycuda._driver.device_attribute.ASYNC_ENGINE_COUNT: 2, pycuda._driver.device_attribute.UNIFIED_ADDRESSING: 1, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_LAYERED_LAYERS: 2048, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_GATHER_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_GATHER_HEIGHT: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_WIDTH_ALTERNATE: 8192, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_HEIGHT_ALTERNATE: 8192, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_DEPTH_ALTERNATE: 32768, pycuda._driver.device_attribute.PCI_DOMAIN_ID: 0, pycuda._driver.device_attribute.TEXTURE_PITCH_ALIGNMENT: 32, pycuda._driver.device_attribute.MAXIMUM_TEXTURECUBEMAP_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURECUBEMAP_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURECUBEMAP_LAYERED_LAYERS: 2046, pycuda._driver.device_attribute.MAXIMUM_SURFACE1D_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_WIDTH: 131072, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_HEIGHT: 65536, pycuda._driver.device_attribute.MAXIMUM_SURFACE3D_WIDTH: 16384, pycuda._driver.device_attribute.MAXIMUM_SURFACE3D_HEIGHT: 16384, pycuda._driver.device_attribute.MAXIMUM_SURFACE3D_DEPTH: 16384, pycuda._driver.device_attribute.MAXIMUM_SURFACE1D_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACE1D_LAYERED_LAYERS: 2048, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_LAYERED_HEIGHT: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_LAYERED_LAYERS: 2048, pycuda._driver.device_attribute.MAXIMUM_SURFACECUBEMAP_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACECUBEMAP_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACECUBEMAP_LAYERED_LAYERS: 2046, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_LINEAR_WIDTH: 268435456, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_LINEAR_WIDTH: 131072, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_LINEAR_HEIGHT: 65000, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_LINEAR_PITCH: 2097120, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_MIPMAPPED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_MIPMAPPED_HEIGHT: 32768, pycuda._driver.device_attribute.COMPUTE_CAPABILITY_MAJOR: 6, pycuda._driver.device_attribute.COMPUTE_CAPABILITY_MINOR: 1, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_MIPMAPPED_WIDTH: 16384, pycuda._driver.device_attribute.STREAM_PRIORITIES_SUPPORTED: 1, pycuda._driver.device_attribute.GLOBAL_L1_CACHE_SUPPORTED: 1, pycuda._driver.device_attribute.LOCAL_L1_CACHE_SUPPORTED: 1, pycuda._driver.device_attribute.MAX_SHARED_MEMORY_PER_MULTIPROCESSOR: 98304, pycuda._driver.device_attribute.MAX_REGISTERS_PER_MULTIPROCESSOR: 65536, pycuda._driver.device_attribute.MANAGED_MEMORY: 1, pycuda._driver.device_attribute.MULTI_GPU_BOARD: 0, pycuda._driver.device_attribute.MULTI_GPU_BOARD_GROUP_ID: 1}), ('GeForce GTX 1080 Ti', (6, 1), 11721506816, {pycuda._driver.device_attribute.MAX_THREADS_PER_BLOCK: 1024, pycuda._driver.device_attribute.MAX_BLOCK_DIM_X: 1024, pycuda._driver.device_attribute.MAX_BLOCK_DIM_Y: 1024, pycuda._driver.device_attribute.MAX_BLOCK_DIM_Z: 64, pycuda._driver.device_attribute.MAX_GRID_DIM_X: 2147483647, pycuda._driver.device_attribute.MAX_GRID_DIM_Y: 65535, pycuda._driver.device_attribute.MAX_GRID_DIM_Z: 65535, pycuda._driver.device_attribute.MAX_SHARED_MEMORY_PER_BLOCK: 49152, pycuda._driver.device_attribute.TOTAL_CONSTANT_MEMORY: 65536, pycuda._driver.device_attribute.WARP_SIZE: 32, pycuda._driver.device_attribute.MAX_PITCH: 2147483647, pycuda._driver.device_attribute.MAX_REGISTERS_PER_BLOCK: 65536, pycuda._driver.device_attribute.CLOCK_RATE: 1620000, pycuda._driver.device_attribute.TEXTURE_ALIGNMENT: 512, pycuda._driver.device_attribute.GPU_OVERLAP: 1, pycuda._driver.device_attribute.MULTIPROCESSOR_COUNT: 28, pycuda._driver.device_attribute.KERNEL_EXEC_TIMEOUT: 0, pycuda._driver.device_attribute.INTEGRATED: 0, pycuda._driver.device_attribute.CAN_MAP_HOST_MEMORY: 1, pycuda._driver.device_attribute.COMPUTE_MODE: pycuda._driver.compute_mode.DEFAULT, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_WIDTH: 131072, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_WIDTH: 131072, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_HEIGHT: 65536, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_WIDTH: 16384, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_HEIGHT: 16384, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_DEPTH: 16384, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_ARRAY_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_ARRAY_HEIGHT: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_ARRAY_NUMSLICES: 2048, pycuda._driver.device_attribute.SURFACE_ALIGNMENT: 512, pycuda._driver.device_attribute.CONCURRENT_KERNELS: 1, pycuda._driver.device_attribute.ECC_ENABLED: 0, pycuda._driver.device_attribute.PCI_BUS_ID: 9, pycuda._driver.device_attribute.PCI_DEVICE_ID: 0, pycuda._driver.device_attribute.TCC_DRIVER: 0, pycuda._driver.device_attribute.MEMORY_CLOCK_RATE: 5505000, pycuda._driver.device_attribute.GLOBAL_MEMORY_BUS_WIDTH: 352, pycuda._driver.device_attribute.L2_CACHE_SIZE: 2883584, pycuda._driver.device_attribute.MAX_THREADS_PER_MULTIPROCESSOR: 2048, pycuda._driver.device_attribute.ASYNC_ENGINE_COUNT: 2, pycuda._driver.device_attribute.UNIFIED_ADDRESSING: 1, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_LAYERED_LAYERS: 2048, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_GATHER_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_GATHER_HEIGHT: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_WIDTH_ALTERNATE: 8192, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_HEIGHT_ALTERNATE: 8192, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_DEPTH_ALTERNATE: 32768, pycuda._driver.device_attribute.PCI_DOMAIN_ID: 0, pycuda._driver.device_attribute.TEXTURE_PITCH_ALIGNMENT: 32, pycuda._driver.device_attribute.MAXIMUM_TEXTURECUBEMAP_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURECUBEMAP_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURECUBEMAP_LAYERED_LAYERS: 2046, pycuda._driver.device_attribute.MAXIMUM_SURFACE1D_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_WIDTH: 131072, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_HEIGHT: 65536, pycuda._driver.device_attribute.MAXIMUM_SURFACE3D_WIDTH: 16384, pycuda._driver.device_attribute.MAXIMUM_SURFACE3D_HEIGHT: 16384, pycuda._driver.device_attribute.MAXIMUM_SURFACE3D_DEPTH: 16384, pycuda._driver.device_attribute.MAXIMUM_SURFACE1D_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACE1D_LAYERED_LAYERS: 2048, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_LAYERED_HEIGHT: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_LAYERED_LAYERS: 2048, pycuda._driver.device_attribute.MAXIMUM_SURFACECUBEMAP_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACECUBEMAP_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACECUBEMAP_LAYERED_LAYERS: 2046, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_LINEAR_WIDTH: 268435456, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_LINEAR_WIDTH: 131072, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_LINEAR_HEIGHT: 65000, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_LINEAR_PITCH: 2097120, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_MIPMAPPED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_MIPMAPPED_HEIGHT: 32768, pycuda._driver.device_attribute.COMPUTE_CAPABILITY_MAJOR: 6, pycuda._driver.device_attribute.COMPUTE_CAPABILITY_MINOR: 1, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_MIPMAPPED_WIDTH: 16384, pycuda._driver.device_attribute.STREAM_PRIORITIES_SUPPORTED: 1, pycuda._driver.device_attribute.GLOBAL_L1_CACHE_SUPPORTED: 1, pycuda._driver.device_attribute.LOCAL_L1_CACHE_SUPPORTED: 1, pycuda._driver.device_attribute.MAX_SHARED_MEMORY_PER_MULTIPROCESSOR: 98304, pycuda._driver.device_attribute.MAX_REGISTERS_PER_MULTIPROCESSOR: 65536, pycuda._driver.device_attribute.MANAGED_MEMORY: 1, pycuda._driver.device_attribute.MULTI_GPU_BOARD: 0, pycuda._driver.device_attribute.MULTI_GPU_BOARD_GROUP_ID: 2}), ('GeForce GTX 1080 Ti', (6, 1), 11721506816, {pycuda._driver.device_attribute.MAX_THREADS_PER_BLOCK: 1024, pycuda._driver.device_attribute.MAX_BLOCK_DIM_X: 1024, pycuda._driver.device_attribute.MAX_BLOCK_DIM_Y: 1024, pycuda._driver.device_attribute.MAX_BLOCK_DIM_Z: 64, pycuda._driver.device_attribute.MAX_GRID_DIM_X: 2147483647, pycuda._driver.device_attribute.MAX_GRID_DIM_Y: 65535, pycuda._driver.device_attribute.MAX_GRID_DIM_Z: 65535, pycuda._driver.device_attribute.MAX_SHARED_MEMORY_PER_BLOCK: 49152, pycuda._driver.device_attribute.TOTAL_CONSTANT_MEMORY: 65536, pycuda._driver.device_attribute.WARP_SIZE: 32, pycuda._driver.device_attribute.MAX_PITCH: 2147483647, pycuda._driver.device_attribute.MAX_REGISTERS_PER_BLOCK: 65536, pycuda._driver.device_attribute.CLOCK_RATE: 1620000, pycuda._driver.device_attribute.TEXTURE_ALIGNMENT: 512, pycuda._driver.device_attribute.GPU_OVERLAP: 1, pycuda._driver.device_attribute.MULTIPROCESSOR_COUNT: 28, pycuda._driver.device_attribute.KERNEL_EXEC_TIMEOUT: 0, pycuda._driver.device_attribute.INTEGRATED: 0, pycuda._driver.device_attribute.CAN_MAP_HOST_MEMORY: 1, pycuda._driver.device_attribute.COMPUTE_MODE: pycuda._driver.compute_mode.DEFAULT, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_WIDTH: 131072, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_WIDTH: 131072, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_HEIGHT: 65536, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_WIDTH: 16384, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_HEIGHT: 16384, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_DEPTH: 16384, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_ARRAY_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_ARRAY_HEIGHT: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_ARRAY_NUMSLICES: 2048, pycuda._driver.device_attribute.SURFACE_ALIGNMENT: 512, pycuda._driver.device_attribute.CONCURRENT_KERNELS: 1, pycuda._driver.device_attribute.ECC_ENABLED: 0, pycuda._driver.device_attribute.PCI_BUS_ID: 10, pycuda._driver.device_attribute.PCI_DEVICE_ID: 0, pycuda._driver.device_attribute.TCC_DRIVER: 0, pycuda._driver.device_attribute.MEMORY_CLOCK_RATE: 5505000, pycuda._driver.device_attribute.GLOBAL_MEMORY_BUS_WIDTH: 352, pycuda._driver.device_attribute.L2_CACHE_SIZE: 2883584, pycuda._driver.device_attribute.MAX_THREADS_PER_MULTIPROCESSOR: 2048, pycuda._driver.device_attribute.ASYNC_ENGINE_COUNT: 2, pycuda._driver.device_attribute.UNIFIED_ADDRESSING: 1, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_LAYERED_LAYERS: 2048, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_GATHER_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_GATHER_HEIGHT: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_WIDTH_ALTERNATE: 8192, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_HEIGHT_ALTERNATE: 8192, pycuda._driver.device_attribute.MAXIMUM_TEXTURE3D_DEPTH_ALTERNATE: 32768, pycuda._driver.device_attribute.PCI_DOMAIN_ID: 0, pycuda._driver.device_attribute.TEXTURE_PITCH_ALIGNMENT: 32, pycuda._driver.device_attribute.MAXIMUM_TEXTURECUBEMAP_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURECUBEMAP_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURECUBEMAP_LAYERED_LAYERS: 2046, pycuda._driver.device_attribute.MAXIMUM_SURFACE1D_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_WIDTH: 131072, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_HEIGHT: 65536, pycuda._driver.device_attribute.MAXIMUM_SURFACE3D_WIDTH: 16384, pycuda._driver.device_attribute.MAXIMUM_SURFACE3D_HEIGHT: 16384, pycuda._driver.device_attribute.MAXIMUM_SURFACE3D_DEPTH: 16384, pycuda._driver.device_attribute.MAXIMUM_SURFACE1D_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACE1D_LAYERED_LAYERS: 2048, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_LAYERED_HEIGHT: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACE2D_LAYERED_LAYERS: 2048, pycuda._driver.device_attribute.MAXIMUM_SURFACECUBEMAP_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACECUBEMAP_LAYERED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_SURFACECUBEMAP_LAYERED_LAYERS: 2046, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_LINEAR_WIDTH: 268435456, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_LINEAR_WIDTH: 131072, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_LINEAR_HEIGHT: 65000, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_LINEAR_PITCH: 2097120, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_MIPMAPPED_WIDTH: 32768, pycuda._driver.device_attribute.MAXIMUM_TEXTURE2D_MIPMAPPED_HEIGHT: 32768, pycuda._driver.device_attribute.COMPUTE_CAPABILITY_MAJOR: 6, pycuda._driver.device_attribute.COMPUTE_CAPABILITY_MINOR: 1, pycuda._driver.device_attribute.MAXIMUM_TEXTURE1D_MIPMAPPED_WIDTH: 16384, pycuda._driver.device_attribute.STREAM_PRIORITIES_SUPPORTED: 1, pycuda._driver.device_attribute.GLOBAL_L1_CACHE_SUPPORTED: 1, pycuda._driver.device_attribute.LOCAL_L1_CACHE_SUPPORTED: 1, 
pycuda._driver.device_attribute.MAX_SHARED_MEMORY_PER_MULTIPROCESSOR: 98304, 
pycuda._driver.device_attribute.MAX_REGISTERS_PER_MULTIPROCESSOR: 65536, 
pycuda._driver.device_attribute.MANAGED_MEMORY: 1, 
pycuda._driver.device_attribute.MULTI_GPU_BOARD: 0, 
pycuda._driver.device_attribute.MULTI_GPU_BOARD_GROUP_ID: 3
})]

nfrasser · March 9, 2020, 4:37pm

@MHB thanks, that helped.

In the Patch Motion job, try setting the “Override knots” parameters to the following:

Override knots Z: 5
Override knots Y: 6
Override knots X: 8

MHB · March 9, 2020, 5:59pm

Thanks, this still gives the same memory error.

hsnyder · May 13, 2020, 10:18pm

For those experiencing these memory allocation failures, a patch is now available that will address this problem. After you have updated to CryoSPARC v2.15, please see the following section of the new CryoSPARC guide regarding how to apply the patch (depending on whether your setup is single or multiple nodes)
https://guide.cryosparc.com/setup-configuration-and-management/software-updates#apply-patches

After the patch is applied, these jobs should complete successfully. You’ll notice a new switch in the job builder for patch motion correction, called “Reduce GPU memory usage”. It is on by default, but we consider this a beta feature and if you have any trouble with it, you can turn it off and revert to the previous behaviour.