Dear all,
With our current installation, we can do patch motion correction from movies in .mrc format collected with a K3 camera without using super resolution mode. However, with a dataset containing movies collected in super resolution mode on the K3, in .tiff format, patch motion correction fails.
We are using G Force RTX 2080 and 2080 Titanium.
CUDA version 10.0
The SLURM script is the following:
#SBATCH --job-name cryosparc_P12_J21
#SBATCH -n 12
#SBATCH --gres=gpu:2
#SBATCH -p gpu
#SBATCH --mem=64000MB
#SBATCH -o /home/cryosparc_user/cryo_em/cryosparc_projects/mkiller/P12/J21/output.log
#SBATCH -e /home/cryosparc_user/cryo_em/cryosparc_projects/mkiller/P12/J21/error.log
export MODULEPATH=/opt/ohpc/pub/modulefiles:/g/easybuild/x86_64/CentOS/7/haswell/modules/all
source /opt/ohpc/admin/lmod/lmod/init/bash
echo CUDA_VISIBLE_DEVICE=$CUDA_VISIBLE_DEVICES
echo hostname=`hostname`
module load cryosparc2_worker/2.14.2-foss-2018b
module list
cryosparcw run --project P12 --job J21 --master_hostname cryosparc.embl-hamburg.de --master_command_core_port 39002 > /home/cryosparc_user/cryo_em/cryosparc_projects/mkiller/P12/J21/job.log 2>&1
The error message is the following:
[CPU: 498.3 MB] Traceback (most recent call last):
File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
run_old(*args, **kw)
File "/g/easybuild/x86_64/CentOS/7/haswell/software/cryosparc2_worker/2.14.2-foss-2018b/deps/anaconda/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "cryosparc2_compute/jobs/pipeline.py", line 153, in thread_work
work = processor.process(item)
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 157, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 160, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 161, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 446, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 197, in cryosparc2_compute.engine.cuda_core.transfer_ndarray_to_cudaarray
MemoryError: cuArrayCreate failed: out of memory
[CPU: 163.6 MB] Outputting partial results now...
[CPU: 163.7 MB] Traceback (most recent call last):
File "cryosparc2_worker/cryosparc2_compute/run.py", line 82, in cryosparc2_compute.run.main
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 349, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi
AssertionError: Child process with PID 64102 has terminated unexpectedly!
Has anyone experienced the same issue and have a solution ?
Thank you in advance,
Maxime.