Workaround to set dynamic GPU allocation using the environment variable for DeepEMhancer


we are using RTX 2070 super cards (8 GB RAM) for cryoEM reconstructions in Cryosparc.
When we run the DeepEMhancer outside of Cryosparc we get this type of error:

(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node conv3d_1/convolution}}]]
0 successful operations.
0 derived errors ignored.

which can be dealt with (as recommended on github: GitHub - rsanchezgarc/deepEMhancer: Deep learning for cryo-EM maps post-processing), by setting this value TF_FORCE_GPU_ALLOW_GROWTH=‘true’

prior to invoking deepemhancer command with all its options:

TF_FORCE_GPU_ALLOW_GROWTH=‘true’ deepemhancer -p highRes

In Cryosparc DeepEMhancer wrapper however, at this moment we cannot see a way to use this option, so instead, we modified the script in the cryosparc_worker installation.

In our case this script is found in the following path:

we added the following line

os.environ[‘TF_FORCE_GPU_ALLOW_GROWTH’] = ‘true’

in the following context:

import subprocess
os.environ[‘TF_FORCE_GPU_ALLOW_GROWTH’] = ‘true’

def run_deepemhancer_wrapper(job):

The GPU error does not appear anymore. It may be wiped be a new CS update though.

I hope this could be useful to someone.
All the best
Lukasz K


Hi @Lukasz_K, thank you for pointing this out! We’ve recorded this and will likely add the dynamic GPU allocation option to the wrapper in a future update.