After upgrading to 3.2.0, I’m experiencing to execute the topaz extract job with the following reproducible error at the early stage of Preprocessing stage.
========= monitor process now starting main process MAINPROCESS PID 42032 MAIN PID 42032 topaz.run_topaz cryosparc_compute.jobs.jobregister Traceback (most recent call last): File "<string>", line 1, in <module> File "cryosparc_worker/cryosparc_compute/run.py", line 168, in cryosparc_compute.run.run File "/home/cryosparcuser/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py", line 1886, in get_instance_information cpufreq = psutil.cpu_freq() File "/home/cryosparcuser/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/psutil/__init__.py", line 1857, in cpu_freq ret = _psplatform.cpu_freq() File "/home/cryosparcuser/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/psutil/_pslinux.py", line 702, in cpu_freq "can't find current frequency file") NotImplementedError: can't find current frequency file slurmstepd: error: Step 45161.0 exceeded memory limit (8322468 > 8192000), being killed slurmstepd: error: *** STEP 45161.0 ON gpu026 CANCELLED AT 2021-07-05T20:28:27 *** srun: Job step aborted: Waiting up to 32 seconds for job step to finish. srun: error: gpu026: task 0: Killed
If anyone has any suggestions to look into, it would be very much appreciated.
hee jong kim