Dear CryoSPARC community,
I am having a problem with NU refinement where the job fails with a no heart beat error.
“Job is unresponsive - no heartbeat received in 30 seconds.”
The input is the map and particle from a class in heterogenous refinement.
Job log is just filled with:
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
…
Seems to me the job is still running, but its flagged as failed (red). I have this issue with all of the NU refinement jobs.
Using latest CS version (3.2.0) on a Centos 8 machine.
4x 1080 Ti GPUs
40 CPU cores
128 GB
[cryosparc_user@localhost ~]$ cryosparcm log command_core
Changed job P1.J88 status launched
Running project UID P1 job UID J88
Running job on worker type node
Running job using: /home/cryosparc_user/cryosparc/cryosparc_worker/bin/cryosparcw
---------- Scheduler finished ---------------
Changed job P1.J88 status started
---- Deleting project UID P1 job UID J85
Now clearing job…
[EXPORT_JOB] : Request to export P1 J85
[EXPORT_JOB] : Exporting job to /data/cryosparc/USERS/omid/P1/J85
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/cryosparc/USERS/omid/P1/J85/gridfs_data…
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile…
[EXPORT_JOB] : Done. Exported in 0.00s
[EXPORT_JOB] : Updating job manifest…
[EXPORT_JOB] : Done. Updated in 0.05s
[EXPORT_JOB] : Exported P1 J85 in 0.06s
[EXPORT_JOB] : Request to export P1 J85
[EXPORT_JOB] : Exporting job to /data/cryosparc/USERS/omid/P1/J85
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/cryosparc/USERS/omid/P1/J85/gridfs_data…
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile…
[EXPORT_JOB] : Done. Exported in 0.00s
[EXPORT_JOB] : Updating job manifest…
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P1 J85 in 0.01s
Changed job P1.J88 status running
------------- Heartbeat check -------------------
deadline: 2021-10-01 17:49:49.016333
Overdue jobs : [{’_id’: ObjectId(‘61574033013c10123d6d6338’), ‘project_uid’: ‘P1’, ‘uid’: ‘J86’}, {’_id’: ObjectId(‘61574046013c10123d6d6a95’), ‘project_uid’: ‘P1’, ‘uid’: ‘J87’}]
Marking job P1.J86 as failed
Changed job P1.J86 status failed
Marking job P1.J87 as failed
Changed job P1.J87 status failed
------------- Heartbeat check done -------------
Best,
Omid