T20 Job failed - I can run these models with g4dn compute nodes but with p3dn it is failing

Troubleshooting
#1 
Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 84, in cryosparc_compute.run.main
  File "/apps/software/cryosparc/cryosparc-v3.2_master/cryosparc_compute/jobs/workflows/buildrun_bench.py", line 285, in run_extensive_workflow
    juids['inspect_blob_picks'] = get_curated_blob_picks(job, juids['patch_ctf_est']+'.exposures')
  File "/apps/software/cryosparc/cryosparc-v3.2_master/cryosparc_compute/jobs/workflows/buildrun_bench.py", line 160, in get_curated_blob_picks
    assert jstatus == 'waiting', "Inspect picks did not start within 120 seconds!"
AssertionError: Inspect picks did not start within 120 seconds!

I would like to know more about the message “Inspect picks did not start within 120 seconds!”

#2 
================= CRYOSPARCW =======  2021-10-20 13:50:02.588177  =========
Project P9 Job J7
Master XXXXXXXX Port 39002
===========================================================================
========= monitor process now starting main process
MAINPROCESS PID 11498
========= monitor process now waiting for main process
MAIN PID 11498
manual_picker.run_inspect_simple cryosparc_compute.jobs.jobregister
***************************************************************
INTERACTIVE JOB STARTED ===  2021-10-20 13:50:05.375682  ==========================
 * Serving Flask app "select_2D" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
#3

What does it mean “sending heartbeat” , is it waiting for another machine?

#4

Hi @pvalappil,

Can you try cloning this Inspect Picks job and run it manually? Please also send me the contents of the “Overview” tab for the Inspect Picks job. The T20s workflow is pretty lightweight, it shouldn’t be taking more that 2 minutes to start up.

#5

Hi Stephan, I am able to solve it. please close this task