SLURM job does not start

Install, Configure and Update
#1

Hi,

I am having issues with running jobs - it just gets lunched and never starts. It looks the installation is fine but it could be problem with the cluster configuration or submission script. Any suggestions would be really helpful. I did try running the jobs with default cpu numbers but it didn’t help.

Screen Shot 2022-01-17 at 18.23.37
Screen Shot 2022-01-17 at 18.23.371109×499 37.6 KB

Screen Shot 2022-01-17 at 17.59.20
Screen Shot 2022-01-17 at 17.59.201172×603 76.6 KB

Screen Shot 2022-01-18 at 15.42.18
Screen Shot 2022-01-18 at 15.42.18753×514 36.9 KB

Thanks
Sudeep

#2

@sudeepka What does the Overview tab of a “Launched” job show?

#3

Hi @wtwmpel Here is the screenshot of the overview tab of the launched job.

Screen Shot 2022-01-18 at 18.54.57
Screen Shot 2022-01-18 at 18.54.57921×358 19.9 KB

#4

Hello @sudeepka ,

did you run cryosparcm cluster connect?
Do you see Lane "something" (cluster) on the Resource Manager?
How does your cluster_info.json and cluster_script.sh look like?

Best,

Juan

#5

@sudeepka If you haven’t yet resolved the problem, please post the output of
cryosparcm log command_core
shortly after scheduling a job, to ensure the log includes a recent record of a cluster job.

#6

@jucastil ran the connect again and this is how it looks

Screen Shot 2022-01-20 at 10.41.06
Screen Shot 2022-01-20 at 10.41.06905×708 232 KB

Do you see Lane "something" (cluster) on the Resource Manager? -I don´t see any thing like this.

Screen Shot 2022-01-20 at 10.44.38
Screen Shot 2022-01-20 at 10.44.381069×597 62.1 KB

.sh I already shared in the earlier converstaion in this thread. Here is my .json file

"name" : "2002260-gpu 1d",

"worker_bin_path" : "/sudeep/cryosparc_worker/bin/cryosparcw",

"cache_path" : "$LOCAL_SCRATCH",

"send_cmd_tpl" : "{{ command }}",

"qsub_cmd_tpl" : "sbatch {{ script_path_abs }}",

"qstat_cmd_tpl" : "squeue -j {{ cluster_job_id }}",

"qdel_cmd_tpl" : "scancel {{ cluster_job_id }}",

"qinfo_cmd_tpl" : "sinfo",

"transfer_cmd_tpl" : "cp {{ src_path }} {{ dest_path }}"
#7

@wtempel No I have not resolved the problem yet. If I run cryosparcm log command_core I this this detail and I don´t see the recently started job there … tried few times running the command but no I don´t see the recent record. Also I see this error: OSError: [Errno 98] Address already in use. Here is the details :

Screen Shot 2022-01-20 at 10.57.23
Screen Shot 2022-01-20 at 10.57.23907×697 146 KB

Screen Shot 2022-01-20 at 10.57.33
Screen Shot 2022-01-20 at 10.57.33913×187 31.8 KB

Screen Shot 2022-01-20 at 11.03.38

#8

Hi @sudeepka @wtempel for me looks like a communication problem (python issue)

Cryosparc tries to launch the job but the port is not open or not available…

I would try stopping the cryosparc instance and starting it again pointing to a different port…

#9

@sudeepka Is there another cryoSPARC master instance running on the same computer, perhaps installed by another user and relying on the same range of network ports?
Multiple cryoSPARC master instances on the same computer must be carefully isolated from each other.

#10

@wtempel Thanks again for reaching me. I managed to fix it. My master was in bad state so I killed all the ongoing jobs and restarted the cryosparcm and ran cryosparcm cluster connect that perfectly helped.