Jobs get queued, but don't start

Troubleshooting
1

Hi Ali,

I am trying to use CryoSPARCv2 on a new workstation (CentOS7). Though all the GPUs, memory and SSD are available for running CryoSPARC jobs, none of my jobs get started and they remain queued.
I noticed in the forum that others have faced similar issue, but didn’t find a solution that helped.

Following is the output I am getting for cryosparcm joblog PX JX :

Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc2_compute/client.py", line 83, in <module>
    print eval("cli."+command)
  File "<string>", line 1, in <module>
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc2_compute/client.py", line 57, in func
    assert False, res['error']
AssertionError: {u'message': u"OtherError: argument of type 'NoneType' is not iterable", u'code': 500, u'data': None, u'name': u'OtherError'}

The output for cryosparcm log command_core is:

Base path:  /data/work/bojian
Indexing /data/work/bojian/stone_colab ----
Base path:  /data/work/bojian/stone_colab
Indexing /data/work/bojian/stone_colab/YS7_cryo_manu_08242019 ----
Base path:  /data/work/bojian/stone_colab/YS7_cryo_manu_08242019
Indexing /data/work/bojian/stone_colab/YS7_cryo_manu_08242019/relion_processing ----
Base path:  /data/work/bojian/stone_colab/YS7_cryo_manu_08242019/relion_processing
Indexing /data/work/bojian/stone_colab/YS7_cryo_manu_08242019/relion_processing/Extract ----
Base path:  /data/work/bojian/stone_colab/YS7_cryo_manu_08242019/relion_processing/Extract
Indexing /data/work/bojian/stone_colab/YS7_cryo_manu_08242019/relion_processing/Extract/Extract_motioncorr2_bin4 ----
Base path:  /data/work/bojian/stone_colab/YS7_cryo_manu_08242019/relion_processing/Extract/Extract_motioncorr2_bin4
Indexing /data/work/bojian/stone_colab/YS7_cryo_manu_08242019/relion_processing/Extract/Extract_motioncorr2_bin4/particles.star ----
Base path:  /data/work/bojian/stone_colab/YS7_cryo_manu_08242019/relion_processing/Extract/Extract_motioncorr2_bin4
Indexing /data/work/bojian/stone_colab/YS7_cryo_manu_08242019/relion_processing/Extract/Extract_motioncorr2_bin4/particles.star ----
Base path:  /data/work/bojian/stone_colab/YS7_cryo_manu_08242019/relion_processing/Extract/Extract_motioncorr2_bin4
particle_meta_path path
Setting parameter J1.particle_meta_path with value /data/work/bojian/stone_colab/YS7_cryo_manu_08242019/relion_processing/Extract/Extract_motioncorr2_bin4/particles.star of type <type 'str'>
failed to connect link
---- Deleting project UID P13 job UID J1
     Now clearing job..
COMMAND CORE STARTED ===  2019-09-06 00:57:23.815447  ==========================
*** BG WORKER START
COMMAND CORE STARTED ===  2019-09-13 21:49:04.137190  ==========================
*** BG WORKER START
COMMAND CORE STARTED ===  2019-09-13 21:49:14.709424  ==========================
*** BG WORKER START
COMMAND CORE STARTED ===  2019-09-13 21:49:50.838601  ==========================
*** BG WORKER START
[EXPORT_PROJECT] : Exporting project P1...
[EXPORT_PROJECT] : /data/work/saikat/test/P1 does not exist. Skipping export of P1
[EXPORT_PROJECT] : Exporting project P2...
[EXPORT_PROJECT] : /data/work/saikat/test/P1/P2 does not exist. Skipping export of P2
[EXPORT_PROJECT] : Exporting project P3...
[EXPORT_PROJECT] : /data/work/saikat/test/P3 does not exist. Skipping export of P3
[EXPORT_PROJECT] : Exporting project P4...
[EXPORT_PROJECT] : /data/work/bojian/test/P4 does not exist. Skipping export of P4
[EXPORT_PROJECT] : Exporting project P5...
[EXPORT_PROJECT] : /data/work/saikat/P5 does not exist. Skipping export of P5
[EXPORT_PROJECT] : Exporting project P6...
[EXPORT_PROJECT] : /data/work/cryosparc_data/bojian/test/P6 does not exist. Skipping export of P6
[EXPORT_PROJECT] : Exporting project P7...
[EXPORT_PROJECT] : /data/work/cryosparc_data/bojian/sc_test/P7 does not exist. Skipping export of P7
[EXPORT_PROJECT] : Exporting project P8...
[EXPORT_PROJECT] : /data/work/cryosparc_data/bojian/stone_colab/YS7_cryo/P8 does not exist. Skipping export of P8
[EXPORT_PROJECT] : Exporting project P9...
[EXPORT_PROJECT] : /data/work/cryosparc_data/bojian/stone_colab/YS7_cryo/P9 does not exist. Skipping export of P9
[EXPORT_PROJECT] : Exporting project P10...
[EXPORT_PROJECT] : /data/work/cryosparc_data/saikat/P10 does not exist. Skipping export of P10
[EXPORT_PROJECT] : Exporting project P11...
[EXPORT_PROJECT] : Exported project P11 to /data/work/cryosparc_data/saikat/test/P11/project.json in 0.02s
[EXPORT_PROJECT] : Exporting project P12...
[EXPORT_PROJECT] : Exported project P12 to /data/work/bojian/stone_colab/YS7_cryo_manu_08242019/relion_processing/P12/project.json in 0.00s
[EXPORT_PROJECT] : Exporting project P13...
[EXPORT_PROJECT] : Exported project P13 to /data/work/cryosparc_data/saikat/test/P13/project.json in 0.00s
[EXPORT_JOB] : Request to export P13 J2
[EXPORT_JOB] :    Exporting job to /data/work/cryosparc_data/saikat/test/P13/J2
[EXPORT_JOB] :    Exporting all of job's images in the database to /data/work/cryosparc_data/saikat/test/P13/J2/gridfs_data...
[EXPORT_JOB] :    Done. Exported 0 images in 0.00s
[EXPORT_JOB] :    Exporting all job's streamlog events...
[EXPORT_JOB] :    Done. Exported 1 files in 0.00s
[EXPORT_JOB] :    Exporting job metafile...
[EXPORT_JOB] :    Done. Exported in 0.00s
[EXPORT_JOB] :    Updating job manifest...
[EXPORT_JOB] :    Done. Updated in 0.00s
[EXPORT_JOB] : Exported P13 J2 in 0.02s
particle_meta_path path
Setting parameter J2.particle_meta_path with value /data/work/bojian/stone_colab/YS7_cryo_manu_08242019/relion_processing/Extract/Extract_motioncorr2_bin4/particles.star of type <type 'str'>
[EXPORT_JOB] : Request to export P13 J2
[EXPORT_JOB] :    Exporting job to /data/work/cryosparc_data/saikat/test/P13/J2
[EXPORT_JOB] :    Exporting all of job's images in the database to /data/work/cryosparc_data/saikat/test/P13/J2/gridfs_data...
[EXPORT_JOB] :    Done. Exported 0 images in 0.00s
[EXPORT_JOB] :    Exporting all job's streamlog events...
[EXPORT_JOB] :    Done. Exported 1 files in 0.00s
[EXPORT_JOB] :    Exporting job metafile...
[EXPORT_JOB] :    Done. Exported in 0.00s
[EXPORT_JOB] :    Updating job manifest...
[EXPORT_JOB] :    Done. Updated in 0.00s
[EXPORT_JOB] : Exported P13 J2 in 0.01s
failed to connect link
---- Deleting project UID P13 job UID J2
     Now clearing job..
[EXPORT_JOB] : Request to export P13 J2
[EXPORT_JOB] :    Exporting job to /data/work/cryosparc_data/saikat/test/P13/J2
[EXPORT_JOB] :    Exporting all of job's images in the database to /data/work/cryosparc_data/saikat/test/P13/J2/gridfs_data...
[EXPORT_WORKSPACES] : Exporting all workspaces in P13...
[EXPORT_WORKSPACES] : Exported all workspaces in P13 to /data/work/cryosparc_data/saikat/test/P13/workspaces.json in 0.00s
---- Deleting project UID P7 job UID J1
     Now clearing job..
[EXPORT_JOB] : Request to export P7 J1
[EXPORT_JOB] :    Exporting job to /data/work/cryosparc_data/bojian/sc_test/P7/J1
[EXPORT_JOB] :    Exporting all of job's images in the database to /data/work/cryosparc_data/bojian/sc_test/P7/J1/gridfs_data...
[EXPORT_JOB] :    Done. Exported 0 images in 0.00s
[EXPORT_JOB] :    Exporting all job's streamlog events...
[EXPORT_JOB] :    Done. Exported 1 files in 0.00s
[EXPORT_JOB] :    Exporting job metafile...
[EXPORT_JOB] :    Done. Exported in 0.00s
[EXPORT_JOB] :    Updating job manifest...
[EXPORT_JOB] :    Done. Updated in 0.00s
[EXPORT_JOB] : Exported P7 J1 in 0.02s
[EXPORT_WORKSPACES] : Exporting all workspaces in P7...
[EXPORT_WORKSPACES] : Exported all workspaces in P7 to /data/work/cryosparc_data/bojian/sc_test/P7/workspaces.json in 0.00s
[JSONRPC ERROR  2019-09-15 20:20:41.885038  at  get_job ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 113, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 3124, in get_job
    return sanitize_id(com.get_job(mongo.db, project_uid, job_uid, *args, **kwargs))
  File "cryosparc2_command/command_core/__init__.py", line 163, in sanitize_id
    if '_id' in obj:
TypeError: argument of type 'NoneType' is not iterable
-----------------------------------------------------
[JSONRPC ERROR  2019-09-15 20:20:41.885923  at  get_job_dir_abs ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 113, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 4048, in get_job_dir_abs
    job_doc = get_job(project_uid, job_uid, 'job_dir')
  File "cryosparc2_command/command_core/__init__.py", line 122, in wrapper
    raise e
TypeError: argument of type 'NoneType' is not iterable
-----------------------------------------------------
[JSONRPC ERROR  2019-09-15 20:20:41.885968  at  get_job_log_path_abs ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 113, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 4054, in get_job_log_path_abs
    job_dir_abs = get_job_dir_abs(project_uid, job_uid)
  File "cryosparc2_command/command_core/__init__.py", line 122, in wrapper
    raise e
TypeError: argument of type 'NoneType' is not iterable
-----------------------------------------------------
[JSONRPC ERROR  2019-09-15 20:23:35.870524  at  get_job ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 113, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 3124, in get_job
    return sanitize_id(com.get_job(mongo.db, project_uid, job_uid, *args, **kwargs))
  File "cryosparc2_command/command_core/__init__.py", line 163, in sanitize_id
    if '_id' in obj:
TypeError: argument of type 'NoneType' is not iterable
-----------------------------------------------------
[JSONRPC ERROR  2019-09-15 20:23:35.870711  at  get_job_dir_abs ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 113, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 4048, in get_job_dir_abs
    job_doc = get_job(project_uid, job_uid, 'job_dir')
  File "cryosparc2_command/command_core/__init__.py", line 122, in wrapper
    raise e
TypeError: argument of type 'NoneType' is not iterable
-----------------------------------------------------
[JSONRPC ERROR  2019-09-15 20:23:35.870811  at  get_job_log_path_abs ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 113, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 4054, in get_job_log_path_abs
    job_dir_abs = get_job_dir_abs(project_uid, job_uid)
  File "cryosparc2_command/command_core/__init__.py", line 122, in wrapper
    raise e
TypeError: argument of type 'NoneType' is not iterable
-----------------------------------------------------
[JSONRPC ERROR  2019-09-15 20:41:16.077544  at  get_job ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 113, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 3124, in get_job
    return sanitize_id(com.get_job(mongo.db, project_uid, job_uid, *args, **kwargs))
  File "cryosparc2_command/command_core/__init__.py", line 163, in sanitize_id
    if '_id' in obj:
TypeError: argument of type 'NoneType' is not iterable
-----------------------------------------------------
[JSONRPC ERROR  2019-09-15 20:41:16.077734  at  get_job_dir_abs ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 113, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 4048, in get_job_dir_abs
    job_doc = get_job(project_uid, job_uid, 'job_dir')
  File "cryosparc2_command/command_core/__init__.py", line 122, in wrapper
    raise e
TypeError: argument of type 'NoneType' is not iterable
-----------------------------------------------------
[JSONRPC ERROR  2019-09-15 20:41:16.077831  at  get_job_log_path_abs ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 113, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 4054, in get_job_log_path_abs
    job_dir_abs = get_job_dir_abs(project_uid, job_uid)
  File "cryosparc2_command/command_core/__init__.py", line 122, in wrapper
    raise e
TypeError: argument of type 'NoneType' is not iterable
-----------------------------------------------------

Any suggestions or fix will be highly appreciated.

Thanks,
-Saikat

2

Hi @saikat,

Can you send us the output of the following commands:
cryosparcm status
cryosparcm log database

Is this a brand new installation of cryoSPARC?

3

Hi Stephan,

Output for cryosparcm status is:

CryoSPARC System master node installed at

/home/cryosparc_user/software/cryosparc/cryosparc2_master

Current cryoSPARC version: v2.11.0

The output for cryosparcm log database is:

2019-09-16T13:50:30.601-0400 I NETWORK  [conn49919] received client metadata from 129.49.104.55:59976 conn49919: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.6.1810 Core", architecture: "x86_64", version: "3.10.0-957.27.2.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2019-09-16T13:50:30.641-0400 I -        [conn49919] end connection 129.49.104.55:59976 (11 connections now open)
2019-09-16T13:50:35.328-0400 I NETWORK  [thread1] connection accepted from 129.49.104.55:60000 #49920 (11 connections now open)
2019-09-16T13:50:35.329-0400 I NETWORK  [conn49920] received client metadata from 129.49.104.55:60000 conn49920: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.6.1810 Core", architecture: "x86_64", version: "3.10.0-957.27.2.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2019-09-16T13:50:35.367-0400 I -        [conn49920] end connection 129.49.104.55:60000 (11 connections now open)
2019-09-16T13:50:40.019-0400 I NETWORK  [thread1] connection accepted from 129.49.104.55:60028 #49921 (11 connections now open)
2019-09-16T13:50:40.020-0400 I NETWORK  [conn49921] received client metadata from 129.49.104.55:60028 conn49921: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.6.1810 Core", architecture: "x86_64", version: "3.10.0-957.27.2.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2019-09-16T13:50:40.056-0400 I -        [conn49921] end connection 129.49.104.55:60028 (11 connections now open)
2019-09-16T13:50:44.735-0400 I NETWORK  [thread1] connection accepted from 129.49.104.55:60054 #49922 (11 connections now open)
2019-09-16T13:50:44.736-0400 I NETWORK  [conn49922] received client metadata from 129.49.104.55:60054 conn49922: { driver: { name: "PyMongo", version: "3.4.0" }, os: { type: "Linux", name: "CentOS Linux 7.6.1810 Core", architecture: "x86_64", version: "3.10.0-957.27.2.el7.x86_64" }, platform: "CPython 2.7.15.final.0" }
2019-09-16T13:50:44.773-0400 I -        [conn49922] end connection 129.49.104.55:60054 (11 connections now open)

This is a brand new installation of cryosparc on a new GPU workstation.

Thanks,
-Saikat

4

Hi Stephan,

Any suggestions for troubleshooting?

I reinstalled cryosparc v2.11 today and have the same problem.

Thanks,
-Saikat

5

Hi @saikat, is this still an issue?

6

Hi Sarah,

I was able to resolve this issue by uninstalling CryoSPARC and doing a new reinstall.

Thanks,
-Saikat

7

Thanks @saikat! Glad it was resolved.