Hello,
We were trying to remove intermediate results in several of our projects.
We get several progress bars for the clean up jobs, but they seem to be stuck for days. We do not see reduction in used space. Is there any way to debug what is wrong?
We are using cryosparc 2.15.0.
Kind regards.
Miguel
same here, was this ever resolved?
The most likely case is that the process got interrupted and stopped, but the notification didn’t get cleared. You should be fine to clear the notification at this point (in the Notification Manager found in the Resource Manager) and restart the “Clear Intermediate Results” function, which will continue where it left off.
I have tried that a few times but it just start the clearance again and then get stuck at some point. not sure how to continue. I have also started clearing the jobs one at a time (over 700 of them…) but that didn’t help either when i later tried clearing the project.
Hi @zalkr,
Can you try the command again, and while it’s happening/when it freezes, run the command:
cryosparcm log command_core so we can get some logs from the server?
Sorry for the late reply. This never got solved. But since I opened this issue, we have migrated our cryosparc installation and database into a new server. We have not seen the issue yet, but I am unsure if anybody has cleared intermediate results since we moved our instance.
I can agree with @zalkr, even after clearing notifications, restarting the clean-up would always get stuck at the same stage.
Hi @vatese,
Can you send us logs from command core when you notice this problem?
cryosparcm log command_core
Hi,
seems like I have the same issue, everytime a user cleans a certain project, I get a stuck progress bar…
Here is the last portion of the requested logs. J47 was a Local Resolution job. J48 does not exist in this workspace.
[EXPORT_JOB] : Writing 17 database images to [SOMEPATH]/P39/J47/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 17 images in 0.13s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for volume
[EXPORT_JOB] : Done. Exported in 0.02s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P39 J47 in 0.17s
[POST-RESPONSE-THREAD ERROR 2020-10-28 09:25:17.675362 at clear_intermediate_results_run ]
-----------------------------------------------------
Traceback (most recent call last):
File "cryosparc2_command/command_core/__init__.py", line 146, in run
self.target(*self.args)
File "cryosparc2_command/command_core/__init__.py", line 4135, in clear_intermediate_results_run
rc.clear_intermediate_results(project_uid, job['uid'], always_keep_final=True)
File "cryosparc2_compute/jobs/runcommon.py", line 625, in clear_intermediate_results
output_result = com.query(job['output_results'], lambda r: r['group_name'] == slot['group_name'] and r['name'] == slot['result_name'] , error="No match for %s.%s" % (slot['group_name'], slot['result_name']))
File "cryosparc2_compute/jobs/common.py", line 357, in query
assert res != default, error
AssertionError: No match for imported_particles.blob
-----------------------------------------------------
**custom thread exception hook caught something
**** handle exception rc
Exception in thread Thread-18752:
Traceback (most recent call last):
File "/opt/cryosparc_cluster/cryosparc2_master/deps/anaconda/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "cryosparc2_compute/jobs/runcommon.py", line 1689, in run_with_except_hook
handle_exception(*sys.exc_info())
File "cryosparc2_compute/jobs/runcommon.py", line 1653, in handle_exception
set_job_status('failed')
File "cryosparc2_compute/jobs/runcommon.py", line 139, in set_job_status
cli.set_job_status(_project_uid, _job_uid, status)
AttributeError: 'NoneType' object has no attribute 'set_job_status'
set status to failed
Since this is easily reproducible in this case, let me know if you need more info.
Hi @lkat,
This is very helpful, thank you for sharing. Looks like we need better error handling for this function!
[EXPORT_JOB] : Request to export P2 J19
[EXPORT_JOB] : Exporting job to /mnt/Data/Shimon_Berstein/P1/J19
[EXPORT_JOB] : Exporting all of job's images in the database to /mnt/Data/Shimon_Berstein/P1/J19/gridfs_data...
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Done. Exported in 0.01s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P2 J19 in 0.01s
[EXPORT_JOB] : Request to export P2 J190
[EXPORT_JOB] : Exporting job to /mnt/Data/Shimon_Berstein/P1/J190
[EXPORT_JOB] : Exporting all of job's images in the database to /mnt/Data/Shimon_Berstein/P1/J190/gridfs_data...
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for exposures_accepted
[EXPORT_JOB] : Creating .csg file for exposures_uncategorized
[EXPORT_JOB] : Done. Exported in 0.02s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P2 J190 in 0.03s
[EXPORT_JOB] : Request to export P2 J191
[EXPORT_JOB] : Exporting job to /mnt/Data/Shimon_Berstein/P1/J191
[EXPORT_JOB] : Exporting all of job's images in the database to /mnt/Data/Shimon_Berstein/P1/J191/gridfs_data...
[EXPORT_JOB] : Writing 16 database images to /mnt/Data/Shimon_Berstein/P1/J191/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 16 images in 0.04s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.01s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for particles
[EXPORT_JOB] : Creating .csg file for micrographs
[EXPORT_JOB] : Done. Exported in 0.02s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P2 J191 in 0.07s
[EXPORT_JOB] : Request to export P2 J192
[EXPORT_JOB] : Exporting job to /mnt/Data/Shimon_Berstein/P1/J192
[EXPORT_JOB] : Exporting all of job's images in the database to /mnt/Data/Shimon_Berstein/P1/J192/gridfs_data...
[EXPORT_JOB] : Writing 18 database images to /mnt/Data/Shimon_Berstein/P1/J192/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 18 images in 0.06s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for micrographs
[EXPORT_JOB] : Creating .csg file for particles
[EXPORT_JOB] : Done. Exported in 0.01s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P2 J192 in 0.09s
[EXPORT_JOB] : Request to export P2 J193
[EXPORT_JOB] : Exporting job to /mnt/Data/Shimon_Berstein/P1/J193
[EXPORT_JOB] : Exporting all of job's images in the database to /mnt/Data/Shimon_Berstein/P1/J193/gridfs_data...
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Done. Exported in 0.00s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P2 J193 in 0.01s
[EXPORT_JOB] : Request to export P2 J194
[EXPORT_JOB] : Exporting job to /mnt/Data/Shimon_Berstein/P1/J194
[EXPORT_JOB] : Exporting all of job's images in the database to /mnt/Data/Shimon_Berstein/P1/J194/gridfs_data...
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.01s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for particles
[EXPORT_JOB] : Done. Exported in 0.02s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.01s
[EXPORT_JOB] : Exported P2 J194 in 0.04s
[EXPORT_JOB] : Request to export P2 J195
[EXPORT_JOB] : Exporting job to /mnt/Data/Shimon_Berstein/P1/J195
[EXPORT_JOB] : Exporting all of job's images in the database to /mnt/Data/Shimon_Berstein/P1/J195/gridfs_data...
[EXPORT_JOB] : Writing 232 database images to /mnt/Data/Shimon_Berstein/P1/J195/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 232 images in 0.58s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.02s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for particles
[EXPORT_JOB] : Creating .csg file for class_averages
[EXPORT_JOB] : Done. Exported in 0.02s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.01s
[EXPORT_JOB] : Exported P2 J195 in 0.62s
[EXPORT_JOB] : Request to export P2 J196
[EXPORT_JOB] : Exporting job to /mnt/Data/Shimon_Berstein/P1/J196
[EXPORT_JOB] : Exporting all of job's images in the database to /mnt/Data/Shimon_Berstein/P1/J196/gridfs_data...
[EXPORT_JOB] : Writing 59 database images to /mnt/Data/Shimon_Berstein/P1/J196/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 59 images in 0.11s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for particles_selected
[EXPORT_JOB] : Creating .csg file for templates_selected
[EXPORT_JOB] : Creating .csg file for particles_excluded
[EXPORT_JOB] : Creating .csg file for templates_excluded
[EXPORT_JOB] : Done. Exported in 0.02s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P2 J196 in 0.14s
[EXPORT_JOB] : Request to export P2 J197
[EXPORT_JOB] : Exporting job to /mnt/Data/Shimon_Berstein/P1/J197
[EXPORT_JOB] : Exporting all of job's images in the database to /mnt/Data/Shimon_Berstein/P1/J197/gridfs_data...
[EXPORT_JOB] : Writing 232 database images to /mnt/Data/Shimon_Berstein/P1/J197/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 232 images in 0.51s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.01s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for particles
[EXPORT_JOB] : Creating .csg file for class_averages
[EXPORT_JOB] : Done. Exported in 0.01s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.01s
[EXPORT_JOB] : Exported P2 J197 in 0.54s
[EXPORT_JOB] : Request to export P2 J198
[EXPORT_JOB] : Exporting job to /mnt/Data/Shimon_Berstein/P1/J198
[EXPORT_JOB] : Exporting all of job's images in the database to /mnt/Data/Shimon_Berstein/P1/J198/gridfs_data...
[EXPORT_JOB] : Writing 59 database images to /mnt/Data/Shimon_Berstein/P1/J198/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 59 images in 0.10s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for particles_selected
[EXPORT_JOB] : Creating .csg file for templates_selected
[EXPORT_JOB] : Creating .csg file for particles_excluded
[EXPORT_JOB] : Creating .csg file for templates_excluded
[EXPORT_JOB] : Done. Exported in 0.02s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P2 J198 in 0.13s
[EXPORT_JOB] : Request to export P2 J199
[EXPORT_JOB] : Exporting job to /mnt/Data/Shimon_Berstein/P1/J199
[EXPORT_JOB] : Exporting all of job's images in the database to /mnt/Data/Shimon_Berstein/P1/J199/gridfs_data...
[EXPORT_JOB] : Writing 222 database images to /mnt/Data/Shimon_Berstein/P1/J199/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 222 images in 0.40s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.10s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for particles_all_classes
[EXPORT_JOB] : Creating .csg file for particles_class_0
[EXPORT_JOB] : Creating .csg file for volume_class_0
[EXPORT_JOB] : Creating .csg file for particles_class_1
[EXPORT_JOB] : Creating .csg file for volume_class_1
[EXPORT_JOB] : Creating .csg file for particles_class_2
[EXPORT_JOB] : Creating .csg file for volume_class_2
[EXPORT_JOB] : Done. Exported in 0.04s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P2 J199 in 0.55s
[EXPORT_JOB] : Request to export P2 J2
[EXPORT_JOB] : Exporting job to /mnt/Data/Shimon_Berstein/P1/J2
[EXPORT_JOB] : Exporting all of job's images in the database to /mnt/Data/Shimon_Berstein/P1/J2/gridfs_data...
[EXPORT_JOB] : Writing 23 database images to /mnt/Data/Shimon_Berstein/P1/J2/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 23 images in 0.03s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.02s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for micrographs
[JSONRPC ERROR 2020-10-28 16:11:57.388779 at dump_job_database ]
-----------------------------------------------------
Traceback (most recent call last):
File "cryosparc2_command/command_core/__init__.py", line 108, in wrapper
res = func(*args, **kwargs)
File "cryosparc2_command/command_core/__init__.py", line 2787, in dump_job_database
rc.dump_job_database(project_uid = project_uid, job_uid = job_uid, job_completed = job_completed, migration = migration, abs_export_dir = abs_export_dir)
File "cryosparc2_compute/jobs/runcommon.py", line 415, in dump_job_database
with open(abs_csg_file_path, 'w') as openfile:
IOError: [Errno 13] Permission denied: u'/mnt/Data/Shimon_Berstein/P1/J2/P2_J2_micrographs.csg'
-----------------------------------------------------
[POST-RESPONSE-THREAD ERROR 2020-10-28 16:11:57.389668 at clear_intermediate_results_run ]
-----------------------------------------------------
Traceback (most recent call last):
File "cryosparc2_command/command_core/__init__.py", line 139, in run
self.target(*self.args)
File "cryosparc2_command/command_core/__init__.py", line 4153, in clear_intermediate_results_run
dump_job_database(project_uid, job['uid'])
File "cryosparc2_command/command_core/__init__.py", line 117, in wrapper
raise e
IOError: [Errno 13] Permission denied: u'/mnt/Data/Shimon_Berstein/P1/J2/P2_J2_micrographs.csg'
-----------------------------------------------------
**custom thread exception hook caught something
**** handle exception rc
Exception in thread Thread-3535840:
Traceback (most recent call last):
File "/home/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "cryosparc2_compute/jobs/runcommon.py", line 1682, in run_with_except_hook
handle_exception(*sys.exc_info())
File "cryosparc2_compute/jobs/runcommon.py", line 1646, in handle_exception
set_job_status('failed')
File "cryosparc2_compute/jobs/runcommon.py", line 132, in set_job_status
cli.set_job_status(_project_uid, _job_uid, status)
AttributeError: 'NoneType' object has no attribute 'set_job_status'
set status to failed
Is there an easy fix for this?
Hi @zalkr,
Can you check the permissions for the folder /mnt/Data/Shimon_Berstein/P1/J2/? Does it have the correct write permissions for the user running the cryoSPARC application?
Hi @stephan.
Sorry for the late reply. Here is how it looks like:
P42 had bad permissions.
P32 returned these at run/command_core.log
Clearing intermediate results for all jobs in P32
[EXPORT_JOB] : Request to export P32 J1
[EXPORT_JOB] : Exporting job to /stornext/HPCScratch/cryoSPARC_data/uploads/HYBRID/P32/J1
[EXPORT_JOB] : Exporting all of job's images in the database to /stornext/HPCScratch/cryoSPARC_data/uploads/HYBRID/P32/J1/gridfs_data...
[EXPORT_JOB] : Writing 7 database images to /stornext/HPCScratch/cryoSPARC_data/uploads/HYBRID/P32/J1/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 7 images in 0.12s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.03s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for imported_movies
[EXPORT_JOB] : Done. Exported in 0.02s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.01s
[EXPORT_JOB] : Exported P32 J1 in 0.18s
...
[EXPORT_JOB] : Request to export P32 J599
[EXPORT_JOB] : Exporting job to /stornext/HPCScratch/cryoSPARC_data/uploads/HYBRID/P32/J599
[EXPORT_JOB] : Exporting all of job's images in the database to /stornext/HPCScratch/cryoSPARC_data/uploads/HYBRID/P32/J599/gridfs_data...
[EXPORT_JOB] : Writing 1134 database images to /stornext/HPCScratch/cryoSPARC_data/uploads/HYBRID/P32/J599/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 1134 images in 9.20s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 1.03s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for particles_all_classes
[EXPORT_JOB] : Creating .csg file for particles_class_0
[EXPORT_JOB] : Creating .csg file for volume_class_0
[EXPORT_JOB] : Creating .csg file for particles_class_1
[EXPORT_JOB] : Creating .csg file for volume_class_1
[EXPORT_JOB] : Creating .csg file for particles_class_2
[EXPORT_JOB] : Creating .csg file for volume_class_2
[EXPORT_JOB] : Creating .csg file for particles_class_3
[EXPORT_JOB] : Creating .csg file for volume_class_3
[EXPORT_JOB] : Creating .csg file for particles_class_4
[EXPORT_JOB] : Creating .csg file for volume_class_4
[EXPORT_JOB] : Creating .csg file for particles_class_5
[EXPORT_JOB] : Creating .csg file for volume_class_5
[EXPORT_JOB] : Done. Exported in 0.07s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P32 J599 in 10.30s
[POST-RESPONSE-THREAD ERROR 2020-12-08 13:13:36.658626 at clear_intermediate_results_run ]
-----------------------------------------------------
Traceback (most recent call last):
File "cryosparc2_command/command_core/__init__.py", line 139, in run
self.target(*self.args)
File "cryosparc2_command/command_core/__init__.py", line 4152, in clear_intermediate_results_run
rc.clear_intermediate_results(project_uid, job['uid'], always_keep_final=True)
File "cryosparc2_compute/jobs/runcommon.py", line 618, in clear_intermediate_results
output_result = com.query(job['output_results'], lambda r: r['group_name'] == slot['group_name'] and r['name'] == slot['result_name'] , error="No match for %s.%s" % (slot['group_name'], slot['result_name']))
File "cryosparc2_compute/jobs/common.py", line 351, in query
assert res != default, error
AssertionError: No match for imported_particles.blob
-----------------------------------------------------
**custom thread exception hook caught something
**** handle exception rc
Exception in thread Thread-1141475:
Traceback (most recent call last):
File "/stornext/System/data/structbio/cryosparc/cryosparc2_master/deps/anaconda/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "cryosparc2_compute/jobs/runcommon.py", line 1682, in run_with_except_hook
handle_exception(*sys.exc_info())
File "cryosparc2_compute/jobs/runcommon.py", line 1646, in handle_exception
set_job_status('failed')
File "cryosparc2_compute/jobs/runcommon.py", line 132, in set_job_status
cli.set_job_status(_project_uid, _job_uid, status)
AttributeError: 'NoneType' object has no attribute 'set_job_status'
set status to failed
[EXPORT_PROJECT] : Exporting project P32...
[EXPORT_PROJECT] : Exported project P32 to /stornext/HPCScratch/cryoSPARC_data/uploads/HYBRID/P32/project.json in 0.00s
Kind regards.
thanx, indeed an easy fix. now I know where to look for the bug :^)
Unfortunately this issue still persists on 3.2 with the latest patch:
[POST-RESPONSE-THREAD ERROR 2021-05-18 10:26:33.691674 at clear_intermediate_results_run ]
-----------------------------------------------------
Traceback (most recent call last):
File "/cryosparc/cryosparc-v3/cryosparc_master/cryosparc_command/command_core/__init__.py", line 155, in run
self.target(*self.args)
File "/cryosparc/cryosparc-v3/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4437, in clear_intermediate_results_run
rc.clear_intermediate_results(project_uid, job['uid'], always_keep_final=True)
File "/cryosparc/cryosparc-v3/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 622, in clear_intermediate_results
output_result = com.query(job['output_results'], lambda r: r['group_name'] == slot['group_name'] and r['name'] == slot['result_name'] , error="No match for %s.%s" % (slot['group_name'], slot['result_name']))
File "/cryosparc/cryosparc-v3/cryosparc_master/cryosparc_compute/jobs/common.py", line 615, in query
assert res != default, error
AssertionError: No match for imported_particles.blob
-----------------------------------------------------
**custom thread exception hook caught something
**** handle exception rc
set status to failed
---- Deleting project UID P18 job UID J932
Now clearing job..Hi all,
I’m in the same situations these days…
the “cryosparcm log command_core” command gives the following output:
[EXPORT_JOB] : Writing 4 database images to /users/cryouser/P1/J136/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 4 images in 0.06s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for templates
[EXPORT_JOB] : Done. Exported in 0.01s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P1 J136 in 0.08s
[EXPORT_JOB] : Request to export P1 J137
[EXPORT_JOB] : Exporting job to /users/cryouser/P1/J137
[EXPORT_JOB] : Exporting all of job's images in the database to /users/cryouser/P1/J137/gridfs_data...
[EXPORT_JOB] : Writing 57 database images to /users/cryouser/P1/J137/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 57 images in 0.22s
[EXPORT_JOB] : Exporting all job's streamlog events...
[EXPORT_JOB] : Done. Exported 1 files in 0.01s
[EXPORT_JOB] : Exporting job metafile...
[EXPORT_JOB] : Creating .csg file for templates_selected
[EXPORT_JOB] : Creating .csg file for templates_excluded
[EXPORT_JOB] : Done. Exported in 0.01s
[EXPORT_JOB] : Updating job manifest...
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P1 J137 in 0.25s
[POST-RESPONSE-THREAD ERROR 2021-07-18 13:42:33.053990 at clear_intermediate_results_run ]
-----------------------------------------------------
Traceback (most recent call last):
File "/users/cryouser/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 146, in run
self.target(*self.args)
File "/users/cryouser/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4300, in clear_intermediate_results_run
rc.clear_intermediate_results(project_uid, job['uid'], always_keep_final=True)
File "/users/cryouser/cryosparc/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 628, in clear_intermediate_results
output_result = com.query(job['output_results'], lambda r: r['group_name'] == slot['group_name'] and r['name'] == slot['result_name'] , error="No match for %s.%s" % (slot['group_name'], slot['result_name']))
File "/users/cryouser/cryosparc/cryosparc_master/cryosparc_compute/jobs/common.py", line 394, in query
assert res != default, error
AssertionError: No match for micrographs.mscope_params
-----------------------------------------------------
**custom thread exception hook caught something
**** handle exception rc
set status to failed
Waiting for data... (interrupt to abort)
All directories have the proper permission and I didn’t quite get what do in this situation.
Thanks,
Donna
Has this been addressed in a patch yet?