Hi All,
Having a new issue when updating cryosparc to the latest version. Seem to have gone smoothly, but when I tried to do a run I got this:
Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 43, in cryosparc2_compute.run.main
AssertionError: Version mismatch! Worker and master versions are not the same. Please update.
I tried forced option on the worker but again that went fine. Yet I still get the error. Any ideas?
My setup: Centos with 4xNvidia RTX 2080
Status:
CryoSPARC System master node installed at
/data/CRYOSPARC/cryosparc2_master
Current cryoSPARC version: v2.15.0
cryosparcm process status:
app STOPPED Not started
app_dev STOPPED Not started
command_core RUNNING pid 14422, uptime 0:11:08
command_proxy RUNNING pid 14470, uptime 0:11:05
command_rtp STOPPED Not started
command_vis RUNNING pid 14455, uptime 0:11:06
database RUNNING pid 14341, uptime 0:11:10
watchdog_dev STOPPED Not started
webapp RUNNING pid 14474, uptime 0:11:03
webapp_dev STOPPED Not started
global config variables:
CRYOSPARC_FORCE_USER=true
export CRYOSPARC_LICENSE_ID=“XXX”
export CRYOSPARC_MASTER_HOSTNAME=“XXX”
export CRYOSPARC_DB_PATH="/data/CRYOSPARC/cryosparc2_database"
export CRYOSPARC_BASE_PORT=XXX
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
Log:
[EXPORT_JOB] : Request to export P8 J31
[EXPORT_JOB] : Exporting job to /data/CRYOSPARC/USERS/Omid/P8/J31
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/CRYOSPARC/USERS/Omid/P8/J31/gridfs_data…
[EXPORT_JOB] : Writing 23 database images to /data/CRYOSPARC/USERS/Omid/P8/J31/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 23 images in 0.03s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[EXPORT_JOB] : Done. Exported 1 files in 0.03s
[EXPORT_JOB] : Exporting job metafile…
[EXPORT_JOB] : Creating .csg file for particles
[EXPORT_JOB] : Done. Exported in 0.01s
[EXPORT_JOB] : Updating job manifest…
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P8 J31 in 0.07s
[EXPORT_JOB] : Request to export P8 J32
[EXPORT_JOB] : Exporting job to /data/CRYOSPARC/USERS/Omid/P8/J32
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/CRYOSPARC/USERS/Omid/P8/J32/gridfs_data…
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile…
[EXPORT_JOB] : Done. Exported in 0.00s
[EXPORT_JOB] : Updating job manifest…
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P8 J32 in 0.01s
[EXPORT_JOB] : Request to export P8 J33
[EXPORT_JOB] : Exporting job to /data/CRYOSPARC/USERS/Omid/P8/J33
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/CRYOSPARC/USERS/Omid/P8/J33/gridfs_data…
[EXPORT_JOB] : Writing 242 database images to /data/CRYOSPARC/USERS/Omid/P8/J33/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 242 images in 0.77s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[EXPORT_JOB] : Done. Exported 1 files in 0.10s
[EXPORT_JOB] : Exporting job metafile…
[EXPORT_JOB] : Creating .csg file for particles_all_classes
[EXPORT_JOB] : Creating .csg file for particles_class_0
[EXPORT_JOB] : Creating .csg file for volume_class_0
[EXPORT_JOB] : Creating .csg file for particles_class_1
[EXPORT_JOB] : Creating .csg file for volume_class_1
[EXPORT_JOB] : Done. Exported in 0.02s
[EXPORT_JOB] : Updating job manifest…
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P8 J33 in 0.90s
[EXPORT_JOB] : Request to export P8 J34
[EXPORT_JOB] : Exporting job to /data/CRYOSPARC/USERS/Omid/P8/J34
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/CRYOSPARC/USERS/Omid/P8/J34/gridfs_data…
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile…
[EXPORT_JOB] : Done. Exported in 0.00s
[EXPORT_JOB] : Updating job manifest…
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P8 J34 in 0.00s
[EXPORT_JOB] : Request to export P8 J35
[EXPORT_JOB] : Exporting job to /data/CRYOSPARC/USERS/Omid/P8/J35
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/CRYOSPARC/USERS/Omid/P8/J35/gridfs_data…
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile…
[EXPORT_JOB] : Done. Exported in 0.00s
[EXPORT_JOB] : Updating job manifest…
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P8 J35 in 0.01s
[EXPORT_JOB] : Request to export P8 J36
[EXPORT_JOB] : Exporting job to /data/CRYOSPARC/USERS/Omid/P8/J36
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/CRYOSPARC/USERS/Omid/P8/J36/gridfs_data…
[EXPORT_JOB] : Writing 81 database images to /data/CRYOSPARC/USERS/Omid/P8/J36/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 81 images in 0.31s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[EXPORT_JOB] : Done. Exported 1 files in 0.01s
[EXPORT_JOB] : Exporting job metafile…
[EXPORT_JOB] : Creating .csg file for particles
[EXPORT_JOB] : Creating .csg file for volume
[EXPORT_JOB] : Creating .csg file for mask
[EXPORT_JOB] : Done. Exported in 0.03s
[EXPORT_JOB] : Updating job manifest…
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P8 J36 in 0.36s
[EXPORT_JOB] : Request to export P8 J37
[EXPORT_JOB] : Exporting job to /data/CRYOSPARC/USERS/Omid/P8/J37
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/CRYOSPARC/USERS/Omid/P8/J37/gridfs_data…
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile…
[EXPORT_JOB] : Done. Exported in 0.00s
[EXPORT_JOB] : Updating job manifest…
[EXPORT_JOB] : Done. Updated in 0.02s
[EXPORT_JOB] : Exported P8 J37 in 0.03s
[EXPORT_JOB] : Request to export P8 J38
[EXPORT_JOB] : Exporting job to /data/CRYOSPARC/USERS/Omid/P8/J38
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/CRYOSPARC/USERS/Omid/P8/J38/gridfs_data…
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile…
[EXPORT_JOB] : Done. Exported in 0.01s
[EXPORT_JOB] : Updating job manifest…
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P8 J38 in 0.02s
[EXPORT_JOB] : Request to export P8 J39
[EXPORT_JOB] : Exporting job to /data/CRYOSPARC/USERS/Omid/P8/J39
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/CRYOSPARC/USERS/Omid/P8/J39/gridfs_data…
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[JSONRPC ERROR 2020-08-24 11:18:32.894830 at dump_job_database ]
Traceback (most recent call last):
File “cryosparc2_command/command_core/init.py”, line 115, in wrapper
res = func(*args, **kwargs)
File “cryosparc2_command/command_core/init.py”, line 2770, in dump_job_database
rc.dump_job_database(project_uid = project_uid, job_uid = job_uid, job_completed = job_completed, migration = migration, abs_export_dir = abs_export_dir)
File “cryosparc2_compute/jobs/runcommon.py”, line 381, in dump_job_database
with open(abs_export_events_path, ‘w’) as openfile:
IOError: [Errno 13] Permission denied: u’/data/CRYOSPARC/USERS/Omid/P8/J39/events.bson’
[POST-RESPONSE-THREAD ERROR 2020-08-24 11:18:32.903091 at clear_intermediate_results_run ]
Traceback (most recent call last):
File “cryosparc2_command/command_core/init.py”, line 146, in run
self.target(*self.args)
File “cryosparc2_command/command_core/init.py”, line 4136, in clear_intermediate_results_run
dump_job_database(project_uid, job[‘uid’])
File “cryosparc2_command/command_core/init.py”, line 124, in wrapper
raise e
IOError: [Errno 13] Permission denied: u’/data/CRYOSPARC/USERS/Omid/P8/J39/events.bson’
**custom thread exception hook caught something
**** handle exception rc
Exception in thread Thread-43:
Traceback (most recent call last):
File “/data/CRYOSPARC/cryosparc2_master/deps/anaconda/lib/python2.7/threading.py”, line 801, in __bootstrap_inner
self.run()
File “cryosparc2_compute/jobs/runcommon.py”, line 1689, in run_with_except_hook
handle_exception(*sys.exc_info())
File “cryosparc2_compute/jobs/runcommon.py”, line 1653, in handle_exception
set_job_status(‘failed’)
File “cryosparc2_compute/jobs/runcommon.py”, line 139, in set_job_status
cli.set_job_status(_project_uid, _job_uid, status)
AttributeError: ‘NoneType’ object has no attribute ‘set_job_status’
set status to failed
[EXPORT_JOB] : Request to export P8 J47
[EXPORT_JOB] : Exporting job to /data/CRYOSPARC/USERS/Omid/P8/J47
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/CRYOSPARC/USERS/Omid/P8/J47/gridfs_data…
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile…
[EXPORT_JOB] : Done. Exported in 0.01s
[EXPORT_JOB] : Updating job manifest…
[EXPORT_JOB] : Done. Updated in 0.05s
[EXPORT_JOB] : Exported P8 J47 in 0.06s
[EXPORT_JOB] : Request to export P8 J36
[EXPORT_JOB] : Exporting job to /data/CRYOSPARC/USERS/Omid/P8/J36
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/CRYOSPARC/USERS/Omid/P8/J36/gridfs_data…
[EXPORT_JOB] : Request to export P8 J36
[EXPORT_JOB] : Exporting job to /data/CRYOSPARC/USERS/Omid/P8/J36
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/CRYOSPARC/USERS/Omid/P8/J36/gridfs_data…
[EXPORT_JOB] : Writing 81 database images to /data/CRYOSPARC/USERS/Omid/P8/J36/gridfs_data/gridfsdata_0
[EXPORT_JOB] : Done. Exported 81 images in 0.17s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[EXPORT_JOB] : Done. Exported 1 files in 0.01s
[EXPORT_JOB] : Exporting job metafile…
[EXPORT_JOB] : Creating .csg file for particles
[EXPORT_JOB] : Creating .csg file for volume
[EXPORT_JOB] : Creating .csg file for mask
[EXPORT_JOB] : Done. Exported in 0.03s
[EXPORT_JOB] : Updating job manifest…
[EXPORT_JOB] : Done. Updated in 0.00s
[EXPORT_JOB] : Exported P8 J36 in 0.21s
[EXPORT_JOB] : Request to export P8 J47
[EXPORT_JOB] : Exporting job to /data/CRYOSPARC/USERS/Omid/P8/J47
[EXPORT_JOB] : Exporting all of job’s images in the database to /data/CRYOSPARC/USERS/Omid/P8/J47/gridfs_data…
[EXPORT_JOB] : Done. Exported 0 images in 0.00s
[EXPORT_JOB] : Exporting all job’s streamlog events…
[EXPORT_JOB] : Done. Exported 1 files in 0.00s
[EXPORT_JOB] : Exporting job metafile…
[EXPORT_JOB] : Done. Exported in 0.00s
[EXPORT_JOB] : Updating job manifest…
[EXPORT_JOB] : Done. Updated in 0.02s
[EXPORT_JOB] : Exported P8 J47 in 0.02s
---------- Scheduler running ---------------
Jobs Queued: [(u’P8’, u’J47’)]
Licenses currently active : 0
Now trying to schedule J47
Need slots : {u’GPU’: 1, u’RAM’: 3, u’CPU’: 4}
Need fixed : {u’SSD’: True}
Master direct : False
Scheduling job to c105445
Failed to connect link: HTTP Error 502: Bad Gateway
Not a commercial instance - heartbeat set to 12 hours.
Launchable! – Launching.
Changed job P8.J47 status launched
Running project UID P8 job UID J47
Running job on worker type node
Running job using: /data/CRYOSPARC/cryosparc2_worker/bin/cryosparcw
---------- Scheduler finished ---------------
Changed job P8.J47 status started
Changed job P8.J47 status failed
Thanks,
Omid