No jobs showing up after recreating database

Data Management
#1

Hi there,
I have recently had a hard disk failure with the disk containing the home directory and the cryosparc database. I have recovered the disk following this guide (recovering the superblock from a backup on the disk), but the cryosparc_database got corrupted. There was no suitable backup of the database (huge mistake, I know), and I was unable to restore it with mongod. I tried to delete the database and creating an empty one following this solution, which seems to have worked. Trying to attach P1 failed due to a live session which was running at the time of the disk failure, so I have deleted the session folder from the project directory. After this, P1 was successfully (?) attached and the workspaces and live sessions appear:

Screenshot from 2023-08-30 21-59-55
Screenshot from 2023-08-30 21-59-551920×1080 177 KB

The jobs, however, do not show up within the workspaces, even though the statistics (number of completed/failed/killed jobs) are correctly showing in the workspace details:
Screenshot from 2023-08-30 21-56-51
Screenshot from 2023-08-30 21-56-511920×1080 164 KB

Trying to run any new jobs fails as an existing job ID (starting at J1) is assigned to them. Running cryosparcm test workers also fails for the same reason.
Screenshot from 2023-08-30 21-56-47
Screenshot from 2023-08-30 21-56-471920×1080 259 KB

Is there a way to salvage my earlier jobs? I think starting a new project should work for any later data processing, but if possible, it would be nice to get my earlier results back.
Job log from the failed worker test:


================= CRYOSPARCW =======  2022-07-25 11:01:55.677029  =========
Project P1 Job J1
Master biopc6.bcbp.gu.se Port 61002
===========================================================================
========= monitor process now starting main process
MAINPROCESS PID 14836
MAIN PID 14836
imports.run cryosparc_compute.jobs.jobregister
========= monitor process now waiting for main process
========= sending heartbeat
========= sending heartbeat
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
TIFFReadDirectory: Warning, Bogus "StripByteCounts" field, ignoring and calculating from imagelength.
========= sending heartbeat
TIFFFetchDirectory: Can not read TIFF directory count.
TIFFReadDirectory: Failed to read directory at offset 1572864.
/data1/szabolcs/2022-07-04_Westenhoff_PaPhy-FL-initial_EPU/Images-Disc1/GridSquare_3686077/Data/FoilHole_4773925_Data_3688194_3688196_20220705_190027_Fractions.tif: Not a TIFF or MDI file, bad magic number 0 (0x0).
/data1/szabolcs/2022-07-04_Westenhoff_PaPhy-FL-initial_EPU/Images-Disc1/GridSquare_3686077/Data/FoilHole_4773937_Data_3688194_3688196_20220705_190546_Fractions.tif: Not a TIFF or MDI file, bad magic number 0 (0x0).
***************************************************************
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
min: 60566.393555 max: 73381.872070
min: 1030643.386719 max: 1112529.113281
min: 1030643.386719 max: 1112529.113281
min: 252635.568848 max: 283566.743652
min: 60659.390869 max: 73373.327881
min: 60553.894531 max: 73431.792969
***************************************************************
Traceback (most recent call last):
  File "/home/analys/cryosparc/cryosparc_master/cryosparc_compute/jobs/imports/run.py", line 870, in run_import_movies_or_micrographs
    result.get()
  File "/home/analys/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
  File "/home/analys/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/analys/cryosparc/cryosparc_master/cryosparc_compute/jobs/imports/run.py", line 788, in header_check_worker
    eer_upsamp_factor=eer_upsamp_factor)
  File "/home/analys/cryosparc/cryosparc_master/cryosparc_compute/jobs/imports/run.py", line 677, in read_movie_header
    shape = tiff.read_tiff_shape(abs_path)
  File "/home/analys/cryosparc/cryosparc_master/cryosparc_compute/blobio/tiff.py", line 69, in read_tiff_shape
    tif = libtiff.TIFF.open(path)
  File "/home/analys/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/libtiff/libtiff_ctypes.py", line 501, in open
    raise TypeError('Failed to open file ' + repr(filename))
TypeError: Failed to open file b'/data1/szabolcs/2022-07-04_Westenhoff_PaPhy-FL-initial_EPU/Images-Disc1/GridSquare_3686077/Data/FoilHole_4773923_Data_3688194_3688196_20220705_185941_Fractions.tif'
Traceback (most recent call last):
  File "/home/analys/cryosparc/cryosparc_master/cryosparc_compute/jobs/imports/run.py", line 870, in run_import_movies_or_micrographs
    result.get()
  File "/home/analys/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
  File "/home/analys/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/analys/cryosparc/cryosparc_master/cryosparc_compute/jobs/imports/run.py", line 788, in header_check_worker
    eer_upsamp_factor=eer_upsamp_factor)
  File "/home/analys/cryosparc/cryosparc_master/cryosparc_compute/jobs/imports/run.py", line 677, in read_movie_header
    shape = tiff.read_tiff_shape(abs_path)
  File "/home/analys/cryosparc/cryosparc_master/cryosparc_compute/blobio/tiff.py", line 69, in read_tiff_shape
    tif = libtiff.TIFF.open(path)
  File "/home/analys/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/libtiff/libtiff_ctypes.py", line 501, in open
    raise TypeError('Failed to open file ' + repr(filename))
TypeError: Failed to open file b'/data1/szabolcs/2022-07-04_Westenhoff_PaPhy-FL-initial_EPU/Images-Disc1/GridSquare_3686077/Data/FoilHole_4773925_Data_3688194_3688196_20220705_190027_Fractions.tif'
Traceback (most recent call last):
  File "/home/analys/cryosparc/cryosparc_master/cryosparc_compute/jobs/imports/run.py", line 870, in run_import_movies_or_micrographs
    result.get()
  File "/home/analys/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
  File "/home/analys/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/analys/cryosparc/cryosparc_master/cryosparc_compute/jobs/imports/run.py", line 788, in header_check_worker
    eer_upsamp_factor=eer_upsamp_factor)
  File "/home/analys/cryosparc/cryosparc_master/cryosparc_compute/jobs/imports/run.py", line 677, in read_movie_header
    shape = tiff.read_tiff_shape(abs_path)
  File "/home/analys/cryosparc/cryosparc_master/cryosparc_compute/blobio/tiff.py", line 69, in read_tiff_shape
    tif = libtiff.TIFF.open(path)
  File "/home/analys/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/libtiff/libtiff_ctypes.py", line 501, in open
    raise TypeError('Failed to open file ' + repr(filename))
TypeError: Failed to open file b'/data1/szabolcs/2022-07-04_Westenhoff_PaPhy-FL-initial_EPU/Images-Disc1/GridSquare_3686077/Data/FoilHole_4773937_Data_3688194_3688196_20220705_190546_Fractions.tif'
========= main process now complete.
========= monitor process now complete.
#2

Hi @Boszlacs ,

Thanks for the detailed report. My initial question would be if you can access job data via the MongoDB command-line:

  1. cryosparcm mongo
  2. > db.jobs.count()
  3. > db.events.count()
  4. exit

In the project directory that you’ve attached, are you able to see folders for each job and a job.json for each?

- Suhail

#3

Hi,

Running the MongoDB commands seems to return the 3 failed jobs (instance testing) from the incorrectly restored P1 and 5 currently running (so far without problems) jobs from a new project I started. I cannot make sense of the number of events, but I had about 700 total jobs in P1 if that helps.

(base) analys@biopc6:~$ cryosparcm mongo
MongoDB shell version v3.6.23
connecting to: mongodb://biopc6.kemi.uu.se:61001/meteor?authSource=admin&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("c0b4d60c-6086-4676-8d9f-5b92553f209a") }
MongoDB server version: 3.6.23
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
meteor:PRIMARY> db.jobs.count()
8
meteor:PRIMARY> db.events.count()
2664
meteor:PRIMARY> exit
bye

I can see the job folders in the project folder a I have found the job.json file in the couple that I have checked.

Screenshot from 2023-08-31 20-11-30
Screenshot from 2023-08-31 20-11-30795×824 179 KB

Screenshot from 2023-08-31 20-10-18
Screenshot from 2023-08-31 20-10-18799×336 73.6 KB

Screenshot from 2023-08-31 20-08-53
Screenshot from 2023-08-31 20-08-53894×342 48.5 KB

#4

You may want to examine the command_core log to check on success or failure of the project’s attachment:

cryosparcm log command_core
cryosparcm filterlog command_core -l ERROR
cryosparcm filterlog command_core -f import_project
cryosparcm filterlog command_core -f import_workspaces
cryosparcm filterlog command_core -f import_jobs
#5

After checking the command_core log, I see this error message being constantly written:

2023-09-29 15:39:05,151 wrapper          	ERROR	| Traceback (most recent call last):
2023-09-29 15:39:05,151 wrapper          	ERROR	|   File "/home/analys/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 195, in wrapper
2023-09-29 15:39:05,151 wrapper          	ERROR	| 	res = func(*args, **kwargs)
2023-09-29 15:39:05,151 wrapper          	ERROR	|   File "/home/analys/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 7203, in get_job_status
2023-09-29 15:39:05,151 wrapper          	ERROR	| 	return get_job(project_uid, job_uid, 'status')['status']
2023-09-29 15:39:05,151 wrapper          	ERROR	|   File "/home/analys/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 186, in wrapper
2023-09-29 15:39:05,151 wrapper          	ERROR	| 	return func(*args, **kwargs)
2023-09-29 15:39:05,151 wrapper          	ERROR	|   File "/home/analys/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5784, in get_job
2023-09-29 15:39:05,151 wrapper          	ERROR	| 	raise ValueError(f"{project_uid} {job_uid} does not exist.")
2023-09-29 15:39:05,151 wrapper          	ERROR	| ValueError: P1 J717 does not exist.
2023-09-29 15:39:06,289 wrapper          	ERROR	| JSONRPC ERROR at get_job_status
2023-09-29 15:39:06,289 wrapper          	ERROR	| Traceback (most recent call last):
2023-09-29 15:39:06,289 wrapper          	ERROR	|   File "/home/analys/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 195, in wrapper
2023-09-29 15:39:06,289 wrapper          	ERROR	| 	res = func(*args, **kwargs)
2023-09-29 15:39:06,289 wrapper          	ERROR	|   File "/home/analys/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 7203, in get_job_status
2023-09-29 15:39:06,289 wrapper          	ERROR	| 	return get_job(project_uid, job_uid, 'status')['status']
2023-09-29 15:39:06,289 wrapper          	ERROR	|   File "/home/analys/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 186, in wrapper
2023-09-29 15:39:06,289 wrapper          	ERROR	| 	return func(*args, **kwargs)
2023-09-29 15:39:06,289 wrapper          	ERROR	|   File "/home/analys/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5784, in get_job
2023-09-29 15:39:06,289 wrapper          	ERROR	| 	raise ValueError(f"{project_uid} {job_uid} does not exist.")
2023-09-29 15:39:06,289 wrapper          	ERROR	| ValueError: P1 J717 does not exist.

This persists even after detaching P1 and deleting it from the database. As far as I can tell, this is all the log contains, and no import_project/workspaces/jobs entry can be found. J717 is a directory containing only an empty job.log file, and it is among the last jobs launched in P1 (720 jobs total).
I have given up on P1 for now and am working in a newly created project directory, but this constant error message worries me.

#6

Hi @Boszlacs, we typically see this issue when a Live session was improperly paused and is in an inconsistent state. To identify this problem, use CryoSPARC’s interactive Python shell:

cryosparcm icli

Enter the following Python code into the prompt

sessions = list(
    db.workspaces.find(
        {
            "status": "paused",
            "$or": [
                {"rtp_childs": {"$exists": True, "$ne": []}},
                {"rtp_workers": {"$exists": True, "$ne": {}}},
            ]
        },
        {"project_uid": 1, "session_uid": 1, "rtp_childs": 1, "rtp_workers": 1},
    )
)
print(sessions)

If you see any output other than [], there are inconsistent sessions.

To fix the issue enter the following code into the prompt:

db.workspaces.update_many(
    {
        "status": "paused",
        "$or": [
            {"rtp_childs": {"$exists": True, "$ne": []}},
            {"rtp_workers": {"$exists": True, "$ne": {}}},
        ],
    },
    {"$set": {"rtp_childs": [], "rtp_workers": {}}},
)

Enter exit() to quit the interactive prompt.

Future versions of CryoSPARC will address this issue so this workaround will no longer be required. Let me know how that goes!

1 Like