Error attaching project: invalid load key, '\x00'

mokca · April 8, 2024, 10:31am

I detached a project from an instance (still running v4.3.1) in order to move it to another instance (running v4.4). I experienced errors in attaching the project to the new instance. I subsequently tried to re-attach it to the original instance. This caused the same errors.

The failed attached project shows the skeleton of the project (number of workspaces, number of jobs) but no jobs.

The project lives on a shared storage system visible to both instances (mounted by NFSv4). The cryosparc accounts on both instances are the same and have the same access the the storage system.

Any assistance gratefully received - I have a 21TB project sitting in limbo and a panicking project owner.

From command_core (I’ve redacted the project name):

2024-04-08 11:15:06,126 import_project       INFO     | Created project P21
2024-04-08 11:15:06,522 import_workspaces    INFO     | Created workspace W1 in P21
2024-04-08 11:15:06,606 import_workspaces    INFO     | Created workspace W2 in P21
2024-04-08 11:15:06,608 import_workspaces    INFO     | CryoSPARC Live session detected. Importing S1 into W2 in P21
2024-04-08 11:15:06,608 import_workspaces    INFO     | CryoSPARC Live S1: Uploading project image data...
2024-04-08 11:15:30,194 import_project_run   ERROR    | Unable to import project from /data/rds/DUDSB/MPREMRNA/EM_PROJECTS/IANDREWS/cryosparc/PROJECTNAME
2024-04-08 11:15:30,194 import_project_run   ERROR    | Traceback (most recent call last):
2024-04-08 11:15:30,194 import_project_run   ERROR    |   File "/local_slow/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 4342, in import_project_run
2024-04-08 11:15:30,194 import_project_run   ERROR    |     import_workspaces(workspaces_doc_data, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id)
2024-04-08 11:15:30,194 import_project_run   ERROR    |   File "/local_slow/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 4453, in import_workspaces
2024-04-08 11:15:30,194 import_project_run   ERROR    |     gridfs_data = pickle.load(openfile)
2024-04-08 11:15:30,194 import_project_run   ERROR    | _pickle.UnpicklingError: invalid load key, '\x00'.
2024-04-08 11:15:30,196 dump_project         INFO     | Exporting project P21
2024-04-08 11:15:30,263 dump_project         INFO     | Exported project P21 to /data/rds/DUDSB/MPREMRNA/EM_PROJECTS/IANDREWS/cryosparc/PROJECTNAME/project.json in 0.07s
2024-04-08 11:15:30,295 run                  ERROR    | POST-RESPONSE-THREAD ERROR at import_project_run
2024-04-08 11:15:30,295 run                  ERROR    | Traceback (most recent call last):
2024-04-08 11:15:30,295 run                  ERROR    |   File "/local_slow/cryosparc/cryosparc2_master/cryosparc_command/commandcommon.py", line 73, in run
2024-04-08 11:15:30,295 run                  ERROR    |     self.target(*self.args)
2024-04-08 11:15:30,295 run                  ERROR    |   File "/local_slow/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 4342, in import_project_run
2024-04-08 11:15:30,295 run                  ERROR    |     import_workspaces(workspaces_doc_data, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id)
2024-04-08 11:15:30,295 run                  ERROR    |   File "/local_slow/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 4453, in import_workspaces
2024-04-08 11:15:30,295 run                  ERROR    |     gridfs_data = pickle.load(openfile)
2024-04-08 11:15:30,295 run                  ERROR    | _pickle.UnpicklingError: invalid load key, '\x00'.
**custom thread exception hook caught something
**** handle exception rc
Traceback (most recent call last):
  File "/local_slow/cryosparc/cryosparc2_master/cryosparc_compute/jobs/runcommon.py", line 2118, in run_with_except_hook
    run_old(*args, **kw)
  File "/local_slow/cryosparc/cryosparc2_master/cryosparc_command/commandcommon.py", line 73, in run
    self.target(*self.args)
  File "/local_slow/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 4342, in import_project_run
    import_workspaces(workspaces_doc_data, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id)
  File "/local_slow/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 4453, in import_workspaces
    gridfs_data = pickle.load(openfile)
_pickle.UnpicklingError: invalid load key, '\x00'.

Traceback (most recent call last):
  File "/local_slow/cryosparc/cryosparc2_master/cryosparc_compute/jobs/runcommon.py", line 2118, in run_with_except_hook
    run_old(*args, **kw)
  File "/local_slow/cryosparc/cryosparc2_master/cryosparc_command/commandcommon.py", line 73, in run
    self.target(*self.args)
  File "/local_slow/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 4342, in import_project_run
    import_workspaces(workspaces_doc_data, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id)
  File "/local_slow/cryosparc/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 4453, in import_workspaces
    gridfs_data = pickle.load(openfile)
_pickle.UnpicklingError: invalid load key, '\x00'.

cs.lock contains this:

e9e21d21-108a-48eb-b6d6-7cc824bb00fe
30178stbdt.icr.ac.uk
39000
2024-04-08T11:35:45.573693

project.json contains this after the failed import:

{
    "uid": "P22",
    "uid_num": 22,
    "title": "SF3b",
    "description": "Enter a description.",
    "project_dir": "/data/rds/DUDSB/MPREMRNA/EM_PROJECTS/IANDREWS/cryosparc/PROJECTNAME",
    "project_params_pdef": {},
    "owner_user_id": "5e40206a8f571221a84d50f2",
    "created_at": {
        "$date": 1700489019481
    },
    "status": "completed",
    "queue_paused": false,
    "deleted": false,
    "users_with_access": [
        "5e40206a8f571221a84d50f2"
    ],
    "size": 0,
    "last_accessed": {},
    "archived": false,
    "detached": false,
    "hidden": true,
    "project_stats": {},
    "generate_intermediate_results_settings": {
        "class_2D": false,
        "class_3D": false,
        "var_3D_disp": false
    },
    "created_at_version": "v4.3.1",
    "import_status": "failed",
    "imported": true,
    "imported_at": {
        "$date": 1712572545487
    }
}

wtempel · April 10, 2024, 4:55pm

The error message suggests that files inside the S1/gridfs_data/ subdirectory of the project directory are corrupted.
You may try:

Caution: The following procedure is outside standard CryoSPARC data management procedure. Please preserve a snapshot of the project directory’s current state before proceeding.
set aside a copy of S1/gridfs_data/ contents, then empty S1/gridfs_data/. As a result, the contents of S1/gridfs_data/ will be missing from the database, but project attachment will not longer be blocked by a failed attempt to upload the presumably corrupt content. CryoSPARC Live session S1 would be missing thumbnails and other images, but the session’s data may still be usable.
attempt attachment of the project directory.
inspect the command_core log for signs of success or failure of the project attachment.
if project attachment succeeded, but the corresponding project card at the /browse/P* URL of the CryoSPARC web app shows No Sessions and/or No Workspaces,

image972×546 48.8 KB

try updating the projects Statistics in the right-hand sidebar.

image716×534 58.1 KB

Does this help?

mokca · April 15, 2024, 9:06am

Thank you - this has helped a lot. All of the jobs outside of the live session have been recovered succesfully and the owner of the project can continue data processing.