Cannot Attach (phantom cs.lock)

Data Management
1

I detached from old instance, copied to new location, attached at new location giving the new location. This worked, but since only a fraction of data was copied it was mostly empty in GUI.

Then I detached new project from new instance, finished the copy, and now I can’t attach as it says the project is locked at the OLD instance.

there is no lock file in old project location. lock file in new location that was copied has old instance id (I expect that’s normal).

ideas?

2

@CryoEM2 Please can you

  1. post the CryoSPARC version(s) of the CryoSPARC instance(s) involved
  2. confirm that you detached, not archived the project.
  3. define new location: same CryoSPARC server/instance, different project directory path?

[edited: question added]

3

old is 4.3.0, new is 4.4.1
detached
new location: from hpc at different institute, synced to AWS, then moved locally to fsx for processing. completely new.

4

I would not expect that given

I would expect:

  1. the first lines of cs.lock files to match for all a given CryoSPARC instance’s project directories
  2. a mismatch between the first lines of cs.lock files from distinct CryoSPARC instances

Unfortunately, all my hypotheses about what may have happened conflict with at least one detail of your account. :person_shrugging:

Can the “old” directory have not been modified in any way between the first, incomplete copy operation and when you

5

the old directory was slightly modified. a new owner was added, and they ran an export job. how they did this when I had detached the project, I dont know. Is there a path out of this?

6

If you are confident in the completeness and currentness of the project directory’s copy, you may want to remove the cs.lock file of questionable origin and try attaching. The usual caveat: the manual removal of a cs.lock file from a CryoSPARC project directory is generally discouraged in the absence of exceptional circumstances.

7

Thanks wtempel,
I removed cs.lock and was able to reattach the project.
However, it “Completed import of project from /fsx-data/processing/ME/P17/P17 into P242 with errors. Please see command_core logs.”

Further, I see cards for all workspaces, but nearly all workspaces are blank - no cards.

how do I access command_core log to find out what went wrong?

8

You can view the log using the command
cryosparcm log command_core (details).
You can also use the more targeted

cryosparcm filterlog command_core -l error

(guide) command, but want to confirm that the applied filter does not omit relevant log entries.
If you are an admin-level user on a v4+ CryoSPARC instance you can view logs under the Instance Logs tab of the Admin screen.
image

9

I downloaded from Instance Logs and command log has this (relevant snippet):
Imported J6007 into P242 in 0.69s…
2024-03-08 13:55:46,134 import_jobs INFO | Uploading image data for J6008…
2024-03-08 13:55:46,955 import_jobs INFO | Done. Uploaded 232 files in 0.82s
2024-03-08 13:55:46,968 import_jobs INFO | Inserted job document in 0.83s…
2024-03-08 13:55:46,968 import_jobs INFO | Inserting streamlogs into jobs…
2024-03-08 13:55:47,214 import_jobs INFO | Done. Inserted 492 streamlogs in 0.25s…
2024-03-08 13:55:47,214 import_jobs INFO | Imported J6008 into P242 in 1.08s…
2024-03-08 13:55:47,218 import_jobs INFO | Uploading image data for J6009…
2024-03-08 13:55:47,834 import_jobs INFO | Done. Uploaded 274 files in 0.62s
2024-03-08 13:55:47,840 import_jobs INFO | Inserted job document in 0.62s…
2024-03-08 13:55:47,840 import_jobs INFO | Inserting streamlogs into jobs…
2024-03-08 13:55:47,856 import_jobs INFO | Done. Inserted 35 streamlogs in 0.02s…
2024-03-08 13:55:47,856 import_jobs INFO | Imported J6009 into P242 in 0.64s…
2024-03-08 13:55:47,860 import_jobs INFO | Uploading image data for J61…
2024-03-08 13:55:47,864 import_jobs INFO | Done. Uploaded 0 files in 0.00s
2024-03-08 13:55:47,868 import_jobs INFO | Inserted job document in 0.01s…
2024-03-08 13:55:47,868 import_jobs INFO | Inserting streamlogs into jobs…
2024-03-08 13:55:47,872 import_jobs INFO | Done. Inserted 0 streamlogs in 0.00s…
2024-03-08 13:55:47,872 import_jobs INFO | Imported J61 into P242 in 0.02s…
2024-03-08 13:55:47,877 layout_tree WARNING | Job J598 parents include non-existent job J581
2024-03-08 13:55:47,878 layout_tree WARNING | Job J592 parents include non-existent job J581
2024-03-08 13:55:47,878 layout_tree WARNING | Job J591 parents include non-existent job J589
2024-03-08 13:55:47,878 layout_tree WARNING | Job J590 parents include non-existent job J581
2024-03-08 13:55:47,878 layout_tree WARNING | Job J590 parents include non-existent job J588
2024-03-08 13:55:47,878 layout_tree WARNING | Job J59 parents include non-existent job J16
2024-03-08 13:55:47,878 layout_tree WARNING | Job J59 parents include non-existent job J8
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5998 parents include non-existent job J5628
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5996 parents include non-existent job J5628
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5993 parents include non-existent job J5708
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5992 parents include non-existent job J4166
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5991 parents include non-existent job J4421
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5990 parents include non-existent job J5307
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5990 parents include non-existent job J5768
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5933 parents include non-existent job J5896
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5932 parents include non-existent job J5896
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5926 parents include non-existent job J5896
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5924 parents include non-existent job J5896
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5923 parents include non-existent job J5896
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5908 parents include non-existent job J5896
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5903 parents include non-existent job J5896
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5902 parents include non-existent job J5896
2024-03-08 13:55:47,878 layout_tree WARNING | Job J5900 parents include non-existent job J5896
2024-03-08 13:55:47,878 layout_tree WARNING | Job J599 parents include non-existent job J581
2024-03-08 13:55:47,878 layout_tree WARNING | Job J593 parents include non-existent job J581
2024-03-08 13:55:47,878 layout_tree WARNING | Job J593 parents include non-existent job J588
2024-03-08 13:55:47,879 import_project_run WARNING | Failed laying out tree in P242: ‘J5628’
2024-03-08 13:55:47,880 import_project_run WARNING | Imported project from /fsx-data/processing/ME/P17/P17 as P242 in 1246.83s with errors.
2024-03-08 13:55:47,880 dump_project INFO | Exporting project P242
2024-03-08 13:55:47,889 dump_project INFO | Exported project P242 to /fsx-data/processing/ME/P17/P17/project.json in 0.01s
2024-03-08 13:55:47,889 update_project_size_run INFO | Beginning update project size for P242

I checked and the non-existent jobs ARE in the proper location and have information in the folders…

10

Does the downloaded log include any errors?

grep -i error /path/to/command_core.log
11

no I dont’ see any errors in command core log pertaining to this. other users are simultaneously deleting and exporting data so the log is chock-full and has some error messages, but none for P242 or related.

12

I am wondering if

  1. “non-existent jobs” failed to be imported
  2. the command_core log has been rotated after the expected
    Importing project from /fsx-data/processing/ME/P17/P17
    INFO-level log entry. In this case, relevant error messages might be present inside the file
    /path/to/cryosparc_master/run/command_core.log.1
    One could search
    grep -B 2 -A 2 J5896 /path/to/cryosparc_master/run/command_core.log*
13

grep yielded this:

command_core.log.3-2024-02-16 13:09:10,717 layout_tree WARNING | Job J5990 parents include non-existent job J5307
command_core.log.3-2024-02-16 13:09:10,717 layout_tree WARNING | Job J5990 parents include non-existent job J5768
command_core.log.3:2024-02-16 13:09:10,717 layout_tree WARNING | Job J5933 parents include non-existent job J5896
command_core.log.3:2024-02-16 13:09:10,717 layout_tree WARNING | Job J5932 parents include non-existent job J5896
command_core.log.3:2024-02-16 13:09:10,717 layout_tree WARNING | Job J5926 parents include non-existent job J5896
command_core.log.3:2024-02-16 13:09:10,717 layout_tree WARNING | Job J5924 parents include non-existent job J5896
command_core.log.3:2024-02-16 13:09:10,717 layout_tree WARNING | Job J5923 parents include non-existent job J5896
command_core.log.3:2024-02-16 13:09:10,717 layout_tree WARNING | Job J5908 parents include non-existent job J5896
command_core.log.3:2024-02-16 13:09:10,717 layout_tree WARNING | Job J5903 parents include non-existent job J5896
command_core.log.3:2024-02-16 13:09:10,717 layout_tree WARNING | Job J5902 parents include non-existent job J5896
command_core.log.3:2024-02-16 13:09:10,717 layout_tree WARNING | Job J5900 parents include non-existent job J5896
command_core.log.3-2024-02-16 13:09:10,717 layout_tree WARNING | Job J599 parents include non-existent job J581
command_core.log.3-2024-02-16 13:09:10,717 layout_tree WARNING | Job J593 parents include non-existent job J581

14

These observations taken together suggest a hypothesis that I had not considered before: The copy of the project directory at AWS (“destination”) might not be an exact replica of the copy on the OLD instance (“source”) because file deletions may not have been synced.
For example, might the following sequence apply in your case:

  1. The source is initially “synced” to the destination while the project is still attached to the old instance.
  2. The source continues to be modified on the OLD instance. Jobs may be deleted.
  3. The source is detached from the OLD instance, cs.lock is automatically removed in the process.
  4. The source is “finally” synced to the destination:
    • new file created after the previous sync are added to the destination
    • files modified on the source after the previous sync are updated on the destination
    • but: files removed from the source after the previous sync (deleted jobs, cs.lock) are not removed from the destination

What methods/commands (including command options) where used for the various copy/sync operations?

I am not sure there would be lines that simultaneously contain the terms ERROR and P242, but I could try a few less restrictive queries. Would you be interested in sharing the logs with the Structura team:

tar zcvf core_logs.tar.gz /path/to/cryosparc_master/run/command_core.log*