I detached my project, move disk to another instance, then attach it. However, only partial of jobs can be seen after attached. I have ~600 jobs before detached, but only ~300 jobs can be found.
I guessed this issue cased by jobs not fully exported during detached process.
can someone help me how to repair it?
As my original project is still in the original instance, can I just somehow modified it back into “attached” status to rescue my project?
Thank you very much! The project was detached, but I do not delete it from database! It can still be view but cannot be calculate any more! I want to mount my disk back to my compute again, then just rescue my project by modify database or something else to set the project back into attached status.
@Meng Please can you confirm on the original instance that the command cryosparcm cli "get_project('P1', 'detached')"
shows True.
If so, you can try:
Detach the project from the “other” instance, then
Re-Attach the project to the original instance.
Confirm the re-attachment was successful. For example, you should be able to clone a job that could not be found on the other instance and run it. If any problems occur, please email us the tgz file produced by the command (on the original instance): cryosparcm snaplogs. Otherwise, please continue.
Detach the project from the original instance, then
Attach the project to the “other” instance. If the attachment does not work as expected, please email us the tgz file produced by the command (on the “other” instance): cryosparcm snaplogs.
In your email (I will send you the email address by direct message), please indicate on which instance and under what circumstances (for example: after failed attachment on “other” instance) that attached tgz file was created.
Was there ever any resolution to this problem? I am having a similar issue, but not entirely sure it is 100 percent the same - anyhow here it is:
I am trying to transfer projects from one CS instance to another, while also transferring to a different drive, which is currently connected to the original drive, or stated differently the new drive is mounted on the old work station.
I have used two different projects to test if transfers are working:
PA: a small one(~25 job, 6GB) and
PB: a larger one, which still isn’t containing many jobs but the size is several times larger than the first.
upon making a copy(scp into a folder with a different name) of the PA, and removing the cs.lock file, I was able to re-attach the project into its original CS-instance, and the new/other CS instance. Initially, project¶ appeared empty, but after closing window an reopening, all contents appeared to be displayed; manage>project data gives a size of 6GB
However, doing the same with PB appears to NOT work, and leaves an apparently empty card/project, however, with the original name, and in the menu “project data”, the project is also displayed in manage>project data and a size is displayed (30GB).
For both PA and PB nothing else have been done, so presumably all the sym-links are the same?!?!
My own comments
cs.lock: yes I know you don’t recommend deleting the cs.lock like that right? however, detaching a project seems risky, since I am trying to find out how it work making copying seem safer at first - not sure if “kosher” or not. Plus, detaching another project has left a bit of a mess since a card remains in the overview, but this isn’t able to respond to any input.
why would there be a difference in CS’s ability to re-attach projects? size, content, something else?
what will happen is I dereference all sym-links as a next test step? only way I know is from your manual, which involves tar with an -h flag, and then extracting in the new location.
I am looking forward to your input and I hope you’ll understand my description - otherwise let me know.
As I remember, scp -r dereferences symlinks, leading to a directory copy that can significantly exceed the size of the copy’s source, which may or may not be desired.
Attachment may fail to complete for a variety of reasons, such as
the source copy of the project directory was corrupt
the copy process failed to complete or introduced a corruption in the “destination” copy
the attachment process is interrupted for other reasons
Please inspect the command_core log for relevant messages: cryosparcm log command_core.
Assuming that the project directory was copied using a method that maintains symbolic links, link targets can be updated as described for a similar use case.
As I remember, scp -r dereferences symlinks, leading to a directory copy that can significantly exceed the size of the copy’s source, which may or may not be desired
Yes I agree - it isn’t entirely clear what to prefer, and what happens if one uses the dereference option.
Are you aware of the [Delete Project from Database]
yes, and that isn’t possible after detaching; and it deletes content of sub-folders in the projects, so it doesn’t really replace the function of the “detach” function. Basically, I am saying that in this trouble-shooting phase it leaves footprints that I can’t get rid of.
Attachment may fail to complete for a variety of reasons, such as
the source copy of the project directory was corrupt
the copy process failed to complete or introduced a corruption in the “destination” copy
the attachment process is interrupted for other reasons
Please inspect the command_core log for relevant messages:
cryosparcm log command_core.
DO you know a convenient way to check integrity of a copy, and do you know if it is possible to check for corruption in the original project/project directory?
Assuming that the project directory was copied using a method that maintains symbolic links, link targets can be updated as described for a similar use case.
Yes I actually saw that. Seems like it may be the way to go. That still doesn’t solve the current problem of a blank project (the sym-link should currently still work, since the refer to locations/files that are still connected to CS isntance number 2(new one))
It seems to me that you are describing the Delete Project action, which is distinct from the Delete Project from Database action.
This may not be convenient, but a way: When errors occur during the attachment a copy of the original project directory, the command_core log should contain hints about the specific job whose import failed, and possibly specific “failing” components of the job directory. Depending on the specific component and error, one can further analyze the copy, then check whether the same corruption already exists in the original.
It seems to me that you are describing the Delete Project action, which is distinct from the Delete Project from Database action.
I think you’re right! However, I have no idea where to find the functionality “delete project from database”; is that what one should do to get rid of the card with a project saying “detached”?
Also, it was suggested to me by someone else that the function “Archive” should be used when trying to do what I wanna do, instead of doing the “detach and re-attach” (including changing the sym-links) work flow.
(just rehashing that what I wanna do is to move a project from one CS instance, to another CS-instance with a different drive…)
I currently doubt it, since the manual seems to say that detach-reattach is the way to go, whereas archive is more about putting a project up for storage on the same CS-instance, but just wanted to check?
does “archive” change all sym-links to “hard”-files?
@jaj2024 Your hunch is correct, archive-unarchive is not intended for the transferral of a project to another CryoSPARC instance, because the CryoSPARC project unarchive action assumes that the metadata associated with the project are already present in the “target” instance’s database, and that those database records are consistent with the contents of the project directory.
The CryoSPARC project attach action, on the other hand, creates new database records based on the contents of the project directory.
Symbolic links are not converted to regular files during the CryoSPARC project archive action.
