Is 5 minutes or so for attaching a 10TB project typical? What happens when a project is attached ? What resources are accessed the most during this process? Looking to see if I can improve bottlenecks in my I/O . Cryosparc mogodb and projects are coming from network drives but I don’t see high throughput utilization during attach.
Also it’s not possible to reattach a detached project from the interface? If I need to reattach it creates a new Project. Is it safe to remove the old detached project?
Hi @francis.reyes,
I believe that currently with project attaching the most time-consuming activity would be re-uploading GridFS files back into MongoDB. If this is the case, contributing factors would include processing power and file I/O speeds on the master node, as well as a good connection with the database. For attaching a 10TB project, I think a few minutes is within the range of expectations. Please let us know if any adjustments on your end lead to a noticeable improvement, as this would be useful knowledge.
It is not possible to detach and re-attach a project as the same project in CryoSPARC. Detaching a project is intended for fully transferring a project to another instance. If the intention is to move the project temporarily to another location, archiving the project may be a better solution. Please see Guide: Data Management in CryoSPARC (v4.0+) - CryoSPARC Guide for more details.
Thanks for the guidance. What qualifies as moving the project to another instance? In my workflow the headnode containing the cryosparc installation may be terminated or rebuilt, but the database remains the same ( i will turn on maintenance mode and ensure that no jobs are running prior to terminating the instance) Am i safe in leaving all my projects in their current state? Or before the instance is destroyed (but again keeping the DB intact), should I archive my projects? Or should I detach them?
In this case, “instance” refers to another installation of CryoSPARC. Detaching a project is intended for those attempting to transfer a project from one installation of CryoSPARC to another installation on a different computer.
Between restarts of CryoSPARC involving cryosparcm stop, shutting the system down, then cryosparcm start, or updating the system with cryosparcm update, it is not necessary to archive or detach existing projects to maintain their integrity. Any existing projects before a shutdown will still be available after restarting CryoSPARC.
If your workflow involves completely tearing down the system which CryoSPARC is running on, as long as all files before teardown and restored and the database is kept intact, it is expected for projects to remain intact as well.
When updating CryoSPARC, please refer to Software Updates and Patches - CryoSPARC Guide for more information. For keeping your projects safe in case of an issue with updating, it is recommended to back up your database before attempting the update. This is detailed in the linked guide as well.