We are about to install v2.11 and are concerned that the one data database migration might made existing data/projects unusable if our v2.9 instance is invoked.
Is this a valid concern?
Is the metadata only written to cryosparc_db directory, or user project directories as well?
If only the DB directory, simple enough to checkpoint the file system, if user project directories less so as we are talking potentially Tbytes of data.
If in user directory space (non-DB space) are there certain sub-directories we can concentrate are archiving efforts on?
The database migration will create three new files in each project directory: project.json (no more than a few kb’s) workspaces.json (no more than a few kb’s) job_manifest.json (no more than a few kb’s)
The database migration will also create a few new files in each job directory: job.json (no more than a few kb’s) events.bson (no more than a few kb’s) gridfs_data/gridfsdata_* (anywhere between 0-1000MB based on how many plots/images in the job) *.csg (no more than a few kb’s)
If you’d like to save space/time during the migration, I’d recommend deleting any jobs that you don’t want in your instance (i.e. deleting killed/ failed/ unwanted building jobs) before updating to v2.11. After upgrading to v2.11, you can use the “Clear Intermediate Results” feature (detailed here) to save even more space in your instance.
I take it that since there are new files being created, not currently existing files altered, that if we needed for some reason to run the 2.9 instance that the new files would simply be ignored and not impede 2.9’s ability to run. Is that correct?
If you don’t explicitly update to v2.11 (i.e. by running cryosparcm update), none of these files will be created, and you will be able to use cryoSPARC normally as you would. Is that what you mean?
The update page, https://cryosparc.com/docs/tutorials/data-management/ in the first section says
Maintenance of valid project directories is accomplished via “continuous project export” functionality in v2.11+. Project directories are constantly being updated without interruption for the user. During the initial update to v2.11, a one-time migration will occur to update all projects with their metadata. Thereafter, certain changes that happen within a project triggers an update of the on-disk project information.
We are looking at a new install, move Cuda from 8 to 10.1 and move from cryosparcm to cryosparc.
And the department’s technical person is concerned that if we need to run v2.9 again these new files will cause an issue.
I see what you mean. You’re correct, downgrading from v2.11 back to v2.9 will not cause an issue given that 2.11 will create new files inside the project and job directories.