Best way to move archived project directories

Data Management
1

Hi,

I am trying to clean up cryosparc. I am trying to archive old, useful, projects and then remove them to bulk storage.

After using the UI to archive the project, what is the most suitable way to move the directory to a different disk? Is it OK to just copy and paste via file manager or is there a command/UI button to move it?

Thanks,
Matt

1 Like
2

After a project has been archived in the CryoSPARC sense, the underlying project directory needs to be copied/moved using methods/tools outside CryoSPARC. Those methods need to be chosen and used carefully for various reasons. One, but not the only, reason is the use of symbolic links inside project directories. Whether to dereference or not those links is an important consideration. Other considerations:

  • should the copy operation be resumable in case of a disruption?
  • how will the integrity of the copied data be verified?
3

Do you have a recommendation for a method to move archives?

4

I would be interested in hearing from forum users/practitioners who recently dealt with this problem. @mjmcleod64 For more applicable suggestions, please can you provide additional details:

  • are project and the destination “bulk” storage mounted on the same computer?
  • if networked storage is involved, what are the protocols and network speeds involved?
5

Ideally, I would move to a Network connected NAS. But I think it would be also very important to know how to move archived files to external usb-connected storage.

6

rsync is one option for moving data around. rsync supports local and network transfers, and can be resumed after interruption. See here for alternatives.
You may want to verify that checksums of original and copied files match to detect potential data corruption that may have occurred during transfer.
When initiating transfers on a remote computer, I recommend running your commands inside a “re-attachable” screen or tmux session (on the host were the data transfer is running) that would survive network disconnections.
Data management is a sensitive task. I further recommend

  • familiarizing yourself with the programs you end up using
  • being aware of your organization’s policies on data retention, availability and integrity
  • reaching out to your organization’s IT support for help
1 Like