Restack particles before export yields tar file containing original particle stack

Import and Export
1

Trying to save space by consolidating good particles and throwing out bad particles from an initial local motion correction job. From NU refinement, I restack particles resulting in 1.9 GB file. Then reconstruct those particles, export, and tar with dereference option. The resulting file is >100 GB, same as the original local motion correction job.

2

@cryofun Please can you post the output of the command

find /path/to/project_dir/exports/jobs/job_dir -type d

and also list the job types of all job IDs shown in the find command’s output.

3

Here it is:

find /path/to/project_dir/exports/jobs/job_dir/ -type d
/path/to/project_dir/exports/jobs/job_dir/
/path/to/project_dir/exports/jobs/job_dir/J4155_volume
/path/to/project_dir/exports/jobs/job_dir/J4155_volume/J4155
/path/to/project_dir/exports/jobs/job_dir/J4155_particles
/path/to/project_dir/exports/jobs/job_dir/J4155_particles/J4153
/path/to/project_dir/exports/jobs/job_dir/J4155_particles/J4153/restack
/path/to/project_dir/exports/jobs/job_dir/J4155_particles/J1371
/path/to/project_dir/exports/jobs/job_dir/J4155_particles/J1371/motioncorrected
/path/to/project_dir/exports/jobs/job_dir/J4155_particles/J1410
/path/to/project_dir/exports/jobs/job_dir/J4155_particles/J1410/localmotioncorrected
/path/to/project_dir/exports/jobs/job_dir/J4155_mask
/path/to/project_dir/exports/jobs/job_dir/J4155_mask/J4155
/path/to/project_dir/exports/jobs/job_dir/gridfs_data

4

Thanks @cryofun. Please can you also post the outputs of

cd /path/to/project_dir/
ls J1371/motioncorrected | wc -l
ls J1410/localmotioncorrected | wc -l
find exports/jobs/job_dir -type d  -exec sh -c "du -shL {}; ls {} | wc -l" \;
5

Hi, I do not have a very good understanding of the way particle images are stored, but I had similar issues when trying to transfer particle sets between instances. What worked for me was to take the restacked particle set, and plug it into a Particle Sets job. There, you can open the input group and remove the location, which will result in your selected particles being written into a single .mrcs file, instead of just referencing their location on the micrograph.

6

Thanks for the help. Here is the output:

ls J1371/motioncorrected | wc -l
26705

ls J1410/localmotioncorrected | wc -l
10196

find /path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/ -type d -exec sh -c "du -shL {}; ls {} | wc -l" \;
235G	/path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/
6
463M	/path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/J4155_volume
3
463M	/path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/J4155_volume/J4155
6
235G	/path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/J4155_particles
5
2.3G	/path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/J4155_particles/J4153
1
2.3G	/path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/J4155_particles/J4153/restack
1
232G	/path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/J4155_particles/J1371
1
232G	/path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/J4155_particles/J1371/motioncorrected
4221
291M	/path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/J4155_particles/J1410
1
291M	/path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/J4155_particles/J1410/localmotioncorrected
4221
85M	/path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/J4155_mask
3
84M	/path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/J4155_mask/J4155
1
3.0M	/path/to/project_dir/exports/jobs/job_dir/J4155_homo_reconstruct/gridfs_data
1