Project size is a fantasy number

Application
1

Does the project size function follow symlinks? Because it thinks one recent project is 40+TB… on a 14TB RAID array…

Screenshot 2023-12-29 at 13.04.15
Screenshot 2023-12-29 at 13.04.15718×378 22 KB

2

@rbs_sci Symlinks should not be dereferenced when the space is calculated. Please can you

  • post a screenshot of the expanded Data Management Details.
    image
  • screenshots of the Cleanup Project Data preview window with the various options (Clear preprocessing jobs, Clear intermediate results, etc) selected, to show how data usage is distributed among various types of data. Caution: Do not proceed to actual Cleanup; Cancel instead.
    image
    image1063×701 65.6 KB
  • outputs of the commands
    df -h /path/to/project_dir
stat -f /path/to/project_dir
3

Hi @wtempel

Seems like it’s a UI bug in Data Management. I think it might be following the symlinks present in the movies folder, which I did to make handling this project easier, as the raw data are scattered all over the place and I could pre-curate (read: not process) the absolute mountain of data collected by an ex-lab-member. When I next get the chance, I’ll test breaking the symlinks by unmounting the remote storage.

Data management screenshot:

Screenshot from 2024-01-03 22-25-32
Screenshot from 2024-01-03 22-25-32370×1179 79 KB

Cleanup project data screenshot:
Before recalculating:

Screenshot from 2024-01-03 22-26-00
Screenshot from 2024-01-03 22-26-001079×762 87.9 KB

After recalculating:

Screenshot from 2024-01-03 22-30-34
Screenshot from 2024-01-03 22-30-341081×729 85.8 KB

Output from du -h:
$ du -h CS-censoredProjectName/ 4.0K CS-censoredProjectName/J54/gridfs_data 60K CS-censoredProjectName/J54 14M CS-censoredProjectName/J31/gridfs_data 327M CS-censoredProjectName/J31 4.0K CS-censoredProjectName/J51/gridfs_data 76K CS-censoredProjectName/J51 31M CS-censoredProjectName/J62/gridfs_data 84G CS-censoredProjectName/J62 104M CS-censoredProjectName/J11/gridfs_data 139M CS-censoredProjectName/J11 4.0K CS-censoredProjectName/J46/gridfs_data 60K CS-censoredProjectName/J46 2.5M CS-censoredProjectName/J69/gridfs_data 1.2G CS-censoredProjectName/J69 4.0K CS-censoredProjectName/J55/gridfs_data 60K CS-censoredProjectName/J55 35M CS-censoredProjectName/J114/gridfs_data 7.2G CS-censoredProjectName/J114 41M CS-censoredProjectName/J66/gridfs_data 107G CS-censoredProjectName/J66 41M CS-censoredProjectName/J118/gridfs_data 8.6G CS-censoredProjectName/J118 4.0K CS-censoredProjectName/J56/gridfs_data 44K CS-censoredProjectName/J56 38M CS-censoredProjectName/J70/gridfs_data 6.0G CS-censoredProjectName/J70 4.0K CS-censoredProjectName/J79/gridfs_data 40K CS-censoredProjectName/J79 42M CS-censoredProjectName/J99/gridfs_data 8.6G CS-censoredProjectName/J99 4.0K CS-censoredProjectName/J84/gridfs_data 32K CS-censoredProjectName/J84 11M CS-censoredProjectName/J23/gridfs_data 176M CS-censoredProjectName/J23 4.0K CS-censoredProjectName/J50/gridfs_data 76K CS-censoredProjectName/J50 4.0K CS-censoredProjectName/J29/gridfs_data 36K CS-censoredProjectName/J29 6.2M CS-censoredProjectName/J19/gridfs_data 204M CS-censoredProjectName/J19 4.0K CS-censoredProjectName/J81/gridfs_data 28K CS-censoredProjectName/J81 37M CS-censoredProjectName/J128/gridfs_data 64G CS-censoredProjectName/J128 4.0K CS-censoredProjectName/J52/gridfs_data 68K CS-censoredProjectName/J52 4.0K CS-censoredProjectName/J85/gridfs_data 52K CS-censoredProjectName/J85 66M CS-censoredProjectName/J107/gridfs_data 1.1G CS-censoredProjectName/J107 96K CS-censoredProjectName/J43/gridfs_data 2.0G CS-censoredProjectName/J43 92K CS-censoredProjectName/J44/gridfs_data 2.0G CS-censoredProjectName/J44 123M CS-censoredProjectName/J58/gridfs_data 15G CS-censoredProjectName/J58 3.3M CS-censoredProjectName/J1/gridfs_data 98M CS-censoredProjectName/J1/imported 108M CS-censoredProjectName/J1 34M CS-censoredProjectName/J63/gridfs_data 15G CS-censoredProjectName/J63 96K CS-censoredProjectName/J42/gridfs_data 2.0G CS-censoredProjectName/J42 76K CS-censoredProjectName/J40/gridfs_data 2.0G CS-censoredProjectName/J40 41M CS-censoredProjectName/J119/gridfs_data 8.6G CS-censoredProjectName/J119 3.8M CS-censoredProjectName/J13/gridfs_data 1.1G CS-censoredProjectName/J13 96K CS-censoredProjectName/J45/gridfs_data 2.0G CS-censoredProjectName/J45 41M CS-censoredProjectName/J101/gridfs_data 8.6G CS-censoredProjectName/J101 284M CS-censoredProjectName/J126/gridfs_data 480M CS-censoredProjectName/J126 8.4M CS-censoredProjectName/J121/gridfs_data 243G CS-censoredProjectName/J121/reconstructed 247G CS-censoredProjectName/J121 2.1M CS-censoredProjectName/J91/gridfs_data 1.4G CS-censoredProjectName/J91 4.0K CS-censoredProjectName/J80/gridfs_data 28K CS-censoredProjectName/J80 424K CS-censoredProjectName/J68/gridfs_data 54G CS-censoredProjectName/J68/downsample 54G CS-censoredProjectName/J68 71M CS-censoredProjectName/J94/gridfs_data 61G CS-censoredProjectName/J94/extract 61G CS-censoredProjectName/J94 4.0K CS-censoredProjectName/J49/gridfs_data 76K CS-censoredProjectName/J49 104K CS-censoredProjectName/J39/gridfs_data 2.0G CS-censoredProjectName/J39 624K CS-censoredProjectName/J77/gridfs_data 4.0G CS-censoredProjectName/J77 4.0K CS-censoredProjectName/J53/gridfs_data 24K CS-censoredProjectName/J53 4.0M CS-censoredProjectName/J12/gridfs_data 36M CS-censoredProjectName/J12 4.0K CS-censoredProjectName/J36/gridfs_data 56K CS-censoredProjectName/J36 47M CS-censoredProjectName/J104/gridfs_data 10G CS-censoredProjectName/J104 35M CS-censoredProjectName/J35/gridfs_data 132G CS-censoredProjectName/J35/extract 132G CS-censoredProjectName/J35 204K CS-censoredProjectName/J110/gridfs_data 217M CS-censoredProjectName/J110 176K CS-censoredProjectName/J113/gridfs_data 217M CS-censoredProjectName/J113 3.2M CS-censoredProjectName/J3/gridfs_data 48M CS-censoredProjectName/J3/imported 55M CS-censoredProjectName/J3 13M CS-censoredProjectName/J18/gridfs_data 952M CS-censoredProjectName/J18 71M CS-censoredProjectName/J93/gridfs_data 61G CS-censoredProjectName/J93/extract 61G CS-censoredProjectName/J93 42M CS-censoredProjectName/J125/gridfs_data 75G CS-censoredProjectName/J125 2.1M CS-censoredProjectName/J96/gridfs_data 1.4G CS-censoredProjectName/J96 568K CS-censoredProjectName/J72/gridfs_data 4.0G CS-censoredProjectName/J72 616K CS-censoredProjectName/J76/gridfs_data 4.0G CS-censoredProjectName/J76 3.7M CS-censoredProjectName/J8/gridfs_data 18M CS-censoredProjectName/J8 4.0K CS-censoredProjectName/J82/gridfs_data 28K CS-censoredProjectName/J82 180K CS-censoredProjectName/J112/gridfs_data 217M CS-censoredProjectName/J112 35M CS-censoredProjectName/J98/gridfs_data 7.2G CS-censoredProjectName/J98 54M CS-censoredProjectName/J65/gridfs_data 142G CS-censoredProjectName/J65 82M CS-censoredProjectName/J15/gridfs_data 25G CS-censoredProjectName/J15/extract 25G CS-censoredProjectName/J15 628K CS-censoredProjectName/J74/gridfs_data 4.0G CS-censoredProjectName/J74 17M CS-censoredProjectName/J34/gridfs_data 1.9G CS-censoredProjectName/J34 141M CS-censoredProjectName/J20/gridfs_data 236M CS-censoredProjectName/J20 36M CS-censoredProjectName/J61/gridfs_data 15G CS-censoredProjectName/J61 4.0K CS-censoredProjectName/J37/gridfs_data 28K CS-censoredProjectName/J37 34M CS-censoredProjectName/J64/gridfs_data 15G CS-censoredProjectName/J64 277M CS-censoredProjectName/J21/gridfs_data 387M CS-censoredProjectName/J21 72K CS-censoredProjectName/J122/gridfs_data 193M CS-censoredProjectName/J122 4.0K CS-censoredProjectName/J59/gridfs_data 136M CS-censoredProjectName/J59 632K CS-censoredProjectName/J75/gridfs_data 4.0G CS-censoredProjectName/J75 12M CS-censoredProjectName/J14/gridfs_data 444M CS-censoredProjectName/J14 4.0K CS-censoredProjectName/J123/gridfs_data 80K CS-censoredProjectName/J123 285M CS-censoredProjectName/J16/gridfs_data 538M CS-censoredProjectName/J16 41M CS-censoredProjectName/J100/gridfs_data 8.6G CS-censoredProjectName/J100 11M CS-censoredProjectName/J17/gridfs_data 77M CS-censoredProjectName/J17 4.0K CS-censoredProjectName/J47/gridfs_data 80K CS-censoredProjectName/J47 200K CS-censoredProjectName/J108/gridfs_data 217M CS-censoredProjectName/J108 8.1M CS-censoredProjectName/J28/gridfs_data 99M CS-censoredProjectName/J28 4.0K CS-censoredProjectName/J48/gridfs_data 76K CS-censoredProjectName/J48 36M CS-censoredProjectName/J33/gridfs_data 33G CS-censoredProjectName/J33/extract 34G CS-censoredProjectName/J33 14M CS-censoredProjectName/J32/gridfs_data 327M CS-censoredProjectName/J32 71M CS-censoredProjectName/J90/gridfs_data 61G CS-censoredProjectName/J90/extract 62G CS-censoredProjectName/J90 2.6G CS-censoredProjectName/J5/thumbnails 7.1M CS-censoredProjectName/J5/gridfs_data 10T CS-censoredProjectName/J5/motioncorrected 10T CS-censoredProjectName/J5 360K CS-censoredProjectName/J7/gridfs_data 78M CS-censoredProjectName/J7 36M CS-censoredProjectName/movies/20220706 195M CS-censoredProjectName/movies/20220701 6.7M CS-censoredProjectName/movies/20220712 95M CS-censoredProjectName/movies/20220708 332M CS-censoredProjectName/movies 71M CS-censoredProjectName/J92/gridfs_data 61G CS-censoredProjectName/J92/extract 62G CS-censoredProjectName/J92 580K CS-censoredProjectName/J73/gridfs_data 4.0G CS-censoredProjectName/J73 204K CS-censoredProjectName/J111/gridfs_data 217M CS-censoredProjectName/J111 11M CS-censoredProjectName/J9/gridfs_data 21M CS-censoredProjectName/J9 42M CS-censoredProjectName/J115/gridfs_data 8.6G CS-censoredProjectName/J115 168K CS-censoredProjectName/J30/gridfs_data 31M CS-censoredProjectName/J30 4.0K CS-censoredProjectName/J78/gridfs_data 28K CS-censoredProjectName/J78 7.9M CS-censoredProjectName/J6/gridfs_data 257G CS-censoredProjectName/J6/ctfestimated 257G CS-censoredProjectName/J6 6.4M CS-censoredProjectName/J57/gridfs_data 176M CS-censoredProjectName/J57 200K CS-censoredProjectName/J109/gridfs_data 217M CS-censoredProjectName/J109 117M CS-censoredProjectName/J25/gridfs_data 1.8G CS-censoredProjectName/J25 76K CS-censoredProjectName/J41/gridfs_data 2.0G CS-censoredProjectName/J41 4.0K CS-censoredProjectName/J83/gridfs_data 28K CS-censoredProjectName/J83 4.1M CS-censoredProjectName/J10/gridfs_data 2.5G CS-censoredProjectName/J10/extract 2.5G CS-censoredProjectName/J10 2.9M CS-censoredProjectName/J71/gridfs_data 4.0K CS-censoredProjectName/J71/reconstructed 139M CS-censoredProjectName/J71/hyp_opt_trajs 606M CS-censoredProjectName/J71 4.0K CS-censoredProjectName/J87/gridfs_data 32K CS-censoredProjectName/J87 2.9M CS-censoredProjectName/J106/gridfs_data 4.0K CS-censoredProjectName/J106/reconstructed 136M CS-censoredProjectName/J106/hyp_opt_trajs 610M CS-censoredProjectName/J106 35M CS-censoredProjectName/J116/gridfs_data 7.2G CS-censoredProjectName/J116 2.2M CS-censoredProjectName/J95/gridfs_data 1.4G CS-censoredProjectName/J95 2.1M CS-censoredProjectName/J97/gridfs_data 1.4G CS-censoredProjectName/J97 688K CS-censoredProjectName/J67/gridfs_data 4.0G CS-censoredProjectName/J67 35M CS-censoredProjectName/J38/gridfs_data 15G CS-censoredProjectName/J38 28G CS-censoredProjectName/custom 37M CS-censoredProjectName/J26/gridfs_data 719M CS-censoredProjectName/J26 14M CS-censoredProjectName/J24/gridfs_data 327M CS-censoredProjectName/J24 41M CS-censoredProjectName/J105/gridfs_data 8.6G CS-censoredProjectName/J105 41M CS-censoredProjectName/J103/gridfs_data 8.6G CS-censoredProjectName/J103 41M CS-censoredProjectName/J117/gridfs_data 8.6G CS-censoredProjectName/J117 42M CS-censoredProjectName/J27/gridfs_data 884M CS-censoredProjectName/J27 4.0K CS-censoredProjectName/J88/gridfs_data 32K CS-censoredProjectName/J88 5.2M CS-censoredProjectName/J127/gridfs_data 200M CS-censoredProjectName/J127 4.0K CS-censoredProjectName/J4/gridfs_data 28K CS-censoredProjectName/J4 36M CS-censoredProjectName/J60/gridfs_data 15G CS-censoredProjectName/J60 4.0K CS-censoredProjectName/J86/gridfs_data 32K CS-censoredProjectName/J86 4.0K CS-censoredProjectName/J89/gridfs_data 32K CS-censoredProjectName/J89 3.1M CS-censoredProjectName/J2/gridfs_data 18M CS-censoredProjectName/J2/imported 23M CS-censoredProjectName/J2 42M CS-censoredProjectName/J102/gridfs_data 8.6G CS-censoredProjectName/J102 49M CS-censoredProjectName/J124/gridfs_data 7.8G CS-censoredProjectName/J124 4.0K CS-censoredProjectName/J120/gridfs_data 44K CS-censoredProjectName/J120 4.4M CS-censoredProjectName/J22/gridfs_data 80M CS-censoredProjectName/J22 12T CS-censoredProjectName/

Output from stat -f:
stat -f CS-censoredProjectName/ File: "CS-censoredProjectName/" ID: fe16fd9f3b3b7291 Namelen: 255 Type: ext2/ext3 Block size: 4096 Fundamental block size: 4096 Blocks: Total: 3874875559 Free: 543511384 Available: 348210162 Inodes: Total: 488243200 Free: 487229949
Formatting broke when I set preformatted text for console output.

4

Hi @rbs_sci,

Thanks for your feedback! Currently, symlinks created by users inside job directories are followed when calculating the project directory size. (Symlinks created by CryoSPARC during import jobs are ignored). We’ve noted this down as something that may be worked on in the future to improve data management in CryoSPARC.

2 Likes
5

OK, understood. :slight_smile: Thanks for the explanation!

1 Like