ZFS pool strategy for CryoSPARC

Hardware and Performance
1

I’m evaluating a ZFS strategy for a storage pool running on spinning HDDs that is used exclusively for CryoSPARC data processing.

Because CryoSPARC projects generate a massive amount of small metadata files alongside huge movie stacks, I am considering adding an NVMe Mirror as a special allocation class vdev (blocks=128k).

The goal is to offload all directory metadata and small files (\\le 128 KB) to the NVMe vdev, leaving the spinning disks to handle only the large bulk data files.

My questions:

  1. Given that nearly ~50% of files take up less than 0.1% of our pool space, will offloading metadata and files \\le 128 KB to NVMe significantly boost GUI responsiveness and job startup times?

  2. Is blocks=128k the right threshold for CryoSPARC, or would you recommend a different value like 64k or 256k?

2

Interesting idea, if you end up doing this please let us know how it works. Zfs is great, we run it on all our systems but we don’t have any special mirrors setup like this just simple raidz1 or 2.

3

Not carried out any extensive testing (so no nice numbers to plot), but purely anecdotally, on systems running HDD RAID (whether RAIDZ1 or hardware RAID) versus SSD RAID, the single biggest factor for load times is simultaneous load across the CryoSPARC instance (how many people trying to do how much at once) and quality of network connection. Responsiveness/load times of the CryoSPARC interface correlates better with latency and network speed (local/same town/same country/international) than whether it’s been loaded from SSD or HDD.