In short, can we return to the older organization of the scratch space that separated files by project instead of random identifiers without moving to an earlier version of cryoSPARC?
Details:
We use both squeue jobs with RELION and cryoSPARC on the same cluster. Previously, this has not been an issue as we can manually remove files that are taking up too much space in the scratch directory as the files were separated into different folders based on project. With the update to cryoSPARC 4, the scratch space is now organized as shown in the attached image, and is filling up much quicker. There is another image attached showing a bit of the old format (mnt/scratch/“instance”/project/P6/JXXX/extract). Creating a problem for RELION jobs. Is there a way to switch back to the old organizational design for scratch space without returning to an older cryoSPARC version?
Thanks for all your help!
With the new cache system, introduced in v4.4 and the default in v4.5, we aimed at increased reliability and wider cache configuration support. As of v4.5.3, it is still possible to revert to the legacy cache system by (for each worker)
-
defining, inside cryosparc_worker/config.sh
export CRYOSPARC_IMPROVED_SSD_CACHE=false
-
then deleting all contents of the CryoSPARC cache directory
Please be aware that we expect that the code for the legacy cache system will eventually be removed in a software update.
As an alternative to reverting to the old cache system, which we do not generally recommend, you may want to explore the following options:
If the cache storage volume is shared with other applications or uses, you may want to explore the cache quota
(cluster: cache_quota_mb, non-cluster: --ssdquota) and cache reserve
(cache_reserve_mb, --ssdreserve) configuration options.
For the mitigation of cache capacity issues, you may want to consider saving results in 16-bit floating point format and restacking sparsely used particle sets.
The new cache system bumps the modification times of cached files during reads. It is therefore possible to selectively delete files from cache that have not been accessed in a given time. For a motivating example, which would need to be modified according to your needs, please see this discussion and apply caution if you proceed.
[edited 2024-06-10 to add alternative suggestions]
It is a different system. I have looked at cryosparc_master and cryosparc_worker config.sh
following https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-ssd-particle-caching-in-cryosparc
–ssdreserve 4000000 to leave a few TB for other applications
master config.sh
export CRYOSPARC_SSD_CACHE_LIFETIME_DAYS=3
worker config.sh
export CRYOSPARC_CACHE_NUM_THREADS=16
This is a compromise, but keeps the SSD more free at the expense of more read/write cycles. Things do move faster with more threads.
