How to deal with SSD filled with Cache

Install, Configure and Update
1

Hi Everyone,

I am getting the error regarding the space available in /ssd. I am using a latest version of cryosparc.

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 115, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/class2D/newrun.py”, line 88, in cryosparc_master.cryosparc_compute.jobs.class2D.newrun.run_class_2D
File “/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/particles.py”, line 120, in read_blobs
u_blob_paths = cache_run(u_rel_paths)
File “/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/jobs/cache_v2.py”, line 821, in run
return run_with_executor(rel_sources, executor)
File “/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/jobs/cache_v2.py”, line 859, in run_with_executor
state = drive.allocate(sources, active_run_ids=info[“active_run_ids”])
File “/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/jobs/cache_v2.py”, line 582, in allocate
raise RuntimeError(
RuntimeError: SSD cache needs additional 144.70 GiB but drive has 122.37 GiB free. CryoSPARC can only free a maximum of 0.00 B. This may indicate that programs other than CryoSPARC are using the SSD. Please remove any files on the SSD that are outside the /ssd/cryosparc_cache/instance_spgpu:39001 folder or disable SSD cache for this job.

Is there any command or tips or script to deal with cache that quickly covered all space of SSD? I would really appreciate for that.

2

In the cryosparc_master config.sh file sometimes the actual path to the SSD can change depending on what is happening on the file system.

The file structure in SSD-scratch changed recently. It used to by project and you could just go in and manually delete

$rm -rf project 23

which is safe https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-ssd-particle-caching-in-cryosparc

but now you may want to change the File lifetime or perhaps clean up some jobs. I do not think clear intermediate results will clear the SDD cache.

In your case, if the SSD is full it would be good to know how to manually clear it. The old way was easier since you could just remove caches for old projects. Most of the scratch space was taken by particle stacks anyways.

3

Thank yo so much…Mark. Will be helpful.

4

@Suresh Under normal circumstances, file copying to and file removal from
/ssd/cryosparc_cache/instance_spgpu\:39001 should happen automatically. Manual removal should not be required.
If the filesystem that stores the /ssd/cryosparc_cache/instance_spgpu\:39001 directory is shared with other uses/functions, in addition to particle caching for the CryoSPARC instance on spgpu at port 39000, those other uses may reduce the space available for particle caching.
To provide a clearer picture of the situation, please can you run these commands on the CryoSPARC worker computer and post their outputs:

sudo du -sh /ssd
sudo du -sh /ssd/cryosparc_cache/
du -sh /ssd/cryosparc_cache/instance_spgpu\:39001
df -h /ssd/cryosparc_cache/instance_spgpu\:39001
5

Thank you @wtempel.
I have noticed our SSDs filling too.
In v4.5.3 is there a safe way to manually purge the SSD/scratch cache ?

6

@Mark-A-Nakasone In CryoSPARC v4.5.3, if no CryoSPARC jobs are running, it is safe, but should normally not be required, to delete the instance_${CRYOSPARC_MASTER_HOSTNAME}:$((CRYOSPARC_BASE_PORT+1))/ subdirectory under the Cache Path of a CryoSPARC worker.

1 Like
7

Thank you @wtempel I will give that a try :grinning:

8

/ssd/cs_scratch/instance_worker3:39001/store-v2
Can files in SSD cache locations be manually deleted?

9

@niejiawei The deletion of files inside instance_*:*/ or instance_*:*/store-v2 should generally be left up to CryoSPARC. However, it is safe to manually delete any files and folders under instance_*:*/ when no jobs are running.