SSD cache issue_2

Install, Configure and Update
1

Dear colleagues,

I receive the following error whenever I try to run my processes with a cache.

[CPU: 1.05 GB Avail: 506.25 GB]
Using random seed of 907570191

[CPU: 1.15 GB Avail: 506.15 GB]
Loading a ParticleStack with 947974 items…

[CPU: 1.15 GB Avail: 506.09 GB]
SSD cache : cache successfully synced in_use

[CPU: 1.15 GB Avail: 506.07 GB]
SSD cache : cache successfully synced, found 0.00 MB of files on SSD.

[CPU: 1.15 GB Avail: 506.03 GB]
SSD cache : cache successfully requested to check 9585 files.

[CPU: 1.16 GB Avail: 505.99 GB]
SSD cache : cache requires 148,059 MB more on the SSD for files to be downloaded.

[CPU: 1.16 GB Avail: 505.99 GB]
SSD cache : cache may not have enough space for download

[CPU: 1.16 GB Avail: 505.99 GB]
Needed | 148,058.59 MB
Available | 114,435.87 MB
Disk size | 374,389.52 MB
Usable space | 364,389.52 MB (reserve 10,000 MB)

[CPU: 1.16 GB Avail: 505.99 GB]
SSD cache : There are no files that can be deleted at cache location /scratch/cryosparc_cache

[CPU: 1.16 GB Avail: 505.99 GB]
SSD cache : This could be because other jobs are running and using files, or because a different program has used up space on the SSD.

[CPU: 1.16 GB Avail: 505.99 GB]
SSD cache : Cache full for past 30s
free: 114,436 MB, total: 364,390 MB (reserve 10,000 MB)
There may be other large files pending download preventing this job from using the cache
Checking again in 30 seconds for space to become available…

Could you please prompt how to fix that?

Thank you.

Sincerely,
Dmitry

2

cryoSPARC thinks that there is insufficient space available in the cache location to hold your images. Perhaps have a rummage in /scratch and free up some more space? Bear in mind that space may be being utilised by processes other than cryoSPARC.

Depending on the typical size of your image stacks, a 350GB scratch partition may generally be limiting if running multiple, disparate jobs each with unique input data–although float16 can help to mitigate some of the pressure.

Cheers,
Yang

1 Like
3

Hello Yang,

Yes, indeed you are right. I noticed that issue.
My question is how to extend the currently available SSD scratch space?
I found this option

How to update a worker’s configuration (ssd path) - Install, Configure and Update - CryoSPARC Discuss

But I don’t see how I can set up the new SSD scratch with the updated size.

Could you please prompt on that?

Thank you.

Kind regards,
Dmitry

4

Was the worker set up initially with a restricted cache quota?

If so, you can reconfigure that particular worker using cryosparcw connect with the relevant options, e.g. --ssdquota, and include the --update flag. Refer to the documentation for more information.

Cheers,
Yang

1 Like
5

I am not sure how the initial setup of the CS was done. Is it possible to check for the space restructuins somehow?

If there are no restrictions, is there anything that can be done to fix the SSD scratch insufficient issue?

Thank you.

6

You can get a read out of the worker’s current volume setup with the shell command df -H.

In the absence of an explicit --ssdquota flag during setup, cryoSPARC will assign all available space in the defined location as usable cache space. This is the most common use case. If true in your case, the only way to increase the available space is to define a different location that has more space, e.g. a second SSD or by swapping out the existing SSD for one with a higher capacity.

Cheers,
Yang

1 Like
7

I wanted to provide you with an update regarding the issue.

The problem arose when the scratch disk, approximately 3.7TB in size, was not mounted. Consequently, the CS utilized another smaller SSD disk, where the available space was insufficient.

To address this, we took the following steps:

  1. Identified the correct SSD disk.
  2. Mounted it using the command: mount /dev/tsdfse0e/scratch.
  3. Restarted the CS.

Thank you once again for your understanding and patience.

Sincerely,
Dmitry

1 Like