I have an issue with many jobs using the SSD cache.
When copying data to the cache, the jobs complains about missing space and states that there are no files, that can be deleted.
Looking at the SSD directly, I can see that there are several hundered GB from other Projects causing the SSD to be full. However, Cryosparc should be able to remove those, right?
I know, how to empty the cache manually, but that solves it only until the next two jobs used it.
Is there something in the 4.5 update, that may help here or do you have any idea how to solve this permanently?
I am not sure, but I believe, that this behavior started after I restructured our queues, but I don’t see why that should be related.
Example error:
[2024-05-15 8:48:30.57] [CPU: 4.14 GB Avail: 506.23 GB]
SSD cache : cache may not have enough space for download
[2024-05-15 8:48:30.57] [CPU: 4.14 GB Avail: 506.23 GB]
Needed | 757,193.52 MB
Available | 57,100.25 MB
Disk size | 1,875,677.55 MB
Usable space | 1,865,677.55 MB (reserve 10,000 MB)
[2024-05-15 8:48:30.58] [CPU: 4.14 GB Avail: 506.23 GB]
SSD cache : There are no files that can be deleted at cache location /scratch
[2024-05-15 8:48:30.58] [CPU: 4.14 GB Avail: 506.23 GB]
SSD cache : This could be because other jobs are running and using files, or because a different program has used up space on the SSD.
[2024-05-15 8:48:30.58] [CPU: 4.14 GB Avail: 506.23 GB]
SSD cache : Cache full for past 6,370s
free: 57,100 MB, total: 1,865,678 MB (reserve 10,000 MB)
There may be other large files pending download preventing this job from using the cache
Checking again in 30 seconds for space to become available...