One of our users encountered an issue when they were trying to run 2D classification with SSD caching. We had configured the SSD cache quota to be 200000 MB but the job stated it needed 232609.88 MB. Clearly the issue was the required SSD cache needed for the job exceeded the SSD cache quota. Below I am including the messages the user received:
Summary
Using random seed of 1170246694
[CPU: 485.9 MB] Loading a ParticleStack with 385049 items..
[CPU: 485.9 MB] SSD cache : cache successfuly synced in_use
[CPU: 485.9 MB] SSD cache : cache successfuly synced, found 0.00MB of files on SSD.
[CPU: 486.2 MB] SSD cache : cache successfuly requested to check 6809 files.
[CPU: 486.9 MB] SSD cache : cache requires 232609.88MB more on the SSD for files to be downloaded.
[CPU: 486.9 MB] SSD cache : cache does not have enough space for download
[CPU: 486.9 MB] SSD cache : but there are no files that can be deleted.
[CPU: 486.9 MB] SSD cache : This could be because other jobs are running and using files, or because a different program has used up space on the SSD.
[CPU: 486.9 MB] SSD cache : Waiting 30 seconds for space to become available...
Would it be possible to add a feature where if the required SSD cache space needed for a job exceeds the available SSD cache quota that the message reflects this? A bonus would be if the job would report this and wait a minute or two then run without SSD caching assuming the user did not realize how much space would be needed but would still like to run the job.