Hi
Our instance runs around 2500 jobs per months. So the lanes are quite busy. I discovered that the RAM allocation for some jobs like Deep Picker Train or basically every job with Topaz are using a lot of memory. For example currently a Deep Picker Train is using 160GB of RAM.
For example the above mention job does only allocate:
[CPU: 89.3 MB Avail: 250.20 GB] CPU : [12]
[CPU: 89.3 MB Avail: 250.20 GB] GPU : [3]
[CPU: 89.3 MB Avail: 250.19 GB] RAM : [9, 10]
Which is way to low.
Each of our systems has 4 GPUs inside and therefore would support up to 4 jobs simultaneous. Due to the low RAM allocation in these jobs, cryosparc will start more jobs on the system and this leads in the worst case that the system crashes or is unresponsive because the RAM is 100% full.
My suggestion would be to revise the Memory allocation for such jobs to ensure system stability across all lanes.
Best wishes