Hi @olibclarke, this definitely looks like a database inconsistency, can you try the following:
First kill every running cryoSPARC job.
In a terminal, enter cryosparcm mongo to run cryoSPARC’s interactive MongoDB shell.
Enter the following command to see how many cache files are locked.
db.cache_files.find({status: {$nin: ['hit', 'miss']}}).length()
If the printed number is greater than zero, then some files are incorrectly locked.
To unlock them, enter this command
db.cache_files.updateMany({status: {$nin: ['hit', 'miss']}}, {$set: {status: 'miss'}})
Then try running your jobs.
Note that you will still see the “cache waiting” message if there are multiple jobs trying to cache the same files, but it should now proceed once another job finishes caching. Do not kill cache-waiting jobs while this is happening.
Let me know if you run into any issues with this.
Nick