The workspace shows blank tiles for specific workspaces.
log webapp shows the following error
==== [workspace] project query user 60401371d0666b30081a63ab user true
==== [jobs] project query user 60401371d0666b30081a63ab user true
Not Found → [61c37b6f63bf700b684c58e0]
Not Found → [61c3784b9c4ec2556a71e805]
Not Found → [61c37d045287f34f57f332d1]
Not Found → [61c37fc1ee44e2e88a49be92]
Not Found → [61c37d2795e00b94cde887dd]
I want to get more information about what is not found
Hi @neeraj,
As you suggested the ‘Not Found’ logs refer to MongoDB IDs of PNG images uploaded representing various job output images (we refer to them as tile images) you see when browsing the user interface. These images are uploaded as the job progresses and shouldn’t be removed unless the job is cleared or deleted.
Is this workspace you’re viewing from an imported job? Did you modify the database in any way or perform any actions (clear job, clear intermediate workspaces)?
Additionally, if you could provide me with the database log file, that would be helpful:
cryosparcm log database > cryosparc_database.log
- Suhail
The jobs, workspace, and project are not imported. The workspace has been periodically cleared of intermediates, and specific jobs have been cleared and requeued, but there is no obvious pattern to it since it affects all workspaces from the project simultaneously. Given a chance to load the tile images, they eventually appear and remain until the workspace is refreshed again. Smaller workspaces in the same project will not have this issue (unless I previously attempted to load a larger workspace first, then it seems the smaller workspace will wait for the larger workspace to load first before showing tile images). Other workspaces from other projects are exhibiting the same behavior, once the tile images are stuck, no matter which workspace or project I navigate to in the CryoSparc instance, tile images will not appear.
Thanks for the additional information and logs! The database logs didn’t contain any errors or statements that suggest something is wrong. Are you able to load images/plots when you inspect a job and when running interactive jobs (e.g. exposure curation)?
Additionally, could you please perform the following steps:
cryosparcm mongo
> db.fs.chunks.stats()
> exit
- Suhail
Hi @neeraj , thanks for the log. Are you able to load images/plots when you inspect a job and when running interactive jobs (e.g. exposure curation)?
- Suhail
Once the lagging starts, we can’t open and inspect jobs - no thumbnails or images load. The text is there, but the images don’t load.
@sdawood the same problem has started to occur in a new project I created. It had been fine for weeks until today, and now I’m experiencing the same issue. The only thing I did differently today that I can think of is that I cleared a CryoSparc Live workspace that was in the project (which I had not done up until this point in this project).
I can create jobs and connect inputs to them. I can open running and complete jobs and see all of the log text, but none of the plots or visualizations. If I click Refresh, I have to wait all over again for the tile images to load. It takes less time than the larger project that this problem was first identified on, but still takes approximately 5 minutes. During this lag period, if I try to go to another project and click Refresh the problem spreads to this project as well and I have to now wait for the first set of tile images to load before the page is refreshed in the other project.
Let me know if you need any more information.
@sdawood Just want to update that when the interface is stuck like this I can not inspect interactive jobs like 2D selection.
Hi @neeraj and @navid-paknejad ,
Thank you for the additional information. It’s not immediately clear why this is happening. Thanks for the post. Are you running cryoSPARC in a single workstation configuration, meaning both the master components (web application, database, etc.) are running on the same machine as the jobs? If so, it may be possible that high CPU/memory or network load could cause the web application (and any other process running on the machine) to slow down or become unresponsive.
If you have a utility such as htop installed it can provide more insight into the various processes running on the machine and CPU/memory usage.
- Suhail