MongoDB Performance Monitoring

Troubleshooting
1

We have what is probably a medium-sized CryoSPARC instance running in a VM. We have 16 CPUs, 64 GB RAM, all running on solid state drives directly attached to the VM (not networked). Our current DB size is 157 GB. In any given month, we have 10-15 active projects, and run ~1000 jobs.

What users are encountering is that, inexplicably, the web interface slows down to the point of being nearly unusable. The slowdowns aren’t constant, but they are ubiquitous, and simple tasks like, opening a job, or clicking a new job will hang for 15+ seconds.

I assume this is database related, and wanted to try monitoring DB performance to see if there was a clear bottleneck anywhere. I ran:

mongostat --host localhost:$CRYOSPARC_MONGO_PORT --rowcount 20 1
…just to try and get a mongostat command working, but it failed to authorize me:

2023-08-29T08:57:17.337-0600    Failed: not authorized on admin to execute command { serverStatus: 1, recordStats: 0, $readPreference: { mode: "secondaryPreferred" }, $db: "admin" }

I searched through the documentation, but couldn’t find anything about DB authentication. Are the admin credentials listed anywhere? Or is there a better way to approach this problem altogether?

2

Please can you post additional information, for now:

  • CryoSPARC version and patch
  • When did the problem first occur?
  • Is the system running out of RAM when the slowdowns occur?
3
  • We are on 4.3.0, no patch (our next scheduled downtime is not until 9/16, at which point it’ll get patched).
  • Problem has been occurring since we’ve been using CryoSPARC in one form or another. On this particular system, since installation, which was 3.3.2
  • I think you might be onto something here. I haven’t been able to catch it in the act, but looking right now, I can see that the swap partition is almost completely full (7.5 GB/8 GB), which means the system was swapping at some point. This is the only non-system process that runs on the VM.

In the past, I’ve noticed operations such as launching jobs with lots of particles involved (2-10 million) has chewed through all of the system memory (16 GB or so), on other systems. I never thought we’d fill up 64 GB but maybe that’s the issue.

I’ll see about allocating more memory and maybe that will fix it.

4

You may want to check the size of your command_core process. There is a developing discussion on memory use in v4.3.0. cryoSPARC master unresponsive and high memory load

5

Thanks! I will hold off for that patch and see if it fixes it. In the meantime, I will try to get some monitoring setup on our VM to see if that sheds any light on things.

6

This was precisely our issue. High memory/swap load till the point, where the web interface was very laggy.

@stephan supplied us with a beta patch, which solved the issues. I assume this will be released soon after passing internal QA tests.

2 Likes
7

In a fresh shell, with CryoSPARC v4.3.1, you can

eval $(cryosparcm env)
mongostat --username cryosparc_admin --password $(python -c "from cryosparc_compute import database_management; print(database_management.get_db_admin_password())") --host localhost:$CRYOSPARC_MONGO_PORT  --authenticationDatabase admin --rowcount 20 1
2 Likes