Webinterface lags in projects with big datasets

Troubleshooting
1

Dear Cryosparc Team,

we have recently some issues with the cryosparc webinterface. We have some users with datasets >50,000 movies. When these users are doing jobs, the whole Webinterface and command core lags, meaning jobs can not be cloned, deleted, submitted or edited. Is this due to the size of the datasets or are there other issues? I could not find any errors in the logs so far. There is just “Error: Could not Fetch JXXX” popping up on the web interface. Refreshing or restarting cryosparc does not help. Please let me know which information you need to investigate further.

Best,
Max

2

@mruetter Please can you provide some details about your setup. Please run these commands in a fresh shell from which you should exit after having collected the outputs (assuming that CryoSPARC is not running in a container):

nproc
free -h
eval $(cryosparcm env)
du -sh $CRYOSPARC_DB_PATH
grep "$(df $CRYOSPARC_DB_PATH | tail -n 1 | awk '{print $NF}') " /proc/mounts
cat /sys/kernel/mm/transparent_hugepage/enabled
exit

What is the full url that people type into their web browser’s address bar to access the CryoSPARC UI. Do they specify localhost, or a non-localhost address? Does the URL contain a port number? Do they connect via http, or https?
How many concurrent users does the instance have at any given time?

1 Like
3

Thank you for your fast reply!

We have a cluster setup with the Master instance running on a dedicated node.
Components:
CPUs 1x AMD Ryzen™ Threadripper™ Pro 3975WX 3.50 (4.20) GHz, 32-Core
Memory 256GB (8x 32GB) DDR4 3200MHz
Disk space: 1x 500GB NVME (for OS) 1x 1TB NVME (not used) 1x 2TB NVME (Cryosparc Database)
Operating System: Ubuntu 22.04.5 LTS (Jammy Jellyfish)

I will send you the output and the URL via private message. The users (around 8) connect via a non-localhost adress using http and specifying a port number. Access is not possible outside our local network.

Edit: I do not know if it is related but we just noticed that the import movies job must have a bug. The symlinks of around ~1000 movies were not correct from ~55000 totally imported movies. Specifically, there was a duplication in the path.
This was the path of the correct symlink:

/scratch/tmp/cryosparc/projects/username/exampleproject/S1/import_movies/FoilHole_12193314_Data_12167612_10_20240915_010204_EER.eer -> /cloud/domain/group/cryodata/2024/exampleproject/Images-Disc1/GridSquare_12166570/Data/FoilHole_12193314_Data_12167612_10_20240915_010204_EER.eer

And here the incorrect symlink:

/scratch/tmp/cryosparc/projects/username/exampleproject/S1/import_movies/FoilHole_12193314_Data_12167612_10_20240915_010204_EER.eer -> /cloud/domain/group/cryodata/2024/exampleproject/Images-Disc1/GridSquare_12166570/GridSquare_12166570/Data/FoilHole_12193314_Data_12167612_10_20240915_010204_EER.eer
4

A post was split to a new topic: Incorrect symbolic link for imported data

5

Let’s discuss this issue at Incorrect symbolic link for imported data.

1 Like
6

Please can you confirm the RAM config, as the free -h output you shared privately does not match.

This mode of access may lead to sluggish GUI performance. Please let us know if one of the suggestions in Web UI won't load - #2 by wtempel helps improve the UI response.