Cryosparc on our server work very slow with a lot of delayed communication. Even during the day, it will work well for half a day and suddenly it will not just load. And do we need to restart cryosparc everyday. Is it common? Is there any solution to it? Can you read the error message I received this morning. Can you tell me what could be the problem? Let me know if any further details are required.
Thanks in advance!
Welcome to the forum @Rajnandani.
Please post text output as text.
Please can you post the outputs of these commands:
nvidia-smi --query-gpu=index,name --format=csv
free -h
sudo journalctl | grep -i oom
df -h /data/cryosparc/cryosparc_database
grep "$(df -h /data/cryosparc/cryosparc_database | tail -n 1 | awk '{print $NF}') " /proc/mounts
cat /sys/kernel/mm/transparent_hugepage/enabled
Hello @wtempel
Thanks for your reply! Here is the output:
Last login: Wed Aug 14 15:14:45 2024 from 165.134.35.46
(base) [exx@sn4622115641 ~]$ nvidia-smi --query-gpu=index,name --format=csv
free -h
sudhindex, name
0, NVIDIA RTX A4500
1, NVIDIA RTX A4500
2, NVIDIA RTX A4500
3, NVIDIA RTX A4500
4, NVIDIA RTX A4500
5, NVIDIA RTX A4500
6, NVIDIA RTX A4500
7, NVIDIA RTX A4500
8, NVIDIA RTX A4500
o(base) [exx@sn4622115641 ~]$ free -h
total used free shared buff/cache available
Mem: 503Gi 7.8Gi 46Gi 87Mi 452Gi 495Gi
Swap: 9Gi 2.0Gi 8.0Gi
(base) [exx@sn4622115641 ~]$ sudo journalctl |grep -i oom
(base) [exx@sn4622115641 ~]$ df -h /data/cryosparc/cryosparc_database
Filesystem Size Used Avail Use% Mounted on
/dev/md126 200T 94T 96T 50% /data
(base) [exx@sn4622115641 ~]$ grep "$(df -h /data/cryosparc/cryosparc_database | tail -n 1 | awk '{print $NF}') " /proc/mounts
/dev/md126 /data ext4 rw,seclabel,relatime,stripe=1408 0 0
(base) [exx@sn4622115641 ~]$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
(base) [exx@sn4622115641 ~]$
What is the version of the CryoSPARC software on this this system?
Were there any computations running on the computer when you observed subprocess.TimeoutExpired (in your first post)?
Can you try whether disabling transparent_hugepage by running the command
sudo sh -c "echo never >/sys/kernel/mm/transparent_hugepage/enabled"
just after a reboot resolves this issue (discussion)?