Hi,
Since upgrading to CS version 4.6 we regularly encounter the following error: Queue message [Errno 24] Too many open files: …
Is there is somewhere a setting to increase the number of open files?
1 Like
@Pielhaas Thanks for reporting this issue. Please can you post
- where (which file or UI element) you encounter the error
- if available, additional lines of output leading up to the error
- job type and user action that trigger the error
@Pielhaas, further to @wtempel’s questions, could you also list your linux distribution and post the output of running the ulimit -n command?
It is not clear which file or UI causes this issue as all running jobs are affected. Looking at the Command_core log there are many errors listed, Queue message [Errno 24] Too many open files.
The first occurrence of the error cannot be found within the last 500 latest lines. The only solution is restarting CS.
We use Ubuntu 22.04.3 LTS
The command ulimit -n yields 1024
Hi @Pielhaas, could you send us a list of files currently open by the command_core processes? Here’s how to do this:
List out running command_core processes with ps
ps -eo pid,ppid,start,command | grep "cryosparc_command.command_core"
You should get output that looks similar to this:
899236 823080 Sep 16 /path/to/cryosparc_master/deps/../bin/python /path/to/cryosparc_master/deps/../bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /path/to/cryosparc_master/gunicorn.conf.py
899247 899236 Sep 16 /path/to/cryosparc_master/deps/../bin/python /path/to/cryosparc_master/deps/../bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /path/to/cryosparc_master/gunicorn.conf.py
1233970 3231355 14:27:27 0:00 grep cryosparc_command.command_core:start
Note the PIDs in the two lines that contain python processes, 899236, 899247 in this example.
For each of the two PIDs, list their open files
lsof -p PID
Finally, list out all related CryoSPARC processes, for reference
ps -eo pid,ppid,start,command | grep -e cryosparc -e mongo
Send us the output of last three commands. If required, please obfuscate any lines that contain sensitive information about your file system structure.
Here the output for the commands:
25541 25330 10:34:49 python /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home/lnd/Cryosparc/cryosparc_master/gunicorn.conf.py
25542 25541 10:34:49 python /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home/lnd/Cryosparc/cryosparc_master/gunicorn.conf.py
34276 24947 21:17:23 grep --color=auto cryosparc_command.command_core
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 25541 lnd cwd DIR 259,4 4096 951912 /home/lnd/Cryosparc/cryosparc_master
python 25541 lnd rtd DIR 259,4 4096 2 /
python 25541 lnd txt REG 259,4 17425504 3957019 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/python3.10
python 25541 lnd mem REG 259,4 68160 3943925 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/bson/_cbson.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 1434248 3958493 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_decimal.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 449368 3832374 /home/lnd/Cryosparc/cryosparc_master/cryosparc_tools/cryosparc/core.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 923648 3963901 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_generator.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 66424 3963910 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_sfc64.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 113208 3963905 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_pcg64.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 96512 3963907 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_philox.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 110688 3963903 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_mt19937.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 363184 3963878 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_bounded_integers.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 248320 3963880 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_common.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 773720 3963918 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/mtrand.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 6863304 3956685 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libgfortran.so.5.0.0
python 25541 lnd mem REG 259,4 36518408 3943517 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libopenblasp-r0.3.27.so
python 25541 lnd mem REG 259,4 36064 3946183 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libuuid.so.1.3.0
python 25541 lnd mem REG 259,4 215776 3958477 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_blake2.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 202592 3958495 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_hashlib.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 4890544 3962987 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 213792 3963912 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/bit_generator.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 92112 3963582 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/fft/_pocketfft_internal.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 215384 3963732 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/linalg/_umath_linalg.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 180944 3962986 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/core/_multiarray_tests.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 649032 3956492 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libgcc_s.so.1
python 25541 lnd mem REG 259,4 768280 3956498 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libquadmath.so.0.0.0
python 25541 lnd mem REG 259,4 6210608 3968373 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libcrypto.so.3
python 25541 lnd mem REG 259,4 26800 3958525 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_uuid.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 50480 3949580 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libffi.so.8.1.0
python 25541 lnd mem REG 259,4 543400 3958488 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_ctypes.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 46984 3958541 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/resource.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 26480 3946524 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/psutil/_psutil_posix.abi3.so
python 25541 lnd mem REG 259,4 37184 3946523 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/psutil/_psutil_linux.abi3.so
python 25541 lnd mem REG 259,4 1143408 3968375 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libssl.so.3
python 25541 lnd mem REG 259,4 507136 3958515 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_ssl.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 180952 3962287 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/liblzma.so.5.2.6
python 25541 lnd mem REG 259,4 143568 3958499 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_lzma.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 230272 3955753 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libbz2.so.1.0.8
python 25541 lnd mem REG 259,4 88488 3958478 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_bz2.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 6069952 2621540 /usr/lib/locale/locale-archive
python 25541 lnd mem REG 259,4 138656 3958549 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/zlib.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 135352 3958487 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_csv.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 112800 3950595 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libz.so.1.3.1
python 25541 lnd mem REG 259,4 2220400 2623608 /usr/lib/x86_64-linux-gnu/libc.so.6
python 25541 lnd mem REG 259,4 29952 3958503 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_opcode.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 53712 3958533 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/fcntl.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 209032 3958517 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_struct.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 579792 3958492 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_datetime.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 74784 3958506 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_posixsubprocess.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 55624 3958507 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 103672 3958496 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_heapq.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 520464 3958504 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_pickle.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 161640 3958497 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_json.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 54968 3958534 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/grp.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 266096 3958513 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_socket.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 91712 3958512 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_sha512.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 64936 3958508 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_random.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 244728 3958535 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/math.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 118000 3958531 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/binascii.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 227648 3958529 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/array.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 112688 3958542 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 14664 2623727 /usr/lib/x86_64-linux-gnu/librt.so.1
python 25541 lnd mem REG 259,4 940560 2623638 /usr/lib/x86_64-linux-gnu/libm.so.6
python 25541 lnd mem REG 259,4 14432 2625992 /usr/lib/x86_64-linux-gnu/libutil.so.1
python 25541 lnd mem REG 259,4 17409800 3957190 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.10.so.1.0
python 25541 lnd mem REG 259,4 14432 2623637 /usr/lib/x86_64-linux-gnu/libdl.so.2
python 25541 lnd mem REG 259,4 21448 2623719 /usr/lib/x86_64-linux-gnu/libpthread.so.0
python 25541 lnd mem REG 259,4 58584 3958476 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_bisect.cpython-310-x86_64-linux-gnu.so
python 25541 lnd mem REG 259,4 27002 2627633 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
python 25541 lnd mem REG 259,4 240936 2623094 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
python 25541 lnd 0r FIFO 0,14 0t0 1531109 pipe
python 25541 lnd 1w FIFO 0,14 0t0 1531110 pipe
python 25541 lnd 2w FIFO 0,14 0t0 1531110 pipe
python 25541 lnd 3r CHR 1,9 0t0 10 /dev/urandom
python 25541 lnd 4r FIFO 0,14 0t0 1525455 pipe
python 25541 lnd 5w FIFO 0,14 0t0 1525455 pipe
python 25541 lnd 6u IPv4 1525456 0t0 TCP *:39002 (LISTEN)
python 25541 lnd 7u REG 259,4 0 15204358 /tmp/wgunicorn-1qknf15q (deleted)
and the rest:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 25542 lnd cwd DIR 259,4 4096 951912 /home/lnd/Cryosparc/cryosparc_master
python 25542 lnd rtd DIR 259,4 4096 2 /
python 25542 lnd txt REG 259,4 17425504 3957019 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/python3.10
python 25542 lnd mem REG 259,4 94512 3958536 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/mmap.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 15800 3965874 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Cipher/_raw_aesni.abi3.so
python 25542 lnd mem REG 259,4 32240 3965873 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Cipher/_raw_aes.abi3.so
python 25542 lnd mem REG 259,4 48824 3965882 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Cipher/_raw_des3.abi3.so
python 25542 lnd mem REG 259,4 48824 3965881 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Cipher/_raw_des.abi3.so
python 25542 lnd mem REG 259,4 15760 3965885 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Cipher/_raw_ocb.abi3.so
python 25542 lnd mem REG 259,4 15536 3965993 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Hash/_ghash_clmul.abi3.so
python 25542 lnd mem REG 259,4 596328 3831790 /home/lnd/Cryosparc/cryosparc_master/cryosparc_compute/ioengine/core.so
python 25542 lnd mem REG 259,4 15504 3965994 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Hash/_ghash_portable.abi3.so
python 25542 lnd mem REG 259,4 15312 3966332 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Util/_cpuid_c.abi3.so
python 25542 lnd mem REG 259,4 15560 3966053 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Protocol/_scrypt.abi3.so
python 25542 lnd mem REG 259,4 749064 3950584 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libgmp.so.10.5.0
python 25542 lnd mem REG 259,4 989224 3306793 /home/lnd/Cryosparc/cryosparc_master/cryosparc_command/core.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 15664 3965818 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Cipher/_Salsa20.abi3.so
python 25542 lnd mem REG 259,4 24136 3965954 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Hash/_SHA256.abi3.so
python 25542 lnd mem REG 259,4 19976 3965952 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Hash/_SHA1.abi3.so
python 25542 lnd mem REG 259,4 15704 3965947 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Hash/_BLAKE2s.abi3.so
python 25542 lnd mem REG 259,4 15088 3966337 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Util/_strxor.abi3.so
python 25542 lnd mem REG 259,4 15824 3965880 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Cipher/_raw_ctr.abi3.so
python 25542 lnd mem REG 259,4 15528 3965886 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Cipher/_raw_ofb.abi3.so
python 25542 lnd mem REG 259,4 15736 3965879 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Cipher/_raw_cfb.abi3.so
python 25542 lnd mem REG 259,4 15528 3965878 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Cipher/_raw_cbc.abi3.so
python 25542 lnd mem REG 259,4 171416 3831782 /home/lnd/Cryosparc/cryosparc_master/cryosparc_compute/blobio/prefetch.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 1120880 3968192 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libgomp.so.1.0.0
python 25542 lnd mem REG 259,4 132648 3968385 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libyaml-0.so.2.0.9
python 25542 lnd mem REG 259,4 155400 3944933 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/matplotlib/_image.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 200696 4068865 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/kiwisolver/_cext.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 272232 3949529 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libpng16.so.16.43.0
python 25542 lnd mem REG 259,4 836856 3957002 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libfreetype.so.6.18.3
python 25542 lnd mem REG 259,4 115856 3945105 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/matplotlib/ft2font.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 158744 3944939 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/matplotlib/_path.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 834312 3949274 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libLerc.so.4
python 25542 lnd mem REG 259,4 1178096 3952082 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libzstd.so.1.5.6
python 25542 lnd mem REG 259,4 799832 3968152 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libwebp.so.7.1.9
python 25542 lnd mem REG 259,4 223032 3950703 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libxcb.so.1.1.0
python 25542 lnd mem REG 259,4 617392 3956605 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libtiff.so.5.8.0
python 25542 lnd mem REG 259,4 4287368 4064967 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
python 25542 lnd mem REG 259,4 15184 3965883 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Cipher/_raw_ecb.abi3.so
python 25542 lnd mem REG 259,4 457576 3949300 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libopenjp2.so.2.5.0
python 25542 lnd mem REG 259,4 579648 4064237 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/PIL/_imaging.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 1211752 3958546 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/unicodedata.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 13988848 3968170 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libstdc++.so.6.0.33
python 25542 lnd mem REG 259,4 792984 3952128 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/_brotli.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 18664 2648986 /usr/lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2
python 25542 lnd mem REG 259,4 281984 4073774 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/yaml/_yaml.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 42112 3968149 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libsharpyuv.so.0.1.0
python 25542 lnd mem REG 259,4 271336 3968491 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libjpeg.so.9.5.0
python 25542 lnd mem REG 259,4 68160 3943925 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/bson/_cbson.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 1434248 3958493 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_decimal.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 449368 3832374 /home/lnd/Cryosparc/cryosparc_master/cryosparc_tools/cryosparc/core.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 923648 3963901 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_generator.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 66424 3963910 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_sfc64.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 113208 3963905 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_pcg64.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 96512 3963907 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_philox.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 110688 3963903 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_mt19937.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 363184 3963878 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_bounded_integers.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 248320 3963880 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/_common.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 773720 3963918 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/mtrand.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 6863304 3956685 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libgfortran.so.5.0.0
python 25542 lnd mem REG 259,4 36518408 3943517 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libopenblasp-r0.3.27.so
python 25542 lnd mem REG 259,4 36064 3946183 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libuuid.so.1.3.0
python 25542 lnd mem REG 259,4 215776 3958477 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_blake2.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 202592 3958495 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_hashlib.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 4890544 3962987 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 15872 3965950 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/Crypto/Hash/_MD5.abi3.so
python 25542 lnd mem REG 259,4 213792 3963912 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/random/bit_generator.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 92112 3963582 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/fft/_pocketfft_internal.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 215384 3963732 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/linalg/_umath_linalg.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 180944 3962986 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/numpy/core/_multiarray_tests.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 649032 3956492 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libgcc_s.so.1
python 25542 lnd mem REG 259,4 768280 3956498 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libquadmath.so.0.0.0
python 25542 lnd mem REG 259,4 31056 3968186 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libXdmcp.so.6.0.0
python 25542 lnd mem REG 259,4 17216 3949268 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libXau.so.6.0.0
python 25542 lnd mem REG 259,4 80912 3967945 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libdeflate.so.0
python 25542 lnd mem REG 259,4 16512 3944921 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/matplotlib/_c_internal_utils.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 47240 4074363 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/bcrypt/_bcrypt.abi3.so
python 25542 lnd mem REG 259,4 215232 3965184 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/_cffi_backend.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 81712 3944039 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/pymongo/_cmessage.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 27184 3958485 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_contextvars.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 16576 4074130 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/markupsafe/_speedups.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 203680 3958501 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_multibytecodec.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 6210608 3968373 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libcrypto.so.3
python 25542 lnd mem REG 259,4 26800 3958525 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_uuid.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 50480 3949580 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libffi.so.8.1.0
python 25542 lnd mem REG 259,4 543400 3958488 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_ctypes.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 46984 3958541 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/resource.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 26480 3946524 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/psutil/_psutil_posix.abi3.so
python 25542 lnd mem REG 259,4 37184 3946523 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/psutil/_psutil_linux.abi3.so
python 25542 lnd mem REG 259,4 1143408 3968375 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libssl.so.3
python 25542 lnd mem REG 259,4 507136 3958515 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_ssl.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 180952 3962287 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/liblzma.so.5.2.6
python 25542 lnd mem REG 259,4 143568 3958499 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_lzma.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 230272 3955753 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libbz2.so.1.0.8
python 25542 lnd mem REG 259,4 88488 3958478 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_bz2.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 6069952 2621540 /usr/lib/locale/locale-archive
python 25542 lnd mem REG 259,4 138656 3958549 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/zlib.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 135352 3958487 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_csv.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 112800 3950595 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libz.so.1.3.1
python 25542 lnd mem REG 259,4 2220400 2623608 /usr/lib/x86_64-linux-gnu/libc.so.6
python 25542 lnd mem REG 259,4 29952 3958503 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_opcode.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 53712 3958533 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/fcntl.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 209032 3958517 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_struct.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 579792 3958492 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_datetime.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 74784 3958506 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_posixsubprocess.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 55624 3958507 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 103672 3958496 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_heapq.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 520464 3958504 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_pickle.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 161640 3958497 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_json.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 54968 3958534 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/grp.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 266096 3958513 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_socket.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 91712 3958512 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_sha512.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 64936 3958508 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_random.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 244728 3958535 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/math.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 118000 3958531 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/binascii.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 227648 3958529 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/array.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 112688 3958542 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 14664 2623727 /usr/lib/x86_64-linux-gnu/librt.so.1
python 25542 lnd mem REG 259,4 940560 2623638 /usr/lib/x86_64-linux-gnu/libm.so.6
python 25542 lnd mem REG 259,4 14432 2625992 /usr/lib/x86_64-linux-gnu/libutil.so.1
python 25542 lnd mem REG 259,4 17409800 3957190 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.10.so.1.0
python 25542 lnd mem REG 259,4 14432 2623637 /usr/lib/x86_64-linux-gnu/libdl.so.2
python 25542 lnd mem REG 259,4 21448 2623719 /usr/lib/x86_64-linux-gnu/libpthread.so.0
python 25542 lnd mem REG 259,4 58584 3958476 /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/lib-dynload/_bisect.cpython-310-x86_64-linux-gnu.so
python 25542 lnd mem REG 259,4 27002 2627633 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
python 25542 lnd mem REG 259,4 240936 2623094 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
python 25542 lnd 0r FIFO 0,14 0t0 1531109 pipe
python 25542 lnd 1w FIFO 0,14 0t0 1531110 pipe
python 25542 lnd 2w FIFO 0,14 0t0 1531110 pipe
python 25542 lnd 3r CHR 1,9 0t0 10 /dev/urandom
python 25542 lnd 4r FIFO 0,14 0t0 1525455 pipe
python 25542 lnd 5w FIFO 0,14 0t0 1525455 pipe
python 25542 lnd 6u IPv4 1525456 0t0 TCP *:39002 (LISTEN)
python 25542 lnd 7u REG 259,4 0 15204358 /tmp/wgunicorn-1qknf15q (deleted)
python 25542 lnd 8u a_inode 0,15 0 2090 [eventpoll]
python 25542 lnd 9r FIFO 0,14 0t0 1532977 pipe
python 25542 lnd 10w FIFO 0,14 0t0 1532977 pipe
python 25542 lnd 11u IPv4 1521577 0t0 TCP localhost:48116->localhost:39001 (ESTABLISHED)
python 25542 lnd 12u IPv4 1526447 0t0 TCP localhost:48128->localhost:39001 (ESTABLISHED)
python 25542 lnd 13u IPv4 1860062 0t0 TCP localhost:41132->localhost:39001 (ESTABLISHED)
python 25542 lnd 14u IPv4 1526448 0t0 TCP localhost:48130->localhost:39001 (ESTABLISHED)
python 25542 lnd 15u IPv4 1531117 0t0 TCP localhost:48140->localhost:39001 (ESTABLISHED)
python 25542 lnd 19u IPv4 1550562 0t0 TCP localhost:38080->localhost:39001 (ESTABLISHED)
python 25542 lnd 23u IPv4 2071054 0t0 TCP localhost:34828->localhost:39001 (ESTABLISHED)
python 25542 lnd 26u IPv4 2073177 0t0 TCP localhost:34840->localhost:39001 (ESTABLISHED)
25330 1 10:34:42 python /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/lnd/Cryosparc/cryosparc_master/supervisord.conf
25437 25330 10:34:46 mongod --auth --dbpath /home/lnd/Database/cryosparc_database --port 39001 --oplogSize 64 --replSet meteor --wiredTigerCacheSizeGB 4 --bind_ip_all
25541 25330 10:34:49 python /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home/lnd/Cryosparc/cryosparc_master/gunicorn.conf.py
25542 25541 10:34:49 python /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home/lnd/Cryosparc/cryosparc_master/gunicorn.conf.py
25568 25330 10:34:56 python /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_vis:app -n command_vis -b 0.0.0.0:39003 -c /home/lnd/Cryosparc/cryosparc_master/gunicorn.conf.py
25569 25568 10:34:56 python /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_vis:app -n command_vis -b 0.0.0.0:39003 -c /home/lnd/Cryosparc/cryosparc_master/gunicorn.conf.py
25584 25330 10:34:57 python /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_rtp:start() -n command_rtp -b 0.0.0.0:39005 -c /home/lnd/Cryosparc/cryosparc_master/gunicorn.conf.py
25585 25584 10:34:57 python /home/lnd/Cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_rtp:start() -n command_rtp -b 0.0.0.0:39005 -c /home/lnd/Cryosparc/cryosparc_master/gunicorn.conf.py
25622 25330 10:35:01 /home/lnd/Cryosparc/cryosparc_master/cryosparc_app/nodejs/bin/node ./bundle/main.js
29485 25542 15:00:23 python -c import cryosparc_compute.run as run; run.run() --project P26 --job J227 --master_hostname Cryo-Master.local --master_command_core_port 39002
29506 29485 15:00:24 python -c import cryosparc_compute.run as run; run.run() --project P26 --job J227 --master_hostname Cryo-Master.local --master_command_core_port 39002
34053 25542 21:00:40 python -c import cryosparc_compute.run as run; run.run() --project P28 --job J1075 --master_hostname Cryo-Master.local --master_command_core_port 39002
34074 34053 21:00:40 python -c import cryosparc_compute.run as run; run.run() --project P28 --job J1075 --master_hostname Cryo-Master.local --master_command_core_port 39002
34366 24947 21:25:12 grep --color=auto -e cryosparc -e mongo
Hi @Pielhaas, thank you for sending those over. We didn’t see anything in the output yet that looks suspicious. Could you please wait until you start seeing that queue error message again? When you do, please retrieve those outputs and post them here before restarting CryoSPARC.
Hello,
We face the same issue after 4.6 update, but it seems the issue occurs on 2 of 3 nodes so not sure if restarting CS can help.
It occurs on individual job levels:
and the errors seem like this:
File "/shared/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/psutil/_pslinux.py", line 1777, in _parse_stat_file File "/shared/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/psutil/_common.py", line 840, in bcat File "/shared/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/psutil/_common.py", line 828, in cat File "/shared/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/psutil/_common.py", line 788, in open_binary OSError: [Errno 24] Too many open files: '/proc/52282/stat'
There is a lot of lines but right before, there is this part which seems related:
gpu_id 1
ndims 2
dims 288 288 0
inembed 288 290 0
istride 1
idist 83520
onembed 288 145 0
ostride 1
odist 41760
batch 276
type R2C
wkspc automatic
Python traceback:
<string>:1: UserWarning: Cannot manually free CUDA array; will be freed when garbage collected```Hi @Pielhaas,
I apologize for the delay in responding to you on this issue!
We did make a change in v4.6 that keeps more file descriptors open. We are still looking into how we can make this easier for users in the future, but in the meantime here is some information that may allow you to work around the problem:
There are several things that limit the number of open file descriptors on Linux. First there is the system-wide maximum number of file descriptors. This is usually a very large number and I doubt it would be the problem in this case, but just to be sure, I recommend checking it
cat /proc/sys/fs/file-max
Then there are two per-process limits on the number of open file descriptors: the so-called hard limit and the soft limit. We only need to be concerned with the hard limit, because CryoSPARC automatically sets the soft limit as high as it is allowed to. You can check what the hard limit is by running the following command while logged in as the cryosparc user.
ulimit -Hn
If this is a low number (e.g. 1024), it could be causing problems. The hard limit can be raised by following the instructions here. I would recommend setting both the hard and soft limits to whatever the following command outputs
cat /proc/sys/fs/nr_open
Hopefully this helps. One user who tried these instructions on AWS reported that they also needed to make changes to two files in /etc/security/limits.d: 00_all_limits.conf, and 01_efa.conf, since those were interfering with the values in /etc/security/limits.conf, so that might be something to be aware of depending on your Linux distribution.
–Harris
Thank you for the feedback. It has been a while since we ran into this issue. For whatever reason, the system is currently running just fine.
I have checked the numbers, the hard limit is over 1 million, while the number of file descriptors is basically infinity. I will check these numbers again next time the error occurs.
Today the issue occurred again while we were running multiple instances of Cryosparc Live. I have recorded the output of the command_core processes and downloaded the Cryosparc instance logs. I am happy to send them to you for further analysis. I will restart Cryosparc which appears to be the only solution for now.
Thanks @Pielhaas for this kind offer. Has an email address been shared with you (in a private message) where you can send the information?
I have not yet received a private message.
I see. I sent you that message now.
Hi @Pielhaas,
CryoSPARC v4.6.1, released today, is more conservative about caching open file descriptors than v4.6.0, which should alleviate this problem. For anyone experiencing this issue, we would recommend upgrading.
–Harris
1 Like