Sudden database error (spawn error)

Troubleshooting
1

Hi all, I have a problem with the cryosparcm start command in cryosparc v3.3.2 on Compute Canada. It was working well last week, but since yesterday it has been reporting a database error. The error message is as follows:

Starting cryoSPARC System master process…
CryoSPARC is not already running.
database: ERROR (spawn error)

After reading some discussions, I deleted the contents of the database folder while making a backup of all files. I attempted to run the cryosparcm start command again, but encountered the same error message. Consequently, I moved all the backed up files back to the database folder, and now I am receiving a new error message:

Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1915, in list_database_names
    for doc in self.list_databases(session, nameOnly=True)]
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1896, in list_databases
    res = admin._retryable_read_command(cmd, session=session)
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/database.py", line 756, in _retryable_read_command
    _cmd, read_preference, session)
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1461, in _retryable_read
    read_pref, session, address=address)
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1278, in _select_server
    server = topology.select_server(server_selector)
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/topology.py", line 243, in select_server
    address))
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/topology.py", line 200, in select_servers
    selector, server_timeout, address)
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/topology.py", line 217, in _select_servers_loop
    (self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: localhost:39001: [Errno 111] Connection refused, Timeout: 20.0s, Topology Description: <TopologyDescription id: 6440293d06a3257cfb54bed2, topology_type: Single, servers: [<ServerDescription ('localhost', 39001) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:39001: [Errno 111] Connection refused')>]>

I would greatly appreciate any suggestions. Thank you!
Zhenhua

2

It is possible that prior to this start attempt, older CryoSPARC processes were not properly shutdown. This can happen even during
cryosparcm restart
or even though
cryosparcm stop seemingly ran successfully. To confirm this possibility (under the Linux account that usually runs the CryoSPARC instance):

  1. cryosparcm stop
  2. ps xww | grep -e cryosparc -e mongo

Are there any supervisord or mongod processes still running? If so, after ensuring they do not belong to some other CryoSPARC instance that may also be running on the same server,

  1. kill -TERM (not kill -9) relevant processes
  2. again, for confirmation of process termination
    ps xww | grep -e cryosparc -e mongo
  3. cryosparcm start

Does this work?

1 Like
3

Thank you @wtempel
There were some CryoSPARC processes still running. I killed most of them, but there was one that could not be stopped:

427959 pts/228  S+     0:00 grep --color=auto -e cryosparc -e mongo

So I re-executed the command

cryosparcm start

but received error message as before:

Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1915, in list_database_names
    for doc in self.list_databases(session, nameOnly=True)]
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1896, in list_databases
    res = admin._retryable_read_command(cmd, session=session)
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/database.py", line 756, in _retryable_read_command
    _cmd, read_preference, session)
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1461, in _retryable_read
    read_pref, session, address=address)
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1278, in _select_server
    server = topology.select_server(server_selector)
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/topology.py", line 243, in select_server
    address))
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/topology.py", line 200, in select_servers
    selector, server_timeout, address)
  File "/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/topology.py", line 217, in _select_servers_loop
    (self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: localhost:39001: [Errno 111] Connection refused, Timeout: 20.0s, Topology Description: <TopologyDescription id: 6440a80c05e1093e659c1228, topology_type: Single, servers: [<ServerDescription ('localhost', 39001) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:39001: [Errno 111] Connection refused')>]>

Should I terminate the remaining process before starting CryoSPARC?

4

There is no need to stop this process. It shows as a consequence of the
ps xww | grep -e cryosparc -e mongo
command you used to list the processes.
Please run
ps xww | grep -e cryosparc -e mongo
again under the Linux account that ususally runs CryoSPARC processes and post the output.
Please also post the output of the command
cryosparcm log database.

5

Sure. The output of the ps command is

[zming01@narval3 ~]$ ps xww | grep -e cryosparc -e mongo
 234134 ?        Ss     0:13 python /lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /lustre06/project/6064135/zming01/cryosparc/cryosparc_master/supervisord.conf
3516385 pts/500  S+     0:00 grep --color=auto -e cryosparc -e mongo
[zming01@narval3 ~]$ kill -TERM 234134
[zming01@narval3 ~]$ ps xww | grep -e cryosparc -e mongo
3520795 pts/500  S+     0:00 grep --color=auto -e cryosparc -e mongo

The output of the latter command is

[zming01@narval3 ~]$ cryosparcm log database
char_traitsIcESaIcEEEmb+0x77) [0x56017b3d3087]
 mongod(_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE+0x4C) [0x56017b3d0eec]
 mongod(_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b+0x64) [0x56017b3cd524]
 mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE+0x47C) [0x56017b3cbd4c]
 mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE+0x134) [0x56017b3b9394]
 mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb+0x2C3) [0x56017b2fa543]
 mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0xAA4) [0x56017b2ff324]
 mongod(+0x124816E) [0x56017b3b716e]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x697) [0x56017b2a95d7]
 mongod(+0x7F480C) [0x56017a96380c]
 mongod(main+0x96B) [0x56017a98207b]
 libc.so.6(__libc_start_main+0xE5) [0x7f0bdd017d85]
 mongod(+0x86DC41) [0x56017a9dcc41]
-----  END BACKTRACE  -----

Many thanks! @wtempel

6

Interesting. This is the end of an error message inside
/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/run/database.log
Please can you post the entire error message.

7

@wtempel Yes, of course. But the error message is very long and I fail to break it into smaller parts. Would it be possible for you to inform me of an alternate method for posting the database.log file?

8

Thanks for sending us the database.log file.
It seems the database is corrupted

2023-04-19T22:48:37.220-0400 E STORAGE  [initandlisten] WiredTiger error (0) [1681958917:219981][404942:0x7f0bdcd6dd40], file:collection-4--4025498105836622329.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 1032192: block header checksum of 2753760191 doesn't match expected checksum of 4023257679
2023-04-19T22:48:37.222-0400 E STORAGE  [initandlisten] WiredTiger error (0) [1681958917:222219][404942:0x7f0bdcd6dd40], file:collection-4--4025498105836622329.wt, WT_SESSION.open_cursor: collection-4--4025498105836622329.wt: encountered an illegal file format or internal value

In this case

The cited recommendations apply to CryoSPARC v4. For CryoSPARC v3, use import instead of attachment of CryoSPARC projects.
There is the additional option of “repair”, but one has to keep in mind that option’s limitations.

9

Thanks for your input, @wtempel
Unfortunately, I may need to create a new database since I don’t believe I have a complete backup. Should I remove all the files in the database folder before starting the process? Also, do you have any guidance on how to establish a database?

10

Caution: I have not tested this procedure.

  1. Ensure all CryoSPARC processes are terminated (see above).
  2. Move/rename the database folder, defined by the CRYOSPARC_DB_PATH variable inside
    /lustre06/project/6064135/zming01/cryosparc/cryosparc_master/config.sh
  3. cryosparcm start

Does this procedure create a fresh database directory at the path defined by the CRYOSPARC_DB_PATH variable?

11

@wtempel Yes, you are correct. I renamed the database folder and the command “cryosparcm start” created a new database directory without any error messages. However, the process has been frozen for over 30 minutes after it reported “command_core: started.” Should I stop the process or try something else?

screencapture
screencapture1474×578 56.9 KB

12

You may want to check
/path/to/cryosparc_master/run/command_core.log
for errors.
Is it possible that variables, like $master_host, or directory specifications, like
/home/zming01/projects/def-ximli808/zming01/ versus
/lustre06/project/6064135/zming01,
don’t work as expected?
What is the output of

cryosparcm call /usr/bin/env | grep -v LICENSE_ID

?

13

I noticed that there are some errors in the command_core.log file, but I am not sure whether they are caused by issues with variables or directory specifications. To assist with troubleshooting, I have sent a copy of the command_core.log file to the feedback email address of CryoSPARC for your reference @wtempel
The output of the “cryosparcm call /usr/bin/env | grep -v LICENSE_ID” command is

[zming01@narval2 ~]$ cryosparcm call /usr/bin/env | grep -v LICENSE_ID
SHELL=/bin/bash
TBBROOT=/cvmfs/restricted.computecanada.ca/easybuild/software/2020/Core/intel/2020.1.217/compilers_and_libraries_2020.1.217/linux/tbb
EBROOTIMKL=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217
XDG_CONFIG_DIRS=/cvmfs/soft.computecanada.ca/gentoo/2020/etc/xdg:/etc/xdg
EBVERSIONOPENMPI=4.0.3
HISTCONTROL=ignoredups
no_proxy=ets1.calculquebec.ca,narval.calcul.quebec
CRYOSPARC_MASTER_HOSTNAME=narval2.narval.calcul.quebec
CONDA_EXE=/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/bin/conda
_CE_M=
EBVERSIONICCIFORT=2020.1.217
PKG_CONFIG_PATH=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/blis/0.8.1/share/pkgconfig:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/flexiblas/3.0.4/lib/pkgconfig:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/openmpi/4.0.3/lib/pkgconfig:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/libfabric/1.10.1/lib/pkgconfig:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/ucx/1.8.0/lib/pkgconfig:/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217/mkl/bin/pkgconfig
PYTHONNOUSERSITE=true
EBVERSIONBLIS=0.8.1
EBVERSIONIFORT=2020.1.217
HISTSIZE=1000
HOSTNAME=narval2.narval.calcul.quebec
EBDEVELLIBFABRIC=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/libfabric/1.10.1/easybuild/avx2-Core-libfabric-1.10.1-easybuild-devel
NUMEXPR_NUM_THREADS=1
EBDEVELICCIFORT=/cvmfs/restricted.computecanada.ca/easybuild/software/2020/Core/intel/2020.1.217/easybuild/Core-intel-2020.1.217-easybuild-devel
LMOD_FAMILY_BASE_OS=gentoo
_ModuleTable011_=MDEvLmxvY2FsL2Vhc3lidWlsZC9tb2R1bGVzLzIwMjAvQ29yZSIKLCAiL2N2bWZzL3NvZnQuY29tcHV0ZWNhbmFkYS5jYS9lYXN5YnVpbGQvbW9kdWxlcy8yMDIwL2F2eDIvQ29yZSIKLCAiL2N2bWZzL3NvZnQuY29tcHV0ZWNhbmFkYS5jYS9lYXN5YnVpbGQvbW9kdWxlcy8yMDIwL0NvcmUiLCAiL2N2bWZzL3NvZnQuY29tcHV0ZWNhbmFkYS5jYS9jdXN0b20vbW9kdWxlcyIsCn0sCnN5c3RlbUJhc2VNUEFUSCA9ICIvY3ZtZnMvc29mdC5jb21wdXRlY2FuYWRhLmNhL2N1c3RvbS9tb2R1bGVzIiwKfQo=
_ModuleTable007_=LzIwMjAuMS4yMTciLApsb2FkT3JkZXIgPSA1LApwcm9wVCA9IHsKdHlwZV8gPSB7CnRvb2xzID0gMSwKfSwKfSwKc3RhY2tEZXB0aCA9IDEsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJpbnRlbCIsCndWID0gIl4wMDAwMjAyMC4wMDAwMDAwMDEuMDAwMDAwMjE3Lip6ZmluYWwiLAp9LApsaWJmYWJyaWMgPSB7CmZuID0gIi9jdm1mcy9zb2Z0LmNvbXB1dGVjYW5hZGEuY2EvZWFzeWJ1aWxkL21vZHVsZXMvMjAyMC9hdngyL0NvcmUvbGliZmFicmljLzEuMTAuMS5sdWEiLApmdWxsTmFtZSA9ICJsaWJmYWJyaWMvMS4xMC4xIiwKbG9hZE9yZGVyID0gNywKcHJvcFQgPSB7fSwKcmVmX2NvdW50ID0gMSwKc3RhY2tEZXB0aCA9IDIsCnN0YXR1cyA9ICJhY3RpdmUiLAp1
_ModuleTable002_=ewpmbiA9ICIvY3ZtZnMvc29mdC5jb21wdXRlY2FuYWRhLmNhL2N1c3RvbS9tb2R1bGVzL1N0ZEVudi8yMDIwLmx1YSIsCmZ1bGxOYW1lID0gIlN0ZEVudi8yMDIwIiwKbG9hZE9yZGVyID0gMTEsCnByb3BUID0gewpsbW9kID0gewpzdGlja3kgPSAxLAp9LAp9LApzdGFja0RlcHRoID0gMCwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gIlN0ZEVudiIsCndWID0gInMwMDAwMjAyMC4qemZpbmFsIiwKfSwKYmxpcyA9IHsKZm4gPSAiL2N2bWZzL3NvZnQuY29tcHV0ZWNhbmFkYS5jYS9lYXN5YnVpbGQvbW9kdWxlcy8yMDIwL2F2eDIvQ29tcGlsZXIvaW50ZWwyMDIwL2JsaXMvMC44LjEubHVhIiwKZnVsbE5hbWUgPSAiYmxpcy8wLjguMSIsCmxvYWRPcmRlciA9IDEwLApw
CRYOSPARC_PATH=/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/external/mongodb/bin:/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/bin
__LMOD_REF_COUNT_MODULEPATH=/opt/software/modulefiles:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2020/avx2/MPI/intel2020/openmpi4:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2020/avx2/Compiler/intel2020:1;/home/zming01/.local/easybuild/modules/2020/avx2/Core:1;/home/zming01/.local/easybuild/modules/2020/Core:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2020/avx2/Core:1;/cvmfs/soft.computecanada.ca/easybuild/modules/2020/Core:1;/cvmfs/soft.computecanada.ca/custom/modules:1
EPREFIX=/cvmfs/soft.computecanada.ca/gentoo/2020
LMOD_AVAIL_EXTENSIONS=no
CRYOSPARC_INSECURE=false
OMPI_MCA_btl=^openib,ofi
_ModuleTable005_=Z2VudG9vLzIwMjAiLApsb2FkT3JkZXIgPSAyLApwcm9wVCA9IHsKbG1vZCA9IHsKc3RpY2t5ID0gMSwKfSwKfSwKc3RhY2tEZXB0aCA9IDEsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJnZW50b28vMjAyMCIsCndWID0gIjAwMDAwMjAyMC4qemZpbmFsIiwKfSwKaHR0cHByb3h5ID0gewpmbiA9ICIvb3B0L3NvZnR3YXJlL21vZHVsZWZpbGVzL2h0dHBwcm94eS8xLjAubHVhIiwKZnVsbE5hbWUgPSAiaHR0cHByb3h5LzEuMCIsCmxvYWRPcmRlciA9IDEzLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMCwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImh0dHBwcm94eSIsCndWID0gIjAwMDAwMDAwMS4qemZpbmFsIiwKfSwKaW1rbCA9IHsKZm4gPSAiL2N2
EBVERSIONGENTOO=2020
EBROOTOPENMPI=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/openmpi/4.0.3
EBVERSIONMII=1.1.2
LMOD_ADMIN_FILE=/cvmfs/soft.computecanada.ca/config/lmod/admin.list
EBVERSIONUCX=1.8.0
EBROOTBLIS=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/blis/0.8.1
EBROOTGENTOO=/cvmfs/soft.computecanada.ca/gentoo/2020/usr
LMOD_RC=/cvmfs/soft.computecanada.ca/config/lmod/lmodrc.lua:/cvmfs/soft.computecanada.ca/config/lmod/lmodrc/cache_avx2_amd.lua
CRYOSPARC_MONGO_CACHE_GB=4
EBDEVELBLIS=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/blis/0.8.1/easybuild/avx2-Compiler-intel2020-blis-0.8.1-easybuild-devel
CRYOSPARC_COMMAND_VIS_PORT=39003
EBROOTMII=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/mii/1.1.2
LMOD_DIR=/cvmfs/soft.computecanada.ca/custom/software/lmod/lmod/libexec
SACCT_FORMAT=Account,User,JobID,Start,End,AllocCPUS,Elapsed,AllocTRES%30,CPUTime,AveRSS,MaxRSS,MaxRSSTask,MaxRSSNode,NodeList,ExitCode,State%20
RSNT_ARCH=avx2
PWD=/home/zming01
LOGNAME=zming01
MODULERCFILE=:/cvmfs/soft.computecanada.ca/config/lmod//modulerc_narval:/cvmfs/soft.computecanada.ca/config/lmod//modulerc
CONDA_PREFIX=/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env
MODULESHOME=/cvmfs/soft.computecanada.ca/custom/software/lmod/lmod
RSNT_INTERCONNECT=infiniband
MANPATH=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/flexiblas/3.0.4/share/man:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/openmpi/4.0.3/share/man:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/libfabric/1.10.1/share/man:/cvmfs/restricted.computecanada.ca/easybuild/software/2020/Core/intel/2020.1.217/compilers_and_libraries_2020.1.217/linux/man/common:/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/share/man:/cvmfs/soft.computecanada.ca/gentoo/2020/usr/share/gcc-data/x86_64-pc-linux-gnu/9.3.0/man:/cvmfs/soft.computecanada.ca/gentoo/2020/usr/share/binutils-data/x86_64-pc-linux-gnu/2.33.1/man:/cvmfs/soft.computecanada.ca/gentoo/2020/usr/share/man:/cvmfs/soft.computecanada.ca/custom/software/lmod/lmod/share/man::/opt/puppetlabs/puppet/share/man
EBDEVELIMKL=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217/easybuild/Core-imkl-2020.1.217-easybuild-devel
OMPI_MCA_pml=^yalla
_ModuleTable008_=c2VyTmFtZSA9ICJsaWJmYWJyaWMvMS4xMC4xIiwKd1YgPSAiMDAwMDAwMDAxLjAwMDAwMDAxMC4wMDAwMDAwMDEuKnpmaW5hbCIsCn0sCm1paSA9IHsKZm4gPSAiL2N2bWZzL3NvZnQuY29tcHV0ZWNhbmFkYS5jYS9lYXN5YnVpbGQvbW9kdWxlcy8yMDIwL0NvcmUvbWlpLzEuMS4yLmx1YSIsCmZ1bGxOYW1lID0gIm1paS8xLjEuMiIsCmxvYWRPcmRlciA9IDEyLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMCwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gIm1paSIsCndWID0gIjAwMDAwMDAwMS4wMDAwMDAwMDEuMDAwMDAwMDAyLip6ZmluYWwiLAp9LApvcGVubXBpID0gewpmbiA9ICIvY3ZtZnMvc29mdC5jb21wdXRlY2FuYWRhLmNhL2Vhc3lidWlsZC9tb2R1bGVz
TZ=EST5EDT
PIP_CONFIG_FILE=/cvmfs/soft.computecanada.ca/config/python/pip-avx2-gentoo.conf
EBROOTFLEXIBLAS=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/flexiblas/3.0.4
EBDEVELGCCCORE=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/easybuild/Core-gcccore-.9.3.0-easybuild-devel
LMOD_SHORT_TIME=3600
LD_PRELOAD=/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.7m.so:/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libtiff.so
CRYOSPARC_COMMAND_RTP_PORT=39005
EBDEVELOPENMPI=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/openmpi/4.0.3/easybuild/avx2-Compiler-intel2020-openmpi-4.0.3-easybuild-devel
HOME=/home/zming01
_ModuleTable_Sz_=11
LANG=en_CA.UTF-8
CRYOSPARC_ROOT_DIR=/lustre06/project/6064135/zming01/cryosparc/cryosparc_master
LMOD_FAMILY_BASE_OS_VERSION=2020
CRYOSPARC_FORCE_USER=false
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36::su=00:sg=00:ca=00:ow=00:st=00:tw=00:ex=00:or=00::
CRYOSPARC_HTTP_RTP_PORT=39006
SCRATCH=/scratch/zming01
CRYOSPARC_HOSTNAME_CHECK=narval2.narval.calcul.quebec
CRYOSPARC_MONGO_PORT=39001
CRYOSPARC_HTTP_PORT=39000
LMOD_SETTARG_FULL_SUPPORT=no
MII=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/mii/1.1.2/bin/mii
EBROOTLIBFABRIC=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/libfabric/1.10.1
CRYOSPARC_DB_PATH=/project/def-xinli808/zming01/cryosparc/database
CONDA_PROMPT_MODIFIER=(cryosparc_master_env)
TMPDIR=/tmp
OMPI_MCA_mtl=^ofi
CMAKE_PREFIX_PATH=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/mii/1.1.2:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/blis/0.8.1:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/flexiblas/3.0.4:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/openmpi/4.0.3:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/libfabric/1.10.1:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/ucx/1.8.0:/cvmfs/restricted.computecanada.ca/easybuild/software/2020/Core/intel/2020.1.217/compilers_and_libraries_2020.1.217/linux:/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0
CRYOSPARC_CLICK_WRAP=true
EBEXTSLISTFLEXIBLAS=LAPACK-3.9.0
RSNT_CPU_VENDOR_ID=amd
https_proxy=http://squid-compute101:3128
EBVERSIONIMKL=2020.1.217
_ModuleTable010_=bnQgPSAxLApzdGFja0RlcHRoID0gMiwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gInVjeC8xLjguMCIsCndWID0gIjAwMDAwMDAwMS4wMDAwMDAwMDguKnpmaW5hbCIsCn0sCn0sCm1wYXRoQSA9IHsKIi9vcHQvc29mdHdhcmUvbW9kdWxlZmlsZXMiCiwgIi9jdm1mcy9zb2Z0LmNvbXB1dGVjYW5hZGEuY2EvZWFzeWJ1aWxkL21vZHVsZXMvMjAyMC9hdngyL01QSS9pbnRlbDIwMjAvb3Blbm1waTQiCiwgIi9jdm1mcy9zb2Z0LmNvbXB1dGVjYW5hZGEuY2EvZWFzeWJ1aWxkL21vZHVsZXMvMjAyMC9hdngyL0NvbXBpbGVyL2ludGVsMjAyMCIKLCAiL2hvbWUvem1pbmcwMS8ubG9jYWwvZWFzeWJ1aWxkL21vZHVsZXMvMjAyMC9hdngyL0NvcmUiCiwgIi9ob21lL3ptaW5n
LMOD_VERSION=8.7.4
SSH_CONNECTION=137.82.136.3 65202 10.80.49.2 22
EBDEVELFLEXIBLAS=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/flexiblas/3.0.4/easybuild/avx2-Compiler-intel2020-flexiblas-3.0.4-easybuild-devel
EBVERSIONICC=2020.1.217
RSNT_ENABLE_LMOD_CACHE=yes
CC_CLUSTER=narval
__LMOD_Priority_MODULEPATH=/opt/software/modulefiles:-10
LMOD_PACKAGE_PATH=/cvmfs/soft.computecanada.ca/config/lmod/
CRYOSPARC_LIVE_ENABLED=true
_ModuleTable003_=cm9wVCA9IHt9LApzdGFja0RlcHRoID0gMSwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImJsaXMiLAp3ViA9ICIwMDAwMDAwMDAuMDAwMDAwMDA4LjAwMDAwMDAwMS4qemZpbmFsIiwKfSwKZmxleGlibGFzID0gewpmbiA9ICIvY3ZtZnMvc29mdC5jb21wdXRlY2FuYWRhLmNhL2Vhc3lidWlsZC9tb2R1bGVzLzIwMjAvYXZ4Mi9Db21waWxlci9pbnRlbDIwMjAvZmxleGlibGFzLzMuMC40Lmx1YSIsCmZ1bGxOYW1lID0gImZsZXhpYmxhcy8zLjAuNCIsCmxvYWRPcmRlciA9IDksCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAxLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAiZmxleGlibGFzIiwKd1YgPSAiMDAwMDAwMDAzLjAwMDAwMDAwMC4wMDAwMDAwMDQu
INFOPATH=/cvmfs/soft.computecanada.ca/gentoo/2020/usr/share/binutils-data/x86_64-pc-linux-gnu/2.33.1/info::
MODULEPATH_ROOT=/cvmfs/soft.computecanada.ca/easybuild/modules
CRYOSPARC_COMMAND_CORE_PORT=39002
EBROOTGCCCORE=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0
CRYOSPARC_SUPERVISOR_SOCK_FILE=/tmp/cryosparc-supervisor-30833287453c7083e231a256beb206f5.sock
LMOD_PKG=/cvmfs/soft.computecanada.ca/custom/software/lmod/lmod
CRYOSPARC_BASE_PORT=39000
SELINUX_ROLE_REQUESTED=
PYTHONPATH=/lustre06/project/6064135/zming01/cryosparc/cryosparc_master
TERM=xterm
EBROOTICC=/cvmfs/restricted.computecanada.ca/easybuild/software/2020/Core/intel/2020.1.217
_CE_CONDA=
MII_INDEX_FILE=/cvmfs/soft.computecanada.ca/custom/mii/data/avx2_amd
MKL_EXAMPLES=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217/mkl/examples/
FLEXIBLAS=blis
LESSOPEN=|lesspipe %s
USER=zming01
EBROOTIFORT=/cvmfs/restricted.computecanada.ca/easybuild/software/2020/Core/intel/2020.1.217
CURL_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
LIBRARY_PATH=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/blis/0.8.1/lib:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/flexiblas/3.0.4/lib:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/openmpi/4.0.3/lib:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/libfabric/1.10.1/lib:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/ucx/1.8.0/lib:/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217/mkl/lib/intel64:/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217/lib/intel64:/cvmfs/restricted.computecanada.ca/easybuild/software/2020/Core/intel/2020.1.217/compilers_and_libraries_2020.1.217/linux/tbb/lib/intel64/gcc4.8
CONDA_SHLVL=1
CRYOSPARC_DEVELOP=false
RSNT_SLURM_MPI_TYPE=pmix_v3
ARCH=x86_64
EBDEVELUCX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/ucx/1.8.0/easybuild/avx2-Core-ucx-1.8.0-easybuild-devel
LOADEDMODULES=CCconfig:gentoo/2020:gcccore/.9.3.0:imkl/2020.1.217:intel/2020.1.217:ucx/1.8.0:libfabric/1.10.1:openmpi/4.0.3:flexiblas/3.0.4:blis/0.8.1:StdEnv/2020:mii/1.1.2:httpproxy/1.0
EBVERSIONFLEXIBLAS=3.0.4
SELINUX_USE_CURRENT_RANGE=
DISPLAY=localhost:21.0
LMOD_ROOT=/cvmfs/soft.computecanada.ca/custom/software/lmod
RSYNC_PROXY=squid-compute101:3128
EBDEVELMII=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/mii/1.1.2/easybuild/Core-mii-1.1.2-easybuild-devel
SHLVL=2
BASH_ENV=/cvmfs/soft.computecanada.ca/custom/software/lmod/lmod/init/bash
_ModuleTable006_=bWZzL3NvZnQuY29tcHV0ZWNhbmFkYS5jYS9lYXN5YnVpbGQvbW9kdWxlcy8yMDIwL0NvcmUvaW1rbC8yMDIwLjEuMjE3Lmx1YSIsCmZ1bGxOYW1lID0gImlta2wvMjAyMC4xLjIxNyIsCmxvYWRPcmRlciA9IDQsCnByb3BUID0gewp0eXBlXyA9IHsKbWF0aCA9IDEsCn0sCn0sCnN0YWNrRGVwdGggPSAyLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAiaW1rbC8yMDIwLjEuMjE3IiwKd1YgPSAiXjAwMDAyMDIwLjAwMDAwMDAwMS4wMDAwMDAyMTcuKnpmaW5hbCIsCn0sCmludGVsID0gewpmbiA9ICIvY3ZtZnMvc29mdC5jb21wdXRlY2FuYWRhLmNhL2Vhc3lidWlsZC9tb2R1bGVzLzIwMjAvQ29yZS9pbnRlbC8yMDIwLjEuMjE3Lmx1YSIsCmZ1bGxOYW1lID0gImludGVs
LMOD_sys=Linux
__LMOD_STACK_LESSOPEN=fGxlc3NwaXBlICVz:fHwvdXNyL2Jpbi9sZXNzcGlwZS5zaCAlcw==
OMPI_MCA_coll=^hcoll
EBROOTUCX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/ucx/1.8.0
_ModuleTable009_=LzIwMjAvYXZ4Mi9Db21waWxlci9pbnRlbDIwMjAvb3Blbm1waS80LjAuMy5sdWEiLApmdWxsTmFtZSA9ICJvcGVubXBpLzQuMC4zIiwKbG9hZE9yZGVyID0gOCwKcHJvcFQgPSB7CnR5cGVfID0gewptcGkgPSAxLAp9LAp9LApzdGFja0RlcHRoID0gMSwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gIm9wZW5tcGkiLAp3ViA9ICIwMDAwMDAwMDQuMDAwMDAwMDAwLjAwMDAwMDAwMy4qemZpbmFsIiwKfSwKdWN4ID0gewpmbiA9ICIvY3ZtZnMvc29mdC5jb21wdXRlY2FuYWRhLmNhL2Vhc3lidWlsZC9tb2R1bGVzLzIwMjAvYXZ4Mi9Db3JlL3VjeC8xLjguMC5sdWEiLApmdWxsTmFtZSA9ICJ1Y3gvMS44LjAiLApsb2FkT3JkZXIgPSA2LApwcm9wVCA9IHt9LApyZWZfY291
CC_RESTRICTED=true
XDG_SESSION_ID=161229
http_proxy=http://squid-compute101:3128
RSNT_CUDA_DRIVER_VERSION=525.105.17
_ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0gewpiYXNlX29zID0gImdlbnRvbyIsCmNvbXBpbGVyID0gImludGVsIiwKbXBpID0gIm9wZW5tcGkiLAp9LAptVCA9IHsKQ0Njb25maWcgPSB7CmZuID0gIi9jdm1mcy9zb2Z0LmNvbXB1dGVjYW5hZGEuY2EvY3VzdG9tL21vZHVsZXMvQ0Njb25maWcubHVhIiwKZnVsbE5hbWUgPSAiQ0Njb25maWciLApsb2FkT3JkZXIgPSAxLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMSwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gIkNDY29uZmlnIiwKd1YgPSAiTS4qemZpbmFsIiwKfSwKU3RkRW52ID0g
CONDA_PYTHON_EXE=/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/bin/python
EASYBUILD_CONFIGFILES=/cvmfs/soft.computecanada.ca/easybuild/config.cfg
EASYBUILD_BUILDPATH=/tmp/zming01
LMOD_FAMILY_COMPILER=intel
XDG_RUNTIME_DIR=/run/user/3120626
SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt
S_COLORS=auto
_ModuleTable004_=KnpmaW5hbCIsCn0sCmdjY2NvcmUgPSB7CmZuID0gIi9jdm1mcy9zb2Z0LmNvbXB1dGVjYW5hZGEuY2EvZWFzeWJ1aWxkL21vZHVsZXMvMjAyMC9Db3JlL2djY2NvcmUvLjkuMy4wLmx1YSIsCmZ1bGxOYW1lID0gImdjY2NvcmUvLjkuMy4wIiwKbG9hZE9yZGVyID0gMywKcHJvcFQgPSB7fSwKcmVmX2NvdW50ID0gMSwKc3RhY2tEZXB0aCA9IDIsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJnY2Njb3JlLy45LjMuMCIsCndWID0gIjAwMDAwMDAwMC4wMDAwMDAwMDkuMDAwMDAwMDAzLip6ZmluYWwiLAp9LApnZW50b28gPSB7CmZuID0gIi9jdm1mcy9zb2Z0LmNvbXB1dGVjYW5hZGEuY2EvY3VzdG9tL21vZHVsZXMvZ2VudG9vLzIwMjAubHVhIiwKZnVsbE5hbWUgPSAi
LMOD_FAMILY_MPI_VERSION=4.0.3
SSH_CLIENT=137.82.136.3 65202 22
CONDA_DEFAULT_ENV=cryosparc_master_env
MKLROOT=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217/mkl
DEBUGINFOD_URLS=https://debuginfod.centos.org/
OMP_NUM_THREADS=1
LMOD_FAMILY_MPI=openmpi
SQUEUE_SORT=-t,e,S
which_declare=declare -f
CRYOSPARC_HTTP_RTP_LEGACY_PORT=39007
TMOUT=172800
LMOD_FAMILY_COMPILER_VERSION=2020.1.217
SLURM_MPI_TYPE=pmix_v3
XDG_DATA_DIRS=/cvmfs/soft.computecanada.ca/gentoo/2020/usr/share:/usr/local/share:/usr/share
__Init_Default_Modules=1
PATH=/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/external/mongodb/bin:/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/bin:/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin:/lustre06/project/6064135/zming01/cryosparc/cryosparc_master/deps/anaconda/condabin:/home/zming01/.local/bin:/home/zming01/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/mii/1.1.2/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/flexiblas/3.0.4/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/openmpi/4.0.3/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/libfabric/1.10.1/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/ucx/1.8.0/bin:/cvmfs/restricted.computecanada.ca/easybuild/software/2020/Core/intel/2020.1.217/compilers_and_libraries_2020.1.217/linux/bin/intel64:/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/bin:/cvmfs/soft.computecanada.ca/easybuild/bin:/cvmfs/soft.computecanada.ca/custom/bin:/cvmfs/soft.computecanada.ca/gentoo/2020/usr/sbin:/cvmfs/soft.computecanada.ca/gentoo/2020/usr/bin:/cvmfs/soft.computecanada.ca/gentoo/2020/sbin:/cvmfs/soft.computecanada.ca/gentoo/2020/bin:/cvmfs/soft.computecanada.ca/custom/bin/computecanada:/opt/software/slurm/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/home/zming01/projects/def-xinli808/zming01/cryosparc/cryosparc_master/bin
SELINUX_LEVEL_REQUESTED=
MODULEPATH=/opt/software/modulefiles:/cvmfs/soft.computecanada.ca/easybuild/modules/2020/avx2/MPI/intel2020/openmpi4:/cvmfs/soft.computecanada.ca/easybuild/modules/2020/avx2/Compiler/intel2020:/home/zming01/.local/easybuild/modules/2020/avx2/Core:/home/zming01/.local/easybuild/modules/2020/Core:/cvmfs/soft.computecanada.ca/easybuild/modules/2020/avx2/Core:/cvmfs/soft.computecanada.ca/easybuild/modules/2020/Core:/cvmfs/soft.computecanada.ca/custom/modules
SQUEUE_FORMAT=%.15i %.8u %.12a %.14j %.3t %.10L %.5D %.4C %.10b %.7m %N (%r)
CRYOSPARC_HEARTBEAT_SECONDS=60
_LMFILES_=/cvmfs/soft.computecanada.ca/custom/modules/CCconfig.lua:/cvmfs/soft.computecanada.ca/custom/modules/gentoo/2020.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2020/Core/gcccore/.9.3.0.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2020/Core/imkl/2020.1.217.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2020/Core/intel/2020.1.217.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2020/avx2/Core/ucx/1.8.0.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2020/avx2/Core/libfabric/1.10.1.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2020/avx2/Compiler/intel2020/openmpi/4.0.3.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2020/avx2/Compiler/intel2020/flexiblas/3.0.4.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2020/avx2/Compiler/intel2020/blis/0.8.1.lua:/cvmfs/soft.computecanada.ca/custom/modules/StdEnv/2020.lua:/cvmfs/soft.computecanada.ca/easybuild/modules/2020/Core/mii/1.1.2.lua:/opt/software/modulefiles/httpproxy/1.0.lua
INTEL_PYTHONHOME=/cvmfs/restricted.computecanada.ca/easybuild/software/2020/Core/intel/2020.1.217/debugger_2020/python/intel64
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/3120626/bus
EBVERSIONLIBFABRIC=1.10.1
LMOD_CMD=/cvmfs/soft.computecanada.ca/custom/software/lmod/lmod/libexec/lmod
MKL_NUM_THREADS=1
MAIL=/var/spool/mail/zming01
SSH_TTY=/dev/pts/284
INTEL_LICENSE_FILE=/cvmfs/restricted.computecanada.ca/config/licenses/intel/2020/narval.lic
EBVERSIONGCCCORE=9.3.0
CRYOSPARC_CONDA_ENV=cryosparc_master_env
rsync_proxy=squid-compute101:3128
CRYOSPARC_FORCE_HOSTNAME=false
LMOD_SYSTEM_DEFAULT_MODULES=StdEnv mii
CPATH=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/blis/0.8.1/include/blis:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/blis/0.8.1/include:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/flexiblas/3.0.4/include/flexiblas:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/openmpi/4.0.3/include:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/libfabric/1.10.1/include:/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/ucx/1.8.0/include:/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217/mkl/include/fftw:/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217/mkl/include:/cvmfs/restricted.computecanada.ca/easybuild/software/2020/Core/intel/2020.1.217/compilers_and_libraries_2020.1.217/linux/tbb/include
EBROOTICCIFORT=/cvmfs/restricted.computecanada.ca/easybuild/software/2020/Core/intel/2020.1.217
OLDPWD=/lustre06/project/6064135/zming01/cryosparc/cryosparc_master
CMAKE_LIBRARY_PATH=/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/lib64
LMOD_AVAIL_STYLE=grouped:system
BASH_FUNC_ml%%=() {  eval "$($LMOD_DIR/ml_cmd "$@")"
}
BASH_FUNC_which%%=() {  ( alias;
 eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
}
BASH_FUNC_module%%=() {  if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
 case "$-" in
 *v*x*)
 __lmod_sh_dbg='vx'
 ;;
 *v*)
 __lmod_sh_dbg='v'
 ;;
 *x*)
 __lmod_sh_dbg='x'
 ;;
 esac;
 fi;
 if [ -n "${__lmod_sh_dbg:-}" ]; then
 set +$__lmod_sh_dbg;
 echo "Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output" 1>&2;
 fi;
 eval "$($LMOD_CMD bash "$@")" && eval $(${LMOD_SETTARG_CMD:-:} -s sh);
 __lmod_my_status=$?;
 if [ -n "${__lmod_sh_dbg:-}" ]; then
 echo "Shell debugging restarted" 1>&2;
 set -$__lmod_sh_dbg;
 fi;
 unset __lmod_sh_dbg;
 return $__lmod_my_status
}
_=/usr/bin/env

Thank you.

14

Given the complexity of this compute environment and implemented workarounds, I am, unfortunately, unable to pinpoint a specific problem whose correction would restore CryoSPARC function.
Just in case you are attempting to run a CryoSPARC instance in workstation (“standalone”) mode on a cluster compute node, you may reduce complexity by adjusting the following variables inside cryosparc_master/config.sh:

export CRYOSPARC_MASTER_HOSTNAME=127.0.0.1
export CRYOSPARC_HOSTNAME_CHECK=127.0.0.1

and also connecting 127.0.0.1 as a worker (see guide):

cryosparcw connect --worker 127.0.0.1 --master 127.0.0.1 --newlane --lane localhost [...]

This would “insulate” the configuration from hostname changes, but worker information may have to be updated, also with cryosparcw connect, if the host’s CPU, RAM or GPU configuration change.
Also keep in mind that on a shared host, port usage needs to be carefully coordinated to avoid conflicts with other user’s jobs that may also run on the same host.