When I updated from 3.1 to 3.2, Only Databae and Command_core are working

Install, Configure and Update
1

Hi All,

Today I updated my Cryosparc from 3.1 to 3.2 with command cryosparcm update
Master update seemed fine.
But worker wasn’t updated automatically.
So I updated manually using bin/cryosparcw update with update file.
But when update master,
Cryosparc only load database and command_core
like this

qfan@GPU:~> cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/data/softwares/cryosparc/cryosparc2_master
Current cryoSPARC version: v3.2.0
----------------------------------------------------------------------------

CryoSPARC process status:

app                              STOPPED   Not started
app_dev                          STOPPED   Not started
command_core                     RUNNING   pid 4437, uptime 0:03:04
command_rtp                      STOPPED   Not started
command_vis                      STOPPED   Not started
database                         RUNNING   pid 4361, uptime 0:03:09
liveapp                          STOPPED   Not started
liveapp_dev                      STOPPED   Not started
webapp                           STOPPED   Not started
webapp_dev                       STOPPED   Not started

----------------------------------------------------------------------------

global config variables:

export CRYOSPARC_LICENSE_ID=""
export CRYOSPARC_MASTER_HOSTNAME="GPU.cpmc.columbia.edu"
export CRYOSPARC_FORCE_HOSTNAME=true
export CRYOSPARC_DB_PATH="/data/softwares/cryosparc/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false

So I did this command
$cryosparcm log database
These errors came out.

ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.7m.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libtiff.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.7m.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libtiff.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.7m.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libtiff.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.7m.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libtiff.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.7m.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libtiff.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.7m.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libtiff.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.7m.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libtiff.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.7m.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
ERROR: ld.so: object '/data/softwares/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libtiff.so' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
2021-04-23T12:41:19.287-0400 I REPL     [ReplicationExecutor] Entering primary catch-up mode.
2021-04-23T12:41:19.287-0400 I REPL     [ReplicationExecutor] Exited primary catch-up mode.
2021-04-23T12:41:19.442-0400 I NETWORK  [thread1] connection accepted from 127.0.0.1:42179 #1 (1 connection now open)
2021-04-23T12:41:19.443-0400 I NETWORK  [conn1] received client metadata from 127.0.0.1:42179 conn1: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.16.6-2-desktop" }, platform: "CPython 3.7.9.final.0" }
2021-04-23T12:41:19.444-0400 I NETWORK  [thread1] connection accepted from 127.0.0.1:42180 #2 (2 connections now open)
2021-04-23T12:41:19.444-0400 I NETWORK  [conn2] received client metadata from 127.0.0.1:42180 conn2: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.16.6-2-desktop" }, platform: "CPython 3.7.9.final.0" }
2021-04-23T12:41:20.112-0400 I -        [conn1] end connection 127.0.0.1:42179 (2 connections now open)
2021-04-23T12:41:20.112-0400 I -        [conn2] end connection 127.0.0.1:42180 (2 connections now open)
2021-04-23T12:41:20.308-0400 I NETWORK  [thread1] connection accepted from 127.0.0.1:42181 #3 (1 connection now open)
2021-04-23T12:41:20.308-0400 I NETWORK  [conn3] received client metadata from 127.0.0.1:42181 conn3: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.10" }, os: { type: "Linux", name: "openSUSE 13.2 (x86_64)", architecture: "x86_64", version: "Kernel 3.16.6-2-desktop" } }
2021-04-23T12:41:20.310-0400 I REPL     [conn3] replSetInitiate admin command received from client
2021-04-23T12:41:20.313-0400 I -        [conn3] end connection 127.0.0.1:42181 (1 connection now open)
2021-04-23T12:41:20.843-0400 I REPL     [rsSync] transition to primary complete; database writes are now permitted
2021-04-23T12:41:28.325-0400 I NETWORK  [thread1] connection accepted from 156.145.53.241:49204 #4 (1 connection now open)
2021-04-23T12:41:28.326-0400 I NETWORK  [conn4] received client metadata from 156.145.53.241:49204 conn4: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.16.6-2-desktop" }, platform: "CPython 3.7.9.final.0" }
2021-04-23T12:41:28.326-0400 I NETWORK  [thread1] connection accepted from 156.145.53.241:49205 #5 (2 connections now open)
2021-04-23T12:41:28.326-0400 I NETWORK  [conn5] received client metadata from 156.145.53.241:49205 conn5: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.16.6-2-desktop" }, platform: "CPython 3.7.9.final.0" }

I also did patch with cryosparcm patch.
But issues wasn’t solved.

If someone has solutions, please let me know.
I check previous posting seemed same issue with mine. It didn’t work.

Thanks,

Best,

Jinseo

2

Hi @Jinseo, please try the following:

  1. Stop cryoSPARC with command cryosparcm stop
  2. Start cryoSPARC with debug enabled
    DEBUG=1 cryosparcm start
    And send me the output
  3. Get the last log lines from the command_core service
    cryosparcm log command_core | tail -n 250
    And send me the output

After sending me the outputs, if cryoSPARC is still not starting up, try starting the various services manually:

cryosparcm start command_rtp
cryosparcm start webapp
cryosparcm start app
3

Hi @nfrasser,
I encountered exactly the same issue when I update cryosparc from 3.1 to 3.2.
I tried all your suggestions above and the log seems like this:

[2021-05-07 07:56:04,454] ERROR in app: Exception on /startup [GET]
Traceback (most recent call last):
  File "/cryoprograms/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/cryoprograms/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/cryoprograms/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/cryoprograms/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/cryoprograms/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/cryoprograms/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/cryoprograms/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 321, in startup
    instance_information = rc.get_instance_information()
  File "/cryoprograms/cryosparc/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1885, in get_instance_information
    cpufreq = psutil.cpu_freq()
  File "/cryoprograms/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/psutil/__init__.py", line 1857, in cpu_freq
    ret = _psplatform.cpu_freq()
  File "/cryoprograms/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/psutil/_pslinux.py", line 702, in cpu_freq
    "can't find current frequency file")
NotImplementedError: can't find current frequency file

Any more suggestions?

Thanks
Fang K.

4

Hi All,

It’s too late.
I already grinded my workstation.
I installed the fresh Centos 8 and cryosparc 3.2.
Also I installed most latest NVIDIA driver and CUDA 11.2.
Since then, everything looks good.
I think my previous system Opensuse 13.2 might be too old for supporting CUDA.
I already knew there was some issues with ananconda and conda supporting.
Thank you very much.

Best,

Jinseo

5

Same problem here and I finally found the cause:
/sys/devices/system/cpu/cpu%s/cpufreq/cpuinfo_cur_freq is only readable for root but not other users. “chmod a+r” these files will work.