Cryosparc on A100

wxh180 · December 11, 2020, 6:54pm

Dear cryosparc developer,
I am installing the new version 3.0 on one of our servers with A100 gpu cards. The OS is centOS 7 and cuda version 11. I followed the installation reference and I got the following error:

Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
command_core: started
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>
command_vis: started
command_rtp: ERROR (spawn error)

This is a fresh installation this server with A100. Our other servers running 2080Ti have been successfully upgraded to version 3.0 and run well.

Please let me know if any additional information that I can provide to fix this problem. Thanks!

Best regards,
Wei

nfrasser · December 11, 2020, 8:03pm

Hi Wei, can you send me the logs for cryoSPARC’s command_core module? You can do so with this command:

cryosparcm log command_core > command_core.log

Then send the resulting log file to me.

wxh180 · December 11, 2020, 8:26pm

@nfrasser Please see below. Let me know if there is any additional information that you will need. Thanks!

COMMAND CORE STARTED ===  2020-12-11 10:48:05.368682  ==========================
*** BG WORKER START
 * Serving Flask app "command_core" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
[JSONRPC ERROR  2020-12-11 10:48:06.468111  at  set_config_var ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 504, in set_config_var
    mongo.db[colname].update_one({'name' : name}, {op : {'name' : name, 'value' : value}}, upsert=True)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/collection.py", line 1024, in update_one
    hint=hint, session=session),
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/collection.py", line 870, in _update_retryable
    _update, session)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1498, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1384, in _retry_with_session
    return self._retry_internal(retryable, func, session, bulk)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1416, in _retry_internal
    return func(session, sock_info, retryable)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/collection.py", line 866, in _update
    retryable_write=retryable_write)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/collection.py", line 836, in _update
    retryable_write=retryable_write).copy()
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/pool.py", line 699, in command
    self._raise_connection_failure(error)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/pool.py", line 694, in command
    exhaust_allowed=exhaust_allowed)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/network.py", line 162, in command
    parse_write_concern_error=parse_write_concern_error)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/helpers.py", line 140, in _check_command_response
    raise NotMasterError(errmsg, response)
pymongo.errors.NotMasterError: not master, full error: {'ok': 0.0, 'errmsg': 'not master', 'code': 10107, 'codeName': 'NotMaster'}
-----------------------------------------------------
[2020-12-11 10:48:06,471] ERROR in app: Exception on /startup [GET]
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 268, in startup
    set_config_var('running_version', versionfile.read().strip())
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    raise e
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 504, in set_config_var
    mongo.db[colname].update_one({'name' : name}, {op : {'name' : name, 'value' : value}}, upsert=True)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/collection.py", line 1024, in update_one
    hint=hint, session=session),
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/collection.py", line 870, in _update_retryable
    _update, session)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1498, in _retryable_write
    return self._retry_with_session(retryable, func, s, None)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1384, in _retry_with_session
    return self._retry_internal(retryable, func, session, bulk)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1416, in _retry_internal
    return func(session, sock_info, retryable)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/collection.py", line 866, in _update
    retryable_write=retryable_write)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/collection.py", line 836, in _update
    retryable_write=retryable_write).copy()
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/pool.py", line 699, in command
    self._raise_connection_failure(error)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/pool.py", line 694, in command
    exhaust_allowed=exhaust_allowed)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/network.py", line 162, in command
    parse_write_concern_error=parse_write_concern_error)
  File "/data/Programs/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/helpers.py", line 140, in _check_command_response
    raise NotMasterError(errmsg, response)
pymongo.errors.NotMasterError: not master, full error: {'ok': 0.0, 'errmsg': 'not master', 'code': 10107, 'codeName': 'NotMaster'}
[JSONRPC ERROR  2020-12-11 10:48:08.364967  at  get_config_var ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 483, in get_config_var
    assert res is not None, "Variable not found."
AssertionError: Variable not found.
-----------------------------------------------------
[JSONRPC ERROR  2020-12-11 10:48:08.365226  at  get_running_version ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 438, in get_running_version
    version = get_config_var('running_version')
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    raise e
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 483, in get_config_var
    assert res is not None, "Variable not found."
AssertionError: Variable not found.
-----------------------------------------------------
[JSONRPC ERROR  2020-12-11 10:48:08.365395  at  get_system_info ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 462, in get_system_info
    'version' : get_running_version(),
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    raise e
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 438, in get_running_version
    version = get_config_var('running_version')
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    raise e
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 483, in get_config_var
    assert res is not None, "Variable not found."
AssertionError: Variable not found.
-----------------------------------------------------
[JSONRPC ERROR  2020-12-11 10:48:09.992736  at  get_config_var ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 483, in get_config_var
    assert res is not None, "Variable not found."
AssertionError: Variable not found.
-----------------------------------------------------
[JSONRPC ERROR  2020-12-11 10:48:09.992989  at  get_running_version ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 438, in get_running_version
    version = get_config_var('running_version')
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    raise e
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 483, in get_config_var
    assert res is not None, "Variable not found."
AssertionError: Variable not found.
-----------------------------------------------------
[JSONRPC ERROR  2020-12-11 10:48:09.993151  at  get_system_info ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 462, in get_system_info
    'version' : get_running_version(),
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    raise e
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 438, in get_running_version
    version = get_config_var('running_version')
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    raise e
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 483, in get_config_var
    assert res is not None, "Variable not found."
AssertionError: Variable not found.
-----------------------------------------------------
[JSONRPC ERROR  2020-12-11 10:48:12.808171  at  get_config_var ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 483, in get_config_var
    assert res is not None, "Variable not found."
AssertionError: Variable not found.
-----------------------------------------------------
[JSONRPC ERROR  2020-12-11 10:48:12.808423  at  get_running_version ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 438, in get_running_version
    version = get_config_var('running_version')
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    raise e
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 483, in get_config_var
    assert res is not None, "Variable not found."
AssertionError: Variable not found.
-----------------------------------------------------
[JSONRPC ERROR  2020-12-11 10:48:12.808587  at  get_system_info ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 462, in get_system_info
    'version' : get_running_version(),
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    raise e
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 438, in get_running_version
    version = get_config_var('running_version')
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    raise e
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 483, in get_config_var
    assert res is not None, "Variable not found."
AssertionError: Variable not found.
-----------------------------------------------------
[JSONRPC ERROR  2020-12-11 10:48:16.416640  at  get_config_var ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 483, in get_config_var
    assert res is not None, "Variable not found."
AssertionError: Variable not found.
-----------------------------------------------------
[JSONRPC ERROR  2020-12-11 10:48:16.416896  at  get_running_version ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 438, in get_running_version
    version = get_config_var('running_version')
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    raise e
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 483, in get_config_var
    assert res is not None, "Variable not found."
AssertionError: Variable not found.
-----------------------------------------------------
[JSONRPC ERROR  2020-12-11 10:48:16.417063  at  get_system_info ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 462, in get_system_info
    'version' : get_running_version(),
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    raise e
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 438, in get_running_version
    version = get_config_var('running_version')
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    raise e
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "/data/Programs/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 483, in get_config_var
    assert res is not None, "Variable not found."
AssertionError: Variable not found.
-----------------------------------------------------

nfrasser · December 11, 2020, 8:37pm

This looks like an issue similar to that reported here, where changing the base port for a previous cryoSPARC database folder caused an error:

Is this the case for you? Since this is a fresh installation, can you try reinstalling with a different --dbpath argument?

wxh180 · December 12, 2020, 1:45am

@nfrasser I don’t think so.

I tried a different dbpath you suggested. But still have the same error.

wxh180 · December 16, 2020, 12:13am

After I ran a “cryosparm restart”, it appears that command_rtp can start. However, when I ran the test job, it stuck at this step and the job did not run. I wonder what the problem is.

nfrasser · December 16, 2020, 9:49pm

@wxh180, this means that cryoSPARC’s scheduler is having trouble running the job on your worker. Can you send the last few log lines from cryoSPARC’s command_core service again? You can get them like this:

cryosparcm log command_core | tail -n 1000

wxh180 · December 16, 2020, 11:12pm

Yes. It keeps repeating this error:

****** Scheduler Failed ****
---------- Scheduler running ---------------
Jobs Queued: [(‘P1’, ‘J1’)]
Licenses currently active : 0
Now trying to schedule J1
Need slots : {}
Need fixed : {}
Master direct : True
Scheduling directly onto master node c109069
Not a commercial instance - heartbeat set to 12 hours.
Traceback (most recent call last):
File “/data/Programs/cryosparc_v3/cryosparc_master/cryosparc_command/command_core/init.py”, line 189, in background_worker
scheduler_run_core() # sets last run time
File “/data/Programs/cryosparc_v3/cryosparc_master/cryosparc_command/command_core/init.py”, line 1828, in scheduler_run_core
‘lane’ : lanes[0][‘name’],
IndexError: list index out of range
****** Scheduler Failed ****

stephan · December 17, 2020, 3:55pm

Hi @wxh180,

Is it possible that you don’t have any lanes set up?
Did you run the cryosparcw connect command to connect the cryosparc_worker part of cryoSPARC to the master node?
More details here:
https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc#connect-the-standalone-worker-node-to-the-master-node

wxh180 · December 18, 2020, 12:16pm

@stephan No, I did not run this command. But when I ran it, I got the following error:
Warning: conda environment not found; this indicates that a cryoSPARC installation is either incomplete or in progress
Traceback (most recent call last):
File “bin/connect.py”, line 1, in
from builtins import str
ImportError: No module named builtins