Procedure for changing the license key in a running instance

ncoudray · November 26, 2018, 2:48pm

Hi,

We had to install several instances of Cryosparc on our cluster (one for each “lab”) and forgot to request a license for each of them. Now, we have conflicts (Token is invalid. Another cryoSPARC instance is running with the same license ID).
To resolve them, we will request each lab to get their own license key.

Could you please let us know what is the procedure for changing the license key in a running instance without re-installing it?

Thanks,
Best,
Nicolas

sdawood · November 26, 2018, 6:36pm

Hi @ncoudray,

Once you’ve received a license for each node, you can simply:

Stop the instance: cryosparcm stop
Edit the config.sh file in the cryosparc2_master directory (just edit the first line that sets the CRYOSPARC_LICENSE_ID environment variable)
Start the instance again: cryosparcm start

Any jobs you run going forward should proceed normally.

- Suhail

ncoudray · November 26, 2018, 6:49pm

Great, thanks a lot Suhail for the quick feedback!

Best,
Nicolas

ncoudray · November 27, 2018, 2:32pm

Hi @sdawood,

Ah, bad news. Although we have different licences now, we still get the same error message:

Launching job on lane BP_gpu4_short target BP_gpu4_short ...
Token is invalid. Another cryoSPARC instance is running with the same license ID.

Can you please advise how to run several instances of CryoSparc on a shared cluster?

Thanks a lot,
Best,
Nicolas

ncoudray · November 27, 2018, 2:45pm

I notice there is also a config file with the key in “cryosparc2_master” ~ I guess I need to change this one as well.
I haven’t seen any, but is there anything to change in “cryosparc2_database” as well regarding the license key?

spunjani · November 27, 2018, 3:11pm

Hi @ncoudray, the instructions should have been relating to cryosparc2_master, we have updated these in the original post above ^. Please let us know if that works. Thanks!

ncoudray · November 27, 2018, 3:13pm

@spunjani - I will try, thanks.
Just to be clear:
I see 2 config files, one in master, one in worker. You say I should change the config in “master” only? (why not both master and worker config files?)

spunjani · November 27, 2018, 3:17pm

Hi @ncoudray, yes you only need to change the config file in the master node. The worker(s) do not require a license ID because it only works by communicating with the master installation, which verifies you license with our servers.

ncoudray · November 28, 2018, 2:34pm

Hi @spunjani and @sdawood,

I changed the license key and it was working on our cluster. Then, the other installation of CryoSparc (which is running on a different license key now) was re-started, and ours stopped working again. This time, I got different error message and can’t figure out how to correct for it. Here are the messages and log files:

$ cryosparcm start
Starting cryoSPARC System master process…
CryoSPARC is not already running.
database: ERROR (spawn error)
Traceback (most recent call last):
File “”, line 1, in
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py”, line 1149, in database_names
“listDatabases”)[“databases”]]
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/database.py”, line 491, in command
with client._socket_for_reads(read_preference) as (sock_info, slave_ok):
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py”, line 17, in enter
return self.gen.next()
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py”, line 859, in _socket_for_reads
with self._get_socket(read_preference) as sock_info:
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py”, line 17, in enter
return self.gen.next()
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py”, line 823, in _get_socket
server = self._get_topology().select_server(selector)
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py”, line 214, in select_server
address))
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py”, line 189, in select_servers
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:39101: [Errno 111] Connection refused
[svc_bhabhaekiertlabs@ws-0001 svc_bhabhaekiertlabs]$

Then for the logs:

$ cryosparcm log database
2018-11-28T09:05:16.759-0500 I CONTROL [initandlisten] now exiting
2018-11-28T09:05:16.759-0500 I CONTROL [initandlisten] shutting down with code:100
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] MongoDB starting : pid=320003 port=39101 dbpath=/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_database 64-bit host=ws-0001
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] db version v3.4.10-4-g67ee356c6b
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] git version: 67ee356c6be377cda547d16423daef5beb4e8377
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2p 14 Aug 2018
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] allocator: tcmalloc
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] modules: none
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] build environment:
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] distarch: x86_64
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] target_arch: x86_64
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] options: { net: { port: 39101 }, replication: { oplogSizeMB: 64, replSet: “meteor” }, storage: { dbPath: “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_database”, journal: { enabled: false }, wiredTiger: { engineConfig: { cacheSizeGB: 4.0 } } } }
2018-11-28T09:05:19.813-0500 W - [initandlisten] Detected unclean shutdown - /gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_database/mongod.lock is not empty.
2018-11-28T09:05:19.828-0500 I - [initandlisten] Detected data files in /gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_database created by the ‘wiredTiger’ storage engine, so setting the active storage engine to ‘wiredTiger’.
2018-11-28T09:05:19.828-0500 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2018-11-28T09:05:19.828-0500 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4096M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),log=(enabled=false),
2018-11-28T09:05:19.851-0500 E STORAGE [initandlisten] WiredTiger error (122) [1543413919:851902][320003:0x2aaaac3ccd40], connection: /gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_database/WiredTigerLAS.wt: handle-write: pwrite: failed to write 4096 bytes at offset 0: Disk quota exceeded
2018-11-28T09:05:19.853-0500 I - [initandlisten] Assertion: 28595:122: Disk quota exceeded src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 276
2018-11-28T09:05:19.853-0500 I STORAGE [initandlisten] exception in initAndListen: 28595 122: Disk quota exceeded, terminating
2018-11-28T09:05:19.853-0500 I NETWORK [initandlisten] shutdown: going to close listening sockets…
2018-11-28T09:05:19.853-0500 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-39101.sock
2018-11-28T09:05:19.853-0500 I NETWORK [initandlisten] shutdown: going to flush diaglog…

For the database:

$ cryosparcm log> database
2018-11-28T09:05:16.759-0500 I CONTROL [initandlisten] now exiting
2018-11-28T09:05:16.759-0500 I CONTROL [initandlisten] shutting down with code:100
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] MongoDB starting : pid=320003 port=39101 dbpath=/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_database 64-bit host=ws-0001
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] db version v3.4.10-4-g67ee356c6b
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] git version: 67ee356c6be377cda547d16423daef5beb4e8377
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2p 14 Aug 2018
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] allocator: tcmalloc
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] modules: none
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] build environment:
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] distarch: x86_64
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] target_arch: x86_64
2018-11-28T09:05:19.813-0500 I CONTROL [initandlisten] options: { net: { port: 39101 }, replication: { oplogSizeMB: 64, replSet: “meteor” }, storage: { dbPath: “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_database”, journal: { enabled: false }, wiredTiger: { engineConfig: { cacheSizeGB: 4.0 } } } }
2018-11-28T09:05:19.813-0500 W - [initandlisten] Detected unclean shutdown - /gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_database/mongod.lock is not empty.
2018-11-28T09:05:19.828-0500 I - [initandlisten] Detected data files in /gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_database created by the ‘wiredTiger’ storage engine, so setting the active storage engine to ‘wiredTiger’.
2018-11-28T09:05:19.828-0500 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2018-11-28T09:05:19.828-0500 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4096M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),log=(enabled=false),
2018-11-28T09:05:19.851-0500 E STORAGE [initandlisten] WiredTiger error (122) [1543413919:851902][320003:0x2aaaac3ccd40], connection: /gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_database/WiredTigerLAS.wt: handle-write: pwrite: failed to write 4096 bytes at offset 0: Disk quota exceeded
2018-11-28T09:05:19.853-0500 I - [initandlisten] Assertion: 28595:122: Disk quota exceeded src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 276
2018-11-28T09:05:19.853-0500 I STORAGE [initandlisten] exception in initAndListen: 28595 122: Disk quota exceeded, terminating
2018-11-28T09:05:19.853-0500 I NETWORK [initandlisten] shutdown: going to close listening sockets…
2018-11-28T09:05:19.853-0500 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-39101.sock
2018-11-28T09:05:19.853-0500 I NETWORK [initandlisten] shutdown: going to flush diaglog…

And the last one:

$ cryosparc log command_core
if len(self.__data) or self._refresh():
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py”, line 1036, in _refresh
self.__collation))
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py”, line 873, in __send_message
**kwargs)
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py”, line 888, in _send_message_with_response
server = topology.select_server(selector)
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py”, line 214, in select_server
address))
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py”, line 189, in select_servers
self._error_message(selector))
ServerSelectionTimeoutError: ws-0001.cm.cluster:39101: [Errno 111] Connection refused
****** Scheduler Failed ****
****** Heartbeat check failed ****
[JSONRPC ERROR 2018-11-28 09:01:52.188465 at get_config_var ]

Traceback (most recent call last):
File “cryosparc2_command/command_core/init.py”, line 106, in wrapper
res = func(*args, **kwargs)
File “cryosparc2_command/command_core/init.py”, line 317, in get_config_var
res = mongo.db[colname].find_one({‘name’ : name})
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/collection.py”, line 1102, in find_one
for result in cursor.limit(-1):
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py”, line 1114, in next
if len(self.__data) or self._refresh():
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py”, line 1036, in _refresh
self.__collation))
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py”, line 873, in __send_message
**kwargs)
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py”, line 888, in _send_message_with_response
server = topology.select_server(selector)
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py”, line 214, in select_server
address))
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py”, line 189, in select_servers
self._error_message(selector))
ServerSelectionTimeoutError: ws-0001.cm.cluster:39101: [Errno 111] Connection refused

[JSONRPC ERROR 2018-11-28 09:01:52.192107 at get_scheduler_lanes ]

Traceback (most recent call last):
File “cryosparc2_command/command_core/init.py”, line 106, in wrapper
res = func(*args, **kwargs)
File “cryosparc2_command/command_core/init.py”, line 634, in get_scheduler_lanes
return get_config_var(‘lanes’, ‘sched_config’, fail_notset=False, default=[])
File “cryosparc2_command/command_core/init.py”, line 115, in wrapper
raise e
ServerSelectionTimeoutError: ws-0001.cm.cluster:39101: [Errno 111] Connection refused

Traceback (most recent call last):
File “cryosparc2_command/command_core/init.py”, line 146, in background_worker
scheduler_run_core() # sets last run time
File “cryosparc2_command/command_core/init.py”, line 1157, in scheduler_run_core
lanes = get_scheduler_lanes()
File “cryosparc2_command/command_core/init.py”, line 115, in wrapper
raise e
ServerSelectionTimeoutError: ws-0001.cm.cluster:39101: [Errno 111] Connection refused
Traceback (most recent call last):
File “cryosparc2_command/command_core/init.py”, line 151, in background_worker
check_heartbeats()
File “cryosparc2_command/command_core/init.py”, line 1315, in check_heartbeats
‘heartbeat_at’ : {’$lt’ : deadline} }, {‘project_uid’ : 1, ‘uid’ : 1}))
File “/gpfs/data/bhabhaekiertlabs/local_software/CryoSparc/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py”, line 1114, in next

Could you please help with this?

Thanks,
Best,
Nicolas

sdawood · November 28, 2018, 3:29pm

Hi @ncoudray,

It looks like cryoSPARC can’t start because it is having trouble connecting to the database. From your database logs, MongoDB is complaining about a disk quota that is exceeded:

2018-11-28T09:05:19.853-0500 I - [initandlisten] Assertion: 28595:122: Disk quota exceeded src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 276

We haven’t encountered this error with MongoDB before - we will look into it and keep you updated.

Thanks,
Suhail

ncoudray · November 28, 2018, 3:36pm

thanks a lot @sdawood
You’re right, it was related to a lack of space and not the recent changes. The quota has been adjusted and it is now working again.

Thanks!
Nicolas

marcv · December 7, 2018, 4:28pm

Hi @spunjani,

I am running cryosparc in the same cluster that Nicolas, and the program is running amazing, but my lab would like to request our own license key to solve the problem that this topic explains above. We haven been trying to get the license but I have not had replies to our registration. Could you please send us an academic license?

Thank you so much

spunjani · December 7, 2018, 5:11pm

Hi @marcv, our license request system can take up to 24 hours to verify your request. You should likely have received a response now.

donghuachen · February 7, 2020, 3:35am

Hi, on one workstation and under my account, I wanted to run multiple jobs using CryoSPARC. However, I got this message: “Token is invalid. Another cryoSPARC instance is running with the same license ID.”

Does this mean I can only run one job at a time for my account, even for the same Workspace and same Project? Sorry if this question was asked before.

Thanks.

donghuachen · February 7, 2020, 5:36pm

Hi, I stopped then restarted cryoSPARC, the problem seems to be solved. Thanks.