License ID Variable keeps vanishing

merandamasse · May 10, 2022, 4:37am

So I have resolved my previous issues with the error of “no heartbeat detected”, it’s because my license variable keeps vanishing.

My cryosparcm status configuration states that my license is valid, but when I type in: echo $LICENSE_ID it comes up blank. I will then export the license, and then echo will work, but when I check if the license exists (using the curl command) it won’t show up. I am seeing “missing authentication token” so I know that the connection is good, but for some reason I keep losing the license.

merandamasse · May 10, 2022, 4:14pm

This is the output from the job log of the most recent failed job:

Traceback (most recent call last):                                                                                                                                                                                      
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/connection.py", li                                                                                                                                          ne 160, in _new_conn                                                                                                                                            
    (self._dns_host, self.port), self.timeout, **extra_kw                                                                                                                                                                 
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/util/connection.py                                                                                                                                          ", line 84, in create_connection                                                                                                                                                                                
    raise err                                                                                                                       
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/util/connection.py                                                                                                                                          ", line 74, in create_connection                                                                                                                                                                                
    sock.connect(sa)                                                                                                                                            
ConnectionRefusedError: [Errno 111] Connection refused                                                                                                                                                                    

During handling of the above exception, another exception occurred:                                                                                                                                                       

Traceback (most recent call last):                                                                                                                                                                                      
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/connectionpool.py"                                                                                                                                          , line 677, in urlopen                                                                                                                                                  
    chunked=chunked,                                                                                                                                            
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/connectionpool.py"                                                                                                                                          , line 392, in _make_request                                                                                                                                                                    
    conn.request(method, url, **httplib_request_kw)                                                                                                                                                                       
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/http/client.py", line 1262, in request                                                                                                                                            
    self._send_request(method, url, body, headers, encode_chunked)                                                                                                                                                        
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/http/client.py", line 1308, in _send_req                                                                                                                                          uest                                                                                            
    self.endheaders(body, encode_chunked=encode_chunked)                                                                                                                                                                  
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/http/client.py", line 1257, in endheader                                                                                                                                          s                                                                                   
    self._send_output(message_body, encode_chunked=encode_chunked)                                                                                                                                                        
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/http/client.py", line 1028, in _send_out                                                                                                                                          put                                                                                         
    self.send(msg)                                                                                                                                      
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/http/client.py", line 968, in send                                                                                                                                                
    self.connect()                                                                                                                                      
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/connection.py", li                                                                                                                                          ne 187, in connect                                                                                                                                      
    conn = self._new_conn()                                                                                                                                                                 
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/connection.py", li                                                                                                                                          ne 172, in _new_conn                                                                                                                                            
    self, "Failed to establish a new connection: %s" % e                                                                                                                                                                  
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object                                                                                                                                           at 0x7fd3b0ca1590>: Failed to establish a new connection: [Errno 111] Connection
n refused                                                                                                           

During handling of the above exception, another exception occurred:                                                                                                                                                       

Traceback (most recent call last):                                                                                                                                                                                      
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/site-packages/requests/adapters.py", lin                                                                                                                                          e 449, in send                                                                                                                          
    timeout=timeout                                                                                                                                         
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/connectionpool.py"                                                                                                                                          , line 727, in urlopen                                                                                                                                                  
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]                                                                                                                                                       
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/                                                                                                                                          envs/cryosparc_master_env/lib/python3.7/site-packages/urllib3/util/retry.py", line 446, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='c107925', port=39002): Max retries exceeded with url: /api (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd3b0ca1590>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_compute/client.py", line 91, in <module>
    cli = CommandClient(host, int(port))
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_compute/client.py", line 40, in __init__
    self._reload()
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_compute/client.py", line 69, in _reload
    system = self._get_callable('system.describe')()
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/cryosparc_compute/client.py", line 57, in func
    r = requests.post(self.url, data = json.dumps(data, cls=NumpyEncoder), headers = header, timeout=self.timeout)
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='c107925', port=39002): Max retries exceeded with url: /api (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd3b0ca1590>: Failed to establish a new connection: [Errno 111] Connection refused'))

merandamasse · May 10, 2022, 4:16pm

This is the output from the log command_core:

2022-05-10 09:02:36,037 COMMAND.DATA         dump_project         INFO     | Exporting project P3
2022-05-10 09:02:36,038 COMMAND.DATA         dump_project         WARNING  | /data/PIRWT-NCI/Cryo_Sparc_Analysis/P3 does not exist. Skipping export of P3
2022-05-10 09:02:36,039 COMMAND.DATA         dump_project         INFO     | Exporting project P4
2022-05-10 09:02:36,040 COMMAND.DATA         dump_project         WARNING  | /data/PIRWT-NCI/Wanting_Tutorial/P4 does not exist. Skipping export of P4
2022-05-10 09:02:36,041 COMMAND.DATA         dump_project         INFO     | Exporting project P5
2022-05-10 09:02:36,041 COMMAND.DATA         dump_project         WARNING  | /data/PIRWT-NCI/Wanting_Tutorial/P5 does not exist. Skipping export of P5
2022-05-10 09:02:36,043 COMMAND.DATA         dump_project         INFO     | Exporting project P6
2022-05-10 09:02:36,043 COMMAND.DATA         dump_project         WARNING  | /data/UW Madison Arctica/P6 does not exist. Skipping export of P6
2022-05-10 09:02:36,223 COMMAND.DATA         dump_project         INFO     | Exporting project P7
2022-05-10 09:02:36,287 COMMAND.DATA         dump_project         INFO     | Exported project P7 to /data/Arctica_Cryosparc_Analysis/P7/project.json in 0.06s
2022-05-10 09:02:36,290 COMMAND.DATA         dump_project         INFO     | Exporting project P8
2022-05-10 09:02:36,291 COMMAND.DATA         dump_project         WARNING  | /data/Ningkun_T20S_Tutorial/CryoSPARC_Analysis/P8 does not exist. Skipping export of P8
2022-05-10 09:02:36,293 COMMAND.DATA         dump_project         INFO     | Exporting project P9
2022-05-10 09:02:36,294 COMMAND.DATA         dump_project         WARNING  | /data/ApoMb_157_EDC/Cryosparc_Analysis/P9 does not exist. Skipping export of P9
2022-05-10 09:02:37,533 COMMAND.DATA         dump_project         INFO     | Exporting project P10
2022-05-10 09:02:37,595 COMMAND.DATA         dump_project         INFO     | Exported project P10 to /data/ApoMb_157_CryoSparc_analysis/P10/project.json in 0.06s
2022-05-10 09:05:29,923 COMMAND.DATA         dump_job_database    INFO     | Request to export P10 J329
2022-05-10 09:05:29,923 COMMAND.CORE         run                  INFO     | Received task dump_job_database with 2 args and 0 kwargs
2022-05-10 09:05:29,925 COMPUTE.COMMON       param_set_spec_value INFO     | Setting parameter J329.compute_use_ssd with value False of python type <class 'bool'> and param type boolean
2022-05-10 09:05:29,949 COMMAND.DATA         dump_job_database    INFO     |    Exporting job to /data/ApoMb_157_CryoSparc_analysis/P10/J329
2022-05-10 09:05:29,952 COMMAND.DATA         dump_job_database    INFO     |    Exporting all of job's images in the database to /data/ApoMb_157_CryoSparc_analysis/P10/J329/gridfs_data...
2022-05-10 09:05:29,957 COMMAND.DATA         dump_job_database    INFO     |    Done. Exported 0 images in 0.00s
2022-05-10 09:05:29,957 COMMAND.DATA         dump_job_database    INFO     |    Exporting all job's streamlog events...
2022-05-10 09:05:29,960 COMMAND.DATA         dump_job_database    INFO     |    Done. Exported 1 files in 0.00s
2022-05-10 09:05:29,960 COMMAND.DATA         dump_job_database    INFO     |    Exporting job metafile...
2022-05-10 09:05:29,965 COMMAND.DATA         dump_job_database    INFO     |    Done. Exported in 0.01s
2022-05-10 09:05:29,965 COMMAND.DATA         dump_job_database    INFO     |    Updating job manifest...
2022-05-10 09:05:30,068 COMMAND.DATA         dump_job_database    INFO     |    Done. Updated in 0.10s
2022-05-10 09:05:30,068 COMMAND.DATA         dump_job_database    INFO     | Exported P10 J329 in 0.15s
2022-05-10 09:05:30,070 COMMAND.CORE         run                  INFO     | Completed task in 0.14708256721496582 seconds
2022-05-10 09:05:30,070 COMMAND.CORE         run                  INFO     | Received task dump_job_database with 2 args and 0 kwargs
2022-05-10 09:05:30,070 COMMAND.DATA         dump_job_database    INFO     | Request to export P10 J329
2022-05-10 09:05:30,073 COMMAND.DATA         dump_job_database    INFO     |    Exporting job to /data/ApoMb_157_CryoSparc_analysis/P10/J329
2022-05-10 09:05:30,074 COMMAND.DATA         dump_job_database    INFO     |    Exporting all of job's images in the database to /data/ApoMb_157_CryoSparc_analysis/P10/J329/gridfs_data...
2022-05-10 09:05:30,076 COMMAND.DATA         dump_job_database    INFO     |    Done. Exported 0 images in 0.00s
2022-05-10 09:05:30,076 COMMAND.DATA         dump_job_database    INFO     |    Exporting all job's streamlog events...
2022-05-10 09:05:30,077 COMMAND.DATA         dump_job_database    INFO     |    Done. Exported 1 files in 0.00s
2022-05-10 09:05:30,077 COMMAND.DATA         dump_job_database    INFO     |    Exporting job metafile...
2022-05-10 09:05:30,083 COMMAND.DATA         dump_job_database    INFO     |    Done. Exported in 0.01s
2022-05-10 09:05:30,085 COMMAND.DATA         dump_job_database    INFO     |    Updating job manifest...
2022-05-10 09:05:30,104 COMMAND.DATA         dump_job_database    INFO     |    Done. Updated in 0.02s
2022-05-10 09:05:30,104 COMMAND.DATA         dump_job_database    INFO     | Exported P10 J329 in 0.03s
2022-05-10 09:05:30,106 COMMAND.CORE         run                  INFO     | Completed task in 0.035620689392089844 seconds
2022-05-10 09:05:30,106 COMMAND.CORE         run                  INFO     | Received task layout_tree with 4 args and 0 kwargs
2022-05-10 09:05:30,471 COMMAND.CORE         run                  INFO     | Completed task in 0.36505579948425293 seconds
2022-05-10 09:05:30,472 COMMAND.CORE         run                  INFO     | Received task layout_tree with 4 args and 0 kwargs
2022-05-10 09:05:30,677 COMMAND.CORE         run                  INFO     | Completed task in 0.20497751235961914 seconds
2022-05-10 09:05:35,289 COMMAND.SCHEDULER    scheduler_run_core   INFO     | Running...
2022-05-10 09:05:35,289 COMMAND.SCHEDULER    scheduler_run_core   INFO     | Jobs Queued: [('P10', 'J329')]
2022-05-10 09:05:35,291 COMMAND.SCHEDULER    scheduler_run_core   INFO     | Licenses currently active : 0
2022-05-10 09:05:35,291 COMMAND.SCHEDULER    scheduler_run_core   INFO     | Now trying to schedule J329
2022-05-10 09:05:35,292 COMMAND.SCHEDULER    scheduler_run_core   INFO     |    Scheduling job to c107925
2022-05-10 09:05:35,568 COMMAND.CORE         scheduler_run_core   INFO     | Not a commercial instance - heartbeat set to 12 hours.
2022-05-10 09:05:35,642 COMMAND.SCHEDULER    scheduler_run_core   INFO     |      Launchable! -- Launching.
2022-05-10 09:05:35,645 COMMAND.JOBS         set_job_status       INFO     | Status changed for P10.J329 from queued to launched
2022-05-10 09:05:35,665 COMMAND.SCHEDULER    run_job              INFO     |       Running P10 J329
2022-05-10 09:05:35,665 COMMAND.SCHEDULER    run_job              INFO     |         Running job using: /home/cryosparc_user/software/cryosparc/cryosparc_worker/bin/cryosparcw
2022-05-10 09:05:35,675 COMMAND.SCHEDULER    scheduler_run_core   INFO     | Finished
2022-05-10 09:05:36,916 COMMAND.JOBS         set_job_status       INFO     | Status changed for P10.J329 from launched to started
2022-05-10 09:05:41,260 COMMAND.JOBS         set_job_status       INFO     | Status changed for P10.J329 from started to running

wtempel · May 10, 2022, 9:41pm

The LICENSE_ID variable is a convenience for use during installation and should no longer be needed after installation. During installation, the license id should have been captured for persistent storage in the CRYOSPARC_LICENSE_ID variable inside /home/cryosparc_user/software/cryosparc/cryosparc2_master/config.sh. You can check the (installed) license’s validity with the command:
cryosparcm licensestatus

wtempel · May 13, 2022, 3:46pm

@merandamasse Here are some questions to help us narrow down the problem.
Is this a single workstation (combined master and worker) or some other type of setup?
Has this cryoSPARC instance ever worked? If it has, what has changed on the computer(s) or on the network?
Did the problem first occur after an “unclean” shutdown of the cryoSPARC software or the computer?

merandamasse · May 13, 2022, 4:00pm

Hi @wtempel, the issues started to sporadically occur when I updgraded to version 3 of CryoSPARC. The instance had worked on and off since then, and more recently has been throwing errors on every job. Unfortunately, I think that every time I try to run a job now, an unclean shutdown of CryoSPARC occurs.

wtempel · May 13, 2022, 4:55pm

Because I don’t know more about the setup, use(r) patterns and workloads, I can only provide general pointers to

network issues
- computer or network-wide firewalls
- network stability
- network services (like DNS)
workload
- computer may be overwhelmed by demands on memory, processing, storage

If any of these issues results in an “unclean” disruption of cryoSPARC, additional issues may arise, such as “zombie” processes or stale socket file(s). The latter problem would manifest itself in the presence of /tmp/mongodb-*.sock and/or /tmp/cryosparc-supervisor-*.sock files owned by the linux user running the cryoSPARC instance even though the cryoSPARC instance is down and the supervisord, mongod processes have exited.