Webapp not starting after v3.0.1 update

asente · December 17, 2020, 7:09pm

Hi,

After updating from 2.15 to 3.0.1 (without any errors) I cannot start the webapp. Overriding the worker update does not help.

Current cryoSPARC version: v3.0.1

cryosparcm process status:

app STOPPED Not started
app_dev STOPPED Not started
command_core RUNNING pid 440832, uptime 0:15:53
command_rtp RUNNING pid 440885, uptime 0:15:50
command_vis RUNNING pid 440877, uptime 0:15:51
database RUNNING pid 440750, uptime 0:15:56
liveapp STOPPED Not started
liveapp_dev STOPPED Not started
watchdog_dev STOPPED Not started
webapp STOPPED Not started
webapp_dev STOPPED Not started

cryosparcm log webapp is too long to post here but I don’t see any errors either.

Any suggestions how to overcome this issue would be greatly appreciated.

My best,
Andrija

derlog · December 17, 2020, 8:16pm

Hi,

I have the same problem, except that I was upgrading from 3.0 to 3.0.1. The update process stopped halfway exactly as when I upgraded from 2.15 to 3.0 and I had to use the tip provided at that time to continue the installation:

This trick works again this time. However the web app does not respond. “cryosparcm restart” gives me:

CryoSPARC is running.
Stopping cryoSPARC 
command_core: stopped
command_rtp: stopped
command_vis: stopped
database: stopped
Shut down
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
command_core: started
command_core connection succeeded
command_vis: started
command_rtp: started
command_rtp connection succeeded

but the web app still doesn’t respond. After reading the above post I checked “cryosparcm status” and it looks identical to the above post apart from the PIDs of course. I even restarted the whole computer but that didn’t help. Any ideas?

Derek

stephan · December 17, 2020, 8:27pm

Hi @derlog, @asente,

Is it possible if you can replace the req() function inside cryosparc2_master/bin/minicurl.py with the following:

def req(method, url, timeout=3):
    verify = True
    if (os.environ.get('CRYOSPARC_INSECURE', 'false') == 'true'):
        verify=False

    if method == 'POST':
        return requests.post(url, timeout=timeout, verify = verify)
    elif method == 'PUT':
        return requests.put(url, timeout=timeout, verify = verify)
    elif method == 'DELETE':
        return requests.delete(url, timeout=timeout, verify = verify)
    else:
        return requests.get(url, timeout=timeout, verify = verify)

Save the file, then restart cryoSPARC cryosparcm restart and let me know if that works

derlog · December 17, 2020, 8:55pm

stephan:

def req(method, url, timeout=3):
    verify = True
    if (os.environ.get('CRYOSPARC_INSECURE', 'false') == 'true'):
        verify=False

    if method == 'POST':
        return requests.post(url, timeout=timeout, verify = verify)
    elif method == 'PUT':
        return requests.put(url, timeout=timeout, verify = verify)
    elif method == 'DELETE':
        return requests.delete(url, timeout=timeout, verify = verify)
    else:
        return requests.get(url, timeout=timeout, verify = verify)

Hi Stephan,

Thanks for the rapid feedback. Unfortunately that didn’t work. I get the same messages on restart and the same 4 processes running upon checking the status.

Let me know if I can provide any more diagnostic information.

/Derek

asente · December 17, 2020, 9:03pm

Hi @stephan,

Many thanks for such a quick reply. I can confirm it didn’t work for me either.

My best,
Andrija

stephan · December 17, 2020, 9:05pm

Hey @derlog, @asente,

Thanks for trying that out. If cryoSPARC is mission critical, I suggest updating to cryoSPARC v3.0.0, since this bug is one that’s been introduced in cryoSPARC v3.0.1. You can do so by running the command cryosparcm update --version=v3.0.0

Is it possible if you can also send the output of cryosparcm log command_rtp?

derlog · December 17, 2020, 9:09pm

Hi @stephan,

I will certainly go back to 3.0.0. How would you like me/us to send the output of cryosparcm log command_rtp ? It’s quite long… Or should I just send the last few lines that look like they are from the most recent restart?

/Derek

stephan · December 17, 2020, 9:10pm

Hi @derlog,

If you can, feel free to copy+paste the messages that were added to the log right after the restart.

derlog · December 17, 2020, 9:13pm

OK, here goes:

2020-12-17 22:12:24,196     RTP                   INFO      === STARTED === 
2020-12-17 22:12:24,196     RTP.BACKGROUND_WORKER INFO      === STARTED === 
 * Serving Flask app "rtp_manager" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
2020-12-17 22:12:24,331     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S1
2020-12-17 22:12:25,822     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S2
2020-12-17 22:12:25,986     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S3
2020-12-17 22:12:27,322     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S1
2020-12-17 22:12:28,499     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S2
2020-12-17 22:12:28,503     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S1
2020-12-17 22:12:31,389     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S1
2020-12-17 22:12:32,840     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S2
2020-12-17 22:12:32,845     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S1
2020-12-17 22:12:33,005     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S3
2020-12-17 22:12:33,005     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S2
2020-12-17 22:12:34,258     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S1
2020-12-17 22:12:34,258     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S3
2020-12-17 22:12:35,435     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S2
2020-12-17 22:12:36,683     RTP                   INFO     CREATE LIVE SESSION JOB FOR SESSION S1

stephan · December 17, 2020, 9:22pm

Hi @derlog,

Thanks for supplying this info. By any chance, do you have CRYOSPARC_INSECURE set to true in cryosparc2_master/config.sh?

derlog · December 17, 2020, 9:23pm

Hi @stephan,

No, it’s set to false:

export CRYOSPARC_INSECURE=false

In general I tend not to mess with any parameters

/Derek

stephan · December 17, 2020, 9:38pm

Hi @derlog,

Can you try the following:
Run the line: export DEBUG=true and restart cryoSPARC. Paste any outputs you receive here.

derlog · December 17, 2020, 9:41pm

CryoSPARC is running.
Stopping cryoSPARC 
command_core: stopped
command_rtp: stopped
command_vis: stopped
database: stopped
Shut down
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
command_core: started
command_core connection succeeded

command_vis: started
command_rtp: started
Attempt 1/3 to GET http://cryoem-desktop:39005 failed with exception: HTTPConnectionPool(host='cryoem-desktop', port=39005): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3b61532b10>: Failed to establish a new connection: [Errno 111] Connection refused'))
Retrying...
Attempt 2/3 to GET http://cryoem-desktop:39005 failed with exception: HTTPConnectionPool(host='cryoem-desktop', port=39005): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3b614c8890>: Failed to establish a new connection: [Errno 111] Connection refused'))
Retrying...
Attempt 3/3 to GET http://cryoem-desktop:39005 failed with exception: HTTPConnectionPool(host='cryoem-desktop', port=39005): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3b61502f50>: Failed to establish a new connection: [Errno 111] Connection refused'))
Failed to GET http://cryoem-desktop:39005
Attempt 1/3 to GET http://cryoem-desktop:39005 failed with exception: HTTPConnectionPool(host='cryoem-desktop', port=39005): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f54c48b5a50>: Failed to establish a new connection: [Errno 111] Connection refused'))
Retrying...
Attempt 2/3 to GET http://cryoem-desktop:39005 failed with exception: HTTPConnectionPool(host='cryoem-desktop', port=39005): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f54c484e810>: Failed to establish a new connection: [Errno 111] Connection refused'))
Retrying...
Attempt 3/3 to GET http://cryoem-desktop:39005 failed with exception: HTTPConnectionPool(host='cryoem-desktop', port=39005): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f54c4885e90>: Failed to establish a new connection: [Errno 111] Connection refused'))
Failed to GET http://cryoem-desktop:39005
Attempt 1/3 to GET http://cryoem-desktop:39005 failed with exception: HTTPConnectionPool(host='cryoem-desktop', port=39005): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8c1d41bb50>: Failed to establish a new connection: [Errno 111] Connection refused'))
Retrying...
Attempt 2/3 to GET http://cryoem-desktop:39005 failed with exception: HTTPConnectionPool(host='cryoem-desktop', port=39005): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8c1d3b3890>: Failed to establish a new connection: [Errno 111] Connection refused'))
Retrying...
Attempt 3/3 to GET http://cryoem-desktop:39005 failed with exception: HTTPConnectionPool(host='cryoem-desktop', port=39005): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8c1d3edfd0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Failed to GET http://cryoem-desktop:39005
command_rtp connection succeeded
Attempt 1/3 to GET http://cryoem-desktop:39005/startup failed with exception: HTTPConnectionPool(host='cryoem-desktop', port=39005): Read timed out. (read timeout=3)
Retrying...
Attempt 2/3 to GET http://cryoem-desktop:39005/startup failed with exception: HTTPConnectionPool(host='cryoem-desktop', port=39005): Read timed out. (read timeout=3)
Retrying...
Attempt 3/3 to GET http://cryoem-desktop:39005/startup failed with exception: HTTPConnectionPool(host='cryoem-desktop', port=39005): Read timed out. (read timeout=3)
Failed to GET http://cryoem-desktop:39005/startup

derlog · December 17, 2020, 10:02pm

Hi @stephan,

It’s 11pm here so I will need to pick this up again tomorrow morning. Thanks for your help so far! Can’t work out why it’s trying to use port 39005 - I can’t see that in any config file.

/Derek

stephan · December 18, 2020, 2:13am

Hi @derlog,

Thanks for all your help! I believe I found the solution to this issue, based on your error message. It looks like the timeout we’ve set for this request is too short; it should be increased to 120 seconds.

To do so, edit the file cryosparc2_master/bin/cryosparcm
In the start) function (around line 699), you’ll find a line that looks like this:

python -m bin.minicurl http://$CRYOSPARC_MASTER_HOSTNAME:$CRYOSPARC_COMMAND_RTP_PORT/startup

Append the following to the line: --timeout 120

Which means the line should look like:

python -m bin.minicurl http://$CRYOSPARC_MASTER_HOSTNAME:$CRYOSPARC_COMMAND_RTP_PORT/startup --timeout 120

Save the file and restart cryoSPARC.

If all goes well, cryoSPARC should start up normally (you might still see Max retries exceeded errors, but this is normal). Port 39005 is the port used for command_rtp, which the start function is querying to make sure it has successfully come online before continuing. You can suppress these messages by running unset DEBUG in your shell (unless you’ve already logged out and logged back in).

derlog · December 18, 2020, 5:36am

Hi @stephan,

This works! I get 9 error messages of the type:

Attempt 1/3 to GET http://cryoem-desktop:39005 failed with exception: HTTPConnectionPool(host='cryoem-desktop', port=39005): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f15eb6af150>: Failed to establish a new connection: [Errno 111] Connection refused'))
Failed to GET http://cryoem-desktop:39005
command_rtp connection succeeded

but in the end it works and the webapp starts up. Thanks for sorting this out so quickly. How specific to a given system is this problem?

/Derek

biocit · December 18, 2020, 2:13pm

Same issue here when updating one instance from 3.0 to 3.0.1. Adding – timeout 120 seemed to solve the webapp startup problem

stephan · December 18, 2020, 5:09pm

Hi @derlog, @biocit,

Thank you both for the update. It looks like this happens when you have a lot of cryoSPARC Live sessions, since the startup function we are calling in the start) command loops through every Live session to create a “Live Session” job. The original timeout was only 3 seconds, which might not be enough on slower networks. We’ve re-build the v3.0.1 package to include this fix, so any users updating to this version from this point on will have it.

biocit · December 18, 2020, 5:15pm

Hi @stephan
yep, confirm. The problems showed up on the one (out of 3) systems where users already had quite a lot of sessions. Update/upgrade on the other 2 systems went smoothly.

derlog · December 20, 2020, 8:52am

Hi @stephan, @biocit,

FYI I can say that we should in principle have zero cryoSPARC Live sessions because we have never used it I had tried to access it a few times after the 3.0.0 update but never succeeded in connecting to the server. I later discovered that this was because it’s only available on the local network via localhost:39006 and not via the external IP.

Don’t know if that helps but I wanted to put the information out there. This is a standalone cryoSPARC installation on Ubuntu Linux.

/Derek