Change ROOT_URL for Meteor Webapp

kmdalton · December 16, 2021, 6:15pm

It would be very helpful if there was a way to change the ROOT_URL in the CryoSPARC webapp. This would allow CryoSPARC to be easily served with Open OnDemand in an HPC setting. Right now all the Open OnDemand solutions rely on a NoVNC connection (AFAIK). This means users interact with CryoSPARC inside a web-browser inside of a virtual desktop inside of their local web-browser. It’s not a great user experience. However, if it were possible to change the ROOT_URL of the CryoSPARC webapp, it could be served directly to users’ local browser tab seamlessly through Open OnDemand.

It seems to me like the only thing preventing this from working is the fact that there are multitude of hardcoded paths in the webapp. There are several hundred instances of paths which begin with a “/” in the top-level minified javascript:

[user@computer web.browser]$ grep -oh -e '\"\/[^\"]*\"'  50592286c5f5542d035fd46578678e310bdddf72.js |wc -l
598

Is it possible to change this? It would be really stellar if users could define a ROOT_URL environment variable in the config.sh for cryosparcm. I will note that ROOT_URL is already set in superviorsd.conf

[program:webapp]
command=%(ENV_CRYOSPARC_ROOT_DIR)s/cryosparc_webapp/nodejs/bin/node ./bundle/main.js
directory=%(ENV_CRYOSPARC_ROOT_DIR)s/cryosparc_webapp
autostart=false
autorestart=true
redirect_stderr=true
stdout_logfile=%(ENV_CRYOSPARC_ROOT_DIR)s/run/webapp.log
environment=MONGO_URL="mongodb://%(ENV_CRYOSPARC_MASTER_HOSTNAME)s:%(ENV_CRYOSPARC_MONGO_PORT)s/meteor",MONGO_OPLOG_URL="mongodb://%(ENV_CRYOSPARC_MASTER_HOSTNAME)s:%(ENV_CRYOSPARC_MONGO_PORT)s/local",PORT="%(ENV_CRYOSPARC_HTTP_PORT)s",HTTP_FORWARDED_COUNT="1",ROOT_URL="http://%(ENV_CRYOSPARC_MASTER_HOSTNAME)s:%(ENV_CRYOSPARC_HTTP_PORT)s",NODE_OPTIONS="--max-old-space-size=8192"

However, when I changed it, the sub-URI did not propagate to all the paths in the webapp.

sdawood · December 17, 2021, 4:05pm

Hi @kmdalton,

Thanks for the detailed post, it is very much appreciated. We will look into adding a configurable parameter within config.sh that modifies the ROOT_URL environment variable passed to the Meteor application.

- Suhail

kmdalton · December 17, 2021, 8:55pm

Hi @sdawood,

Many thanks for getting back to me! If it helps convince your team, I would certainly be willing to open source an Open OnDemand app for CryoSPARC. I just need the ROOT_URL support, and I can put something together. I think this would really help a lot of academic users and sysadmins.

Cheers,
Kevin

kmdalton · January 14, 2022, 8:33pm

Hi @sdawood,

Any word from the devs on this? I think installing CryoSPARC without dedicated hardware has stumped a lot of academic sysadmins. This is a possible solution which would certainly work for us at Harvard. My read was that it shouldn’t take much work on your end. Hope I’m right about that!

Thanks,
Kevin

sdawood · January 27, 2022, 2:23pm

Hi @kmdalton ,

Thanks for the ping. I took another look at our code and there shouldn’t be a reason why hosting at a subpath with ROOT_URL changed within cryosparc2_package/cryosparc_master/supervisord.conf should not work. Could you try again and make sure to first stop the cryoSPARC instance? cryosparcm stop

If possible could you let me know if the network requests of the web application are pointing to the right URL (with the subpath)?

kmdalton · January 28, 2022, 4:29am

@sdawood ,

Firstly, thanks for having a look. My colleagues and I strongly appreciate any help you can give us. I am not a web developer, but I will try my best to be helpful.

For starters, I edited the supervisord.conf as follows:

$ diff supervisord.conf original_supervisord.conf
43c43
< environment=MONGO_URL="mongodb://%(ENV_CRYOSPARC_MASTER_HOSTNAME)s:%(ENV_CRYOSPARC_MONGO_PORT)s/meteor",MONGO_OPLOG_URL="mongodb://%(ENV_CRYOSPARC_MASTER_HOSTNAME)s:%(ENV_CRYOSPARC_MONGO_PORT)s/local",PORT="%(ENV_CRYOSPARC_HTTP_PORT)s",HTTP_FORWARDED_COUNT="1",ROOT_URL="http://%(ENV_CRYOSPARC_MASTER_HOSTNAME)s:%(ENV_CRYOSPARC_HTTP_PORT)s/node/%(ENV_CRYOSPARC_MASTER_HOSTNAME)s/%(ENV_CRYOSPARC_HTTP_PORT)s",NODE_OPTIONS="--max-old-space-size=8192"
---
> environment=MONGO_URL="mongodb://%(ENV_CRYOSPARC_MASTER_HOSTNAME)s:%(ENV_CRYOSPARC_MONGO_PORT)s/meteor",MONGO_OPLOG_URL="mongodb://%(ENV_CRYOSPARC_MASTER_HOSTNAME)s:%(ENV_CRYOSPARC_MONGO_PORT)s/local",PORT="%(ENV_CRYOSPARC_HTTP_PORT)s",HTTP_FORWARDED_COUNT="1",ROOT_URL="http://%(ENV_CRYOSPARC_MASTER_HOSTNAME)s:%(ENV_CRYOSPARC_HTTP_PORT)s",NODE_OPTIONS="--max-old-space-size=8192"

This leads to the correctly formatted ROOT_URL for Open OnDemand https://holy7c24101.rc.fas.harvard.edu:9627/node/holy7c24101.rc.fas.harvard.edu/9627. All good so far.

When I try to acess the webapp, I use the Open OnDemand URL https://vdi.rc.fas.harvard.edu/node/holy7c24103.rc.fas.harvard.edu/7565/login which should pass through to the compute node where CryoSPARC is running. The DNS does resolve, and I make it through to the compute node. This is great by the way – it took longer than I care to admit to reach this point. Unfortunately, the webapp fails to load. A progress bar appears for just a moment, but then I quickly get the following page:

It also redirects me to https://vdi.rc.fas.harvard.edu/404, but it is still clearly using CryoSPARC’s css (or whatever other way you have of specifying the style).

In the console, I see a whole bunch of scary errors:

These appear to mostly be to do with fonts and also a favicon which the web server doesn’t know where to find. I do believe that the webapp is using the right ROOT_URL, because my browser is able to find the minified javascript file. However, it gets stuck when trying to find some subset of resources mentioned in the file. I think this is because these are specified with absolute paths in the webapp source. Changing the ROOT_URL environment variable doesn’t have any effect on them. If I were more savvy, perhaps I could hunt down the absolute paths and switch them to relative ones, but I am afraid I don’t know how the 50592286c5f5542d035fd46578678e310bdddf72.js gets generated exactly. I don’t think I can work on the minified file directly. It’s too confusing.

I could be off base about the root cause of these errors. It does feel awfully close to working though.

I’m not 100% sure if I have answered your question. I’m afraid my js and overall knowledge of web technologies is pretty minimal, but I am happy to try running diagnostics for you. You’ll just have to make pretty specific suggestions.

Thanks,
Kevin

kmdalton · February 7, 2022, 11:08pm

Any more thoughts on this @sdawood? It seems to me like some absolute paths are just hardcoded inside the webapp source. I’d think it would be an easy fix for someone familiar with the code. Am I off base?

Cheers,
Kevin

sdawood · February 9, 2022, 3:17pm

Hi @kmdalton,

Thanks very much for the additional details. It looks like the application does load, it’s just that the client-side router operates under the assumption of a root base path. When our team has some time we can test out a modified build that supports ROOT_URL. I’ll keep you updated!

- Suhail

kmdalton · February 18, 2022, 4:12pm

Hey @sdawood ,

Just wanted to ping you to remind you about this ROOT_URL issue. Any progress to report? Anything I can do to help?

Thanks!
Kevin

kmdalton · February 28, 2022, 4:08pm

Hi @sdawood,

Any idea when this ROOT_URL issue will get a fix? There are several labs interested in running CryoSPARC on our cluster. It’ll be much easier to help them if I can give them an OnDemand App. I’d prefer to wait if that’s something that will happen relatively soon.

It does seem like the intention was to support changing ROOT_URL based on CryoSPARC’s supervisord.conf. From that perspective, I’d say this is a bug worth fixing!

Thanks,
Kevin

kmdalton · March 8, 2022, 9:34pm

@team

Friendly reminder about this ROOT_URL issue. I’d really appreciate it if you could take a few moments to write an update!

Thanks again,
Kevin

kmdalton · April 2, 2022, 6:29pm

Hi @sdawood,

Just checking in. Will this issue be patched soon? I saw a recent release, but I don’t think a fix was in it.

Thanks,
Kevin

vallerul · April 4, 2022, 7:44pm

@sdawood : I upvote this request to enable meteor app with a custom ROOT_URL.
It has been a pain to manage cryosparc with multiple users on different machines.
Open on demand would help our users make more use of Cryosparc in a clean way.

Can we please take this request as a priority and see if it can be fixed?

sdawood · April 13, 2022, 4:22pm

Hi all,

Our team looked into a workaround and have determined it’s not technically feasible for us to work on, at least for this version of the cryoSPARC application.

- Suhail

kmdalton · April 13, 2022, 5:28pm

@sdawood

I really appreciate your looking into this. I strongly encourage the team to prioritize this feature in future versions of the webapp. I cannot overstate how much it would benefit academic cluster admins.

Thanks,
Kevin

johrstrom · May 2, 2022, 5:34pm

@kmdalton do you have a github repo you can share? You may be able to use /rnode (as a post to /node) for relative URLS. This allows for origins that can’t change their ROOT_URL like cryosparc and RStudio to name another off the top of my head.

@sdawood I’m a maintainer of Open OnDemand in case you need to toss any questions my way (I noticed you had a few so thought I’d hop on).

vallerul · May 2, 2022, 7:22pm

@johrstrom : I had tried with /rnode and it still was not working as expected.

kmdalton · May 2, 2022, 7:32pm

@johrstrom, thanks for chiming in! I have also tried rnode to no avail. I can follow up with more details later today.

kmdalton · May 2, 2022, 7:55pm

@johrstrom, here’s the OOD Webapp I’ve been testing. It is based on the Jupyter example. Sorry it is a bit haphazard. I more or less abandoned it after realizing that it wasn’t going to be possible to change ROOT_URL.

When I try specifying rnode in the view.html.erb, I am still not able to connect to the webapp. I just get a blank page, and the console complains that some files are missing.

I can confirm that the CryoSPARC webapp is running, and I can connect from a virtual desktop running on a different node:

Perhaps rnode is not configured correctly? I really appreciate any insight you can offer!

kmdalton · May 3, 2022, 9:07pm

Sorry, @johrstrom ! I just realized my repository link was broken. This one works.