Improvements for clusters

open

#1

Hi

I would like to request some features that will make our life easier on clusters.

I like the lane approach that we can have multiple submission scripts as templates. However I think it will be useful to also include some variables that can be passed through GUI to the submission script. For example like in relion, if you can have a XXXextra1XXX variable (actually multiple of them) that can be replaced with any parameter user wants, this variable can be entered during submit to lane process.

Second request is more complicated.
I am using ssh tunnel to interact with GUI. The cluster that I am using at Stanford University offers another browser based method to reach the compute nodes (OnDemand). I would like to use that method for several reasons. However, web browser cannot communicate properly with the GUI. The URL to reach the node is as follows

https:// login sherlock stanford.edu/node/“host-name”/“port-name”/

As I consulted with the cluster admin, the issue is that CryoSPARC tries to load a Javascript file from the root of the webserver: /bee123456789123456789123456789aa.js?meteor_js_resource=true

It works fine with a SSH tunnel, but with OnDemand, it tries to load:

https:// login sherlock stanford.edu/bee123456789123456789123456789a.js?meteor_js_resource=true"

where the file is actually at:

https:// login sherlock stanford.edu/node/<nodename>/<port>/bee123456789123456789123456789aa.js?meteor_js_resource=true

Therefore I see a blank page.

If CryoSPARC does not use absolute paths when loading resources, but relative URLs, this problem should be resolved.

Can you implement these?

Thank you


#2

Hi, as for your second request, can’t use solve it by connecting with VPN rather than using SSH tunnel?


#3

Thank you Marino for your reply.

Standord cluster, probably many other clusters has a saperate 2 tier authentication system. Even on campus we need to tunnel and authenticate. Their web based system allows a computer to stay authenticated for 30days, which cuts down a lot of smartphone interaction. But for that, the javascript on cryosparc web app should load resources properly.
Is not it anyway better to use relative paths for web based applications? (The java script when we try to load cryosparc on web browser asking for an absolute path)

Alpay


#4

More importantly, is it possible to install and run cryosparc host-name free? The nodes on our cluster are stateless, and I would like to be able to run the background instance in any node I want. Many people cannot use cryosparc on our cluster because they do not own a node and they cannot practically request the same node.


#5

Hi @alburse,

Our team is looking into this and will update the thread when we have more information.

Thanks,
Suhail


#6

@alburse We’ve solved this by allowing users to run the master process on the login node. Running on a compute node gives the problems you mentioned with having to update the config.sh file for the master. Plus, running on a compute node results in a lot of wasted cycles since the master process is pretty lightweight and all the serious computation gets submitted through the queue.

The downside is that users do have to choose a non-conflicting port number, and this needs to be accessible from the login node. This is a pretty serious organizational hassle for us and I’d be interested in hearing about other options.


#7

@sbliven Can you share with us how do your run the master process on a login node?

I think running the master process on a compute node is not a big issue. We do that for many other programs. However, running it on a specific compute node is a big issue (you should just get rid of host name if possible). I guess if you can run it on a login node, you can also run it on any node so the problem is I believe resolved (no need to run the master on a specific compute node) Keeping one CPU from a compute node is not a big deal. Accordingly there will be no problem for conflicting port number and its accessibility by running the master on a slurm/pbs/… assigned node. Are these 2 assumptions (can run it on any node and no port problem if run on a compute node) correct? By the way running cryosparc master on a login node may not be a good idea. Login nodes have several restrictions. Therefore cryosparc instance will more likely be terminated and this may cause database issues. Some of the jobs on cryosparc also runs directly on master, those processes are probably too much for login nodes.

Would you also comment on my question about the “javascript on the browser interface of cryosparc is asking for an absolute path” on my first message on this thread.

Thank you.

Alpay


#8

The absolute path to javascript seems like a bug, but the developers would have to address that.

I think that eliminating CRYOSPARC_MASTER_HOSTNAME from config.sh would be a good idea, although it does place additional burden on the users to figure out which node the master is running on (e.g. for running cryosparcm commands, connecting to the webserver, etc). Probably this is a nontrivial feature request, since it would require the master to somehow inform clients as to the correct callback host.

Running the master on a compute node still has the posibility for conflicting ports if two users happen to get assigned cores on the same machine. However unless you have many cryosparc users on the cluster that might be low probability.

Unclean terminations can be a problem on both login nodes and compute nodes. Either way one should call cryosparcm stop rather than killing the process. We have relatively permissive login node policies, so unexpected terminations are more likely on the compute nodes (from preemption or expired allocations).


#9

Great, I hope you can fix the javascript path bug and also share with us how to run cryosparc on login nodes soon.
Thank you

Alpay