Assign a new lane on cluster, Topaz GPU

Install, Configure and Update
1

Hi,

I am using HPC and I would like to know how to add a “new lane” via CLI so that I can use it in cryoSparc. It would of course be another worker lane. We have several GPU node types, featuring nvidia A100, V100, 2080 Ti and others. Since cryosparc supports ampere architecture of GPU, ideally A100 and V100 should work. Is there any measurable benefits of using A100/V100 vs 2080 Ti? What is Topaz particle picker preferred gpu amongst these 3?

I know it’s multiple questions bundled together but it is all referring to single issue.

Thank you very much.

Cheers,
Nebojsa Bogdanovic, PhD

2

Good question, Nebojsa. I can’t add a new lane via CLI in v3.2.0 of cryoSPARC:

$ cryosparcm icli "add_scheduler_lane(boguslane, cluster, title=None, desc='')"
Python 3.7.9 | packaged by conda-forge | (default, Dec  9 2020, 21:08:20)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.20.0 -- An enhanced Interactive Python. Type '?' for help.

 connecting to master2:61002 ...
 cli, rtp, db, gfs and tools ready to use

In [1]: add_scheduler_lane(boguslane, cluster, title=None, desc='')
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-1-0d81014c36f1> in <module>
----> 1 add_scheduler_lane(boguslane, cluster, title=None, desc='')

NameError: name 'add_scheduler_lane' is not defined

In [2]:

Note, desc=’’ has two single quotes (not a double quote), so the syntax looks correct to me.

I ended up editing the cluster_info.json file (from UGE template) to define a new lane name. Then, I did:

cryosparcm cluster connect

That created a new cluster lane (ugecluster) but I can’t add workers to that lane no matter what I try ( --lane option fails).

On the worker node:
$ ./cryosparcw connect --worker "gpu07" --master "master2" --ssdpath /tmp/cryosparc_cache --port 61000 --lane "ugecluster"
...
  Registering worker...
Traceback (most recent call last):
  File "bin/connect.py", line 277, in <module>
    monitor_port = monitor_port,
  File "/cryoSPARC3/cryosparc_worker/cryosparc_compute/client.py", line 62, in func
    assert False, res['error']
AssertionError: {'code': 500, 'data': None, 'message': 'OtherError: Can not add node type target to non-node type lane.', 'name': 'OtherError'}

By the way, ‘–newlane’ option to ‘cryosparcw connect’ is not recognizable by the executable, so, you can’t create a new lane that way either.

I can only add workers to a default lane, before running ‘cryosparcm cluster connect’

Sergio

3

@ponomarevsy Please see https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring/cli#add_scheduler_lane-name-lanetype-title-none-desc for the correct arguments

You can run the function directly on the command line by using cli:
cryosparcm cli "add_scheduler_lane('boguslane', 'clustername')"

If you want to use the interactive python shell (via cryosparcm icli), you need to prefix the command with cli:
In [1]: cli.add_scheduler_lane('boguslane', 'cluster')

Please also note you can’t mix positional and keyword arguments when calling the function this way. It has to be one or the other.

It seems like the lane you’re trying to add a workstation to is a cluster lane. You can only add workstations to non-cluster lanes.
Please see the following section for the correct syntax to connect and update workers:
https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc#update-a-managed-worker-configuration

4

Hi @spunjani,

Thank you for your response. I apologize for asking so many questions (please see below) but I still find this very confusing. Is 'cryosparcm cluster connect' command the same as 'cryosparcm cli "add_scheduler_lane('clustername', 'cluster')"' if clustername (lane) is defined inside the cluster_info.json file? How do I add nodes, not workstations, to an existing cluster lane? The more I do it, the more I get confused. Why are you taking about a workstation? I thought ./cryosparcw connect --worker ... connects a node to a cluster lane (with --lane option). I’ve added 3 GPU nodes via above but it is not queueing: instead, it connects to the 1st node directly, via, ssh, to run a job.

What does this mean: “Please also note you can’t mix positional and keyword arguments when calling the function this way. It has to be one or the other.” What is the difference b/w positional and keyword arguments and how do I use them? I am trying to add a node via 'cryosparcm cli "add_scheduler_target_node' but getting this error:

AssertionError: {'code': 500, 'data': None, 'message': "OtherError: add_scheduler_target_node() missing 7 required positional arguments: 'hostname', 'ssh_str', 'worker_bin_path', 'num_cpus', 'cuda_devs', 'ram_mb', and 'has_ssd'", 'name': 'OtherError'}

Would you have an example, please?

5

On second thought, perhaps, I don’t need to add nodes at all. All I need is to turn the cluster on via cryosparcm cluster connect and the scheduler should do the rest. So what is the point of adding workers, then, in cluster mode? Just kinda confusing…

6

Hi @ponomarevsy,

CryoSPARC uses “lanes” to distinguish resources. There are two types of lanes: workstations (i.e. nodes) or clusters. Workstation lanes are meant for resources where cryoSPARC directly queues jobs on the resources (e.g. it will ssh into the node, and execute the cryoSPARC job itself). You can think of this as a workstation that is at your desk (that has GPUs). Cluster lanes are meant for cluster scheduling systems, where cryoSPARC will submit a job to the cluster (e.g. SLURM, PBS) and the cluster will take responsibility for executing the cryoSPARC job. To add a workstation lane, you have to run the cryosparcw connect command from the workstation itself. To add a cluster lane, you have to create a cluster_info.json and cluster_script.sh and run the cryosparcm cluster connect job. Unlike cluster lanes, workstation lanes can have multiple nodes in them (e.g. four separate GPU workstations in the same lane). Take a look through our Hardware and System requirements to understand the overall topology I’m describing: https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements

To add or delete a cluster lane, see the commands here: https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc#load-the-scripts-and-register-the-integration
To add or delete a worker lane, see the commands here: https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc#connect-a-managed-worker-to-cryosparc

I hope this helps.

2 Likes
7

That DOES help, @stephan! Thanks for the detailed and thoughtful response (and for the links), - it supports my assumption that only cluster configuration will be useful in our case (no need to add workstations). Best regards.

1 Like