Lane usage panel wrong lanes

Hi devTeam,

we recently upgraded from v3.3.2 to v4.3 and I really like the new appearance.

In the home tab, there is the Lane Usage infographic. But this uses very old and obsolete lanes.
In our case, these lanes are plotted:
default imageprocessing cryoA5000 thebeast preprocess
which used to be our lanes before some merging.

This is our current lane configuration, reported by the icli:

In [3]: [n["title"] for n in cli.get_scheduler_lanes()]
Out[3]: ['The Shire', 'The Prancing Pony', 'Minas Tirith']

I get the feeling, I should just reinstall cryoSPARC completely, as we also have the memory issue (see the other thread).

Best
christian

1 Like

I wonder whether the

names are “surviving” elsewhere in your configuration.
Please can you post a screenshot of the Instance tab from the Manage interface (the button with the vertical sliders).
image

here is the Instance information. This is also immediately changed, if I alter names using the icli.

Maybe once can scan the mongo database for the old names, and find out, where they are still saved.

Hi @ctueting ,

Upon further investigation we can confirm there is no need to worry about any misconfiguration or inaccurate state with respect to the scheduler lanes.

The reason why this chart module is displaying ‘old’ lane names is due to the nature of where the data for the chart is sourced from: the CryoSPARC jobs themselves. When a job is launched, the name of the lane it is queued to is recorded. This value does not change after being set initially and is the reason for why a renamed lane could be displayed alongside the old lane name in this view.

Since the time scope of the chart is 7 days, once a week has gone by there should no longer be any reference to the old lane name in CryoSPARC.

Hope that clears things up.

- Suhail

I restarted the server yesterday, and actually the old lanes are not displayed anymore.

But also not the new lanes.


As you can see, there are jobs in the processing history, but the lane usage did not have any information.

Best
Christian

Hi @ctueting ,

For the completed jobs you expect to see in the lane usage chart, could you please inspect each one (select and click ‘view job’ or press spacebar) by searching for the field queued_to_lane in the metadata tab?

- Suhail

Here is an example of a ab initio job, I run on Monday:
grafik

Our cryoSPARC server has also some memory issues (see the other thread), maybe something is generally of with our installation.

I upgraded just last week from v3.3.2 to v4.3 and maybe some files mismatched. I can try a fresh installation to rule this out at some point.

There is still
No relevant data available
in the lane usage overview.

Do you need additional data to debug this issue? It’s not of a big deal, but it would be nice to see how much the workers are used.

Hi @ctueting ,

Apologies for the delayed response. After updating have you encountered this issue again? If so we can take a closer look. Upon initial investigation into the functionality that powers this chart it’s unclear why they would fail populating given the job seems to have correct information stored about what lane it was allocated to.

- Suhail

Hi Suhail,

sorry for the late response. I totally forgot about the issue.

But it’s still buggy. After updating back than, all lanes disappeared (No relevant data available), and now, after months of usage, we have the following:

But default, imageprocessing and preprocess are from pre-update (v3.3), and Shire/MinasTirith are from after the update. Additionally, two lanes are completely missing here (where I queued a lot of jobs last week).

I’m happy to supply you with any logs/information needed to debug this feature.

Best
Christian

Hi @ctueting ,

No problem, thanks for your reply. We will continue to investigate the issue on our end. In the meantime, could you let me know how many lanes in total you have configured in your CryoSPARC instance?

Regards,
Suhail

In total, 4 lanes.

Here is the configuration:

In [1]: cli.get_scheduler_lanes()
Out[1]:
[{'desc': "Within the Shire's green embrace, each digital task queues, shepherded by four hardworking GPUs, guided by a memory as rich as 256 GB of Bilbo's tales, and a scratch space vast as 3.5 TB of Sam's beloved taters.",
  'name': 'Shire',
  'title': 'The Shire',
  'type': 'node'},
 {'desc': "Buzzing with life like Bree's famed 'The Prancing Pony', this lane hosts tasks serviced by two jolly machines, each bearing a 2x2080Ti GPU with 12 GB VRAM, and guided by 128 GB RAM, reminiscent of the inn's many tales and characters, with a 3.5 TB scratch space as plentiful as the inn's storied supplies.",
  'name': 'PrancingPony',
  'title': 'The Prancing Pony',
  'type': 'node'},
 {'desc': "In the shadow of the towering 'White City', Minas Tirith, tasks gather strength from four formidable GPUs, each armed with 24GB VRAM, guided by a memory as vast as 512GB of Gondor's historical scrolls, and a scratch space as grand as 3.5TB mirroring the city's vast treasury.",
  'name': 'MinasTirith',
  'title': 'Minas Tirith',
  'type': 'node'},
 {'desc': 'A small lane with a single 1 GPU workstation for small jobs and training purposes',
  'name': 'training',
  'title': 'Training Lane',
  'type': 'node'}]
1 Like

Perfect, thank you @ctueting . Love the descriptions!!

1 Like