CPU slot allocation overlap across parallel jobs

Install, Configure and Update
1

CPU slot allocation overlap across parallel jobs on the same worker in CryoSPARC 5.0.4

Hi everyone,

I am seeing a CPU scheduling/resource allocation issue after upgrading to CryoSPARC 5.0.4 and would appreciate any insight.

all the Workers were recently disconnected and reconnected after upgrading to 5.0.4.

Problem

On a given Worker node, when I launch multiple jobs in parallel:
multiple jobs are being assigned the same CPU slot IDs , even though many CPUs remain free.

For example:

  • Job 1 → CPU [0,1,2,3], GPU [0]
  • Job 2 → CPU [0,1,2,3], GPU [1]
  • Job 3 → CPU [0,1,2,3], GPU [2]

instead of expected behavior like:

  • Job 1 → CPU [0,1,2,3]
  • Job 2 → CPU [4,5,6,7]
  • Job 3 → CPU [8,9,10,11]

Notes

  • This did not happen in previous CryoSPARC versions .
  • Workers reconnect successfully and report correct resources.
  • Scheduler launches jobs correctly onto the worker.
  • The issue seems specific to CPU slot assignment rather than GPU/RAM scheduling.
  • Worker registration shows correct CPU count and GPUs.

Question

Is this an expected behavior, a scheduler change, or possibly a regression related to CPU affinity/resource allocation?

Any suggestions for additional diagnostics would be very helpful.

Thanks.

3

Thanks for your post @bharat .

The indices in the list do not signify allocation of a specific CPU on the node’s motherboard. This allocation is handled by the operating environment outside CryoSPARC.
Moreover, in the case of connected worker nodes (as opposed to clusters), CPU limitations are not enforced by the CryoSPARC built-in scheduler. In other words, a job could start even if other running CryoSPARC jobs were scheduled to use all installed CPUs.
In contrast, an external cluster workload manager, like slurm, may be used with CryoSPARC and configured to enforce CPU limits.