Hi!
I’m just posting here because I started having issues with submitting jobs on a SLURM scheduler shortly after updating to v3.3. In the last couple days I started getting this error when I try to submit submit jobs that indicate there is an issue allocating memory:
ServerError: Traceback (most recent call last):
File "/usr/local/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 150, in wrapper
res = func(*args, **kwargs)
File "/usr/local/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 2309, in run_job
res = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).decode()
File "/usr/local/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/usr/local/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/subprocess.py", line 488, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/local/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "/usr/local/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/subprocess.py", line 1482, in _execute_child
restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 150, in wrapper
res = func(*args, **kwargs)
File "/usr/local/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 1861, in scheduler_run
scheduler_run_core(do_run)
File "/usr/local/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 2079, in scheduler_run_core
run_job(job['project_uid'], job['uid']) # takes care of the cluster case and the node case
File "/usr/local/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 157, in wrapper
raise ServerError(s.getvalue(), code=400) from e flask_jsonrpc.exceptions.ServerError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 150, in wrapper
res = func(*args, **kwargs)
File "/usr/local/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5121, in enqueue_job
scheduler_run()
File "/usr/local/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 157, in wrapper
raise ServerError(s.getvalue(), code=400) from e flask_jsonrpc.exceptions.ServerError
I’m trying to decipher what might be going on and why CS jobs are not getting submitted so I can work with the HPC manager to get jobs working again.
Thank you,
Russell McFarland