Now I’m stuck with another issue. The previous approach fixed the CS boot up. But when I try to run some job on GPU it gets stuck at ‘launched’ state forever. CS is recognizing the GPUs but I’m not sure why jobs are not starting.
However, my import movie job worked fine with CPUs. Although, extract from micrograph job not working on CPU.
$ cryosparcm restart
CryoSPARC is running.
Stopping cryoSPARC
app: stopped
app_api: stopped
command_core: stopped
command_rtp: stopped
command_vis: stopped
database: stopped
Shut down
Starting cryoSPARC System master process..
CryoSPARC is not already running.
configuring database
configuration complete
database: started
checkdb success
command_core: started
command_core connection succeeded
command_core startup successful
command_vis: started
command_rtp: started
command_rtp connection succeeded
command_rtp startup successful
app: started
app_api: started
-----------------------------------------------------
I got this when I tried to test install:
Running installation tests...
✓ Running as cryoSPARC owner
✓ Running on master node
✓ CryoSPARC is running
✓ Connected to command_core at http://gpu-02.ucsf.edu:61002
✓ CRYOSPARC_LICENSE_ID environment variable is set
✓ License has correct format
✓ Insecure mode is disabled
✓ License server set to "https://get.cryosparc.com"
✓ Connection to license server succeeded
✓ License server returned success status code 200
✓ License server returned valid JSON response
✓ License exists and is valid
✓ CryoSPARC is running v4.1.0
✓ Running the latest version of CryoSPARC
Could not get latest patch (status code 404)
✓ Patch update not required
✓ Admin user has been created
✓ GPU worker connected.
As I’m not sure what’s happening. I thought to try re-connecting worker to CS (thought might be the reason why jobs are not running?!). Here’s what I got while trying:
$ ./bin/cryosparcw connect --worker gpu-02.ucsf.edu --master gpu-02.ucsf.edu --port 61000 --ssdpath /data/
---------------------------------------------------------------
CRYOSPARC CONNECT --------------------------------------------
---------------------------------------------------------------
Attempting to register worker gpu-02.ucsf.edu to command gpu-02.ucsf.edu:61002
Connecting as unix user abhisek
Will register using ssh string: abhisek@gpu-02.ucsf.edu
If this is incorrect, you should re-run this command with the flag --sshstr <ssh string>
---------------------------------------------------------------
Traceback (most recent call last):
File "bin/connect.py", line 76, in <module>
assert cli.test_connection(), "Could not connect to cryoSPARC master at %s:%d" % (master_hostname, command_core_port)
File "/home/emPackages/CS_installation_06152022/cryosparc_worker/cryosparc_compute/client.py", line 65, in func
assert 'error' not in res, f"Encountered error for method \"{key}\" with params {params}:\n{res['error']['message'] if 'message' in res['error'] else res['error']}"
AssertionError: Encountered error for method "test_connection" with params ():
ServerError: Authentication failed - License-ID request header missing.
This may indicate that cryosparc_worker did not update,
cryosparc_worker/config.sh is missing a CRYOSPARC_LICENSE_ID entry,
or CRYOSPARC_LICENSE_ID is not present in your environment.
See https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#command-api-security for more details.
Here’s the content of config.sh (from worker):
export CRYOSPARC_LICENSE_ID="xx"
export CRYOSPARC_USE_GPU=true
export CRYOSPARC_CUDA_PATH="/usr/local/cuda-11.6"
export CRYOSPARC_DEVELOP=false
Please suggest how to fix this issue (I’m assuming jobs are not starting because of that ?!).
Thanks