Change cryosparc hostname

Install, Configure and Update
1

I know this has been covered in the past, but none of the solutions in the forum seem to be working. We got a system from a vendor with CryoSPARC v4.4.1 already installed. It is a standalone system with master/worker combined. During my configuration of the system, I changed the system hostname and now CryoSPARC is not happy. When trying to run a job, the event log complains:

Launching job on lane default target sn4622xxxxxx
Running job on remote worker node hostname sn4622xxxxxx
Failed to launch! 255
ssh: Could not resolve hostname sn4622xxxxxx: Name or service not known

I changed the hostname in CryoSPARC master config.h to match the hostname from the command ‘hostname-f’:
export CRYOSPARC_MASTER_HOSTNAME="mynewhostname.domain.emory.edu"

‘host $(hostname -f)’ resolves properly.
NOTE: host $(hostname) does NOT resolve, but instead yields:
Host mynewhostname not found: 3(NXDOMAIN)

The output of cryosparcm cli "get_scheduler_targets() results in this:

[{'cache_path': '/scr/cryosparc_cache', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 25425608704, 'name': 'NVIDIA RTX A5000'}, {'id': 1, 'mem': 25425608704, 'name': 'NVIDIA RTX A5000'}, {'id': 2, 'mem': 25425608704, 'name': 'NVIDIA RTX A5000'}, {'id': 3, 'mem': 25425608704, 'name': 'NVIDIA RTX A5000'}, {'id': 4, 'mem': 25425608704, 'name': 'NVIDIA RTX A5000'}, {'id': 5, 'mem': 25425608704, 'name': 'NVIDIA RTX A5000'}, {'id': 6, 'mem': 25425608704, 'name': 'NVIDIA RTX A5000'}, {'id': 7, 'mem': 25425608704, 'name': 'NVIDIA RTX A5000'}], 'hostname': 'sn4622xxxxxx', 'lane': 'default', 'monitor_port': None, 'name': 'sn4622xxxxxx', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127], 'GPU': [0, 1, 2, 3, 4, 5, 6, 7], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192]}, 'ssh_str': 'cryosparc_user@sn4622xxxxxx', 'title': 'Worker node sn4622xxxxxx', 'type': 'node', 'worker_bin_path': '/home/cryosparc_user/software/cryosparc/cryosparc_worker/bin/cryosparcw'}]

Where “sn4622xxxxxx” is the original hostname that the vendor assigned to the system.

Based on prior topics about changing the hostname, I found this, but when I try to run the command:
cryosparcm "remove_scheduler_target_node('sn4622xxxxxx')"
It complains:
Unknown cryoSPARC command remove_scheduler_target_node('sn4622xxxxxx')

WHY is it saying that is an unknown command? HELP! :slight_smile:

2

Try insert cli like so:

cryosparcm cli "remove_scheduler_target_node('sn4622xxxxxx')"

This command removes the existing, outdated worker record. You will also need to add a new record based on the new hostname using a command like

/home/cryosparc_user/software/cryosparc/cryosparc_worker/bin/cryosparcw connect \
  --master mynewhostname.domain.emory.edu \
  --worker mynewhostname.domain.emory.edu \
  --ssdpath /scr/cryosparc_cache \
  --port <base_port_number>

The command assumes the combined master/worker. Please review the parameters carefully. Aside from adjusting the hostnames, find the relevant port number by running the command

cryosparcm status | grep "^export CRYOSPARC_BASE_PORT"
3

DOH! Forgot the ‘cli’. I was just blindly following your advice from this post :slight_smile: – New hostname and failed worker nodes – where it didn’t have the ‘cli’ in there! But I now see in THIS one Adding new hostname in user interface that you had it in there – I blame not enough coffee in thinking straight yet this morning! I think we’re good to go. I’ll check with the person that was trying to run the job. Mucho Thanks!

Yes, I think that solved THAT problem. But now there is another issue. I tried to run ‘cryosparcm test workers P1’, but that failed with “Error: [CUresult.CUDA_ERROR_LAUNCH_FAILED] Call to cuDevicePrimaryCtxRetain results in CUDA_ERROR_LAUNCH_FAILED”. I’ll open a new issue if I can’t figure that out.

4 
cryosparc_user@tulasi:~$ cd cryosparc_master/
cryosparc_user@tulasi:~/cryosparc_master$ bin/cryosparcm “remove_scheduler_target_node(‘tulasi.espnoncomp.wadsworth.org’)”
-bash: syntax error near unexpected token `('
cryosparc_user@tulasi:~/cryosparc_master$ bin/cryosparcm remove_scheduler_target_node(‘tulasi.espnoncomp.wadsworth.org’)
-bash: syntax error near unexpected token `('
cryosparc_user@tulasi:~/cryosparc_master$ bin/cryosparcm cli remove_scheduler_target_node(‘tulasi.espnoncomp.wadsworth.org’)
-bash: syntax error near unexpected token `('

Not quite working for me.

We upgraded the host from Suse to Ubuntu, ran cryosparc update AND we also moved the machine, which is both master and worker between networks, meaning we also changed the subdomain name.

I realize I did a lot of things at once, what needs to be done to unravel the knot?

thanks - Brian

cryosparc_user@tulasi:~/cryosparc_master$ bin/cryosparcm cli "remove_scheduler_target_node(‘tulasi.espnoncomp.wadsworth.org’)"
/home/cryosparc_user/cryosparc_master/cryosparc_tools/cryosparc/command.py:135: UserWarning: *** CommandClient: (http://tulasi.espnoncomp.wadsworth.org:39002/api) URL Error [Errno -2] Name or service not known, attempt 1 of 3. Retrying in 30 seconds
  system = self._get_callable("system.describe")()
5

You may want to check the following

  1. Is CRYOSPARC_MASTER_HOSTNAME configured correctly and matching the output of the command
    hostname -f?
    grep HOSTNAME /home/cryosparc_user/cryosparc_master/config.sh
  2. Is the hostname resolving the to the correct computer?
    host $(hostname -f)
  3. Is CryoSPARC running?
    cryosparcm status
    ?
  4. Is the command_core port accessible? In a fresh shell, run
    eval $(cryosparcm env)
curl ${CRYOSPARC_MASTER_HOSTNAME}:39002
exit
    ?
6

I got by the remote worker because I’m also the DNS guy, created a cname for tulasi.noncomp.wadsworth.org that resolves to tulasi.esplab.wadsworth.org

Thought the server started okay after that, but unable to connect the worker, not
finding the license.

However your question was awesome, hostname resolves to tulasi.svc.ny.gov because we are tied to the active directory domain.

This is not an issue with other AD bound Ubuntu clients… and checking one I see that /etc/hostname
is fully qualified, not just the hostname, will give that a try. That is not a Cryosparc issue but a platform config issue.

7

updated /etc/hosts, rebooted

cryosparc_user@tulasi:~$ hostname

tulasi.esplab.wadsworth.org
cryosparc_user@tulasi:~$ cd cryosparc_master/
cryosparc_user@tulasi:~/cryosparc_master$ bin/cryosparcm start
Starting CryoSPARC System master process...
CryoSPARC is not already running.
configuring database...
    configuration complete
database: started
database OK
command_core: started
    command_core connection succeeded

    command_core startup successful
command_vis: ERROR (spawn error)

– not sure why we have a spawn error.
–probably need to fix this before trying to connect the worker.

cryosparc_user@tulasi:~/cryosparc_master$
cryosparc_user@tulasi:~/cryosparc_master$ /home/cryosparc_user/cryosparc_worker/bin/cryosparcw connect --master tulasi.esplab.wadsworth.org --worker tulasi.esplab.wadsworth.org --ssdpath /tmp2 --port 39000 --update
 ---------------------------------------------------------------
  CRYOSPARC CONNECT --------------------------------------------
 ---------------------------------------------------------------
  Attempting to register worker tulasi.esplab.wadsworth.org to command tulasi.esplab.wadsworth.org:39002
  Connecting as unix user cryosparc_user
  Will register using ssh string: cryosparc_user@tulasi.esplab.wadsworth.org
  If this is incorrect, you should re-run this command with the flag --sshstr <ssh string>
 ---------------------------------------------------------------
Traceback (most recent call last):
  File "bin/connect.py", line 90, in <module>
    assert cli.test_connection(), "Could not connect to cryoSPARC master at %s:%d" % (master_hostname, command_core_port)
  File "/home/cryosparc_user/cryosparc_worker/cryosparc2_compute/client.py", line 57, in func
    assert False, res['error']
AssertionError: {u'message': u'ServerError: Authentication failed - License-ID request header missing.\n   This may indicate that cryosparc_worker did not update,\n   cryosparc_worker/config.sh is missing a CRYOSPARC_LICENSE_ID entry,\n   or CRYOSPARC_LICENSE_ID is not present in your environment.\n   See https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#command-api-security for more details.\n', u'code': 403, u'data': None, u'name': u'ServerError'}
cryosparc_user@tulasi:~/cryosparc_master$ hostname
tulasi.esplab.wadsworth.org


cryosparc_user@tulasi:~/cryosparc_master$ bin/cryosparcm env | grep LIC
export "CRYOSPARC_LICENSE_ID=XXXX"

cryosparc_user@tulasi:~/cryosparc_master$ ../cryosparc_worker/bin/cryosparcw env | grep LIC
export "CRYOSPARC_LICENSE_ID=XXXX"
8

updated croysparc_master/config.sh to fix hostname, still hadn the incorrect subdomain name.

Are there other places that needs to be corrected?

cryosparc_user@tulasi:~/cryosparc_master$ bin/cryosparcm env | grep host
cryosparc_user@tulasi:~/cryosparc_master$ bin/cryosparcm env | grep HOST
export “CRYOSPARC_MASTER_HOSTNAME=tulasi.esplab.wadsworth.org”
export “CRYOSPARC_HOSTNAME_CHECK=tulasi.esplab.wadsworth.org”
export “CRYOSPARC_FORCE_HOSTNAME=true”

However we seem to have the same spawn error when restarting cryosparc master.

9

These lines should not be needed for your specific use case, but increase the risk of incorrectly operating your CryoSPARC instance. Consider removing these lines from the cryosparc_master/configure.sh file and restarting CryoSPARC.

Please can you post the outputs of these commands:

ps -eo user:12,pid,ppid,start,command | grep -e cryosparc_master -e mongo
ls -l /tmp/mongo*.sock /tmp/cryosparc*.sock
cryosparcm log command_core | tail -n 20
cryosparcm log command_vis | tail -n 40
cryosparcm log supervisord | tail -n 40

Please can you also confirm that the cryosparc_worker/config.sh file includes a definition for CRYOSPARC_LICENSE_ID, and that the definition matches the CRYOSPARC_LICENSE_ID definition inside cryosparc_master/config.sh

10

Output from most commands below.

Yes, the CRYOSPARC_LICENSE_ID in the config.sh in both master and worker directories match.
Both are the same when viewed with cryosparcm and cryosparcw env commands.

cryosparc_user@tulasi:~$ ps -eo user:12,pid,ppid,start,command | grep -e cryosparc_master -e mongo
cryosparc_u+    4579       1 14:43:20 python /home/cryosparc_user/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/cryosparc_user/cryosparc_master/supervisord.conf
cryosparc_u+    4686    4579 14:43:26 mongod --auth --dbpath /home/cryosparc_user/cryosparc2_database --port 39001 --oplogSize 64 --replSet meteor --wiredTigerCacheSizeGB 4 --bind_ip_all
cryosparc_u+    4790    4579 14:43:31 python /home/cryosparc_user/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home/cryosparc_user/cryosparc_master/gunicorn.conf.py
cryosparc_u+    4791    4790 14:43:31 python /home/cryosparc_user/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home/cryosparc_user/cryosparc_master/gunicorn.conf.py
cryosparc_u+    5376    4148 15:12:32 grep -e cryosparc_master -e mongo


cryosparc_user@tulasi:~$ ls -l /tmp/mongo*.sock /tmp/cryosparc*.sock
srwx------ 1 cryosparc_user systemd-timesync 0 Dec 27 14:43 /tmp/cryosparc-supervisor-aac39d01f9caa09e5b897918f05baddb.sock
srwx------ 1 cryosparc_user systemd-timesync 0 Dec 27 14:43 /tmp/mongodb-39001.sock



cryosparc_user@tulasi:~$ pwd
/home/cryosparc_user
cryosparc_user@tulasi:~$ cd cryosparc_master/
cryosparc_user@tulasi:~/cryosparc_master$ bin/cryosparcm log command_core | tail -n 20
2024-12-27 14:43:38,091 log_scheduler_targets INFO     |           GPU 2 Name     TITAN X (Pascal)
2024-12-27 14:43:38,091 log_scheduler_targets INFO     |           GPU 2 Mem      11.90 GiB
2024-12-27 14:43:38,091 log_scheduler_targets INFO     |           GPU 3 Name     TITAN X (Pascal)
2024-12-27 14:43:38,091 log_scheduler_targets INFO     |           GPU 3 Mem      11.90 GiB
2024-12-27 14:43:38,091 log_scheduler_targets INFO     |
2024-12-27 14:43:38,166 startup              INFO     | CryoSPARC instance ID: d6cb39b3-6a0e-4195-99b6-81e9f1319143
2024-12-27 14:43:38,166 get_gpu_info         INFO     | UPDATING WORKER GPU INFO
2024-12-27 14:43:38,167 update_all_job_sizes INFO     | UPDATING ALL JOB SIZES IN 10s
2024-12-27 14:43:38,167 export_all_projects  INFO     | EXPORTING ALL PROJECTS IN 60s...
2024-12-27 14:43:38,298 get_gpu_info_run     ERROR    | Failed to get GPU info on tulasi
2024-12-27 14:43:38,298 get_gpu_info_run     ERROR    | Traceback (most recent call last):
2024-12-27 14:43:38,298 get_gpu_info_run     ERROR    |   File "/home/cryosparc_user/cryosparc_master/cryosparc_command/command_core/__init__.py", line 1520, in get_gpu_info_run
2024-12-27 14:43:38,298 get_gpu_info_run     ERROR    |     value = subprocess.check_output(full_command, stderr=subprocess.STDOUT, shell=shell, timeout=JOB_LAUNCH_TIMEOUT_SECONDS).decode()
2024-12-27 14:43:38,298 get_gpu_info_run     ERROR    |   File "/home/cryosparc_user/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py", line 421, in check_output
2024-12-27 14:43:38,298 get_gpu_info_run     ERROR    |     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2024-12-27 14:43:38,298 get_gpu_info_run     ERROR    |   File "/home/cryosparc_user/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/subprocess.py", line 526, in run
2024-12-27 14:43:38,298 get_gpu_info_run     ERROR    |     raise CalledProcessError(retcode, process.args,
2024-12-27 14:43:38,298 get_gpu_info_run     ERROR    | subprocess.CalledProcessError: Command '['ssh', 'cryosparc_user@tulasi', 'bash -c "eval $(/home/cryosparc_user/cryosparc_worker/bin/cryosparcw env); python /home/cryosparc_user/cryosparc_worker/cryosparc_compute/get_gpu_info.py"']' returned non-zero exit status 255.
2024-12-27 14:43:48,298 update_all_job_sizes_run INFO     | Finished updating all job sizes (0 jobs updated, 0 projects updated)
2024-12-27 14:44:38,479 export_all_projects_run INFO     | Finished exporting all projects




cryosparc_user@tulasi:~/cryosparc_master$ bin/cryosparcm log command_vis | tail -n 40
    self._reload()
  File "cryosparc2_compute/client.py", line 61, in _reload
    system = self._get_callable('system.describe')()
  File "cryosparc2_compute/client.py", line 49, in func
    r = requests.post(self.url, data = json.dumps(data, cls=NumpyEncoder), headers = header, timeout=self.timeout)
  File "/home/cryosparc_user/cryosparc_master/deps/anaconda/lib/python2.7/site-packages/requests/api.py", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/cryosparc_user/cryosparc_master/deps/anaconda/lib/python2.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/cryosparc_user/cryosparc_master/deps/anaconda/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/cryosparc_user/cryosparc_master/deps/anaconda/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/cryosparc_user/cryosparc_master/deps/anaconda/lib/python2.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='tulasi.wadsworth.org', port=39005): Max retries exceeded with url: /api (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5a65925410>: Failed to establish a new connection: [Errno 111] Connection refused',))
*** client.py: command (http://tulasi.wadsworth.org:39005/api) did not reply within timeout of 300 seconds, attempt 1 of 3
*** client.py: command (http://tulasi.wadsworth.org:39005/api) did not reply within timeout of 300 seconds, attempt 2 of 3
*** client.py: command (http://tulasi.wadsworth.org:39005/api) did not reply within timeout of 300 seconds, attempt 3 of 3
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "cryosparc2_command/command_vis/__init__.py", line 63, in <module>
    rtp = CommandClient(os.environ['CRYOSPARC_MASTER_HOSTNAME'], int(os.environ['CRYOSPARC_COMMAND_RTP_PORT']))
  File "cryosparc2_compute/client.py", line 33, in __init__
    self._reload()
  File "cryosparc2_compute/client.py", line 61, in _reload
    system = self._get_callable('system.describe')()
  File "cryosparc2_compute/client.py", line 49, in func
    r = requests.post(self.url, data = json.dumps(data, cls=NumpyEncoder), headers = header, timeout=self.timeout)
  File "/home/cryosparc_user/cryosparc_master/deps/anaconda/lib/python2.7/site-packages/requests/api.py", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/cryosparc_user/cryosparc_master/deps/anaconda/lib/python2.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/cryosparc_user/cryosparc_master/deps/anaconda/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/cryosparc_user/cryosparc_master/deps/anaconda/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/cryosparc_user/cryosparc_master/deps/anaconda/lib/python2.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='tulasi.wadsworth.org', port=39005): Max retries exceeded with url: /api (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe99a2de410>: Failed to establish a new connection: [Errno 111] Connection refused',))


cryosparc_user@tulasi:~/cryosparc_master$ bin/cryosparcm log supervisord | tail -n 40
2024-12-27 14:19:14,404 INFO spawnerr: unknown error making dispatchers for 'command_vis': EACCES
2024-12-27 14:19:15,406 INFO spawnerr: unknown error making dispatchers for 'command_vis': EACCES
2024-12-27 14:19:17,409 INFO spawnerr: unknown error making dispatchers for 'command_vis': EACCES
2024-12-27 14:19:20,414 INFO spawnerr: unknown error making dispatchers for 'command_vis': EACCES
2024-12-27 14:19:20,414 INFO gave up: command_vis entered FATAL state, too many start retries too quickly
2024-12-27 14:32:56,952 WARN received SIGTERM indicating exit request
2024-12-27 14:32:56,953 INFO waiting for command_core, database to die
2024-12-27 14:32:56,953 WARN ignored SIGHUP indicating restart request (shutdown in progress)
2024-12-27 14:32:57,118 INFO exited: command_core (exit status 0; expected)
2024-12-27 14:32:58,117 INFO stopped: database (exit status 0)
2024-12-27 14:37:09,446 INFO RPC interface 'supervisor' initialized
2024-12-27 14:37:09,447 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2024-12-27 14:37:09,449 INFO daemonizing the supervisord process
2024-12-27 14:37:09,450 CRIT could not write pidfile /home/cryosparc_user/cryosparc_master/run/supervisord.pid
2024-12-27 14:37:16,531 INFO spawned: 'database' with pid 4308
2024-12-27 14:37:17,562 INFO success: database entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-12-27 14:37:21,085 INFO spawned: 'command_core' with pid 4412
2024-12-27 14:37:26,564 INFO success: command_core entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2024-12-27 14:37:33,714 INFO spawnerr: unknown error making dispatchers for 'command_vis': EACCES
2024-12-27 14:37:34,716 INFO spawnerr: unknown error making dispatchers for 'command_vis': EACCES
2024-12-27 14:37:36,720 INFO spawnerr: unknown error making dispatchers for 'command_vis': EACCES
2024-12-27 14:37:39,724 INFO spawnerr: unknown error making dispatchers for 'command_vis': EACCES
2024-12-27 14:37:39,724 INFO gave up: command_vis entered FATAL state, too many start retries too quickly
2024-12-27 14:43:17,080 INFO waiting for command_core to stop
2024-12-27 14:43:17,080 INFO waiting for database to stop
2024-12-27 14:43:17,658 INFO stopped: command_core (exit status 0)
2024-12-27 14:43:18,431 INFO stopped: database (exit status 0)
2024-12-27 14:43:21,061 INFO RPC interface 'supervisor' initialized
2024-12-27 14:43:21,062 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2024-12-27 14:43:21,063 INFO daemonizing the supervisord process
2024-12-27 14:43:21,064 CRIT could not write pidfile /home/cryosparc_user/cryosparc_master/run/supervisord.pid
2024-12-27 14:43:27,406 INFO spawned: 'database' with pid 4686
2024-12-27 14:43:28,425 INFO success: database entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-12-27 14:43:31,943 INFO spawned: 'command_core' with pid 4790
2024-12-27 14:43:36,950 INFO success: command_core entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2024-12-27 14:43:38,726 INFO spawnerr: unknown error making dispatchers for 'command_vis': EACCES
2024-12-27 14:43:39,729 INFO spawnerr: unknown error making dispatchers for 'command_vis': EACCES
2024-12-27 14:43:41,732 INFO spawnerr: unknown error making dispatchers for 'command_vis': EACCES
2024-12-27 14:43:44,736 INFO spawnerr: unknown error making dispatchers for 'command_vis': EACCES
2024-12-27 14:43:44,736 INFO gave up: command_vis entered FATAL state, too many start retries too quickly
cryosparc_user@tulasi:~/cryosparc_master$
cryosparc_user@tulasi:~$ ps -eo user:12,pid,ppid,start,command | grep -e cryosparc_master -e mongo
ls -l /tmp/mongo*.sock /tmp/cryosparc*.sock
cryosparcm log command_core | tail -n 20
cryosparcm log command_vis | tail -n 40
cryosparcm log supervisord | tail -n 40
cryosparc_u+    4579       1 14:43:20 python /home/cryosparc_user/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/cryosparc_user/cryosparc_master/supervisord.conf
cryosparc_u+    4686    4579 14:43:26 mongod --auth --dbpath /home/cryosparc_user/cryosparc2_database --port 39001 --oplogSize 64 --replSet meteor --wiredTigerCacheSizeGB 4 --bind_ip_all
cryosparc_u+    4790    4579 14:43:31 python /home/cryosparc_user/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home/cryosparc_user/cryosparc_master/gunicorn.conf.py
cryosparc_u+    4791    4790 14:43:31 python /home/cryosparc_user/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home/cryosparc_user/cryosparc_master/gunicorn.conf.py
cryosparc_u+    5318    4148 15:12:13 grep -e cryosparc_master -e mongo
srwx------ 1 cryosparc_user systemd-timesync 0 Dec 27 14:43 /tmp/cryosparc-supervisor-aac39d01f9caa09e5b897918f05baddb.sock
srwx------ 1 cryosparc_user systemd-timesync 0 Dec 27 14:43 /tmp/mongodb-39001.sock
11

I commented out the hostname and force hostname commands from the cryosparc_master’s config.sh file but there was no obvious change in behaivor.

cryosparc_user@tulasi:~/cryosparc_master$ vi config.sh
cryosparc_user@tulasi:~/cryosparc_master$ bin/cryosparcm stop
CryoSPARC is running.
Stopping cryoSPARC
command_core: stopped
database: stopped
Shut down
cryosparc_user@tulasi:~/cryosparc_master$ bin/cryosparcm start
Starting CryoSPARC System master process…
CryoSPARC is not already running.
configuring database…
configuration complete
database: started
database OK
command_core: started
command_core connection succeeded
command_core startup successful
command_vis: ERROR (spawn error)
c

12

Thank you for all of your help. I’m all good now.

Issue was that I had log files with the wrong ownership, so the daemons failed to start because they couldn’t open their output files.

Also I had an issue with my browser, couldn’t get to the website, worked fine from a second computer and on the first one from an incognito window, so I completely owned that part of the problem – not that I must not have somehow caused the file ownership problem though I’ve no idea how that could have come about.

I will close this issue now.

Thank you again.

1 Like
13

This problem can occur if one ran cryosparcm commands as a Linux user other than the designated user. This risk is increased if one has enabled CRYOSPARC_FORCE_USER.