SSD cache error related to cache_request_check

Troubleshooting
1

Hi all,

Recently we have encountered a SSD cache problem. Whenever we checked the SSD usage option, the following error message is shown:

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 96, in cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/class3D/run.py", line 89, in cryosparc_compute.jobs.class3D.run.run_class_3D
  File "/home/wcyl/cryosparc/cryosparc_worker/cryosparc_compute/particles.py", line 114, in read_blobs
    u_blob_paths = cache.download_and_return_cache_paths(u_rel_paths)
  File "/home/wcyl/cryosparc/cryosparc_worker/cryosparc_compute/jobs/cache.py", line 105, in download_and_return_cache_paths
    compressed_keys = get_compressed_keys(worker_hostname, rel_paths)
  File "/home/wcyl/cryosparc/cryosparc_worker/cryosparc_compute/jobs/cache.py", line 278, in get_compressed_keys
    compressed_keys = rc.cli.cache_request_check(worker_hostname, rc._project_uid, rc._job_uid, com.compress_paths(rel_paths))
  File "/home/wcyl/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 112, in func
    assert "error" not in res, f'Error for "{key}" with params {params}:\n' + format_server_error(res["error"])
AssertionError: Error for "cache_request_check" with params ('wcyl-4gpu', 'P38', 'J593', '**~a long chunk of compress key here~**'):
ServerError: 
Traceback (most recent call last):
  File "/home/wcyl/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 200, in wrapper
    res = func(*args, **kwargs)
  File "/home/wcyl/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 3208, in cache_request_check
    assert link_path.startswith('/')
AssertionError

Jobs can run if SSD option is unchecked. We have reset the cache system found on cryosparc guide, but no use. Does anyone know how can I solve this problem, as every job without SSD now runs so slow?

2

@kpsleung Please can you post the outputs of these commands:

/home/wcyl/cryosparc/cryosparc_master/bin/cryosparcm cli "get_project('P38', 'project_dir')"
stat $(/home/wcyl/cryosparc/cryosparc_master/bin/cryosparcm cli "get_project_dir_abs('P38')")
3

Here is the output:

{'_id': '6482fda60b7f213a64c686db', 'project_dir': '/media/wcyl/2023_02/20230509/CS-carbon'}

and

File: /media/wcyl/2023_02/20230509/CS-carbon
  Size: 106496    	Blocks: 208        IO Block: 4096   directory
Device: 841h/2113d	Inode: 26193       Links: 1
Access: (0777/drwxrwxrwx)  Uid: ( 1000/    wcyl)   Gid: ( 1000/    wcyl)
Access: 2023-06-09 19:15:33.471368000 +0800
Modify: 2023-06-09 19:01:25.596543500 +0800
Change: 2023-06-09 19:01:25.596543500 +0800
 Birth: -
4

@kpsleung In case you have not yet resolved this issue, please can you also run

stat -f /media/wcyl/2023_02/20230509/CS-carbon
find /media/wcyl/2023_02/20230509/CS-carbon/J593 -type l -ls | tail -n 20

and post the output.