V5.0.7 – Instance Testing / Benchmark job fails with AttributeError: 'str' object has no attribute 'stat' when benchmark data is missing

Troubleshooting
1

v5.0.7 – Instance Testing / Benchmark job fails with AttributeError: 'str' object has no attribute 'stat' when benchmark data is missing

Hi all,

I previously encountered/documented this issue against CryoSPARC v5.0.4 and it appears to still be present in v5.0.7.

Environment

  • CryoSPARC version: v5.0.7
  • Worker revision: dfcba2f3ac0fe600b22b97895e9ca25abbffcee7
  • Deployment: Master + worker on shared /gpfs/shared/filesystem (GPFS)
  • Project directory: /gpfs/shared/filesystem/test/CS-dev-2026-09
  • Benchmark directory: /gpfs/shared/filesystem/test/CS-dev-2026-09/benchmark_data

Issue

When the expected benchmark_data directory does not exist, the Instance Testing / Benchmark job correctly detects that the benchmark data is missing and attempts to automatically download it.

However, the automatic download fails with:

AttributeError: 'str' object has no attribute 'stat'

The job output first reports:

Benchmark data not found: Benchmark directory /gpfs/shared/filesystem/test/CS-dev-2026-09/benchmark_data is not a directory.. Downloading now...

It then attempts to download:

https://s3.us-east-1.wasabisys.com/cryosparc-performance-benchmark-data/performance_benchmark_data_v1.tar.gz

and fails with the following traceback:

Traceback (most recent call last):
  File "/gpfs/shared/filesystem/home/svcacct_homedir/cryosparc_worker/compute/jobs/instance_testing/run.py", line 217, in run_benchmarks
    benchmark_data_dir = get_benchmark_dir(proj_dir_abs, params.benchmark_data_dir)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/shared/filesystem/home/svcacct_homedir/cryosparc_worker/compute/jobs/instance_testing/run.py", line 1099, in get_benchmark_dir
    assert os.path.isdir(benchmark_data_dir), f"Benchmark directory {benchmark_data_dir} is not a directory."
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Benchmark directory /gpfs/shared/filesystem/test/CS-dev-2026-09/benchmark_data is not a directory.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "cli/run.py", line 106, in cli.run.run_job
  File "cli/run.py", line 211, in cli.run.run_job_function
  File "/gpfs/shared/filesystem/home/svcacct_homedir/cryosparc_worker/compute/jobs/instance_testing/run.py", line 221, in run_benchmarks
    benchmark_data_dir = rc.download_benchmark_test_data(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/shared/filesystem/home/svcacct_homedir/cryosparc_worker/core/run_context.py", line 517, in download_benchmark_test_data
    return benchmarks.download_benchmark_test_data(self.get_job(), url, download_dir, download_basename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/shared/filesystem/home/svcacct_homedir/cryosparc_worker/core/benchmarks.py", line 108, in download_benchmark_test_data
    tarball_path = download_file(
                   ^^^^^^^^^^^^^^
  File "/gpfs/shared/filesystem/home/svcacct_homedir/cryosparc_worker/core/benchmarks.py", line 163, in download_file
    if os.path.isfile(dest) and dest.stat().st_size == total_bytes:
                               ^^^^^^^^^
AttributeError: 'str' object has no attribute 'stat'

The failure occurs in:

cryosparc_worker/core/benchmarks.py

at approximately line 163:

if os.path.isfile(dest) and dest.stat().st_size == total_bytes:

dest is a str, but .stat() is being called as though it were a pathlib.Path object.

This is the same underlying error as in v5.0.4.

Workaround

I was able to work around the issue by manually downloading the benchmark dataset and placing it in the expected benchmark_data directory.

For example:

/gpfs/shared/filesystem/test/CS-dev-2026-09/benchmark_data

The problem therefore appears to be specifically with the automatic benchmark-data download path, rather than the benchmark dataset itself.

Possible fix

The problematic line:

if os.path.isfile(dest) and dest.stat().st_size == total_bytes:

could presumably be changed to either:

if os.path.isfile(dest) and os.stat(dest).st_size == total_bytes:

or by ensuring dest is converted to a pathlib.Path before calling .stat().

It would also be helpful if the benchmark setup/download logic provided a clearer indication of the expected directory structure when benchmark data is missing.

Happy to provide additional logs or test a patched version.

Thanks!

2

Thanks for your post @fourzerosix.

If you reported the issue in the forum, please can you post the link?
Please can you post the outputs of these commands

csprojectid="P0" # replace with actual project id
csjobid="J0" # replace with id of the failed benchmark job
cryosparcm cli "api.jobs.find_one('$csprojectid', '$csjobid').params"
cryosparcm cli "api.jobs.find_one('$csprojectid', '$csjobid').started_at"
cryosparcm cli "api.jobs.find_one('$csprojectid', '$csjobid').failed_at"
ls -al /gpfs/shared/filesystem/test/CS-dev-2026-09/benchmark_data
1 Like
3

Thanks @wtempel !

I reviewed my notes and found references to the same issue on v5.0.6 as well (as v5.0.4). I never created a forum post because this is not a work stoppage issue in our environment - only kept my personal notes for reference.

I ran the requested checks, but I think there is an important distinction about the missing benchmark_data directory.

The directory is intentionally absent. That is the condition being tested. The Instance Testing job is supposed to detect that the benchmark dataset is missing, automatically download it, extract it, and then proceed with the benchmark. Instead, the automatic download path is crashing before the download can complete.

Reproduction

I created a new test project with no benchmark_data directory:

/gpfs/shared/filesystem/test/CS-cryo-forum-sandbox/

As expected:

[svcacct@cryosparc_host ~]$ ls -al /gpfs/shared/filesystem/test/CS-cryo-forum-sandbox/benchmark_data
ls: cannot access '/gpfs/shared/filesystem/test/CS-cryo-forum-sandbox/benchmark_data': No such file or directory

I then ran the Instance Testing / Benchmark job.

The job correctly recognized that the benchmark data was missing and entered the automatic-download path:

Benchmark data not found: Benchmark directory /gpfs/shared/filesystem/test/CS-cryo-forum-sandbox/benchmark_data is not a directory.. Downloading now...

Downloading data from
https://s3.us-east-1.wasabisys.com/cryosparc-performance-benchmark-data/performance_benchmark_data_v1.tar.gz

However, the job fails immediately afterward with:

Traceback (most recent call last):
  File "/gpfs/shared/filesystem/home/svcacct_homedir/cryosparc_worker/compute/jobs/instance_testing/run.py", line 217, in run_benchmarks
    benchmark_data_dir = get_benchmark_dir(proj_dir_abs, params.benchmark_data_dir)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/shared/filesystem/home/svcacct_homedir/cryosparc_worker/compute/jobs/instance_testing/run.py", line 1099, in get_benchmark_dir
    assert os.path.isdir(benchmark_data_dir), f"Benchmark directory {benchmark_data_dir} is not a directory."
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Benchmark directory /gpfs/shared/filesystem/test/CS-cryo-forum-sandbox/benchmark_data is not a directory.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "cli/run.py", line 106, in cli.run.run_job
  File "cli/run.py", line 211, in cli.run.run_job_function
  File "/gpfs/shared/filesystem/home/svcacct_homedir/cryosparc_worker/compute/jobs/instance_testing/run.py", line 221, in run_benchmarks
    benchmark_data_dir = rc.download_benchmark_test_data(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/shared/filesystem/home/svcacct_homedir/cryosparc_worker/core/run_context.py", line 517, in download_benchmark_test_data
    return benchmarks.download_benchmark_test_data(self.get_job(), url, download_dir, download_basename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/shared/filesystem/home/svcacct_homedir/cryosparc_worker/core/benchmarks.py", line 108, in download_benchmark_test_data
    tarball_path = download_file(
                   ^^^^^^^^^^^^^^
  File "/gpfs/shared/filesystem/home/svcacct_homedir/cryosparc_worker/core/benchmarks.py", line 163, in download_file
    if os.path.isfile(dest) and dest.stat().st_size == total_bytes:
                                ^^^^^^^^^
AttributeError: 'str' object has no attribute 'stat'

Why I believe this is a CryoSPARC bug . . .

The traceback shows the expected control flow:

run_benchmarks()
    -> get_benchmark_dir()
       -> benchmark_data missing
    -> download_benchmark_test_data()
       -> download_file()
          -> AttributeError

The actual failure is:

if os.path.isfile(dest) and dest.stat().st_size == total_bytes:

At this point, dest is a Python str, but .stat() is being called on it as though it were a pathlib.Path.

That results in:

AttributeError: 'str' object has no attribute 'stat'

So the automatic download never gets a chance to finish.

Additional testing

I independently downloaded the benchmark archive from the exact URL shown in the CryoSPARC job output and verified that the archive is valid and contains the expected directory structure:

benchmark_data/
benchmark_data/class2D_test/
benchmark_data/gpu_fsc_test/
benchmark_data/movies/
benchmark_data/movies/mrc/
benchmark_data/movies/tiff/
benchmark_data/movies/eer/
benchmark_data/picking_test/
benchmark_data/gpu_engine_test/
...

Therefore, this does not appear to be a problem with the benchmark archive itself or with the filesystem path.

I also reproduced the same AttributeError previously on v5.0.4 and v5.0.6. The worker has now been upgraded to v5.0.7, and the same failure is still present.

Current job config (from request)

For completeness, I queried the job directly:

[svcacct@cryosparc_host ~]$ cryosparcm cli "api.jobs.find_one('$csprojectid', '$csjobid').params"
{"bench_cpu": true, "bench_fs": true, "bench_gpu": true, "test_sequential": true, "test_particle": true, "num_cpus": 16, "use_all_gpus": true, "gpu_num_gpus": 1, "use_ssd": true, "send_data": false}

The job started and failed within approximately 10 seconds:

started_at:
2026-09-01T19:35:20.719000+00:00

failed_at:
2026-09-01T19:35:30.597000+00:00

I don’t know how common it is for sites to use the builtin benchmark - but we use it as part of a validation suite at least once/month but oftentimes more if adding servers and/or adjusting configurations.

This does not affect us anymore as i’ve manually downloaded the benchmark data and staged it on shared (GPFS) storage for relevant lab members to easily access. To solve this in future releases, you may want to see if engineering could review the automatic benchmark-data download implementation in:

cryosparc_worker/core/benchmarks.py

specifically download_file() around line 163.

The current code appears to be:

if os.path.isfile(dest) and dest.stat().st_size == total_bytes:

If dest is intentionally a string, I believe this needs to be something equivalent to:

if os.path.isfile(dest) and os.stat(dest).st_size == total_bytes:

Alternatively, if dest is intended to be a pathlib.Path, it should be converted to a Path before this operation.

The important point is that pre-populating benchmark_data is not a valid reproduction workaround, because doing so bypasses the code path under investigation. The failure occurs specifically when the benchmark data is absent and CryoSPARC attempts to perform its advertised automatic download.

This is reproducible on v5.0.7 and appears to be the same defect previously observed on v5.0.4 and v5.0.6.

4

Thanks @fourzerosix . We confirmed this is a bug in CryoSPARC and made a note to fix it in a future software release.

1 Like
5

@wtempel you rock - thanks so much!