Unable to download empiar_10025_subset.tar for Extensive Validation

Troubleshooting
1

Hi,

It appears that there’s an issue with the file hosting of empiar_10025_subset.tar for Extensive Validation.
The download fails, whether during the execution of Extensive Validation job, using cryosparcm downloadtest, or using curl https://s3.wasabisys.com/cryosparc-test-data-dist/empiar_10025_subset.tar, even from multiple locations/machines.

cryoSPARC version: v4.7.1

Thanks

2

I see the same when trying to access the above file, except I get fed an Access Denied XML format file.

3

Thanks for reporting; this is now fixed!

4

Thank you for the quick fix!
I have confirmed the tar file is downloadable, and it runs through Extensive Validation if I manually place it and give it through “Path to Dataset Data”.
I would like to let you know that it seems the SHA256 of the file has changed from the original written in run_extensive_validation.py, which causes an interruption after automatic downloading during an Extensive Validation job.

[CPU:   85.9 MB]
Downloading data from https://s3.wasabisys.com/cryosparc-test-data-dist/empiar_10025_subset.tar... (99% - verifying /home/username/cryosparc_prjs/CS-extensive-validation-testing-1/empiar_10025_subset.tar)

[CPU:   93.9 MB]
Traceback (most recent call last):
  File "/home/username/cryosparc/cryosparc_worker/cryosparc_compute/jobs/workflows/run_extensive_validation.py", line 179, in run_extensive_validation
    dataset_data_dir = get_benchmark_dir(proj_dir_abs, dataset_info[dataset_selected]['basename'], params['dataset_data_dir'])
  File "/home/username/cryosparc/cryosparc_worker/cryosparc_compute/jobs/workflows/run_extensive_validation.py", line 46, in get_benchmark_dir
    assert os.path.isdir(benchmark_data_dir), f"Benchmark directory {benchmark_data_dir} is not a directory."
AssertionError: Benchmark directory /bulk5/data/EMPIAR/10025/data/empiar_10025_subset is not a directory.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 129, in cryosparc_master.cryosparc_compute.run.main
  File "/home/username/cryosparc/cryosparc_worker/cryosparc_compute/jobs/workflows/run_extensive_validation.py", line 182, in run_extensive_validation
    dataset_data_dir = rc.download_test_data(
  File "/home/username/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 2580, in download_test_data
    tarball_path = download_and_verify_url(
  File "/home/username/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 2549, in download_and_verify_url
    assert verify_sha256(dest, checksum_sha256), \
AssertionError: Could not verify checksum for `/home/username/cryosparc_prjs/CS-extensive-validation-testing-1/empiar_10025_subset.tar` (downloaded from `https://s3.wasabisys.com/cryosparc-test-data-dist/empiar_10025_subset.tar`)

sha256sum empiar_10025_subset.tar
3abc91780c9810cac4bbeec6517fce731ea8b29237fd5b66a45413e4dea51b22  empiar_10025_subset.tar

    '10025': {
        'name': 'EMPIAR-10025',
        'description': 'A subset of the EMPIAR-10025 dataset, containing 20 movies.',
        'basename': 'empiar_10025_subset',
        'link': 'https://s3.wasabisys.com/cryosparc-test-data-dist/empiar_10025_subset.tar',
        'checksum': 'a02f56d40c597c5059671e6e518c62892568d02f51a7f869855f2ee718280890',
5

Thanks @ntsu for reporting the incorrect checksum. The file with the expected checksum is again available for download at the relevant link.

6

Thanks. Confirmed to run through an Extensive Validation job automatically. The issue is resolved.

2 Likes