Inconsistent Map Level Values Between Download and Local File

Application
1

Hi everyone,

I’ve encountered an issue where the level values in a map downloaded from the CryoSPARC website (followed picture,left) are significantly different from the original map file located in the job directory(followed picture,right).

屏幕截图 2024-11-19 103220
屏幕截图 2024-11-19 103220736×522 112 KB

I’m using CryoSPARC version 4.5.3 and have tried downloading the map using both Chrome and Firefox, but the issue persists. Has anyone know the reason?

2

It may be that the map download via the UI was truncated in a way that did not prevent the viewing program from displaying the truncated file.
Please can you post the outputs of the commands

ls -l mapfile.mrc
shasum mapfile.mrc

for both versions of the map and email us the tgz file created by the command
cryosparcm snaplogs.

3

Thanks for your reply and this is the output of the commands:

ls -l  J595_013_volume_map.mrc
-rw-rw-r-- 1 jclabem jclabem 442369024 11 16 13:06 J595_013_volume_map.mrc

shasum  J595_013_volume_map.mrc
084da8a7685f5ea5afa126c486a13c06fcf9fa22  J595_013_volume_map.mrc

ls -l cryosparc_P1_J595_013_volume_map.mrc 
-rw-rw-r-- 1 spuser spuser 442369024 Nov 25 20:01 cryosparc_P1_J595_013_volume_map.mrc

shasum cryosparc_P1_J595_013_volume_map.mrc
71cb94a43dd36eea59370c29c47beedd24e552b6  cryosparc_P1_J595_013_volume_map.mrc

I have emailed the tgz file and please let me know if further infomration required.

4

Thanks @tosyl . The file names in your latest post differ from those in the screenshot from Inconsistent Map Level Values Between Download and Local File. Do you observe inconsistencies within each pair of map files, where one file was downloaded via the web UI, the other directly from the job directory?
Please can you post the outputs of these commands

cryosparcm cli "get_job('P1', 'J213', 'job_type', 'version', 'instance_information', 'status',  'params_spec', 'errors_run', 'created_at', 'started_at')"
cryosparcm cli "get_job('P1', 'J595', 'job_type', 'version', 'instance_information', 'status',  'params_spec', 'errors_run', 'created_at', 'started_at')"
5

Thanks @wtempel .

Do you observe inconsistencies within each pair of map files

Yes, the map files downloaded via the UI differ from their corresponding files in the job directory.

Here is the outputs:

$ cryosparcm cli "get_job('P1', 'J213', 'job_type', 'version', 'instance_information', 'status',  'params_spec', 'errors_run', 'created_at', 'started_at')"
{'_id': '67317b814e86c7a434a3459c', 'created_at': 'Mon, 11 Nov 2024 03:35:29 GMT', 'errors_run': [], 'instance_information': {'CUDA_version': '11.8', 'available_memory': '441.96GB', 'cpu_model': 'Intel(R) Xeon(R) Platinum 8468V', 'driver_version': '12.4', 'gpu_info': [{'id': 0, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090', 'pcie': '0000:49:00'}], 'ofd_hard_limit': 51200, 'ofd_soft_limit': 1024, 'physical_cores': 96, 'platform_architecture': 'x86_64', 'platform_node': 'gpu1', 'platform_release': '5.15.0-94-generic', 'platform_version': '#104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024', 'total_memory': '503.52GB', 'used_memory': '57.65GB'}, 'job_type': 'nonuniform_refine_new', 'params_spec': {'compute_use_ssd': {'value': False}, 'crg_do_anisomag': {'value': True}, 'crg_do_spherical': {'value': True}, 'crg_do_tetrafoil': {'value': True}, 'refine_ctf_global_refine': {'value': True}}, 'project_uid': 'P1', 'started_at': 'Mon, 11 Nov 2024 04:11:26 GMT', 'status': 'completed', 'uid': 'J213', 'version': 'v4.6.0'}

$ cryosparcm cli "get_job('P1', 'J595', 'job_type', 'version', 'instance_information', 'status',  'params_spec', 'errors_run', 'created_at', 'started_at')"
{'_id': '673807dc4c41e373a06f1395', 'created_at': 'Sat, 16 Nov 2024 02:47:56 GMT', 'errors_run': [], 'instance_information': {'CUDA_version': '11.8', 'available_memory': '947.48GB', 'cpu_model': 'Intel(R) Xeon(R) Platinum 8468V', 'driver_version': '12.4', 'gpu_info': [{'id': 0, 'mem': 47576711168, 'name': 'NVIDIA L40', 'pcie': '0000:29:00'}], 'ofd_hard_limit': 51200, 'ofd_soft_limit': 51200, 'physical_cores': 96, 'platform_architecture': 'x86_64', 'platform_node': 'gpu31', 'platform_release': '5.15.0-94-generic', 'platform_version': '#104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024', 'total_memory': '1007.52GB', 'used_memory': '53.52GB'}, 'job_type': 'nonuniform_refine_new', 'params_spec': {}, 'project_uid': 'P1', 'started_at': 'Sat, 16 Nov 2024 02:49:10 GMT', 'status': 'completed', 'uid': 'J595', 'version': 'v4.5.3'}
6

@tosyl Thanks for these details. Please could you share one inconsistent pair of map files with us so that we may inspect more closely. Ideally, one of the pair’s files should be have bee downloaded recently/now and the files should be accomapined by a freshly created tgz logs archive from the command

cryosparcm snaplogs

to ensure the logs include the time window when the file was downloaded via the web UI.
If you can, please share the files with us privately via your institution’s file sharing service. Otherwise, we can make arrangements for you to upload the files. Please let us know.

7

I have prepared these three files and sent you a private message with the details, please find the link and download them. Thanks for your help @wtempel !

8

Thanks @tosyl. I downloaded the files.

9

@tosyl May we ask for additional information that may help us troubleshoot. Please:

  1. email us the job report for job J595 in project P1.
  2. post the outputs of these command on the CryoSPARC master node
    uname -a
free -h
cat /proc/1/sched | head -1
cat /sys/fs/cgroup/memory.high
grep "$(df $(cryosparcm cli "get_project_dir_abs('P1')") | tail -n 1 | awk '{print $NF}') " /proc/mounts
  3. What are the brand and version of the web browser that you used for map download?
  4. On what OS is the web browser running?
10

@wtempel

  1. I have emailed the job report and hope it finds you well.
  2. Here is the output:
  $ uname -a
  Linux login 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
  $ free -h
                 total        used        free      shared  buff/cache   available
  Mem：      503Gi        72Gi       132Gi        36Mi       298Gi       427Gi
  Swap：       31Gi        61Mi        31Gi
  $ cat /proc/1/sched | head -1
  systemd (1, #threads: 1)
  $ cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
  486843453440
  $ grep "$(df $(cryosparcm cli "get_project_dir_abs('P1')") | tail -n 1 | awk '{print $NF}') " /proc/mounts
  netdata_hdd_hdd_fs /public storage rw,noatime 0 0
  1. Both Firefox115.12.0esr (64-bit) and Google (Version 126.0.6478.126 (Official Build) (64-bit)) I’ve tried.
  2. Ubuntu 22.04.

Hope these can help!

11

Thanks @tosyl for sending the job report and posting the information.
Please can you also post

  • the URL in the web browser’s address bar when you access the CryoSPARC web app
  • whether you set up ssh port forwarding or use a vpn to access the CryoSPARC web app
  • wether volumes downloaded through the CryoSPARC web app appear corrupted every single time. If not, are there any patterns? Does volume file corruption via CryoSPARC web app download occur for specific
    • project(s)
    • job type(s)
    • map type(s)
12

@wtempel

  • The URL in my browser’s address bar is http://192.168.1.232:39000/
  • I access CryoSPARC directly without SSH port forwarding or VPN.
  • Regarding volume downloads, yes - they are corrupted every time regardless of:
    Project type
    Job types (2D Classification, Homogeneous Refinement, Non-uniform Refinement)
    Map types

Additionally, when downloading series of volumes from Heterogeneous Refinement, 3D Variability or 3D Classification, all I can get is a ~40MB zip file (named like cryosparc_P1_J2171_final_volume_series.zip) that fails to decompress.
When attempting to download event logs, I get the error message “Could not download event log pdf”, and any PDF files downloaded through the interface are corrupted as well.

13

Thanks @tosyl For posting this information. To help narrow down the possible failure sources, please can you try

  1. Downloading and unpacking the Instance Logs report:
    Admin → Instance Logs → Download Report. Can the downloaded system_report_*.zip file by unpacked and does it contain valid *.log and *.json files?
  2. Establish a port forwarding channel to the CryoSPARC master computer so that you can connect to the web app using the URL http://127.0.0.1:39000 and perform the steps below using that connection for the tests below, essentially testing downloads via port-forwarded access that failed via http://192.168.1.232:39000:
  3. a map download. Is the map downloaded via web app at http://127.0.0.1:39000 corrupted?
  4. a cs file download: Download the All particles metadata under the Outputs tab
    image
    image905×373 47.5 KB
    of a Homogeneous Refinement job and compare the sizes and checksums of corresponding cs files between the downloaded file and the file (final iteration) inside the job directory (replace X, Y, Z with actual project, job, iteration numbers):
    # downloaded
ls -lh cryosparc_PX_JY_ZZZ_particles.cs
md5sum cryosparc_PX_JY_ZZZ_particles.cs
# CryoSPARC job directory
ls -lh JY_ZZZ_particles.cs
md5sum JY_ZZZ_particles.cs # Linux
md5 JY_ZZZ_particles.cs # macOS

Please can you also provide more details on the project directory storage. I am unfamiliar with the output

What is the output of the command

stat -f /public

on the CryoSPARC master computer?

14

@wtempel Thanks for your continued help. I’ve performed the tests you requested, and the results are below.

  1. system_report_2024_12_25T02_34_34.447Z.zip has been downloaded and unpacked successfully with no error.It contains valid *.log and *.json files.
  2. I accessed http://127.0.0.1:39000/browse/P4-W10-J*#job(P4-J181) via Chrome for downloading.
  3. Unfortunately, maps downloaded via http://127.0.0.1:39000 are also corrupted.
  4. Here is the output regarding the .cs files:
# downloaded via http://127.0.0.1:39000
ls -lh cryosparc_P4_J181_013_particles.cs
-rw-r--r-- 1 root root 36M 12 25 10:29 cryosparc_P4_J181_013_particles.cs
md5sum cryosparc_P4_J181_013_particles.cs
dad6174e1fd4491e64aa3ec515e0fc8b  cryosparc_P4_J181_013_particles.cs
# downloaded via http://192.168.1.232:39000
ls -lh cryosparc_P4_J181_013_particles.cs
-rw-r--r-- 1 root root 36M 12月 25 10:40 cryosparc_P4_J181_013_particles.cs
md5sum cryosparc_P4_J181_013_particles.cs
dad6174e1fd4491e64aa3ec515e0fc8b  cryosparc_P4_J181_013_particles.cs
# CryoSPARC job directory
ls -lh J181_013_particles.cs
-rw-rw-r-- 1 tianxiaowen jclabem 36M  9 27 17:36 J181_013_particles.cs
md5sum J181_013_particles.cs
a86f07d6c5c583d8bb193c1271014128  J181_013_particles.cs

We are using a new storage system named ParaStor300S, so maybe this is a incompatibility issue? Here is the output of stat -f /public:

  File："/public"
    ID: 1200000000 Namelen：255     Type：UNKNOWN (0x797083)
Block size：4096       Fundamental block size：4096
    Blocks：Total：1582265204736 Free：1414038191454 Available：1414038191454
Inodes: Total：9223372036854775807 Free：9223372036826378150

I have no further information about ParaStor300S at the moment, but I’m trying to find more internal documentation on it. Please let me know if there are any other tests you’d like me to run or any other information I can provide.

15

That is a possibility. Would it be possible for you to try if this is issue can be resolved by using storage for CryoSPARC project directories that is shared between the CryoSPARC instance’s nodes using “standard” nfs?

16

@wtempel the NFS thing - users think it might hurt performance, so we’re gonna skip that for now. I’m gonna see if I can reproduce this on a different machine with regular storage though. Will keep you posted and thanks for all the help!

1 Like