Hi, I am a new user of CryoSPARC and I need help with the following error:
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/class2D/newrun.py”, line 73, in cryosparc_master.cryosparc_compute.jobs.class2D.newrun.run_class_2D
File “/media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC/cryosparc_worker/cryosparc_compute/particles.py”, line 120, in read_blobs
u_blob_paths = cache_run(u_rel_paths)
File “/media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC/cryosparc_worker/cryosparc_compute/jobs/cache.py”, line 115, in download_and_return_cache_paths
used_mb = sync_hits(worker_hostname, ssd_cache_path, instance_id)
File “/media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC/cryosparc_worker/cryosparc_compute/jobs/cache.py”, line 168, in sync_hits
os.makedirs(this_instance_cache_path, exist_ok=True)
File “/media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/os.py”, line 223, in makedirs
mkdir(name, mode)
OSError: [Errno 22] Invalid argument: ‘/media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC/scratch/cryosparc_cache/instance_comino:39001’
I made a directory with the name ‘instance_comino’ but it is not helping me to solve the issue. Any suggestion to resolve this issue will be highly appreciated.
Welcome to the forum @sabdulmohid .
Please can you confirm that the directory /media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC/scratch/cryosparc_cache/
exists and is suitable for caching.
- What are the outputs of the commands
lsblk stat -f /media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC/scratch/cryosparc_cache/ - How is the device on which the
cryosparc_cache/directory is stored connected to your computer? Is it an internal or external device?
Hi @wtempel! Thanks for your response.
- Yes the directory exists. what do you mean by suitable for caching? The server where the cryoSPARC is installed has three partition disks (1 TiB for boot, 8 TiB where the cryoSPARC is installed and another 2 TiB which is unused till now. Alongwith this server, a synology NAS storage is attached and mounted with the server.
Here I have copied the entire path and the response that i obtained after entering the command that you mentioned.
zhanglab_cwru@comino:/media/zhanglab_cwru/00E0143AE014387C$ cd CRYOSPARC/
zhanglab_cwru@comino:/media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC$ ls
cryosparc_database cryosparc_master cryosparc_master.tar.gz cryosparc_worker cryosparc_worker.tar.gz scratch
zhanglab_cwru@comino:/media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC$ cd scratch/
zhanglab_cwru@comino:/media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC/scratch$ ls
cryosparc_cache
zhanglab_cwru@comino:/media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC/scratch$ cd cryosparc_cache/
zhanglab_cwru@comino:/media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC/scratch/cryosparc_cache$ ls
instance_comino
zhanglab_cwru@comino:/media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC/scratch/cryosparc_cache$ lsblk
stat -f /media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC/scratch/cryosparc_cache/
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 4K 1 loop /snap/bare/5
loop2 7:2 0 63,5M 1 loop /snap/core20/2015
loop3 7:3 0 63,9M 1 loop /snap/core20/2105
loop4 7:4 0 73,9M 1 loop /snap/core22/864
loop5 7:5 0 55,7M 1 loop /snap/core18/2812
loop6 7:6 0 240,3M 1 loop /snap/firefox/3358
loop7 7:7 0 349,7M 1 loop /snap/gnome-3-38-2004/143
loop8 7:8 0 485,5M 1 loop /snap/gnome-42-2204/120
loop9 7:9 0 497M 1 loop /snap/gnome-42-2204/141
loop10 7:10 0 91,7M 1 loop /snap/gtk-common-themes/1535
loop11 7:11 0 12,3M 1 loop /snap/snap-store/959
loop12 7:12 0 53,3M 1 loop /snap/snapd/19457
loop13 7:13 0 40,9M 1 loop /snap/snapd/20290
loop14 7:14 0 452K 1 loop /snap/snapd-desktop-integration/83
loop15 7:15 0 242,9M 1 loop /snap/firefox/3504
loop16 7:16 0 321,1M 1 loop /snap/vlc/3721
loop17 7:17 0 74,1M 1 loop /snap/core22/1033
nvme0n1 259:0 0 931,5G 0 disk
├─nvme0n1p1 259:4 0 512M 0 part /boot/efi
└─nvme0n1p2 259:5 0 931G 0 part /var/snap/firefox/common/host-hunspell
/
nvme1n1 259:1 0 1,8T 0 disk
├─nvme1n1p1 259:2 0 16M 0 part
└─nvme1n1p2 259:3 0 1,8T 0 part /media/zhanglab_cwru/6E5A0F9B5A0F5EE9
nvme2n1 259:6 0 7,3T 0 disk
├─nvme2n1p1 259:7 0 16M 0 part
└─nvme2n1p2 259:8 0 7,3T 0 part /media/zhanglab_cwru/00E0143AE014387C
File: “/media/zhanglab_cwru/00E0143AE014387C/CRYOSPARC/scratch/cryosparc_cache/”
ID: e014387c00e0143a Namelen: 255 Type: UNKNOWN (0x7366746e)
Block size: 4096 Fundamental block size: 4096
Blocks: Total: 1953502207 Free: 1946750866 Available: 1946750866
Inodes: Total: 0 Free: 0
It seems that cryosparc_cache/ exists and is stored on an nvme device. This s good. For proper cache function, I am concerned about the UNKNOWN file system type shown by the stat -f command. Do you know the filesystem type?
What are the outputs of the commands
/sbin/blkid
mount -v | grep /dev/nvme2n1p2
?
Another, longer term concern is that the CryoSPARC database and cryosparc_cache/ share the same device. Many SSD devices’ lifetime is specified in terms of data amount written. Caching may write large amounts of data, which could, depending on the storage’s implementation, accelerate the device’s failure. It may be better to select a device for exclusive scratch/cache use so that if/when the device fails, more valuable data, like the CryoSPARC database, are not affected.
Hi, thanks again for your response
The file system type of the device we are using is ‘tmpfs’. The details are as follows:
(base) zhanglab_cwru@comino:~/Downloads$ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 51G 32M 51G 1% /run
/dev/nvme0n1p2 916G 44G 825G 6% /
tmpfs 252G 343M 252G 1% /dev/shm
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
/dev/nvme0n1p1 511M 6,1M 505M 2% /boot/efi
tmpfs 51G 3,2M 51G 1% /run/user/1001
/dev/nvme1n1p2 1,9T 159M 1,9T 1% /media/zhanglab_cwru/6E5A0F9B5A0F5EE9
/dev/nvme2n1p2 7,3T 26G 7,3T 1% /media/zhanglab_cwru/00E0143AE014387C
129.22.210.8:/volume1/CRYOEM_DATA 175T 259G 175T 1% /zhanglab_cryoem_data
The outputs of the command is as follows:
(base) zhanglab_cwru@comino:~$ /sbin/blkid
mount -v | grep /dev/nvme2n1p2
/dev/nvme0n1p2: UUID="e2841943-7ac7-4a3e-ad24-8f36c18d6393" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="9a453089-a4da-4abb-bd5b-cbb55c9a472c"
/dev/nvme2n1p2 on /media/zhanglab_cwru/00E0143AE014387C type ntfs3 (rw,nosuid,nodev,relatime,uid=1001,gid=1001,iocharset=utf8,windows_names,uhelper=udisks2)
So, my understanding from your reply is, we should re-install the cryosparc in such a way so that the cryosparc database and the cryosparc_cache/ stays in different storage device. Am I correct?
Do you think the longer name of the mounting devices
(i.e. /dev/nvme1n1p2 1,9T 159M 1,9T 1% /media/zhanglab_cwru/6E5A0F9B5A0F5EE9
/dev/nvme2n1p2 7,3T 26G 7,3T 1% /media/zhanglab_cwru/00E0143AE014387C
could create a problem in future?
Correct. But this would not resolve the problem for which you opened this forum topic.
It seems that the cache path resides on an ntfs3 filesystem, and I have not tested whether ntfs3 is suitable for this purpose. You may want to use a filesystem like ext4 or xfs instead. Caution: Creating a filesystem on a physical or logical volume can (you can almost say will) destroy data currently stored on that volume.
Then could you please let me know how should I proceed if a fresh install of cryosparc in the boot drive and creating the cache in other location does not solve this problem?
I don’t know if I’ll be able to covert the format of the 8 TB storage from ntfs3 to ext4. I’ll check and confirm.
The primary issue at this time is likely the filesystem type, the issue of sharing scratch space with the database is secondary. Your “boot drive” seems to have a compatible filesystem type, but you should be aware that the database may grow to hundreds of gigabytes in size; a size exceeding 1 TB is possible for “busy” instances. You may want to consider an additional (2TB?) device for the database or, if 129.22.210.8:/volume1/CRYOEM_DATA is reliable, reasonably fast and meets database requirements, you could transfer (see precautions for copying below) the database to that shared filesystem. In any case, you want to monitor remaining storage and never allow the database to run out of space.
“Converting” the ntfs3 filesystem entails destroying the data it holds. Be sure to copy all data as needed before you replace the filesystem.
Because the filesystem also holds the database and in case you would like to continue using that database, I recommend two (redundant for extra reliability) precautions:
- backup the database.
- additionally, prepare a copy the database directory after CryoSPARC has been stopped completely to ensure a valid copy.
I recommend seeking help from your IT support regarding the dismounting of the old, creation and mounting of the new filesystem, if needed.
Hi, I changed the filesystem from ntfs3 to ext4 and re-installed the cryosparc and tested the software using the ‘empiar_10025_subset’ dataset. Everything is running well now. Thanks for your help and suggestion.