Motion correction of tif files in a sshfs mount

Dear all,

we are testing a setup where we mounted a folder with the ‘sshfs’ command and now we wish to use this folder as the source of movies in tif format. CS version 3.2.2. SLURM cluster setup.
I can execute Import movies and the symbolic links are created. The links seem to work fine when I test them in the terminal. The files have read permissions:

stat FoilHole_10182344_Data_10180841_10180843_20191010_130455_Fractions.tif 
  File: FoilHole_10182344_Data_10180841_10180843_20191010_130455_Fractions.tif
  Size: 643139681 	Blocks: 1256136    IO Block: 4096   regular file
Device: 47h/71d	Inode: 9           Links: 1
Access: (0644/-rw-r--r--)  Uid: (383400030/ UNKNOWN)   Gid: (383400030/ UNKNOWN)
Access: 2022-09-13 12:16:20.000000000 +0200
Modify: 2022-09-13 12:21:32.000000000 +0200
Change: 2022-09-13 12:21:32.000000000 +0200

But when I setup a Motion correction job I get this error

[CPU: 217.6 MB]  Error occurred while processing J97/imported/016082983820181942327_FoilHole_10185797_Data_10180841_10180843_20191010_160357_Fractions.tif
Traceback (most recent call last):
  File "/d/hpc/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 60, in exec
    return self.process(item)
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 117, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 125, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "/d/hpc/software/cryosparc/cryosparc_worker/cryosparc_compute/blobio/tiff.py", line 65, in read_tiff
    mov = read_tiff_frames_threaded(path, num_threads)
  File "/d/hpc/software/cryosparc/cryosparc_worker/cryosparc_compute/blobio/tiff.py", line 44, in read_tiff_frames_threaded
    tif = libtiff.TIFF.open(tifpath, mode='r')
  File "/d/hpc/software/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/libtiff/libtiff_ctypes.py", line 501, in open
    raise TypeError('Failed to open file ' + repr(filename))
TypeError: Failed to open file b'/ab/ab/ab/cryosparc/P4/J97/imported/016082983820181942327_FoilHole_10185797_Data_10180841_10180843_20191010_160357_Fractions.tif'

Marking J97/imported/016082983820181942327_FoilHole_10185797_Data_10180841_10180843_20191010_160357_Fractions.tif as incomplete and continuing...

All the movies fail in this way. These same movies were once before imported and motion corrected on a different cluster so the movies themselves should be OK.

Any ideas? Is it a CS issue?

The movie import is performed on the master node and creates soft links to a path that is accessible by said node. Is the motion correction job being performed on the same node? If it’s being assigned to a separate worker/cluster node, does that node have access to the raw .tif files using the same path?

Hi @leetleyang,

I think you are right. It is a cluster and since I only mount this on the master it only works on the master.
Great, thank you!!

The SSHFS mount must be replicated on the workers with the same path, also the allow_other flag must be set.