Hi!
I have been processing my data on a single workstation setup, but recently decided to move to a cluster. I have two datasets that I moved to the cluster using the following command:
rsync -rltv /home/szabolcs/directory szabolcs@***:/home/szabolcs
In dataset 1, there should be 16.056 micrographs in TIFF format, and on my workstation (running Cryosparc 4.3.0, no patch applied) I was able to import all of them with the job Import Movies as expected. Here is an example file path and the path I used in the job as the Movies data path:
/home/szabolcs/cem_00535_00062/k3_frames_a/Images-Disc1/GridSquare_25523892/Data/FoilHole_27691559_Data_25530222_25530224_20230827_080032_fractions.tiff
/home/szabolcs/cem_00535_00062/k3_frames_a/Images-Disc1/*/Data/*.tiff
The cluster is running Cryosparc 4.3.1, and when trying to import the movies from either dataset, I get way less micrographs than expected (6.528 in the case of dataset 1). The absolute paths of the dataset directories (and the Movies data path given to the jobs) match. I have compared the size of the directories and they match; I have re-run the rsync command and it did not transfer any additional files; and I have run
find -type f -name "*.tiff" | head
in the dataset directory, which returned 16056 on both machines. Checking the head and the tail of the list of TIFF files shows the same absolute path for all files (except different GridSquare directories).
I have tried restarting the Import Movies on both instances, the workstation still sees all micrographs and the cluster still does not. I have replaced the more specific subdirectory names with wildcards with no effect.
Am I missing something, or could this be a bug in 4.3.1? I could try confirming this by updating the single workstation and restarting the job, but I would rather avoid that.