Import movies located outside project directory tree fails

Troubleshooting
1

For import to work, do the movies need to be located under the same directory tree as the project dirs?

I just tried importing the same file from 3 different locations:

  • FSx: /fsx/movie.tif
  • S3: /mymountedS3bucket/movie.tif
  • scratch partition: /scratch/movie.tif

I cloned the movie import job so the parameters are identical.
File permissions + ownership are identical.
My workspace folders are created under /fsx

CryoSPARC loads the /fsx/movie.tif fine.

With the file on S3 or on /scratch, it throws the error:

Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 84, in cryosparc_compute.run.main
  File "/shared/cryosparc/cryosparc_worker/cryosparc_compute/jobs/imports/run.py", line 585, in run_import_movies_or_micrographs
    assert len(all_abs_paths) > 0, "No files match!"
AssertionError: No files match!

Is that expected behavior?

2

Just to be absolutely clear:

I cloned the movie import job so the parameters are identical.

identical except for the path to the movie! The job log shows the full path of the file it is trying to import and I double-check that it is correct in all 3 instances.

3

Please can somebody point me to a reference if my question is trivial? I have searched and tested for 2 days now and getting nowhere.

Here is the job.log with the error and the command that shows that the file cryosparc cannot open actually exists and is readable by the user running cryosparc:

imports.run cryosparc_compute.jobs.jobregister

TIFFOpen: /mnt/rawdata/21feb08b_gs_uw/21feb08b_00004sq_00002hl_00003en.frames.tif: No such file or directory.

***************************************************************

**** handle exception rc

set status to failed

========= main process now complete.

========= monitor process now complete.

The file cryoSPARC is failing to open exists and is readable by the cryosparc user:

[cryosparc-user]>$ ls -l /mnt/rawdata/21feb08b_gs_uw/21feb08b_00004sq_00002hl_00003en.frames.tif

-rwxr-xr-x 1 root root 313136052 Sep 27 23:03 /mnt/rawdata/21feb08b_gs_uw/21feb08b_00004sq_00002hl_00003en.frames.tif
4

The output from your “ls -l” command shows the file is owned by root, but with r-x permissions that should allow all users to read the file. Are you sure permissions are also set correctly for the /mnt/rawdata/21feb08b_gs_uw/ and /mnt/rawdata/ folders?

5

Thanks for replying, the /mnt/rawdata/21feb08b_gs_uw/ and /mnt/rawdata/ folders have drwxr-xr-x permissions and are also owned by root.

6

If not a problem with permissions, perhaps the file system or mount type can have an effect? Not something I’ve encountered, so I’m just guessing here.

7

Hi @iphan, were you able to get this to work? If not, to help me with further troubleshooting, can you try the following?

  1. Run the job again until it fails

  2. Copy the full job streamlog from the CryoSPARC web interface and send it here

  3. Get the full job debug output log from the command line (substitute PX and JY with the project and job numbers for the import job, e.g,. P1 and J1):

    cryosparcm joblog PX JY > job.log

    Send me the contents of the newly-created job.log file

  4. Add the following line to cryosparc_master/config.sh:

    export CRYOSPARC_DISABLE_IMPORT_ON_MASTER=true

  5. Restart cryoSPARC:

    cryosparcm stop
cryosparcm start

  6. Clear the job and repeat steps 1 - 3.

Let me know if you have any trouble with the above

1 Like
8

Hello @nfrasser , I was not able to fix it, thanks for helping.

My config file already had
export CRYOSPARC_DISABLE_IMPORT_ON_MASTER=true

doh…!

Setting this to ‘false’ allowed the upload from /mnt/rawdata to complete successfully, :roll_eyes:

Thank you so much.

1 Like