Job importing failed

Greeting there,

I came across importing job failed problem which is similar with post from 2 years ago, and after I follow the instruction, it still show same error as “Ubable to find data referenced by cs files”.
Here is how I did it:
tar -cvhf P17_J210_extract_micrographs_multi.tar ./P17_J210_extract_micrographs_multi (I run this command in the export/jobs folder of P17);
mkdir -p my_project/P18/imports/jobs;
tar -xvf P17_J210_extract_micrographs_multi.tar -C my_project/P18/imports/jobs (I run this command in the export/jobs folder of P17).
Was I wrong at any command running?

Thanks and best,
Yanhe

@yanhezhao Is this a duplicate of Importing job failed - #4 by yanhezhao?

If this was the literal command you executed, in which directory did you execute that command?
If my_project was in fact a path beginning with /, would you like to send me a direct message with the output of
find my_project/P18/imports/jobs -ls
?

Greeting there,

Yes, this is a duplicate of another post, sorry for that.
my_project is a subsitution of my absolute path, it is beginning with /, and below is how I run the command and when I run your command, the output said No such file or directory.

[s181081@NucleusC025 /] mkdir -p /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs [s181081@NucleusC025 /] find /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs ls
/project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs
find: ‘ls’: No such file or directory

Thanks to point me out any mistake I made.
Best,
Yanhe

Hi Yanhe,
Please can you re-run the find command with an additional “-” inserted:

find /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs -ls

Edit: Please also post:

  • the output of:
    stat -f /full/path/to/exports/folder
    (path to folder where you ran the tar commands)
  • names and contents of *csg files under P17_J210_extract_micrographs_multi

Good morning,
Thanks a lot for your favor firstly.

1, re-run the find -ls command

[s181081@NucleusC019 /]$ mkdir -p /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs
[s181081@NucleusC019 /]$ find /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs -ls
144120761651560451    4 drwxr-xr-x   2 s181081  Nicastro_lab     4096 Feb 17 15:26 /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs

2, stat -f command output

[s181081@NucleusC019 /]$ stat -f /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P17/exports/jobs
  File: "/project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P17/exports/jobs"
    ID: 20ea7bcc00000000 Namelen: 255     Type: lustre
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 949367607534 Free: 85124964946 Available: 75535295620
Inodes: Total: 1691910144 Free: 821940434

3, names and contents under the export folder

[s181081@NucleusC019 P17_J210_extract_micrographs_multi]$ ls
events.bson  gridfs_data  job.json  P17_J210_micrographs  P17_J210_particles
[s181081@NucleusC019 P17_J210_extract_micrographs_multi]$ cd P17_J210_micrographs/
[s181081@NucleusC019 P17_J210_micrographs]$ ls
J1  J5   J85  P17_J210_micrographs_exported.cs
J4  J84  J87  P17_J210_micrographs_exported.csg
[s181081@NucleusC019 P17_J210_micrographs]$ cd ../P17_J210_particles/
[s181081@NucleusC019 P17_J210_particles]$ ls
J210  J4  J85  P17_J210_particles_exported.cs  P17_J210_particles_exported.csg

Thanks again for your time,
Yanhe

Greeting there,

Sorry for the format porblem during my post.
I just attached one screen shot to clarify the names and contents under the exported folder.

Thanks and cheers,
Yanhe

I am confused: After

and

I would have expected the contents of P17_J210_extract_micrographs_multi.tar inside

/project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs

Did I confuse any paths? Does P17_J210_extract_micrographs_multi.tar contain the expected files:

tar tvf P17_J210_extract_micrographs_multi.tar

Greeting there,

You are totally correct, the P17_J210_extract_micrographs_multi.tar was inside P18/imports/jobs.
I just deleted it from the import/jobs folder when I realized it doesn’t work.
Let me redo the copy command and run the find -ls again and post you the output again.

Thanks for your patient,
Yanhe

Greeting there,

When I redo the extract command as below, something I thought weird happened, as it was built the absolute path of /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P17/exports/jobs/P17_J210_extract_micrographs_multi under the P18/imports/jobs

command to run:
tar -xvf P17_J210_extract_micrographs_multi.tar -C /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs

output example after find -ls
144120761651608170 12 -rw-r–r-- 1 s181081 Nicastro_lab 9736 Oct 27 23:53 /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs/project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P17/exports/jobs/P17_J210_extract_micrographs_multi/P17_J210_micrographs/J4/thumbnails/008319023014213123656_0820_08.53.34_thumb_@1x.png

Thanks and best,
Yanhe

Was your tar file created exactly as described earlier?

If, on the other hand, your archive was created with the full paths to inputs, you can strip leading components during unpacking.

Good afternoon,

Sorry for my late reply. Here are details:

tar -cvhf P17_J210_extract_micrographs_multi.tar ./P17_J210_extract_micrographs_multi (run this command in the export/jobs folder of P17);

mkdir -p imports/jobs (run this command in the folder of P18);

tar -xvf P17_J210_extract_micrographs_multi.tar -C /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs (run this command in the export/jobs folder of P17).

find -ls (run this command in the imports/jobs folder of P18) and output example as below
144120761651629665 40 -rw-r–r-- 1 s181081 Nicastro_lab 39998 Oct 27 15:25 ./P17_J210_extract_micrographs_multi/P17_J210_micrographs/J4/thumbnails/007293578341987832271_0130_20.11.21_thumb_@2x.png

Thanks and cheers,
Yanhe

Greeting,

Hopefully you have a great long weekend.
I had attached details. It still report same error “Ubable to find data referenced by cs files”.

Thanks in advance for your help,
Yanhe

This is not the the content I expected.
Inside the following directories:

  1. the original job directory
  2. the exports/jobs directory of the original project

please run
pwd && find -ls
and post the output. Please also post the output of
tar tvf P17_J210_extract_micrographs_multi.tar

Good afternoon,

Attached please see output. find -ls and tar tvf give large output so I just post example here.

1, the original job directory
pwd /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P17/J210
find -ls example
144120742978621196 38284 -rw-rw-r-- 1 s181081 Nicastro_lab 39201024 Dec 28 21:32 ./extract/013296433599356144510_05315_09.39.53_patch_aligned_doseweighted_particles.mrc

2, the exports/jobs directory of the original project
pwd /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P17/exports/jobs
find -ls example
144120761685119956 4 lrwxrwxrwx 1 s181081 Nicastro_lab 148 Feb 17 20:29 ./P17_J210_extract_micrographs_multi/P17_J210_micrographs/J84/imported/004043529215483168151_04019_15.19.02.mrc → /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data /cryoSPARC/P17/J84/imported/004043529215483168151_04019_15.19.02.mrc

3, tar tvf P17_J210_extract_micrographs_multi.tar example
-rw-rw-r-- s181081/Nicastro_lab 15681024 2022-12-28 21:28 ./P17_J210_extract_micrographs_multi/P17_J210_particles/J210/extract/012065229037015907991_04226_17.15.49_patch_aligned_doseweighted_particles.mrc

Thanks and best,
Yanhe

Good morning,

I’m still struggling with the problem.
As attached shown: after the three command running, the folder (not the link) is under the imports folder of new project P18 and cs files for micrographs and particles are also there.
But when I import it still could not find the cs file.
Could you please point me out?

Thanks and best,
Yanhe

Please can you email us a compressed copy of
/path/to/cryosparc_master/run/command_core.log?

If there is a file
/path/to/P18/J196/job.log,
please can you compress it and email it to us.

Greeting there,

P18/J196 is newest job I built yesterday for Patch CTF, but I already deleted it because of wrong setting.

Thanks and best,
Yanhe

Please can you re-try the job import and look out for a new job that should be created in the workspace.
An actual path to a missing file should be shown at the of the event log (Overview tab on pre-v4 CryoSPARC) of that newly created job.

Greeting there,

Yes, I had it re-try.

It can import particles but not micrographs. And when I check the overview it point me clearly that unable find mrc files in the import movies job.
I had my movies copied and deleted to save space after I done the motion correction and CTF estimation.

Is there a way to save it?

Thanks and cheers,
Yanhe