Job importing failed

yanhezhao · February 16, 2023, 10:47pm

Greeting there,

I came across importing job failed problem which is similar with post from 2 years ago, and after I follow the instruction, it still show same error as “Ubable to find data referenced by cs files”.
Here is how I did it:
tar -cvhf P17_J210_extract_micrographs_multi.tar ./P17_J210_extract_micrographs_multi (I run this command in the export/jobs folder of P17);
mkdir -p my_project/P18/imports/jobs;
tar -xvf P17_J210_extract_micrographs_multi.tar -C my_project/P18/imports/jobs (I run this command in the export/jobs folder of P17).
Was I wrong at any command running?

Thanks and best,
Yanhe

wtempel · February 17, 2023, 12:25am

@yanhezhao Is this a duplicate of Importing job failed - #4 by yanhezhao?

If this was the literal command you executed, in which directory did you execute that command?
If my_project was in fact a path beginning with /, would you like to send me a direct message with the output of
find my_project/P18/imports/jobs -ls
?

yanhezhao · February 17, 2023, 12:47am

Greeting there,

Yes, this is a duplicate of another post, sorry for that.
my_project is a subsitution of my absolute path, it is beginning with /, and below is how I run the command and when I run your command, the output said No such file or directory.

[s181081@NucleusC025 /] mkdir -p /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs [s181081@NucleusC025 /] find /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs ls
/project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs
find: ‘ls’: No such file or directory

Thanks to point me out any mistake I made.
Best,
Yanhe

wtempel · February 17, 2023, 1:34pm

Hi Yanhe,
Please can you re-run the find command with an additional “-” inserted:

find /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs -ls

Edit: Please also post:

the output of:
stat -f /full/path/to/exports/folder
(path to folder where you ran the tar commands)
names and contents of *csg files under P17_J210_extract_micrographs_multi

yanhezhao · February 17, 2023, 3:38pm

Good morning,
Thanks a lot for your favor firstly.

1, re-run the find -ls command

[s181081@NucleusC019 /]$ mkdir -p /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs
[s181081@NucleusC019 /]$ find /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs -ls
144120761651560451    4 drwxr-xr-x   2 s181081  Nicastro_lab     4096 Feb 17 15:26 /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs

2, stat -f command output

[s181081@NucleusC019 /]$ stat -f /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P17/exports/jobs
  File: "/project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P17/exports/jobs"
    ID: 20ea7bcc00000000 Namelen: 255     Type: lustre
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 949367607534 Free: 85124964946 Available: 75535295620
Inodes: Total: 1691910144 Free: 821940434

3, names and contents under the export folder

[s181081@NucleusC019 P17_J210_extract_micrographs_multi]$ ls
events.bson  gridfs_data  job.json  P17_J210_micrographs  P17_J210_particles
[s181081@NucleusC019 P17_J210_extract_micrographs_multi]$ cd P17_J210_micrographs/
[s181081@NucleusC019 P17_J210_micrographs]$ ls
J1  J5   J85  P17_J210_micrographs_exported.cs
J4  J84  J87  P17_J210_micrographs_exported.csg
[s181081@NucleusC019 P17_J210_micrographs]$ cd ../P17_J210_particles/
[s181081@NucleusC019 P17_J210_particles]$ ls
J210  J4  J85  P17_J210_particles_exported.cs  P17_J210_particles_exported.csg

Thanks again for your time,
Yanhe

yanhezhao · February 17, 2023, 3:47pm

Greeting there,

Sorry for the format porblem during my post.
I just attached one screen shot to clarify the names and contents under the exported folder.

Thanks and cheers,
Yanhe

wtempel · February 17, 2023, 4:22pm

yanhezhao:

[s181081@NucleusC019 /]$ find /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs -ls
144120761651560451    4 drwxr-xr-x   2 s181081  Nicastro_lab     4096 Feb 17 15:26 /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs

I am confused: After

and

I would have expected the contents of P17_J210_extract_micrographs_multi.tar inside

/project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs

Did I confuse any paths? Does P17_J210_extract_micrographs_multi.tar contain the expected files:

tar tvf P17_J210_extract_micrographs_multi.tar

yanhezhao · February 17, 2023, 4:44pm

Greeting there,

You are totally correct, the P17_J210_extract_micrographs_multi.tar was inside P18/imports/jobs.
I just deleted it from the import/jobs folder when I realized it doesn’t work.
Let me redo the copy command and run the find -ls again and post you the output again.

Thanks for your patient,
Yanhe

yanhezhao · February 17, 2023, 7:58pm

Greeting there,

When I redo the extract command as below, something I thought weird happened, as it was built the absolute path of /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P17/exports/jobs/P17_J210_extract_micrographs_multi under the P18/imports/jobs

command to run:
tar -xvf P17_J210_extract_micrographs_multi.tar -C /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs

output example after find -ls
144120761651608170 12 -rw-r–r-- 1 s181081 Nicastro_lab 9736 Oct 27 23:53 /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs/project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P17/exports/jobs/P17_J210_extract_micrographs_multi/P17_J210_micrographs/J4/thumbnails/008319023014213123656_0820_08.53.34_thumb_@1x.png

Thanks and best,
Yanhe

wtempel · February 17, 2023, 8:23pm

Was your tar file created exactly as described earlier?

If, on the other hand, your archive was created with the full paths to inputs, you can strip leading components during unpacking.

yanhezhao · February 17, 2023, 9:58pm

Good afternoon,

Sorry for my late reply. Here are details:

tar -cvhf P17_J210_extract_micrographs_multi.tar ./P17_J210_extract_micrographs_multi (run this command in the export/jobs folder of P17);

mkdir -p imports/jobs (run this command in the folder of P18);

tar -xvf P17_J210_extract_micrographs_multi.tar -C /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P18/imports/jobs (run this command in the export/jobs folder of P17).

find -ls (run this command in the imports/jobs folder of P18) and output example as below
144120761651629665 40 -rw-r–r-- 1 s181081 Nicastro_lab 39998 Oct 27 15:25 ./P17_J210_extract_micrographs_multi/P17_J210_micrographs/J4/thumbnails/007293578341987832271_0130_20.11.21_thumb_@2x.png

Thanks and cheers,
Yanhe

yanhezhao · February 21, 2023, 5:54pm

Greeting,

Hopefully you have a great long weekend.
I had attached details. It still report same error “Ubable to find data referenced by cs files”.

Thanks in advance for your help,
Yanhe

wtempel · February 21, 2023, 7:50pm

This is not the the content I expected.
Inside the following directories:

the original job directory
the exports/jobs directory of the original project

please run
pwd && find -ls
and post the output. Please also post the output of
tar tvf P17_J210_extract_micrographs_multi.tar

yanhezhao · February 21, 2023, 8:13pm

Good afternoon,

Attached please see output. find -ls and tar tvf give large output so I just post example here.

1, the original job directory
pwd /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P17/J210
find -ls example
144120742978621196 38284 -rw-rw-r-- 1 s181081 Nicastro_lab 39201024 Dec 28 21:32 ./extract/013296433599356144510_05315_09.39.53_patch_aligned_doseweighted_particles.mrc

2, the exports/jobs directory of the original project
pwd /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data/cryoSPARC/P17/exports/jobs
find -ls example
144120761685119956 4 lrwxrwxrwx 1 s181081 Nicastro_lab 148 Feb 17 20:29 ./P17_J210_extract_micrographs_multi/P17_J210_micrographs/J84/imported/004043529215483168151_04019_15.19.02.mrc → /project/cellbiology/Nicastro_lab/nusr/yanhe/linux/data /cryoSPARC/P17/J84/imported/004043529215483168151_04019_15.19.02.mrc

3, tar tvf P17_J210_extract_micrographs_multi.tar example
-rw-rw-r-- s181081/Nicastro_lab 15681024 2022-12-28 21:28 ./P17_J210_extract_micrographs_multi/P17_J210_particles/J210/extract/012065229037015907991_04226_17.15.49_patch_aligned_doseweighted_particles.mrc

Thanks and best,
Yanhe

yanhezhao · February 22, 2023, 5:06pm

Good morning,

I’m still struggling with the problem.
As attached shown: after the three command running, the folder (not the link) is under the imports folder of new project P18 and cs files for micrographs and particles are also there.
But when I import it still could not find the cs file.
Could you please point me out?

Thanks and best,
Yanhe

wtempel · February 22, 2023, 8:31pm

Please can you email us a compressed copy of
/path/to/cryosparc_master/run/command_core.log?

wtempel · February 23, 2023, 5:47pm

If there is a file
/path/to/P18/J196/job.log,
please can you compress it and email it to us.

yanhezhao · February 23, 2023, 6:23pm

Greeting there,

P18/J196 is newest job I built yesterday for Patch CTF, but I already deleted it because of wrong setting.

Thanks and best,
Yanhe

wtempel · February 23, 2023, 9:56pm

Please can you re-try the job import and look out for a new job that should be created in the workspace.
An actual path to a missing file should be shown at the of the event log (Overview tab on pre-v4 CryoSPARC) of that newly created job.

yanhezhao · February 23, 2023, 10:46pm

Greeting there,

Yes, I had it re-try.

It can import particles but not micrographs. And when I check the overview it point me clearly that unable find mrc files in the import movies job.
I had my movies copied and deleted to save space after I done the motion correction and CTF estimation.

Is there a way to save it?

Thanks and cheers,
Yanhe