CTFFind4 jobs fail

solved

#1

Hi,

I am running T20S tutorial dataset in cryosparc V2.3.2. The CTFFind jobs fail. The following is the output. Do I need to install CTFFind for this, if so how do I specify the path in cryosparc ?

Processed 0 of 20 micrographs in 0.03s

=== Running CTFFIND for micrograph 000000 ( J4/motioncorrected/14sep05c_00024sq_00003hl_00002es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_00024sq_00003hl_00002es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000002 ( J4/motioncorrected/14sep05c_00024sq_00004hl_00002es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_00024sq_00004hl_00002es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000001 ( J4/motioncorrected/14sep05c_00024sq_00003hl_00005es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_00024sq_00003hl_00005es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000003 ( J4/motioncorrected/14sep05c_00024sq_00006hl_00003es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_00024sq_00006hl_00003es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000004 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00002hl_00005es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00002hl_00005es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000006 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00005hl_00002es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00005hl_00002es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000005 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00004hl_00004es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00004hl_00004es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000007 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00005hl_00003es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00005hl_00003es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000008 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00005hl_00005es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00005hl_00005es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000009 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00006hl_00002es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00006hl_00002es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000010 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00006hl_00003es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00006hl_00003es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000011 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00006hl_00005es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00006hl_00005es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000012 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00007hl_00004es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00007hl_00004es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000013 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00007hl_00005es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00007hl_00005es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000014 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00008hl_00005es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00008hl_00005es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000015 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00009hl_00004es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00009hl_00004es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000016 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00010hl_00002es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00010hl_00002es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000017 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00011hl_00002es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00011hl_00002es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000018 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00011hl_00003es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00011hl_00003es.frames_rigid_aligned_ctffind.log

=== Running CTFFIND for micrograph 000019 ( J4/motioncorrected/14sep05c_c_00003gr_00014sq_00011hl_00004es.frames_rigid_aligned.mrc )
CTFFIND did not run correctly. See the log at /home/rakesh/WORK/Projects/4-SPOTSComplex/SParticleProcess/cryoSPARC/P2/J6/ctffind_output/14sep05c_c_00003gr_00014sq_00011hl_00004es.frames_rigid_aligned_ctffind.log

Completed CTF Estimation using CTFFIND.

0 micrographs successfully completed.

20 micrographs failed.

Total time: 0.32s


Compiling job outputs…

Passing through outputs for output group micrographs_success from input group micrographs

This job outputted results [‘ctf’, ‘ctf_plotdata’]

Loaded output dset with 0 items

Passthrough results [‘micrograph_blob’, ‘mscope_params’, ‘background_blob’, ‘rigid_motion’, ‘movie_blob’, ‘gain_ref_blob’]

Loaded passthrough dset with 20 items

Intersection of output and passthrough has 0 items

Passing through outputs for output group micrographs_fail from input group micrographs

This job outputted results [‘ctf’, ‘ctf_plotdata’]

Loaded output dset with 20 items

Passthrough results [‘micrograph_blob’, ‘mscope_params’, ‘background_blob’, ‘rigid_motion’, ‘movie_blob’, ‘gain_ref_blob’]

Loaded passthrough dset with 20 items

Intersection of output and passthrough has 20 items

Checking outputs for output group micrographs_success

Checking outputs for output group micrographs_fail


Job complete. Total time 0.61s

Outputs


#2

Hi @Biorak,

As of v2.3.2, CTFFIND 4.1.8 is bundled with cryoSPARC. You don’t have to install it. You can find the binaries at cryosparc2_worker/deps/external/ctffind/bin
Is it possible if you can post the output of one of the logs created by CTFFIND?


#3

/home/rakesh/WORK/Softwares/CryoEM_Fitting/cryosparcV2/cryosparc2_worker/deps/external/ctffind/bin/ctffind: error while loading shared libraries: libtiff.so.3: cannot open shared object file: No such file or directory

Is it not possibel to upload text or zip files ?


#4

#5

We seem to be having a similar issue with v2.6.4 the ctffind job fails using the test dataset but complains that some file can’t be found?


Processed 0 of 20 micrographs in 0.06s

=== Running CTFFIND for micrograph 000001 ( J2/motioncorrected/14sep05c_00024sq_00003hl_00005es.frames_rigid_aligned.mrc )
Running thread Thread-1…

=== Running CTFFIND for micrograph 000000 ( J2/motioncorrected/14sep05c_00024sq_00003hl_00002es.frames_rigid_aligned.mrc )
Running thread Thread-2…

=== Running CTFFIND for micrograph 000002 ( J2/motioncorrected/14sep05c_00024sq_00004hl_00002es.frames_rigid_aligned.mrc )
Running thread Thread-3…

=== Running CTFFIND for micrograph 000004 ( J2/motioncorrected/14sep05c_c_00003gr_00014sq_00002hl_00005es.frames_rigid_aligned.mrc )
Running thread Thread-5…

=== Running CTFFIND for micrograph 000003 ( J2/motioncorrected/14sep05c_00024sq_00006hl_00003es.frames_rigid_aligned.mrc )
Running thread Thread-4…

=== Running CTFFIND for micrograph 000006 ( J2/motioncorrected/14sep05c_c_00003gr_00014sq_00005hl_00002es.frames_rigid_aligned.mrc )
Running thread Thread-7…

=== Running CTFFIND for micrograph 000005 ( J2/motioncorrected/14sep05c_c_00003gr_00014sq_00004hl_00004es.frames_rigid_aligned.mrc )
Running thread Thread-6…

=== Running CTFFIND for micrograph 000007 ( J2/motioncorrected/14sep05c_c_00003gr_00014sq_00005hl_00003es.frames_rigid_aligned.mrc )
Running thread Thread-8…

Traceback (most recent call last):
File “cryosparc2_compute/jobs/ctf/run.py”, line 173, in work
resample=params[‘resample’]
File “cryosparc2_compute/jobs/ctf/ctf_wrapper.py”, line 398, in run_ctf_wrapper
**kwargs)
File “cryosparc2_compute/jobs/ctf/ctf_wrapper.py”, line 375, in run_ctffind
proc = rc.start_external_process([ctffind_bin_path_abs], log_path_abs) # starts and registers subprocess
File “cryosparc2_compute/jobs/runcommon.py”, line 671, in start_external_process
process = com.start_external_process(command, log_file_path)
File “cryosparc2_compute/jobs/common.py”, line 200, in start_external_process
close_fds = True ) # NOTES close_fds is critical! otherwise child process will keep the webserver’s sockets open and cause all kinds of problems!
File “/home/cryosparc_user/cryosparc2_worker/deps/anaconda/lib/python2.7/subprocess.py”, line 394, in init
errread, errwrite)
File “/home/cryosparc_user/cryosparc2_worker/deps/anaconda/lib/python2.7/subprocess.py”, line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

=== Running CTFFIND for micrograph 000008 ( J2/motioncorrected/14sep05c_c_00003gr_00014sq_00005hl_00005es.frames_rigid_aligned.mrc )
Running thread Thread-3…

Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main
File “cryosparc2_compute/jobs/ctf/run.py”, line 231, in run_ctf_estimation
pool.map(work, selected_exposure_idxs)
File “/home/cryosparc_user/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/pool.py”, line 253, in map
return self.map_async(func, iterable, chunksize).get()
File “/home/cryosparc_user/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/pool.py”, line 572, in get
raise self._value
OSError: [Errno 2] No such file or directory

all the logs are 0b files; but the ctffind bins are in the correct path:
[cryosparc_user@gpu1 root]$ ls /home/cryosparc_user/cryosparc2_worker/deps/ext/ctffind/bin -lah
total 28M
drwxr-xr-x 2 root root 52 Dec 13 09:35 .
drwxrwxr-x 4 cryosparc_user cryosparc_user 55 Dec 13 09:35 …
-rwxr-xr-x 1 cryosparc_user cryosparc_user 28M Jun 7 2017 ctffind
-rwxr-xr-x 1 cryosparc_user cryosparc_user 4.9K Jun 7 2017 ctffind_plot_results.sh


CTF estimation job fails
#6

Hi @nis_it,

Actually, the path you mentioned seems to be the path to the CTFFIND4 binary from a previous version of cryoSPARC (before the update to our dependency manager). Is it possible that an update to your cryosparc2_worker instance somehow failed mid way?
The correct path should be:

sarulthasan@cryoem5:/u/cryosparcprod/production_cryosparc/cryosparc2_worker/deps/external/ctffind-4.1.10/bin$ ll
total 31736
drwxrwxr-x 2 cryosparcprod cryosparcprod        4 Oct 12 16:34 ./
drwxrwxr-x 3 cryosparcprod cryosparcprod        3 Oct 12 16:34 ../
-rwxr-xr-x 1 cryosparcprod cryosparcprod 31545286 Oct 12 16:34 ctffind*
-rwxr-xr-x 1 cryosparcprod cryosparcprod     4963 Oct 12 16:34 ctffind_plot_results.sh*

You can copy the binaries into this directory and it should work again.
I’m currently working on a feature that will allow a user to selectively reinstall dependencies, so until that is released, you might have issues with GCTF as well. A dirty method of fixing this issue is to delete your cryosparc2_worker directory and reinstall it using the “Installation: Standalone Worker” documentation here.


CTF estimation job fails
#7

It must have had some issue with the upgrade, but there was no error messages during the upgrade, as the version of ctffind4 is 4.1.8 not 4.1.10; i created the path and copied the 4.1.10 binary and the job completed correctly.


#8

I had the same issue after updating and when I checked the path, it was correct.
I just fixed this issue by doing a Forced Update for which the instructions are written in the Installation and management guide.


#9

Dear all,

I have installed a recent version of cryosparc and while running the T20 tutorial data and I got the same (CTFFFind4 job fail) error with a log as shown in the screenshot. I carried a forced update and also checked the path of CCTF4 (I hope it is correctly assigned ) (see the screenshot) and I have all the necessary libtiff files.

running on OpenSuse 15.0, Cuda 10.0 with the latest version of cryosparc; installed with a stand-alone option.

Any suggestions are welcome,


#10

Hi @eswar909,

In the streamlog, you can see CTFFIND4 outputted a message with a path to its error log. Can you check that file and see what it says?


#11

Hi@sarulthasan

Please have a look in to the error log

**** handle exception rc
set status to failed
Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main
File “cryosparc2_compute/jobs/ctf/run.py”, line 271, in run_ctf_estimation
rc.output(‘micrographs_fail’, ‘ctf’, outpath_rel, 0, len(exposures_fail))
File “cryosparc2_compute/jobs/runcommon.py”, line 251, in output
result = com.query(job[‘output_results’], lambda r: r[‘group_name’] == group_name and r[‘name’] == name, error=“No output result named %s.%s in job.” % (group_name, name))
File “cryosparc2_compute/jobs/common.py”, line 265, in query
assert res != default, error
AssertionError: No output result named micrographs_fail.ctf in job.
========= main process now complete.
========= monitor process now complete.
Best,
Eswar


#12

Hey @eswar909,

Is this the output of the file /home/takagilab/cryosparc_data/eswar/P2/J30/ctffind_output/14sep05...._rigid_aligned_ctffind.log? or the output of cryosparcm joblog P2 J30?


#13

Hi @sarulthasan
Unfortunately I deleted Job 30, but every time I ran CTF correction I got the same error, for your convenience I am sending the screenshot of Job42 and its log file

/home/takagilab/cryosparc_data/eswar/P2/J26/motioncorrected/14sep05c_c_00003gr_00014sq_00011hl_00003es.frames_rigid_aligned.mrc
/home/takagilab/cryosparc_data/eswar/P2/J42/ctffind_output/a165ef_14sep05c_c_00003gr_00014sq_00011hl_00003es.frames_rigid_aligned_ctffind.mrc
0.6575
300.0
2.7
0.07
512
30.0
5.0
5000.0
50000.0
500.0
no
yes
yes
200.0
no
no
**** handle exception rc
set status to failed
Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main
File “cryosparc2_compute/jobs/ctf/run.py”, line 271, in run_ctf_estimation
rc.output(‘micrographs_fail’, ‘ctf’, outpath_rel, 0, len(exposures_fail))
File “cryosparc2_compute/jobs/runcommon.py”, line 251, in output
result = com.query(job[‘output_results’], lambda r: r[‘group_name’] == group_name and r[‘name’] == name, error=“No output result named %s.%s in job.” % (group_name, name))
File “cryosparc2_compute/jobs/common.py”, line 265, in query
assert res != default, error
AssertionError: No output result named micrographs_fail.ctf in job.
========= main process now complete.
========= monitor process now complete.

Best,
Eswar


#14

Hi @sarulthasan,

I assume the error that I got has to do with OS (Open Suse) that we installed. Based on Cryosparc manual and ccpem bulletin board people suggestions we Installed CentOS7 and running cryosparc quite efficiently without any errors.

Many thanks for your help,
Eswar.


Does cryosparc run on OpenSUSE?