Hi CryoSparc Team:
There is a error when user using cryosparc Patch CTF.
I hope you can help find a solution. Thanks！
Error occurred while processing J2/motioncorrected/007336134006241569583_FoilHole_12814529_Data_12812360_12812362_20250513_143941_EER_patch_aligned.mrc
Traceback (most recent call last):
File "/work/home/admin1/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 59, in exec
return self.process(item)
File "cryosparc_master/cryosparc_compute/jobs/ctf_estimation/run.py", line 137, in cryosparc_master.cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process
ValueError: Encountered non-finite values in result PSA_trim for J2/motioncorrected/007336134006241569583_FoilHole_12814529_Data_12812360_12812362_20250513_143941_EER_patch_aligned.mrc
Marking J2/motioncorrected/007336134006241569583_FoilHole_12814529_Data_12812360_12812362_20250513_143941_EER_patch_aligned.mrc as incomplete and continuing...
And below are some logs for you. If needed, the user will provide some detailed information about the data.
[admin1@login01 ~]$ cryosparcm joblog P6 J3 | tail -n 40
to: 2.523564244260882
/work/home/admin1/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py:59: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
return self.process(item)
HOST ALLOCATION FUNCTION: using numba.cuda.pinned_array
changing fit 20.390399093627927
to: 2.5810631764085987
changing fit 16.71344188002289
to: 4.0297231410331875
changing fit 16.71344188002289
to: 4.110967559199179
changing fit 20.390399093627927
to: 2.529826190276418
/work/home/admin1/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py:59: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
return self.process(item)
HOST ALLOCATION FUNCTION: using numba.cuda.pinned_array
changing fit 20.390399093627927
to: 2.733297465633771
changing fit 16.71344188002289
to: 4.195555369059245
/work/home/admin1/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py:59: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
return self.process(item)
HOST ALLOCATION FUNCTION: using numba.cuda.pinned_array
changing fit 20.390399093627927
to: 2.7704346594603164
changing fit 16.71344188002289
to: 4.078079818725586
changing fit 16.71344188002289
to: 4.265773868959818
changing fit 20.390399093627927
to: 2.56806033924785
========= sending heartbeat at 2025-05-14 10:12:36.936609
========= heartbeat failed at 2025-05-14 10:12:36.942421:
***************************************************************
========= main process now complete at 2025-05-14 10:12:36.798375
Total: 697.160s
MAIN THREAD:
[admin1@login01 ~]$ cryosparcm eventlog P6 J3 | tail -n 40
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 480 MB] Loaded output dset with 954 items
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 480 MB] Passthrough results ['micrograph_blob_non_dw', 'micrograph_blob', 'mscope_params', 'micrograph_blob_non_dw_AB', 'micrograph_thumbnail_blob_1x', 'micrograph_thumbnail_blob_2x', 'movie_blob', 'background_blob', 'rigid_motion', 'spline_motion', 'gain_ref_blob']
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Loaded passthrough dset with 1029 items
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Intersection of output and passthrough has 954 items
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Output dataset contains: ['micrograph_thumbnail_blob_2x', 'gain_ref_blob', 'micrograph_blob_non_dw', 'rigid_motion', 'micrograph_thumbnail_blob_1x', 'movie_blob', 'micrograph_blob', 'background_blob', 'micrograph_blob_non_dw_AB', 'mscope_params', 'spline_motion']
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Outputting passthrough result micrograph_blob_non_dw
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Outputting passthrough result micrograph_blob
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Outputting passthrough result mscope_params
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Outputting passthrough result micrograph_blob_non_dw_AB
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Outputting passthrough result micrograph_thumbnail_blob_1x
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Outputting passthrough result micrograph_thumbnail_blob_2x
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Outputting passthrough result movie_blob
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Outputting passthrough result background_blob
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Outputting passthrough result rigid_motion
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Outputting passthrough result spline_motion
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Outputting passthrough result gain_ref_blob
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Passing through outputs for output group exposures_incomplete from input group exposures
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] This job outputted results ['ctf']
[Wed, 14 May 2025 02:12:35 GMT] [CPU RAM used: 463 MB] Loaded output dset with 75 items
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 463 MB] Passthrough results ['micrograph_blob_non_dw', 'micrograph_blob', 'mscope_params', 'micrograph_blob_non_dw_AB', 'micrograph_thumbnail_blob_1x', 'micrograph_thumbnail_blob_2x', 'movie_blob', 'background_blob', 'rigid_motion', 'spline_motion', 'gain_ref_blob']
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 463 MB] Loaded passthrough dset with 1029 items
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 463 MB] Intersection of output and passthrough has 75 items
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 463 MB] Output dataset contains: ['micrograph_thumbnail_blob_2x', 'gain_ref_blob', 'micrograph_blob_non_dw', 'rigid_motion', 'micrograph_thumbnail_blob_1x', 'movie_blob', 'micrograph_blob', 'background_blob', 'micrograph_blob_non_dw_AB', 'mscope_params', 'spline_motion']
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 463 MB] Outputting passthrough result micrograph_blob_non_dw
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 463 MB] Outputting passthrough result micrograph_blob
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 463 MB] Outputting passthrough result mscope_params
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 463 MB] Outputting passthrough result micrograph_blob_non_dw_AB
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 463 MB] Outputting passthrough result micrograph_thumbnail_blob_1x
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 463 MB] Outputting passthrough result micrograph_thumbnail_blob_2x
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 464 MB] Outputting passthrough result movie_blob
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 464 MB] Outputting passthrough result background_blob
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 464 MB] Outputting passthrough result rigid_motion
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 464 MB] Outputting passthrough result spline_motion
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 464 MB] Outputting passthrough result gain_ref_blob
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 464 MB] Checking outputs for output group exposures
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 465 MB] Checking outputs for output group exposures_incomplete
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 465 MB] Updating job size...
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 466 MB] Exporting job and creating csg files...
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 466 MB] ***************************************************************
[Wed, 14 May 2025 02:12:36 GMT] [CPU RAM used: 466 MB] Job complete. Total time 695.82s
[admin1@login01 ~]$ cryosparcm cli "get_job('P6', 'J3', 'job_type', 'version', 'instance_information', 'status', 'params_spec', 'errors_run')"
{'_id': '6823ed6ec49bfe29f2c26073', 'errors_run': [], 'instance_information': {'CUDA_version': '11.8', 'available_memory': '491.88GB', 'cpu_model': 'Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz', 'driver_version': '12.8', 'gpu_info': [{'id': 0, 'mem': 47709159424, 'name': 'NVIDIA A40', 'pcie': '0000:17:00'}, {'id': 1, 'mem': 47709159424, 'name': 'NVIDIA A40', 'pcie': '0000:65:00'}, {'id': 2, 'mem': 47709159424, 'name': 'NVIDIA A40', 'pcie': '0000:ca:00'}, {'id': 3, 'mem': 47709159424, 'name': 'NVIDIA A40', 'pcie': '0000:e3:00'}], 'ofd_hard_limit': 262144, 'ofd_soft_limit': 262144, 'physical_cores': 48, 'platform_architecture': 'x86_64', 'platform_node': 'gpu03', 'platform_release': '4.18.0-513.5.1.el8_9.x86_64', 'platform_version': '#1 SMP Fri Nov 17 03:31:10 UTC 2023', 'total_memory': '502.82GB', 'used_memory': '6.87GB'}, 'job_type': 'patch_ctf_estimation_multi', 'params_spec': {'compute_num_gpus': {'value': 4}}, 'project_uid': 'P6', 'status': 'completed', 'uid': 'J3', 'version': 'v4.7.0'}