Encountered this fatal error in 3D classification:
========= sending heartbeat at 2024-08-16 22:33:33.115068
<string>:1: UserWarning: Kernel function check_error_for_nans called with very small array <cryosparc_compute.gpu.gpuarray.GPUArray object at 0x7f89b2b98130> (size 2). Array will be passed to kernel as a pointer. Consider modifying the kernel to accept individual scalar arguments instead.
========= sending heartbeat at 2024-08-16 22:33:43.132017
========= sending heartbeat at 2024-08-16 22:33:53.141866
========= sending heartbeat at 2024-08-16 22:34:03.160272
========= sending heartbeat at 2024-08-16 22:34:13.179260
========= sending heartbeat at 2024-08-16 22:34:23.196325
========= sending heartbeat at 2024-08-16 22:34:33.209772
========= sending heartbeat at 2024-08-16 22:34:43.227115
========= sending heartbeat at 2024-08-16 22:34:53.245018
========= sending heartbeat at 2024-08-16 22:35:03.262349
========= sending heartbeat at 2024-08-16 22:35:13.278904
IO request details:
Error ocurred (Input/output error) at line 715 in fread
The file is probably corrupt. If this is a movie, try deleting it and re-importing the movie set. If this is a particle stack, try the 'check for corrupt particles' job (if corrupt particles are found, they will be excluded from the job's output).
filename: /scr/cryosparc_cache/instance_c112384:39001/links/P35-J845-1723783735/e8f2a7d0bffe8d617d431c6f053a2016f9c4f7c3.mrc
filetype: 0
header_only: 0
idx_start: 10
idx_limit: 11
eer_upsampfactor: 2
eer_numfractions: 40
num_threads: 6
buffer: 0x55ef250a04b0
buffer_sz: 589824
nx, ny, nz: 384 384 1
dtype: 12
total_time: -1.000000
io_time: 0.000000
========= sending heartbeat at 2024-08-16 22:35:23.297079
========= sending heartbeat at 2024-08-16 22:35:33.314085
========= sending heartbeat at 2024-08-16 22:35:43.330064
========= sending heartbeat at 2024-08-16 22:35:53.339995
========= sending heartbeat at 2024-08-16 22:36:03.357077
========= sending heartbeat at 2024-08-16 22:36:13.375087
========= sending heartbeat at 2024-08-16 22:36:23.391058
========= sending heartbeat at 2024-08-16 22:36:33.408100
========= sending heartbeat at 2024-08-16 22:36:43.424096
========= sending heartbeat at 2024-08-16 22:36:53.440063
HOST ALLOCATION FUNCTION: using numba.cuda.pinned_array
**** handle exception rc
Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/run.py", line 115, in cryosparc_master.cryosparc_compute.run.main
File "cryosparc_master/cryosparc_compute/jobs/class3D/run.py", line 696, in cryosparc_master.cryosparc_compute.jobs.class3D.run.run_class_3D
File "cryosparc_master/cryosparc_compute/jobs/class3D/run.py", line 1201, in cryosparc_master.cryosparc_compute.jobs.class3D.run.class3D_engine_run
File "cryosparc_master/cryosparc_compute/jobs/class3D/run.py", line 1222, in cryosparc_master.cryosparc_compute.jobs.class3D.run.class3D_engine_run
File "cryosparc_master/cryosparc_compute/engine/newengine.py", line 550, in cryosparc_master.cryosparc_compute.engine.newengine.EngineThread.read_image_data
File "/home/exx/cryosparc/cryosparc_worker/cryosparc_compute/particles.py", line 34, in get_original_real_data
data = self.blob.view()
File "/home/exx/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py", line 145, in view
return self.get()
File "/home/exx/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py", line 140, in get
_, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
File "cryosparc_master/cryosparc_compute/blobio/prefetch.py", line 82, in cryosparc_master.cryosparc_compute.blobio.prefetch.synchronous_native_read
OSError:
IO request details:
Error ocurred (Input/output error) at line 715 in fread
The file is probably corrupt. If this is a movie, try deleting it and re-importing the movie set. If this is a particle stack, try the 'check for corrupt particles' job (if corrupt particles are found, they will be excluded from the job's output).
filename: /scr/cryosparc_cache/instance_c112384:39001/links/P35-J845-1723783735/e8f2a7d0bffe8d617d431c6f053a2016f9c4f7c3.mrc
filetype: 0
header_only: 0
idx_start: 10
idx_limit: 11
eer_upsampfactor: 2
eer_numfractions: 40
num_threads: 6
buffer: 0x55ef250a04b0
buffer_sz: 589824
nx, ny, nz: 384 384 1
dtype: 12
total_time: -1.000000
io_time: 0.000000
set status to failed
========= main process now complete at 2024-08-16 22:37:03.456737.
========= monitor process now complete at 2024-08-16 22:37:03.462796.
The job ran fine until partway through the second half-set of the first full iteration, at which point it died with this error. “Check for Corrupt Particles” does not show any corrupt particles in the input - and in any case, as the O-EM iterations were set to 10 epochs, any corrupt particles should already have been encountered earlier in the job. Rerunning the job now to see if it dies with the same error.
Cheers
Oli