I have submitted a refinement job in Cryosparc in the morning. But suddenly is got disconnected and not able to connect to localhost since then.
Is this related to Cryosparc server issue?
How should I resolve it.
Thank you.
Regards
Dilip
I had this issue once on overnight run - resolved by restarting cryosparc. One job was completed, another one was stopped mid way.
$cryosparc stop
$cryosparc start
Mike
Mike/Dilip, can you find the log file
cryosparc/run/meteor.stdout
and email that to feedback@structura.bio
This might be a problem we’ve seen before.
I had to do a clean install recently, so all my older logs are gone.
–
Mike
Hi Apunjani
When I restarted cryosparc that issue has been resolved. However, I am facing another problem with refinement process. I started refinement for one of my data set. After initialization it just freeze the system run gets aborted with following error:
Engine Started.
Traceback (most recent call last):
File “/home/dillip/cryosparc/cryosparc-compute/sparc/streamlog.py”, line 318, in run_with_except_hook
run_old(*args, **kw)
File “/home/dillip/cryosparc/cryosparc-compute/engine/cuda_core.py”, line 68, in run
self.target(*self.args, dev=self.dev, thidx=self.thidx)
File “/home/dillip/cryosparc/cryosparc-compute/engine/engine.py”, line 502, in work
ET.backproject(accumulators, force_max=force_max) # this will accumulate H, h, sigma2_est
File “/home/dillip/cryosparc/cryosparc-compute/engine/engine.py”, line 319, in backproject
self.toc(‘backproject’)
File “/home/dillip/cryosparc/cryosparc-compute/engine/engine.py”, line 38, in toc
self.wait()
File “/home/dillip/cryosparc/cryosparc-compute/engine/engine.py”, line 34, in wait
self.stream.synchronize()
LaunchError: cuStreamSynchronize failed: the launch timed out and was terminated
Traceback (most recent call last):
File “/home/dillip/cryosparc/cryosparc-compute/sparcjob.py”, line 299, in
superspeed = job_type==‘superrefine’)
File “/home/dillip/cryosparc/cryosparc-compute/refinement.py”, line 273, in refine
cuda_devs=cuda_devs)
File “/home/dillip/cryosparc/cryosparc-compute/engine/engine.py”, line 526, in process
results[k] = cuda_core.ensure_on_cpu(v)
File “/home/dillip/cryosparc/cryosparc-compute/engine/cuda_core.py”, line 91, in ensure_on_cpu
return v.get()
File “/home/dillip/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/gpuarray.py”, line 271, in get
_memcpy_discontig(ary, self, async=async, stream=stream)
File “/home/dillip/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/gpuarray.py”, line 1190, in _memcpy_discontig
drv.memcpy_dtoh(dst, src.gpudata)
LaunchError: cuMemcpyDtoH failed: the launch timed out and was terminated
And at terminal it shows:
[dillip@vina-dock ~]$
Message from syslogd@vina-dock at Feb 22 10:09:06 …
kernel:NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [swapper/8:0]
I called my IT guy and he reboot the system told me its problem with cryosparc.
Please look into it and let me know possible reason behind it.
I shall be highly thankful to you.
Regards
Dilip Kumar
I also got the same cuStreamSynchronize error after an initial round of refinement. This happened after the first set of real space/fourier space/mask space slices were outputted.
Traceback (most recent call last): File "cryosparc2_compute/jobs/runcommon.py", line 705, in run_with_except_hook run_old(*args, **kw) File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 92, in cryosparc2_compute.engine.cuda_core.GPUThread.run File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 93, in cryosparc2_compute.engine.cuda_core.GPUThread.run File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 848, in cryosparc2_compute.engine.engine.process.work File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 559, in cryosparc2_compute.engine.engine.EngineThread.backproject File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 228, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.toc File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 224, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.wait LaunchError: cuStreamSynchronize failed: the launch timed out and was terminated