Denoising fails

olibclarke · May 8, 2024, 12:14am

Hi, when I try the new denoising job (on 200 mics from a fresh Patch Motion job run in v4.5), I get the following error - any suggestions? This happens with either “train from scratch” or the pretrained model selected.

concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <cyfunction work_prep_denoise_train at 0x7fa0e2336c20>: import of module 'cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.data_prep' failed
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 115, in cryosparc_master.cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/micrograph_analysis/run_denoise.py", line 207, in cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.run_denoise.run_train
  File "cryosparc_master/cryosparc_compute/jobs/micrograph_analysis/data_prep.py", line 274, in cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.data_prep.SafePPE.do_work
  File "/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/home/exx/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <cyfunction work_prep_denoise_train at 0x7fa0e2336c20>: import of module 'cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.data_prep' failed

rbs_sci · May 8, 2024, 1:09am

Same here.

concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/cryosparcer/bin/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/home/cryosparcer/bin/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <cyfunction work_prep_denoise_test at 0x719a41541970>: import of module 'cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.data_prep' failed
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 115, in cryosparc_master.cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/micrograph_analysis/run_denoise.py", line 435, in cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.run_denoise.run_train
  File "cryosparc_master/cryosparc_compute/jobs/micrograph_analysis/data_prep.py", line 274, in cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.data_prep.SafePPE.do_work
  File "/home/cryosparcer/bin/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/cryosparcer/bin/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/cryosparcer/bin/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/home/cryosparcer/bin/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <cyfunction work_prep_denoise_test at 0x719a41541970>: import of module 'cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.data_prep' failed

Jadyn · May 8, 2024, 5:54am

Same error!

Is Python 3.10.13 causing the issue? On another system with v.4.4.1, it shows Python 3.8.17 on eval $(~bin/cryosparcm or cw env) . I’m not sure whether Python was updated alongside the CryoSPARC version 4.5 update.


concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/cs/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/home/cs/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <cyfunction work_prep_denoise_train at 0x2b14b763cc70>: import of module 'cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.data_prep' failed
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 115, in cryosparc_master.cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/micrograph_analysis/run_denoise.py", line 207, in cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.run_denoise.run_train
  File "cryosparc_master/cryosparc_compute/jobs/micrograph_analysis/data_prep.py", line 274, in cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.data_prep.SafePPE.do_work
  File "/home/cs/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/cs/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/cs/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/home/cs/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <cyfunction work_prep_denoise_train at 0x2b14b763cc70>: import of module 'cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.data_prep' failed

P.S
When updating from CryoSPARC version 4.4.1 to 4.5.0 on the other workstation, python was also updated from version 3.8.17 to 3.10.13.

myorick · May 8, 2024, 6:58am

Same here.

concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/cryosparcer/bin/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/home/cryosparcer/bin/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <cyfunction work_prep_denoise_test at 0x719a41541970>: import of module 'cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.data_prep' failed
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 115, in cryosparc_master.cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/micrograph_analysis/run_denoise.py", line 435, in cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.run_denoise.run_train
  File "cryosparc_master/cryosparc_compute/jobs/micrograph_analysis/data_prep.py", line 274, in cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.data_prep.SafePPE.do_work
  File "/home/cryosparcer/bin/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/cryosparcer/bin/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/cryosparcer/bin/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/home/cryosparcer/bin/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <cyfunction work_prep_denoise_test at 0x719a41541970>: import of module 'cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.data_prep' failed

parrot · May 8, 2024, 2:31pm

Got the exactly same error. I rerun the Patch motion and CTF. After setup the resolution cutoff, the accepted micrographs were the input for denoising. All the parameters are default.

yodamoppet · May 8, 2024, 3:41pm

We are seeing the same error here as well.

apunjani · May 8, 2024, 4:54pm

Hi all, thank you for reporting - we have reproduced the error and are working on a fix. Thanks!

apunjani · May 8, 2024, 7:40pm

This issue is now fixed in v4.5.1, please see the change log here: Latest Updates | CryoSPARC

olibclarke · May 9, 2024, 1:48am

Thanks, yep working now in 4.5.1!

olibclarke · May 9, 2024, 2:06am

Now that I have it running - the training plot in the log does not have a label for the Y axis. What does it represent?

rposert · May 9, 2024, 2:58pm

Hi @olibclarke!

Good question — the Y axis here represents loss (lower is better). The units are arbitrary.

The Micrograph Denoiser job uses a specialized Noise2Noise network to denoise micrographs. In this architecture, the model uses half-micrograph A to produce a prediction of what half-micrograph B looks like. This graph compares the error between that prediction and the true half-micrograph B as training proceeds.

Please let me know if you still have questions about this plot!