Error during 2D class of manual picked particles

[ctueting]

Hi all,

we encounter a similar issue, as discussed here:

Error while 2D Classification

Hi @hsnyder, thank you all for this thread. Like @jr10, also for me re-extracting the particles solved the problem. I had encountered the same issue described by the other users when doing either ab initio or hetero refinement. This happened after merging 2 different “extract from micrographs” jobs located in two different workspaces of the same project. But as I said re-extracting the particles was the solution. By any chance is there now a way to avoid this problem all together, so to be ab…

but our errors says, to report the bug:

[CPU: 1.49 GB]   Traceback (most recent call last):
  File "/home/cryosparc_user/Software/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 1844, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
  File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run
  File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 1028, in cryosparc_compute.engine.engine.process.work
  File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 87, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
  File "/home/cryosparc_user/Software/cryosparc_worker/cryosparc_compute/particles.py", line 22, in get_original_real_data
    return self.blob.view().copy()
  File "/home/cryosparc_user/Software/cryosparc_worker/cryosparc_compute/blobio/mrc.py", line 127, in view
    return self.get()
  File "/home/cryosparc_user/Software/cryosparc_worker/cryosparc_compute/blobio/mrc.py", line 122, in get
    _, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
  File "cryosparc_worker/cryosparc_compute/blobio/prefetch.py", line 69, in cryosparc_compute.blobio.prefetch.synchronous_native_read
RuntimeError: mrc_readmic (1) 481
Please report this as a bug.: Invalid argument
[arguments] /scratch/instance_cryo801m:39001/projects/P14/J1491/extract/Tony__0034_patch_aligned_doseweighted_particles.mrc, 0, 0, 41, 42, 2, 40

The job log is not very helpful in this case:

================= CRYOSPARCW =======  2022-04-28 11:45:18.491011  =========
Project P14 Job J1492
Master cryo801m Port 39002
===========================================================================
========= monitor process now starting main process
MAINPROCESS PID 56017
========= monitor process now waiting for main process
MAIN PID 56017
class2D.run cryosparc_compute.jobs.jobregister
***************************************************************
Running job  J1492  of type  class_2D
Running job on hostname %s cryoem01
Allocated Resources :  {'fixed': {'SSD': True}, 'hostname': 'cryoem01', 'lane': 'server18', 'lane_type': 'server18', 'license': True, 'licenses_acquired': 2, 'slots': {'CPU': [0, 1], 'GPU': [0, 1], 'RAM': [0, 1, 2]}, 'target': {'cache_path': '/scratch', 'cache_quota_mb': 1000000, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 11554848768, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 1, 'mem': 11551571968, 'name': 'NVIDIA GeForce RTX 2080 Ti'}], 'hostname': 'cryoem01', 'lane': 'server18', 'monitor_port': None, 'name': 'cryoem01', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27], 'GPU': [0, 1], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]}, 'ssh_str': 'cryosparc_user@cryoem01', 'title': 'Worker node cryoem01', 'type': 'node', 'worker_bin_path': '/home/cryosparc_user/Software/cryosparc_worker/bin/cryosparcw'}}
HOST ALLOCATION FUNCTION: using cudrv.pagelocked_empty
HOST ALLOCATION FUNCTION: using cudrv.pagelocked_empty
HOST ALLOCATION FUNCTION: using cudrv.pagelocked_empty
HOST ALLOCATION FUNCTION: using cudrv.pagelocked_empty
**custom thread exception hook caught something
**** handle exception rc
set status to failed
========= main process now complete.
========= monitor process now complete.

To add some more information to this, this only happens after extract particles after manual picking.

Using the same micrographs, but picking with blob picker, using the same settings during extracting and 2D classification, blob picker derived particles could be classified, manual picked particles cannot.

Any ideas on how to overcome this issue?

If you need any additional information, please ask and I will supply them ASAP.

Best
Christian

[wtempel]

@ctueting Thank you for reporting. Just to confirm: This happened during 2D classification of (exclusively) hand-picked particles (as the title suggests)?
Please also let us know the job inputs’ job type(s) and the following:

@stephan

Environment

• cryoSPARC version from cryosparcm status
• CUDA toolkit and pycuda driver information
  eval $(<path-to-cryosparc_worker>/bin/cryosparcw env) ${CRYOSPARC_CUDA_PATH}/bin/nvcc --version python -c "import pycuda.driver; print(pycuda.driver.get_version())"
• Output of uname -a on master node
• Output of uname -a && free && nvidia-smi on relevant worker node(s)

Thanks.

[ctueting]

Yes. In this workspace, we have successfully 2D classified blob and template picked particles without issues, just the manual picked failed.

Here the required information (sorry for not posting them in first place):

CryoSPARC System master node installed at
/home/cryosparc_user/cryosparc/cryosparc_master
Current cryoSPARC version: v3.3.2
----------------------------------------------------------------------------

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Nov_30_19:08:53_PST_2020
Cuda compilation tools, release 11.2, V11.2.67
Build cuda_11.2.r11.2/compiler.29373293_0

Master:

Linux cryo801m 3.10.0-1160.59.1.el7.x86_64 #1 SMP Wed Feb 23 16:47:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Worker:

Linux cryoem01 3.10.0-1160.59.1.el7.x86_64 #1 SMP Wed Feb 23 16:47:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
              total        used        free      shared  buff/cache   available
Mem:      131523804     3305188     6126668       44252   122091948   127378288
Swap:      26214396      431616    25782780
Thu Apr 28 17:37:19 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.54       Driver Version: 510.54       CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:17:00.0 Off |                  N/A |
| 31%   29C    P8    22W / 250W |      1MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:65:00.0 Off |                  N/A |
| 31%   31C    P8     2W / 250W |    106MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    1   N/A  N/A      2871      G   /usr/bin/X                         72MiB |
|    1   N/A  N/A      4495      G   /usr/bin/gnome-shell               31MiB |
+-----------------------------------------------------------------------------+

For the input types:
Negative stain EM images (*.mrc) from a 300 kEV microscope (px size 1.895)
Patch Motion and Patch CTF performed.

Prior to picking exposures curated.

For extracting the particles, we used a boxsize of 280 px.

2D classification tried with 10, 20, and up to 200 classes, which runs fine on blob picked particles, but failed on manual picked.

Thanks
Christian

[hsnyder]

Hi Christian,

Thanks for reporting this, I’ll be looking into it. When extracting the particles after manual picking, did you use the CPU extraction job or the GPU extraction job? Also, what happens when you try a “check for corrupt particles” job on the extracted manual picks?

–Harris

[ctueting]

Hi Harris,

I got the same error in each scenario:

Manual picker → Extract Particles (GPU) → 2D class
Manual picker → Extract Particles (CPU) → 2D class

Manual picker → Extract Particles (GPU) → Check for corrupt particles → 2D class
Manual picker → Extract Particles (CPU) → Check for corrupt particles → 2D class

In fact, the

This is the GUI error msg

Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 85, in cryosparc_compute.run.main
  File "cryosparc_worker/cryosparc_compute/jobs/class2D/run.py", line 159, in cryosparc_compute.jobs.class2D.run.run_class_2D
  File "/home/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/particles.py", line 32, in get_prepared_fspace_data
    return fourier.resample_fspace(fourier.fft(self.get_prepared_real_data()), self.dataset.N)
  File "/home/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/particles.py", line 27, in get_prepared_real_data
    return (self.dataset.prepare_normalization_factor * self['blob/sign']) * (self.dataset.prepare_real_window * (self.get_original_real_data()))
  File "/home/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/particles.py", line 22, in get_original_real_data
    return self.blob.view().copy()
  File "/home/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py", line 127, in view
    return self.get()
  File "/home/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py", line 122, in get
    _, data, total_time = prefetch.synchronous_native_read(self.fname, idx_start = self.page, idx_limit = self.page+1)
  File "cryosparc_worker/cryosparc_compute/blobio/prefetch.py", line 69, in cryosparc_compute.blobio.prefetch.synchronous_native_read
RuntimeError: mrc_readmic (1) 481
Please report this as a bug.: Invalid argument
[arguments] /DATA_500/cryosparc_projects/P30/J1526/extract/Tony__0035_patch_aligned_doseweighted_particles.mrc, 0, 0, 31, 32, 2, 40

Best
Christian

[ctueting]

Update:

I created a new workspace, reimported the images, run patch motion and ctf estimation, and this resolved the issue.

I am still confused, why in first place, blob picker was functional, and manual picker was not.

If you need any additional information or logs from the two different workspaces, I can send them to you to understand the issue.

Thanks, Christian

EDIT2:

As we wanted to use the initial 2000 manual picked particles, I used “reassign particles” to the newly imported micrographs, but than, 2D classification failed again.

EDIT3:
I found a solution:

1. Reassign particles with the newly imported images and the manually picked particles
2. Manual picker with the new images and the particles from (1)
3. Extraction & 2D class

[hsnyder]

Hi @ctueting,

Hmmm… I’m glad you found a workaround, but at the moment, I’m just as confused as you are about the root cause of the issue you’re experiencing.

Could you please specify one full chain of jobs that caused the error (from import to 2d class), and also one full chain that worked correctly?

Also, this is probably not related to your particular issue, but it looks to me like you’re using CentOS 7, is that correct? Many users have reported problems on that operating system; Structura recommends an LTS release of Ubuntu instead.

– Harris

[ctueting]

Hi Harris,

how do I report the full chain of jobs best? Just in plain text every job or somehow from the Tree View? Or should I also add the job.log / job.json files or send them privately to you?

And yes, we’re running Centos7 on all our machines currently. This was already partially set in my group when I joined, so we keep things harmonized. But actually, within the next 2 years, the support of Centos7 will be over, so we have to switch. But we have never faced any OS relating issues so far with any of the programs we’re running on the machines (cryoSPARC, Scipion/Relion, AlphaFold, Phenix/Coot, etc.). It’s just sometimes a bit tricky, e.g. with default gcc 4 on Centos7, but besides that it’s running fine.

Best
Christian

[hsnyder]

Hi @ctueting,

Either screenshots of the tree view or a plain text version of the tree view would be fine, thanks very much. I don’t need the log files.

–Harris