I/O error when accessing particle images in CryoSPARC v4.6.0 if SSD caching is turned off

[meier]

Hi everyone,
we ran into a issue with CryoSPARC v4.6.0 immediately after the update from v4.5.3+240807.

We ran two different “Helix Refine” jobs and both terminated with a lot of these errors:

[CPU: 3.93 GB Avail: 177.79 GB]
Traceback (most recent call last):
File “/software/share/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 2304, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 135, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 136, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1090, in cryosparc_master.cryosparc_compute.engine.engine.process.work
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 138, in cryosparc_master.cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “/software/share/cryosparc/cryosparc_worker/cryosparc_compute/ioengine/cmdbuf.py”, line 82, in wait
raise IOError(‘\n\n’.join(errs))
OSError: I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000114_downsample.mrc
data type: 0x10
frames: [7:8]
eer upsample factor: 2
eer number of fractions: 40

The same jobs ran OK in the previous version (v4.5.3+240807) of CryoSPARC. The same jobs run OK on CryoSPARC v4.6.0 if SSD particle caching is turned on.
=> The errors occur only if SSD particle caching is turned off. I assume it is a bug in the new I/O system of CryoSPARC v4.6.0.

---

We did the following trouble shooting:
Verified that the path to the images exist: confirmed
Verified that the CryoSPARC user has permission to read them: confirmed
Verified that the correct CryoSPARC user is running on the worker node: confirmed
Verified that the job is started on the correct GPU: confirmed
Verified that there is enough disk space available: confirmed
Tried to run the job on different worker node: Same errors.
Turned on SSD particle caching: No errors, job completes. Cache was empty in the beginning and the particle images transferred successfully on the SSD.
Turned off SSD particle caching: Job fails with above error.

---

Additional information:

CryoSPARC instance information: master-worker

uname -a && free -g (on master, which is also the worker in this particular example)
Linux luna 5.10.216-gentoo #2 SMP Tue Jul 2 12:13:56 CDT 2024 x86_64 Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz GenuineIntel GNU/Linux

•           total        used        free      shared  buff/cache   available*
  

Mem: 188 11 51 0 127 177
Swap: 255 1 254

eval $(/software/share/cryosparc/cryosparc_worker/bin/cryosparcw env)
env | grep PATH
CRYOSPARC_PATH=/software/share/cryosparc/cryosparc_worker/bin
PYTHONPATH=/software/share/cryosparc/cryosparc_worker
NUMBA_CUDA_INCLUDE_PATH=/software/share/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include
LD_LIBRARY_PATH=/home/jscluster/.local/lib64:/software/lib:/usr/local/lib:/software/share/IMOD/lib
PATH=/software/share/cryosparc/cryosparc_worker/bin:/software/share/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/software/share/cryosparc/cryosparc_worker/deps/anaconda/condabin:/usr/x86_64-pc-linux-gnu/gcc-bin/11:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/bin:/usr/lib/llvm/17/bin:/opt/intel/oneapi/mkl/2023.0.0/bin/intel64:/etc/eselect/wine/bin:/opt/cuda/bin:/opt/cuda/libnvvp:/software/share/moe2022/bin

/sbin/ldconfig -p | grep -i cuda
libtorch_cuda_linalg.so (libc6,x86-64) => /usr/lib64/libtorch_cuda_linalg.so

• libtorch_cuda.so (libc6,x86-64) => /usr/lib64/libtorch_cuda.so*
• libtensorpipe_cuda.so (libc6,x86-64) => /usr/lib64/libtensorpipe_cuda.so*
• libpcsamplingutil.so (libc6,x86-64) => /opt/cuda/extras/CUPTI/lib64/libpcsamplingutil.so*
• libnvvm.so.4 (libc6,x86-64) => /opt/cuda/nvvm/lib64/libnvvm.so.4*
• libnvvm.so (libc6,x86-64) => /opt/cuda/nvvm/lib64/libnvvm.so*
• libnvrtc.so.11.2 (libc6,x86-64) => /opt/cuda/lib64/libnvrtc.so.11.2*
• libnvrtc.so (libc6,x86-64) => /opt/cuda/lib64/libnvrtc.so*
• libnvrtc-builtins.so.11.8 (libc6,x86-64) => /opt/cuda/lib64/libnvrtc-builtins.so.11.8*
• libnvrtc-builtins.so (libc6,x86-64) => /opt/cuda/lib64/libnvrtc-builtins.so*
• libnvperf_target.so (libc6,x86-64) => /opt/cuda/extras/CUPTI/lib64/libnvperf_target.so*
• libnvperf_host.so (libc6,x86-64) => /opt/cuda/extras/CUPTI/lib64/libnvperf_host.so*
• libnvjpeg.so.11 (libc6,x86-64) => /opt/cuda/lib64/libnvjpeg.so.11*
• libnvjpeg.so (libc6,x86-64) => /opt/cuda/lib64/libnvjpeg.so*
• libnvblas.so.11 (libc6,x86-64) => /opt/cuda/lib64/libnvblas.so.11*
• libnvblas.so (libc6,x86-64) => /opt/cuda/lib64/libnvblas.so*
• libnvToolsExt.so.1 (libc6,x86-64) => /opt/cuda/lib64/libnvToolsExt.so.1*
• libnvToolsExt.so (libc6,x86-64) => /opt/cuda/lib64/libnvToolsExt.so*
• libnpps.so.11 (libc6,x86-64) => /opt/cuda/lib64/libnpps.so.11*
• libnpps.so (libc6,x86-64) => /opt/cuda/lib64/libnpps.so*
• libnppitc.so.11 (libc6,x86-64) => /opt/cuda/lib64/libnppitc.so.11*
• libnppitc.so (libc6,x86-64) => /opt/cuda/lib64/libnppitc.so*
• libnppisu.so.11 (libc6,x86-64) => /opt/cuda/lib64/libnppisu.so.11*
• libnppisu.so (libc6,x86-64) => /opt/cuda/lib64/libnppisu.so*
• libnppist.so.11 (libc6,x86-64) => /opt/cuda/lib64/libnppist.so.11*
• libnppist.so (libc6,x86-64) => /opt/cuda/lib64/libnppist.so*
• libnppim.so.11 (libc6,x86-64) => /opt/cuda/lib64/libnppim.so.11*
• libnppim.so (libc6,x86-64) => /opt/cuda/lib64/libnppim.so*
• libnppig.so.11 (libc6,x86-64) => /opt/cuda/lib64/libnppig.so.11*
• libnppig.so (libc6,x86-64) => /opt/cuda/lib64/libnppig.so*
• libnppif.so.11 (libc6,x86-64) => /opt/cuda/lib64/libnppif.so.11*
• libnppif.so (libc6,x86-64) => /opt/cuda/lib64/libnppif.so*
• libnppidei.so.11 (libc6,x86-64) => /opt/cuda/lib64/libnppidei.so.11*
• libnppidei.so (libc6,x86-64) => /opt/cuda/lib64/libnppidei.so*
• libnppicc.so.11 (libc6,x86-64) => /opt/cuda/lib64/libnppicc.so.11*
• libnppicc.so (libc6,x86-64) => /opt/cuda/lib64/libnppicc.so*
• libnppial.so.11 (libc6,x86-64) => /opt/cuda/lib64/libnppial.so.11*
• libnppial.so (libc6,x86-64) => /opt/cuda/lib64/libnppial.so*
• libnppc.so.11 (libc6,x86-64) => /opt/cuda/lib64/libnppc.so.11*
• libnppc.so (libc6,x86-64) => /opt/cuda/lib64/libnppc.so*
• libicudata.so.74 (libc6,x86-64) => /usr/lib64/libicudata.so.74*
• libicudata.so.74 (libc6) => /usr/lib/libicudata.so.74*
• libicudata.so (libc6,x86-64) => /usr/lib64/libicudata.so*
• libicudata.so (libc6) => /usr/lib/libicudata.so*
• libgstcuda-1.0.so.0 (libc6,x86-64) => /usr/lib64/libgstcuda-1.0.so.0*
• libgstcuda-1.0.so.0 (libc6) => /usr/lib/libgstcuda-1.0.so.0*
• libgstcuda-1.0.so (libc6,x86-64) => /usr/lib64/libgstcuda-1.0.so*
• libgstcuda-1.0.so (libc6) => /usr/lib/libgstcuda-1.0.so*
• libc10_cuda.so (libc6,x86-64) => /usr/lib64/libc10_cuda.so*
• libcusparse.so.11 (libc6,x86-64) => /opt/cuda/lib64/libcusparse.so.11*
• libcusparse.so (libc6,x86-64) => /opt/cuda/lib64/libcusparse.so*
• libcusolverMg.so.11 (libc6,x86-64) => /opt/cuda/lib64/libcusolverMg.so.11*
• libcusolverMg.so (libc6,x86-64) => /opt/cuda/lib64/libcusolverMg.so*
• libcusolver.so.11 (libc6,x86-64) => /opt/cuda/lib64/libcusolver.so.11*
• libcusolver.so (libc6,x86-64) => /opt/cuda/lib64/libcusolver.so*
• libcurand.so.10 (libc6,x86-64) => /opt/cuda/lib64/libcurand.so.10*
• libcurand.so (libc6,x86-64) => /opt/cuda/lib64/libcurand.so*
• libcupti.so.11.8 (libc6,x86-64) => /opt/cuda/extras/CUPTI/lib64/libcupti.so.11.8*
• libcupti.so (libc6,x86-64) => /opt/cuda/extras/CUPTI/lib64/libcupti.so*
• libcuinj64.so.11.8 (libc6,x86-64) => /opt/cuda/lib64/libcuinj64.so.11.8*
• libcuinj64.so (libc6,x86-64) => /opt/cuda/lib64/libcuinj64.so*
• libcufftw.so.10 (libc6,x86-64) => /opt/cuda/lib64/libcufftw.so.10*
• libcufftw.so (libc6,x86-64) => /opt/cuda/lib64/libcufftw.so*
• libcufft.so.10 (libc6,x86-64) => /opt/cuda/lib64/libcufft.so.10*
• libcufft.so (libc6,x86-64) => /opt/cuda/lib64/libcufft.so*
• libcudnn_ops_train.so.8 (libc6,x86-64) => /opt/cuda/lib64/libcudnn_ops_train.so.8*
• libcudnn_ops_train.so (libc6,x86-64) => /opt/cuda/lib64/libcudnn_ops_train.so*
• libcudnn_ops_infer.so.8 (libc6,x86-64) => /opt/cuda/lib64/libcudnn_ops_infer.so.8*
• libcudnn_ops_infer.so (libc6,x86-64) => /opt/cuda/lib64/libcudnn_ops_infer.so*
• libcudnn_cnn_train.so.8 (libc6,x86-64) => /opt/cuda/lib64/libcudnn_cnn_train.so.8*
• libcudnn_cnn_train.so (libc6,x86-64) => /opt/cuda/lib64/libcudnn_cnn_train.so*
• libcudnn_cnn_infer.so.8 (libc6,x86-64) => /opt/cuda/lib64/libcudnn_cnn_infer.so.8*
• libcudnn_cnn_infer.so (libc6,x86-64) => /opt/cuda/lib64/libcudnn_cnn_infer.so*
• libcudnn_adv_train.so.8 (libc6,x86-64) => /opt/cuda/lib64/libcudnn_adv_train.so.8*
• libcudnn_adv_train.so (libc6,x86-64) => /opt/cuda/lib64/libcudnn_adv_train.so*
• libcudnn_adv_infer.so.8 (libc6,x86-64) => /opt/cuda/lib64/libcudnn_adv_infer.so.8*
• libcudnn_adv_infer.so (libc6,x86-64) => /opt/cuda/lib64/libcudnn_adv_infer.so*
• libcudnn.so.8 (libc6,x86-64) => /opt/cuda/lib64/libcudnn.so.8*
• libcudnn.so (libc6,x86-64) => /opt/cuda/lib64/libcudnn.so*
• libcudart.so.11.0 (libc6,x86-64) => /opt/cuda/lib64/libcudart.so.11.0*
• libcudart.so (libc6,x86-64) => /opt/cuda/lib64/libcudart.so*
• libcudadebugger.so.1 (libc6,x86-64) => /usr/lib64/libcudadebugger.so.1*
• libcuda.so.1 (libc6,x86-64) => /usr/lib64/libcuda.so.1*
• libcuda.so.1 (libc6) => /usr/lib/libcuda.so.1*
• libcuda.so (libc6,x86-64) => /usr/lib64/libcuda.so*
• libcuda.so (libc6) => /usr/lib/libcuda.so*
• libcublasLt.so.11 (libc6,x86-64) => /opt/cuda/lib64/libcublasLt.so.11*
• libcublasLt.so (libc6,x86-64) => /opt/cuda/lib64/libcublasLt.so*
• libcublas.so.11 (libc6,x86-64) => /opt/cuda/lib64/libcublas.so.11*
• libcublas.so (libc6,x86-64) => /opt/cuda/lib64/libcublas.so*
• libcheckpoint.so (libc6,x86-64) => /opt/cuda/extras/CUPTI/lib64/libcheckpoint.so*
• libaccinj64.so.11.8 (libc6,x86-64) => /opt/cuda/lib64/libaccinj64.so.11.8*
• libaccinj64.so (libc6,x86-64) => /opt/cuda/lib64/libaccinj64.so*
• libOpenCL.so.1 (libc6,x86-64) => /opt/cuda/lib64/libOpenCL.so.1*
• libOpenCL.so (libc6,x86-64) => /opt/cuda/lib64/libOpenCL.so*

uname -a
Linux luna 5.10.216-gentoo #2 SMP Tue Jul 2 12:13:56 CDT 2024 x86_64 Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz GenuineIntel GNU/Linux

free -g
total used free shared buff/cache available
Mem: 188 11 51 0 127 177
Swap: 255 1 254

nvidia-smi
Mon Sep 16 13:07:08 2024
±----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.05 Driver Version: 525.85.05 CUDA Version: 12.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 Off | 00000000:03:00.0 Off | 0 |
| N/A 39C P8 9W / 70W | 6MiB / 15360MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 1 Quadro P4000 Off | 00000000:04:00.0 Off | N/A |
| 46% 39C P8 5W / 105W | 169MiB / 8192MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 2 Tesla T4 Off | 00000000:A1:00.0 Off | 0 |
| N/A 40C P8 9W / 70W | 6MiB / 15360MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 4124171 G /usr/bin/X 4MiB |
| 1 N/A N/A 4124171 G /usr/bin/X 166MiB |
| 2 N/A N/A 4124171 G /usr/bin/X 4MiB |
±----------------------------------------------------------------------------+

[meier]

Full job log (edited):
License is valid.

Launching job on lane Luna target luna …

Running job on master node hostname luna
[CPU: 93.2 MB Avail: 181.61 GB]

Job J143 Started
[CPU: 93.2 MB Avail: 181.61 GB]

Master running v4.6.0, worker running v4.6.0
[CPU: 93.7 MB Avail: 181.61 GB]

Working in directory: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J143
[CPU: 93.7 MB Avail: 181.61 GB]

Running on lane Luna
[CPU: 93.7 MB Avail: 181.61 GB]

Resources allocated:
[CPU: 93.7 MB Avail: 181.61 GB]

Worker: luna
[CPU: 93.7 MB Avail: 181.61 GB]

CPU : [0, 1, 2, 3]
[CPU: 93.7 MB Avail: 181.61 GB]

GPU : [0]
[CPU: 93.7 MB Avail: 181.60 GB]

RAM : [0, 1, 2]
[CPU: 93.7 MB Avail: 181.60 GB]

SSD : False
[CPU: 93.7 MB Avail: 181.60 GB]

---

[CPU: 93.7 MB Avail: 181.60 GB]

Importing job module for job type helix_refine…
[CPU: 310.6 MB Avail: 181.39 GB]

Job ready to run
[CPU: 310.6 MB Avail: 181.39 GB]

---

[CPU: 750.6 MB Avail: 180.96 GB]

Using random seed of 580465034
[CPU: 827.8 MB Avail: 180.88 GB]

Loading a ParticleStack with 750439 items…
[CPU: 1.18 GB Avail: 180.52 GB]

Done.
[CPU: 1.18 GB Avail: 180.52 GB]

Windowing particles
[CPU: 1.18 GB Avail: 180.52 GB]

Done.
[CPU: 1.18 GB Avail: 180.52 GB]

Will generate initial model.
[CPU: 1.18 GB Avail: 180.52 GB]

====== Gold Standard Split ======
[CPU: 1.18 GB Avail: 180.52 GB]

Particles do not have input alignments3D/split assigned, so will be randomly split into two halves.
[CPU: 1.25 GB Avail: 180.46 GB]

Splitting input particles by filament uid.
[CPU: 1.26 GB Avail: 180.45 GB]

Split A has 375220 particles
[CPU: 1.26 GB Avail: 180.45 GB]

Split B has 375219 particles
[CPU: 1.26 GB Avail: 180.45 GB]

====== Refinement ======
[CPU: 1.26 GB Avail: 180.45 GB]

Input particles have box size 422
[CPU: 1.26 GB Avail: 180.45 GB]

Input particles have pixel size 0.9677
[CPU: 1.26 GB Avail: 180.45 GB]

Particles will be zeropadded/truncated to size 422 during alignment
[CPU: 1.26 GB Avail: 180.45 GB]

Volume refinement will be done with effective box size 422
[CPU: 1.26 GB Avail: 180.45 GB]

Volume refinement will be done with pixel size 0.9677
[CPU: 1.26 GB Avail: 180.45 GB]

Particles will be zeropadded/truncated to size 422 during backprojection
[CPU: 1.26 GB Avail: 180.45 GB]

Particles will be backprojected with box size 422
[CPU: 1.26 GB Avail: 180.45 GB]

Volume will be internally cropped and stored with box size 422
[CPU: 1.26 GB Avail: 180.45 GB]

Volume will be interpolated with box size 422 (zeropadding factor 1.00)
[CPU: 1.26 GB Avail: 180.45 GB]

DC components of images will be ignored and volume will be floated at each iteration.
[CPU: 1.26 GB Avail: 180.45 GB]

Spherical windowing of maps is enabled
[CPU: 1.26 GB Avail: 180.45 GB]

Premultiplying will be skipped.
[CPU: 1.26 GB Avail: 180.45 GB]

====== Symmetry ======
[CPU: 1.26 GB Avail: 180.45 GB]

Refining with [edited out] point-group symmetry enforced.
[CPU: 1.26 GB Avail: 180.45 GB]

Limiting alignment tilt to 20.00 degrees.
[CPU: 1.26 GB Avail: 180.45 GB]

Refining with initial helical rise of [edited out] Angstrom ([edited out] pixels) enforced.
[CPU: 1.26 GB Avail: 180.45 GB]

Refining with initial helical twist of [edited out] degrees ([edited out] radians) enforced.
[CPU: 1.26 GB Avail: 180.45 GB]

Limiting alignment shifts to +/- 0.5 helical rises (once full dataset is seen).
[CPU: 1.26 GB Avail: 180.45 GB]

Will perform at least one extra iteration because limit shifts is enabled.
[CPU: 1.27 GB Avail: 180.45 GB]

Refining with helical symmetry order of [edited out] enforced.
[CPU: 1.27 GB Avail: 180.45 GB]

If enabled, the central 211 z-slices (17 rises) will be used for real space symmetry enforcement/searching.
[CPU: 1.27 GB Avail: 180.45 GB]

Will enforce helical symmetry in real space when the resolution reaches 8.000 A.
[CPU: 1.27 GB Avail: 180.45 GB]

Will search helical symmetry parameters when the resolution reaches 6.000 A, with rise extent 1.200 A and twist extent 3.000.
[CPU: 1.27 GB Avail: 180.45 GB]

Setting input per-particle scale factors to 1.0
[CPU: 1.27 GB Avail: 180.45 GB]

Starting at initial resolution 20.000A (radwn 20.419).
[CPU: 1.27 GB Avail: 180.45 GB]

Will perform at most 12 iterations while the GSFSC is worse than 6A.
[CPU: 1.27 GB Avail: 180.45 GB]

====== Masking ======
[CPU: 2.49 GB Avail: 179.23 GB]

No mask input was connected, so dynamic masking will be enabled.
[CPU: 2.49 GB Avail: 179.23 GB]

Dynamic mask threshold: 0.3000
[CPU: 2.49 GB Avail: 179.23 GB]

Dynamic mask near (A): 6.00
[CPU: 2.49 GB Avail: 179.23 GB]

Dynamic mask far (A): 14.00
[CPU: 2.49 GB Avail: 179.23 GB]

Dynamic mask z-clip fraction: 0.850
[CPU: 2.49 GB Avail: 179.23 GB]

====== Initial Model ======
[CPU: 2.49 GB Avail: 179.23 GB]

Using in-plane rotation estimates from filament field…
[CPU: 2.49 GB Avail: 179.23 GB]

Generating initial density from particle images…
[CPU: 3.93 GB Avail: 177.79 GB]

Traceback (most recent call last):
File “/software/share/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 2304, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 135, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 136, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1090, in cryosparc_master.cryosparc_compute.engine.engine.process.work
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 138, in cryosparc_master.cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “/software/share/cryosparc/cryosparc_worker/cryosparc_compute/ioengine/cmdbuf.py”, line 82, in wait
raise IOError(‘\n\n’.join(errs))
OSError: I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000114_downsample.mrc
data type: 0x10
frames: [7:8]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000263_downsample.mrc
data type: 0x10
frames: [800:801]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000069_downsample.mrc
data type: 0x10
frames: [488:489]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000696_downsample.mrc
data type: 0x10
frames: [711:712]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000069_downsample.mrc
data type: 0x10
frames: [494:495]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000547_downsample.mrc
data type: 0x10
frames: [650:651]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000280_downsample.mrc
data type: 0x10
frames: [651:652]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001033_downsample.mrc
data type: 0x10
frames: [15:16]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000193_downsample.mrc
data type: 0x10
frames: [585:586]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000575_downsample.mrc
data type: 0x10
frames: [978:979]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001059_downsample.mrc
data type: 0x10
frames: [243:244]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000359_downsample.mrc
data type: 0x10
frames: [705:706]
eer upsample factor: 2
eer number of fractions: 40
[CPU: 3.27 GB Avail: 178.50 GB]

Traceback (most recent call last):
File “/software/share/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 2304, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 135, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
File “cryosparc_master/cryosparc_compute/gpu/gpucore.py”, line 136, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1090, in cryosparc_master.cryosparc_compute.engine.engine.process.work
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 138, in cryosparc_master.cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “/software/share/cryosparc/cryosparc_worker/cryosparc_compute/ioengine/cmdbuf.py”, line 82, in wait
raise IOError(‘\n\n’.join(errs))
OSError: I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000098_downsample.mrc
data type: 0x10
frames: [852:853]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000826_downsample.mrc
data type: 0x10
frames: [870:871]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001152_downsample.mrc
data type: 0x10
frames: [968:969]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000087_downsample.mrc
data type: 0x10
frames: [120:121]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000917_downsample.mrc
data type: 0x10
frames: [221:222]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000222_downsample.mrc
data type: 0x10
frames: [566:567]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001134_downsample.mrc
data type: 0x10
frames: [360:361]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000767_downsample.mrc
data type: 0x10
frames: [693:694]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000325_downsample.mrc
data type: 0x10
frames: [797:798]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001427_downsample.mrc
data type: 0x10
frames: [846:847]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000255_downsample.mrc
data type: 0x10
frames: [974:975]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000910_downsample.mrc
data type: 0x10
frames: [901:902]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000503_downsample.mrc
data type: 0x10
frames: [577:578]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000877_downsample.mrc
data type: 0x10
frames: [538:539]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000603_downsample.mrc
data type: 0x10
frames: [600:601]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000329_downsample.mrc
data type: 0x10
frames: [819:820]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000317_downsample.mrc
data type: 0x10
frames: [933:934]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000703_downsample.mrc
data type: 0x10
frames: [133:134]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001112_downsample.mrc
data type: 0x10
frames: [101:102]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000895_downsample.mrc
data type: 0x10
frames: [202:203]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000743_downsample.mrc
data type: 0x10
frames: [515:516]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000367_downsample.mrc
data type: 0x10
frames: [678:679]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000547_downsample.mrc
data type: 0x10
frames: [75:76]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000403_downsample.mrc
data type: 0x10
frames: [785:786]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000025_downsample.mrc
data type: 0x10
frames: [128:129]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000349_downsample.mrc
data type: 0x10
frames: [116:117]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001440_downsample.mrc
data type: 0x10
frames: [785:786]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001333_downsample.mrc
data type: 0x10
frames: [358:359]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000480_downsample.mrc
data type: 0x10
frames: [810:811]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000185_downsample.mrc
data type: 0x10
frames: [539:540]
eer upsample factor: 2
eer number of fractions: 40

[meier]

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000976_downsample.mrc
data type: 0x10
frames: [515:516]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000187_downsample.mrc
data type: 0x10
frames: [183:184]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001298_downsample.mrc
data type: 0x10
frames: [209:210]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001427_downsample.mrc
data type: 0x10
frames: [203:204]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001221_downsample.mrc
data type: 0x10
frames: [511:512]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001427_downsample.mrc
data type: 0x10
frames: [723:724]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001333_downsample.mrc
data type: 0x10
frames: [241:242]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001380_downsample.mrc
data type: 0x10
frames: [66:67]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001303_downsample.mrc
data type: 0x10
frames: [551:552]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000222_downsample.mrc
data type: 0x10
frames: [931:932]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000014_downsample.mrc
data type: 0x10
frames: [477:478]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000687_downsample.mrc
data type: 0x10
frames: [267:268]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000813_downsample.mrc
data type: 0x10
frames: [780:781]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000946_downsample.mrc
data type: 0x10
frames: [760:761]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001383_downsample.mrc
data type: 0x10
frames: [451:452]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000863_downsample.mrc
data type: 0x10
frames: [387:388]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000287_downsample.mrc
data type: 0x10
frames: [210:211]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000475_downsample.mrc
data type: 0x10
frames: [815:816]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000547_downsample.mrc
data type: 0x10
frames: [950:951]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000191_downsample.mrc
data type: 0x10
frames: [200:201]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001064_downsample.mrc
data type: 0x10
frames: [674:675]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000724_downsample.mrc
data type: 0x10
frames: [714:715]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001445_downsample.mrc
data type: 0x10
frames: [326:327]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000936_downsample.mrc
data type: 0x10
frames: [568:569]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001427_downsample.mrc
data type: 0x10
frames: [462:463]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000909_downsample.mrc
data type: 0x10
frames: [228:229]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000000_downsample.mrc
data type: 0x10
frames: [787:788]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000976_downsample.mrc
data type: 0x10
frames: [315:316]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000542_downsample.mrc
data type: 0x10
frames: [379:380]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000014_downsample.mrc
data type: 0x10
frames: [495:496]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000030_downsample.mrc
data type: 0x10
frames: [608:609]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001333_downsample.mrc
data type: 0x10
frames: [305:306]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000143_downsample.mrc
data type: 0x10
frames: [380:381]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001249_downsample.mrc
data type: 0x10
frames: [101:102]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001109_downsample.mrc
data type: 0x10
frames: [85:86]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000327_downsample.mrc
data type: 0x10
frames: [948:949]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000585_downsample.mrc
data type: 0x10
frames: [972:973]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000722_downsample.mrc
data type: 0x10
frames: [374:375]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000772_downsample.mrc
data type: 0x10
frames: [498:499]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000760_downsample.mrc
data type: 0x10
frames: [512:513]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000471_downsample.mrc
data type: 0x10
frames: [689:690]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000821_downsample.mrc
data type: 0x10
frames: [726:727]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000471_downsample.mrc
data type: 0x10
frames: [708:709]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000760_downsample.mrc
data type: 0x10
frames: [140:141]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000902_downsample.mrc
data type: 0x10
frames: [494:495]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000178_downsample.mrc
data type: 0x10
frames: [811:812]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000294_downsample.mrc
data type: 0x10
frames: [676:677]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000968_downsample.mrc
data type: 0x10
frames: [993:994]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001285_downsample.mrc
data type: 0x10
frames: [13:14]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000772_downsample.mrc
data type: 0x10
frames: [749:750]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001228_downsample.mrc
data type: 0x10
frames: [45:46]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000856_downsample.mrc
data type: 0x10
frames: [786:787]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001329_downsample.mrc
data type: 0x10
frames: [541:542]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000963_downsample.mrc
data type: 0x10
frames: [932:933]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000446_downsample.mrc
data type: 0x10
frames: [434:435]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000113_downsample.mrc
data type: 0x10
frames: [467:468]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001348_downsample.mrc
data type: 0x10
frames: [241:242]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000895_downsample.mrc
data type: 0x10
frames: [456:457]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000860_downsample.mrc
data type: 0x10
frames: [253:254]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000240_downsample.mrc
data type: 0x10
frames: [250:251]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_001147_downsample.mrc
data type: 0x10
frames: [635:636]
eer upsample factor: 2
eer number of fractions: 40

I/O error, iosys_open_cacheable_fds line 1863: Operation not permitted
Could not open file. Make sure the path exists and the CryoSPARC user has permission to read it.

I/O request details:
filename: /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/J124/downsample/batch_000772_downsample.mrc
data type: 0x10
frames: [390:391]
eer upsample factor: 2
eer number of fractions: 40
[CPU: 3.06 GB Avail: 178.71 GB]

Done in 45.792s.

[CPU: 3.06 GB Avail: 178.71 GB]

Resampling initial model to specified volume representation size and pixel-size…

**** Kill signal sent by CryoSPARC (ID: ) ****

Job is unresponsive - no heartbeat received in 180 seconds.

cryosparcm joblog P47 J143
:1: UserWarning: Cannot manually free CUDA array; will be freed when garbage collected
========= sending heartbeat at 2024-09-16 09:24:04.450884
========= heartbeat failed at 2024-09-16 09:24:04.460607: *** (http://luna:39002, code 400) Encountered ServerError from JSONRPC function “heartbeat” with params (‘P47’, ‘J143’):
ServerError: [Errno 1] Operation not permitted: ‘/net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock’
Traceback (most recent call last):
File “/software/share/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py”, line 196, in wrapper
res = func(*args, **kwargs)
File “/software/share/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py”, line 252, in wrapper
project_instance_uid = read_lockfile_instance_uid(lockfile_path_abs)
File “/software/share/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py”, line 187, in wrapper
return func(*args, **kwargs)
File “/software/share/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py”, line 329, in read_lockfile_instance_uid
with open(lockfile_path_abs) as lockfile:
PermissionError: [Errno 1] Operation not permitted: ‘/net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock’

========= sending heartbeat at 2024-09-16 09:24:14.470931
========= heartbeat failed at 2024-09-16 09:24:14.481456: *** (http://luna:39002, code 400) Encountered ServerError from JSONRPC function “heartbeat” with params (‘P47’, ‘J143’):
ServerError: [Errno 1] Operation not permitted: ‘/net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock’
Traceback (most recent call last):
File “/software/share/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py”, line 196, in wrapper
res = func(*args, **kwargs)
File “/software/share/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py”, line 252, in wrapper
project_instance_uid = read_lockfile_instance_uid(lockfile_path_abs)
File “/software/share/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py”, line 187, in wrapper
return func(*args, **kwargs)
File “/software/share/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py”, line 329, in read_lockfile_instance_uid
with open(lockfile_path_abs) as lockfile:
PermissionError: [Errno 1] Operation not permitted: ‘/net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock’

************* Connection to cryosparc command lost. Heartbeat failed 3 consecutive times at 2024-09-16 09:24:14.481615.
/software/share/cryosparc/cryosparc_worker/bin/cryosparcw: line 151: 302763 Killed python -c “import cryosparc_compute.run as run; run.run()” “$@”

[wtempel]

Welcome to the forum @meier .
Please can you post the outputs of these commands

ls -l /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock
ls -Z /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock
grep "$(df /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock | tail -n 1 | awk '{print $NF}') " /proc/mounts

[meier]

Hi Wolfram,
thank you very much for your fast response!
OK, here is the output. I ran the commands as the CryoSPARC user (jscluster):

ls -l /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock
-rw-rw-r-- 1 jscluster jscluster 75 Jun 6 10:05 /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock

ls -Z /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock
? /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock

grep "$(df /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock | tail -n 1 | awk ‘{print $NF}’) " /proc/mounts
systemd-1 /net/este/cryoEM_tapuy/jscluster autofs rw,relatime,fd=124,pgrp=1,timeout=60,minproto=5,maxproto=5,direct,pipe_ino=268 0 0
jscluster@este:/media/cryoEM_tapuy/jscluster /net/este/cryoEM_tapuy/jscluster fuse.sshfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0

cat /net/este/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock
344037a6-87cd-4976-a8c2-a490713e6a49
luna
39000
2024-06-06T10:05:15.751322

With best regards,
Markus

[wtempel]

Thanks @meier .
What are the outputs of these commands on host luna:

hostname -s
ls -ld software/share/cryosparc/cryosparc_master
ps -eo user:20,command | grep cryosparc_
id jscluster
ssh jscluster@este id
ssh jscluster@este ls -l /media/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock

[meier]

hostname -s
luna

ls -ld /software/share/cryosparc/cryosparc_master
drwxrwxr-x 15 admin admin 4096 Sep 15 14:55 /software/share/cryosparc/cryosparc_master
As you can see, I use privilege separation. The installation is owned by admin:admin, all runtime files and folders are owned by jscluster:jscluster. The CryoSPARC supervisor runs as jscluster:jscluster.

ps -eo user:20,command | grep cryosparc_
jscluster python /software/share/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /software/share/cryosparc/cryosparc_master/supervisord.conf
jscluster python /software/share/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /software/share/cryosparc/cryosparc_master/supervisord.conf
jscluster mongod --auth --dbpath /net/luna/users/jscluster/cryosparc_database --port 39001 --oplogSize 64 --replSet meteor --wiredTigerCacheSizeGB 4 --bind_ip_all
jscluster python /software/share/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /software/share/cryosparc/cryosparc_master/gunicorn.conf.py
jscluster python /software/share/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /software/share/cryosparc/cryosparc_master/gunicorn.conf.py
jscluster python /software/share/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_vis:app -n command_vis -b 0.0.0.0:39003 -c /software/share/cryosparc/cryosparc_master/gunicorn.conf.py
jscluster python /software/share/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_vis:app -n command_vis -b 0.0.0.0:39003 -c /software/share/cryosparc/cryosparc_master/gunicorn.conf.py
jscluster python /software/share/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_rtp:start() -n command_rtp -b 0.0.0.0:39005 -c /software/share/cryosparc/cryosparc_master/gunicorn.conf.py
jscluster python /software/share/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_rtp:start() -n command_rtp -b 0.0.0.0:39005 -c /software/share/cryosparc/cryosparc_master/gunicorn.conf.py
jscluster /software/share/cryosparc/cryosparc_master/cryosparc_app/nodejs/bin/node ./bundle/main.js
jscluster grep --color=auto cryosparc_

id jscluster
uid=3000(jscluster) gid=3000(jscluster) groups=3000(jscluster),11(floppy),14(uucp),18(audio),19(cdrom),20(dialout),26(tape),27(video),80(cdrw),85(usb),442(plugdev),35(games),502(scanner),509(qemu)
(Master and all worker nodes are Gentoo boxes)

ssh jscluster@este id
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
uid=3000(jscluster) gid=3000(jscluster) groups=3000(jscluster),4(adm),7(lp),10(uucp),20(dialout),21(fax),24(cdrom),25(floppy),26(tape),29(audio),30(dip),44(video),46(plugdev),110(netdev),120(lpadmin),125(scanner),128(sambashare)
(that’s an Ubuntu box. It hosts the data for this project.)

ssh jscluster@este ls -l /media/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
-rw-rw-r-- 1 jscluster jscluster 75 Jun 6 10:05 /media/cryoEM_tapuy/jscluster/pauline/CS-myProject/cs.lock

[wtempel]

The error reading cs.lock is likely unrelated to the update, but may indicate incompatibility between the sshfs configuration and CryoSPARC.
We can unfortunately not explain how enabling caching has resolved the IOError during Helix Refinement.

[meier]

Hi Wolfram,
thank you for confirming that this is not a bug in CryoSPARC, but rather a local configuration issue! This brought me on the right track.
We never had an issue with sshfs before, until the update to CryoSPARC 4.6.0.
Apparently, the new version opens all particle images at once while running the helix_refine job. Since there were 1452 particle images, it required 1452 file handles. However, the maximum allowed number of file handles per user on Ubuntu is just 1024 per default. Since the CryoSPARC project folder was hosted on an Ubuntu box, it ran out of file handles. This caused an I/O error on sshfs on the worker node.
However, if CryoSPARC transferred the particle images on the local SSD cache instead, it only opened a limited number of particle images on the Ubuntu box at one time to do the transfer, so sufficient file handles were available. The maximum allowed number of file handles per user on the worker node was set to 12288. Once all particle images had been transferred to the SSD, the worker node could open all of them simultaneously as there were sufficient file handles, and the job could complete.
I now changed the maximum allowed number of file handles for the CryoSPARC user to 65536 on the master node, all worker nodes and all data servers. This resolved the issue.

[meier]

To change the the maximum allowed number of file handles per user, create a new text file “/etc/security/limits.d/00_user_limits.conf” as root and fill it analogous to the following content:

#<domain>        <type>  <item>         <value>>
*                hard    stack           unlimited
*                soft    stack           unlimited
*                hard    nofile          32768
*                soft    nofile          32768
@jscluster       hard    nofile          65536
@jscluster       soft    nofile          65536 
# End of file

This sets the maximum available file handles for a default user (*) to 32768. Users in the group “jscluster” may allocate 65536 file handles. It also allows an unlimited stack size.

Changes only take effect on new logins. Therefore, stop CryoSPARC and wait for all sshfs automount points to unmount.

More information can be found in the Linux-PAM Manual:

info limits.conf