RBMC error platform_read_file: fopen: Operation not permitted (version: 4.7)

Indrajit · June 4, 2026, 11:00am

Hi

I am trying to run RBMC on a rather old dataset of ours (initially processed before path motion correction was available on Cryosprc) and am getting the following error:

I have checked the symbolic links and the permissions to the movie files and they seem to be fine. I have tried removing the movie that Cryosprc flagged and then restart RBMC. In that case cryosparc flagged another movie and the job died. I have re-imported the movies –> rerun patch motion correction –> reassigned the particles to the new patch motion correction and then tried RBMC. However the problem persists.

Since I could not think of anything else, I manually cleared the SSD cache but still no luck.

I am using Cryosparc version 4.7.0. I was wondering if any of you have faced a similar issue or have any suggestion to figure out what might be going on.

Thanks,

Indrajit

wtempel · June 4, 2026, 5:41pm

Hi @Indrajit:

Please can you post the outputs of these commands:

csprojectid=P99 # replace with actual project ID
csjobid=J199 # replace with actual rbmc job id
cryosparcm joblog $csprojectid $csjobid | tail -n 40
cryosparcm cli "get_job('$csprojectid', '$csjobid', 'job_type', 'version', 'input_slot_groups.connections', 'params_spec', 'status')"

Indrajit · June 5, 2026, 2:36pm

Here is the output.

spuser@spgpu5:/data/spgpu2_lnk/data/Users/Anamika/eBIC_bi28576-21_m02_28_04_2023/CS-ebic-bi28576-21-appol$ cryosparcm joblog $csprojectid $csjobid | tail -n 40

data type: 0x10

frames:    \[0:-1\]

eer upsample factor: 1

eer number of fractions: 40.

movie 682382087437757538: J401/imported/000682382087437757538_FoilHole_28363737_Data_28306427_28306429_20230429_060038_fractions.tif

Total: 3956.513s

  iosys:2:

             iocb processing \[hits     660\]: 0.077303s 0.00% (1791.083554s 45.27% with children)

             iosys_sort_jobs \[hits     660\]: 0.004530s 0.00%

    iosys_open_cacheable_fds \[hits     660\]: 9.854870s 0.25%

    iosys_process_cached_fd_reads \[hits     660\]: 0.020064s 0.00% (38.355984s 0.97% with children)

            io_uring fill sq \[hits     660\]: 0.838919s 0.02%

              io_uring block \[hits     660\]: 37.496299s 0.95%

           io_uring drain cq \[hits     660\]: 0.000703s 0.00%

                   slow_path \[hits     660\]: 1742.790867s 44.05%

  iosys:1:

             iocb processing \[hits     716\]: 0.149255s 0.00% (1549.867631s 39.17% with children)

             iosys_sort_jobs \[hits     717\]: 0.006034s 0.00%

    iosys_open_cacheable_fds \[hits     717\]: 15.510005s 0.39%

    iosys_process_cached_fd_reads \[hits     717\]: 0.003144s 0.00% (54.042359s 1.37% with children)

            io_uring fill sq \[hits     717\]: 1.260170s 0.03%

              io_uring block \[hits     717\]: 52.778047s 1.33%

           io_uring drain cq \[hits     717\]: 0.000998s 0.00%

                   slow_path \[hits     716\]: 1480.159978s 37.41%

  iosys:0:

             iocb processing \[hits     654\]: 0.147551s 0.00% (1814.945699s 45.87% with children)

             iosys_sort_jobs \[hits     654\]: 0.004339s 0.00%

    iosys_open_cacheable_fds \[hits     654\]: 10.461533s 0.26%

    iosys_process_cached_fd_reads \[hits     654\]: 0.002326s 0.00% (39.847742s 1.01% with children)

            io_uring fill sq \[hits     654\]: 0.797075s 0.02%

              io_uring block \[hits     654\]: 39.047629s 0.99%

           io_uring drain cq \[hits     654\]: 0.000711s 0.00%

                   slow_path \[hits     654\]: 1764.484534s 44.60%

  MAIN THREAD:

========= main process now complete at 2026-06-04 13:56:53.199024.

========= monitor process now complete at 2026-06-04 13:56:53.991904.


spuser@spgpu5:/data/spgpu2_lnk/data/Users/Anamika/eBIC_bi28576-21_m02_28_04_2023/CS-ebic-bi28576-21-appol$ cryosparcm cli "get_job('$csprojectid', '$csjobid', 'job_type', 'version', 'input_slot_groups.connections', 'params_spec', 'status')"

{'\_id': '6a181eac8800c32cbee0d66a', 'input_slot_groups': \[{'connections': \[{'group_name': 'micrographs', 'job_uid': 'J403', 'slots': \[{'group_name': 'micrographs', 'job_uid': 'J403', 'result_name': 'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name': 'movie_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J403', 'result_name': 'gain_ref_blob', 'result_type': 'exposure.gain_ref_blob', 'slot_name': 'gain_ref_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J403', 'result_name': 'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name': 'mscope_params', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J403', 'result_name': 'rigid_motion', 'result_type': 'exposure.motion', 'slot_name': 'rigid_motion', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J403', 'result_name': 'spline_motion', 'result_type': 'exposure.motion', 'slot_name': 'spline_motion', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J403', 'result_name': 'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name': 'background_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J403', 'result_name': 'micrograph_blob_non_dw', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J403', 'result_name': 'micrograph_blob_non_dw_AB', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J403', 'result_name': 'micrograph_thumbnail_blob_1x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J403', 'result_name': 'micrograph_thumbnail_blob_2x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J403', 'result_name': 'micrograph_blob', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}\]}\]}, {'connections': \[\]}, {'connections': \[{'group_name': 'particles', 'job_uid': 'J409', 'slots': \[{'group_name': 'particles', 'job_uid': 'J409', 'result_name': 'blob', 'result_type': 'particle.blob', 'slot_name': 'blob', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J409', 'result_name': 'location', 'result_type': 'particle.location', 'slot_name': 'location', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J409', 'result_name': 'alignments3D', 'result_type': 'particle.alignments3D', 'slot_name': 'alignments3D', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J409', 'result_name': 'ctf', 'result_type': 'particle.ctf', 'slot_name': 'ctf', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J409', 'result_name': 'alignments2D', 'result_type': 'particle.alignments2D', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J409', 'result_name': 'pick_stats', 'result_type': 'particle.pick_stats', 'slot_name': None, 'version': 'F'}\]}\]}, {'connections': \[{'group_name': 'volume', 'job_uid': 'J409', 'slots': \[{'group_name': 'volume', 'job_uid': 'J409', 'result_name': 'map', 'result_type': 'volume.blob', 'slot_name': 'map', 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J409', 'result_name': 'map_half_A', 'result_type': 'volume.blob', 'slot_name': 'map_half_A', 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J409', 'result_name': 'map_half_B', 'result_type': 'volume.blob', 'slot_name': 'map_half_B', 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J409', 'result_name': 'mask_refine', 'result_type': 'volume.blob', 'slot_name': 'mask_refine', 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J409', 'result_name': 'map_sharp', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J409', 'result_name': 'mask_fsc', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J409', 'result_name': 'mask_fsc_auto', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J409', 'result_name': 'precision', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}\]}\]}, {'connections': \[{'group_name': 'mask', 'job_uid': 'J409', 'slots': \[{'group_name': 'mask', 'job_uid': 'J409', 'result_name': 'mask_refine', 'result_type': 'volume.blob', 'slot_name': 'mask', 'version': 'F'}\]}\]}\], 'job_type': 'reference_motion_correction', 'params_spec': {}, 'project_uid': 'P14', 'status': 'failed', 'uid': 'J399', 'version': 'v4.7.0'}

wtempel · June 5, 2026, 5:44pm

Thanks @Indrajit. Please can you post the outputs of these commands (after correctly defining the PROJECT_DIR variable):

PROJECT_DIR="/path/to/project_dir" # edit
df -hT ${PROJECT_DIR}/
ls -lhs ${PROJECT_DIR}/J403/motioncorrected/{00877323,00068238208}*_rigid_traj.npy

Indrajit · June 8, 2026, 9:02am

Hi @wtempel

Here is the output

spuser@spgpu5:\~$ PROJECT_DIR="/data/Users/Anamika/eBIC_bi28576-21_m02_28_04_2023/CS-ebic-bi28576-21-appol"

spuser@spgpu5:\~$ df -hT ${PROJECT_DIR}/

Filesystem      Type        Size  Used Avail Use% Mounted on

spuser@spgpu2:/ fuse.sshfs   29T   19T  9.9T  66% /data/spgpu2_lnk

spuser@spgpu5:\~$ ls -lhs ${PROJECT_DIR}/J403/motioncorrected/{00877323,00068238208}\*\_rigid_traj.npy

4.0K -rw-rw-r-- 1 spuser spuser 848 May 29 20:39 /data/Users/Anamika/eBIC_bi28576-21_m02_28_04_2023/CS-ebic-bi28576-21-appol/J403/motioncorrected/000682382087437757538_FoilHole_28363737_Data_28306427_28306429_20230429_060038_fractions_rigid_traj.npy

4.0K -rw-rw-r-- 1 spuser spuser 848 May 29 18:45 /data/Users/Anamika/eBIC_bi28576-21_m02_28_04_2023/CS-ebic-bi28576-21-appol/J403/motioncorrected/008773238890307695612_FoilHole_28350528_Data_28306427_28306429_20230429_032227_fractions_rigid_traj.npy

Thanks

Indrajit

wtempel · June 8, 2026, 9:49pm

Thanks @Indrajit for posting this information.

Given the project directories sshfs implementation, I wonder whether you ran into an issue similar to I/O error when accessing particle images in CryoSPARC v4.6.0 if SSD caching is turned off - #9 by meier.
What are the outputs of the following commands on the sgpu2 computer (as user spuser):

id
uname -a
ulimit -Hs
ulimit -Ss

?

Indrajit · June 9, 2026, 9:46am

@wtempel thank you for looking into this!

Here is the output:

(base) spuser@spgpu2:~$ id

uid=1000(spuser) gid=1000(spuser) groups=1000(spuser),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),108(lpadmin),126(sambashare)

(base) spuser@spgpu2:~$ uname -a

Linux spgpu2 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

(base) spuser@spgpu2:~$ ulimit -Hs

unlimited

(base) spuser@spgpu2:~$ ulimit -Ss

8192

A few more bits of information that may help in understanding what is going on.

I had successfully run RBMC on this same project with the same file locations (i.e. frames and output located on spgpu2 computer and calculations done on spgpu5). I believe that RBMC run was done in Cryosparc v4.5. As far as I can tell the way I have mounted spgpu2 on spgpu5 has not changed.
Only RBMC jobs are getting affected. I have rerun motion correction and that did not give any issues. So, Cryosparc can access the frames just fine.
Just to see if this was a one off issue with a specific job id and set up a new RBMC job (essentially the same input as the RBMC job that failed but set up from scratch). This time the job failed at a different point. I cleared the new job and reran but the point of failure remained the same. Here is the log from the newly failed RBMC job.

-Indrajit

hsnyder · June 18, 2026, 9:39pm

Hi @Indrajit

Unfortunately, when using sshfs, “Operation not permitted” doesn’t mean the operation wasn’t permitted. It means “something went wrong with IO on the server side, but we don’t know exactly what”. I’m not surprised that the symlinks and file permissions all check out. If it were a genuine permission error, the error message would be “Permission denied”. For the same reason, I can’t be exactly sure what the problem is. The main difference that I can think of between RBMC and other jobs that read movies is the fact that RBMC may do more concurrent IO per process. That may be hitting some sort of limit either on the server side or within the server’s sshd daemon, but I’m speculating here. There may be information in the server’s logs (particularly the sshd logs) that help shed light on the actual error.

Partly for these sorts of reasons, and partly for performance reasons, we don’t generally recommend sshfs. NFS is more reliable and easier to debug from our perspective as a software vendor.

If you’re curious for a longer answer…

“Operation not permitted” is different from “Permission denied”. The former means that Linux returned the error code EPERM while the latter indicates EACCES. EACCES is what you get when the permission bits are wrong on a file (the chmod case).

Unfortunately, sshfs maps error codes in a bizarre way. On the server side, both EACCES and EPERM become an internal error code SSH2_FX_PERMISSION_DENIED which is sent over the wire. The generic catch-all is SSH2_FX_FAILURE. On the client side, SSH2_FX_FAILURE becomes EPERM and SSH2_FX_PERMISSION_DENIED becomes EACCES. So if you get “Operation not permitted” (EPERM, remember) over sshfs, the only thing you know is that it is not a permission error from the server’s perspective (i.e. it is neither EPERM nor EACCES) and instead it’s some error that the sshd implementers didn’t anticipate or chose not to classify in a descriptive way.

I don’t love it either, but unfortunately that seems to be how it works.

–Harris

Indrajit · June 19, 2026, 9:05am

@hsnyder

Many thanks for the detailed explanation. This is the only project for which we have the raw frames and CS output folder on a computer that is different from where we are running cryosparc. In the short run we are moving the frames and CS output to the computer that is running cryosparc and will give the RBMC another go (other projects that we are seem to be happy about RBMC jobs). But in the long run we will move away from sshfs to nfs. We certainly don’t what this disruption again. I will post what we find here.

Thanks

Indrajit

Indrajit · June 22, 2026, 10:14am

An update on the issue. Once we moved the frames and the Cryosprc output directory on the same computer that is running Cryosparc RBMC worked without any error. This is perhaps unsurprising if the issue is with the network file system. @hsnyder and @wtempel thank you once again for the detailed explanations.

-Indrajit