Losing my mind w/ permissions issue

jfscheuren · July 26, 2024, 7:31pm

I am having an issue where my worker node cannot create files in a project folder, but it can read and write to files. Here’s my setup:
I have a master and a worker node. Both have a local cryosparc user with the same uid/gid.
I created a folder owned by the cryosparc user on a ceph mount point. Both users can create/read/write to files in the folder. …/cryosparc
When I run a job I see an error like this after the ssh passwordless login completes bash: …/cryosparc/CS-test2/J1/job.log: Permission denied
Cryosparc does something to the permissions and the worker cannot create files in the CS-test2 folder or any folders in it. The worker can still write to files and read files. Permissions w/ ls -latr look like this: drwxrwsrwx. 7 cryosparc cryosparc 10 Jul 26 13:22 CS-test2

Any help would be greatly appreciated. I’ve been working on this for too long.

wtempel · July 29, 2024, 9:47pm

Welcome to the forum @jfscheuren .

permissions may

be unsafe
still not ensure desired access. The required/desired access may fail for reasons other than the configured permissions, such as
- the nfs server may export the directory in read-only mode
- the nfs client may have mounted the filesystem read-only; this and previous point seemingly ruled out by
  
  jfscheuren:
  
  it can read and write to files.
- the nfs client may have mounted the share under the incorrect mount point
- the Linux user’s access to the shared file may be blocked at any level in the directory hierarchy due to incorrect directory permissions and/or directory ownership
- additional access controls are in place. The dot . character at the and of drwxrwsrwx. suggests that possibility (wikipedia). What are the outputs of these commands:
```
ls -ld /abs/path/to/cryosparc/CS-test2/J1
ls -lZ /abs/path/to/cryosparc/CS-test2/J1
```