Hello,
When CryoSPARC checks that the user running the application can write to a directory on an NFSv4 mounted filesystem with ACLs Python returns the incorrect answer.
This is occurring on multiple master node after upgrading the host from RHEL 7 to RHEL 8.
Notes
- This is being done as the user that runs the CryoSPARC application.
- The user has read and write permissions to the directory (I’ll demonstrate this).
- I have obfuscated some paths, usernames, UIDs, GIDs, and hostnames.
CryoSPARC instance information
- Type: master with cluster
- Software version from
cryosparcm status$ cryosparcm status | grep version Current cryoSPARC version: v4.4.1+240110 - Output of
uname -a && free -gon master node$ uname -a && free -g Linux hostname.fqdn 4.18.0-513.18.1.el8_9.x86_64 #1 SMP Thu Feb 1 03:51:05 EST 2024 x86_64 x86_64 x86_64 GNU/Linux total used free shared buff/cache available Mem: 376 3 250 0 121 370 Swap: 3 0 3 - Operating System:
$ cat /etc/system-release Red Hat Enterprise Linux release 8.9 (Ootpa)
Problem with steps of troubleshooting
This sequence was done to get information about the issue but it occurs at more than this step.
When trying to create a new project called ‘test’ via the web within the directory /nfs/mount/data/
I initially see the message “Missing write permissions for project container directory /nfs/mount/data”.
So I modify the file /usr/local/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py as shown in the
output of diff -c below:
*** __init__.py 2024-02-29 17:57:23.646083466 -0500
--- __init__.py.org 2024-02-29 17:50:23.087832163 -0500
***************
*** 3794,3802 ****
valid = False
message = f"Missing read permissions for project container directory {expanded_project_container_dir}"
elif not os.access(expanded_project_container_dir, os.W_OK):
! #valid = False
! #message = f"Missing write permissions for project container directory {expanded_project_container_dir}"
! pass
return {
"slug" : title_slug,
--- 3794,3801 ----
valid = False
message = f"Missing read permissions for project container directory {expanded_project_container_dir}"
elif not os.access(expanded_project_container_dir, os.W_OK):
! valid = False
! message = f"Missing write permissions for project container directory {expanded_project_container_dir}"
return {
"slug" : title_slug,
I then restart the CryoSPARC application and try to create a new project called test in the directory /nfs/mount/data/
I receive this error in the browser:
“Unable to create project: ServerError Error: new project directory not writable /nfs/mount/data/CS-test”
In the command_core log is the following:
2024-02-29 16:32:12,662 wrapper ERROR | JSONRPC ERROR at create_empty_project
2024-02-29 16:32:12,662 wrapper ERROR | Traceback (most recent call last):
2024-02-29 16:32:12,662 wrapper ERROR | File "/usr/local/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 195, in wrapper
2024-02-29 16:32:12,662 wrapper ERROR | res = func(*args, **kwargs)
2024-02-29 16:32:12,662 wrapper ERROR | File "/usr/local/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 3852, in create_empty_project
2024-02-29 16:32:12,662 wrapper ERROR | project_dir = create_and_return_new_project_dir(project_container_dir, project_dir_name)
2024-02-29 16:32:12,662 wrapper ERROR | File "/usr/local/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 3744, in create_and_return_new_project_dir
2024-02-29 16:32:12,662 wrapper ERROR | assert False, "Error: new project directory not writable %s" % expanded_new_project_dir
2024-02-29 16:32:12,662 wrapper ERROR | AssertionError: Error: new project directory not writable /nfs/mount/data/CS-test
The directory is created:
But the directory is created:
$ stat /nfs/mount/data/CS-test
File: /nfs/mount/data/CS-test
Size: 0 Blocks: 64 IO Block: 1048576 directory
Device: 40h/64d Inode: 7301205695 Links: 2
Access: (2770/drwxrws---) Uid: (#######/cryosparc_user) Gid: (#######/some_group)
Context: system_u:object_r:nfs_t:s0
Access: 2024-02-29 16:32:12.660183000 -0500
Modify: 2024-02-29 16:32:12.660183000 -0500
Change: 2024-02-29 16:32:12.660183000 -0500
Birth: -
And the directory is writable:
$ touch /nfs/mount/data/CS-test/write-test
$ stat /nfs/mount/data/CS-test/write-test
File: /nfs/mount/data/CS-test/write-test
Size: 0 Blocks: 48 IO Block: 1048576 regular empty file
Device: 40h/64d Inode: 7290082035 Links: 1
Access: (0770/-rwxrwx---) Uid: (#######/cryosparc_user) Gid: (#######/some_group)
Context: system_u:object_r:nfs_t:s0
Access: 2024-02-29 16:38:37.428924000 -0500
Modify: 2024-02-29 16:38:37.428924000 -0500
Change: 2024-02-29 16:38:37.429056000 -0500
Birth: -
I make some further changes to the file /usr/local/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py
as shown in the output of diff -c below:
$ diff -c __init__.py __init__.py.org
*** __init__.py 2024-02-29 18:25:12.083844409 -0500
--- __init__.py.org 2024-02-29 17:50:23.087832163 -0500
***************
*** 3714,3721 ****
assert False, f"Error: could not create project container directory {full_project_container_dir} due to {exc}"
# now we know path exists and is directory, now check writable:
if not os.access(full_project_container_dir, os.W_OK):
! #assert False, "Error: project container directory not writable."
! pass
return full_project_container_dir
def create_and_return_new_project_dir(project_container_dir, project_name):
--- 3714,3720 ----
assert False, f"Error: could not create project container directory {full_project_container_dir} due to {exc}"
# now we know path exists and is directory, now check writable:
if not os.access(full_project_container_dir, os.W_OK):
! assert False, "Error: project container directory not writable."
return full_project_container_dir
def create_and_return_new_project_dir(project_container_dir, project_name):
***************
*** 3741,3748 ****
assert False, f"Error: could not create project directory {expanded_new_project_dir} due to {exc}"
# now we know path exists and is directory, now check writable:
if not os.access(expanded_new_project_dir, os.W_OK):
! #assert False, "Error: new project directory not writable %s" % expanded_new_project_dir
! pass
return new_project_dir # this one has shell vars
def check_project_dir(project_dir: str):
--- 3740,3746 ----
assert False, f"Error: could not create project directory {expanded_new_project_dir} due to {exc}"
# now we know path exists and is directory, now check writable:
if not os.access(expanded_new_project_dir, os.W_OK):
! assert False, "Error: new project directory not writable %s" % expanded_new_project_dir
return new_project_dir # this one has shell vars
def check_project_dir(project_dir: str):
***************
*** 3751,3757 ****
full_project_dir = os.path.expandvars(project_dir)
assert os.path.isdir(full_project_dir), "Project directory does not exist"
assert os.access(full_project_dir, os.R_OK), "Project directory is not readable."
! #assert os.access(full_project_dir, os.W_OK), "Project directory is not writable."
return project_dir
@extern
--- 3749,3755 ----
full_project_dir = os.path.expandvars(project_dir)
assert os.path.isdir(full_project_dir), "Project directory does not exist"
assert os.access(full_project_dir, os.R_OK), "Project directory is not readable."
! assert os.access(full_project_dir, os.W_OK), "Project directory is not writable."
return project_dir
@extern
***************
*** 3796,3804 ****
valid = False
message = f"Missing read permissions for project container directory {expanded_project_container_dir}"
elif not os.access(expanded_project_container_dir, os.W_OK):
! #valid = False
! #message = f"Missing write permissions for project container directory {expanded_project_container_dir}"
! pass
return {
"slug" : title_slug,
--- 3794,3801 ----
valid = False
message = f"Missing read permissions for project container directory {expanded_project_container_dir}"
elif not os.access(expanded_project_container_dir, os.W_OK):
! valid = False
! message = f"Missing write permissions for project container directory {expanded_project_container_dir}"
return {
"slug" : title_slug,
***************
*** 4433,4441 ****
com.error_notification(mongo.db, notification_id, "Unable to import project: directory %s does not exist"%(abs_path_export_project_dir))
assert False, "[IMPORT_PROJECT] : Directory %s does not exist"%(abs_path_export_project_dir)
if not os.access(abs_path_export_project_dir, os.W_OK):
! #com.error_notification(mongo.db, notification_id, "Unable to import project: directory %s not writable."%(abs_path_export_project_dir))
! #assert False, "[IMPORT_PROJECT] Error: project container directory not writable."
! pass
all_projects = list(mongo.db['projects'].find({}, {'uid' : 1, 'project_dir' : 1, 'deleted' : 1, 'detached' : 1}))
--- 4430,4437 ----
com.error_notification(mongo.db, notification_id, "Unable to import project: directory %s does not exist"%(abs_path_export_project_dir))
assert False, "[IMPORT_PROJECT] : Directory %s does not exist"%(abs_path_export_project_dir)
if not os.access(abs_path_export_project_dir, os.W_OK):
! com.error_notification(mongo.db, notification_id, "Unable to import project: directory %s not writable."%(abs_path_export_project_dir))
! assert False, "[IMPORT_PROJECT] Error: project container directory not writable."
all_projects = list(mongo.db['projects'].find({}, {'uid' : 1, 'project_dir' : 1, 'deleted' : 1, 'detached' : 1}))
***************
*** 5976,5983 ****
assert False, f"Error: could not create job directory {full_new_job_dir} due to {exc}"
# now we know path exists and is directory, now check writable:
if not os.access(full_new_job_dir, os.W_OK):
! #assert False, "Error: new job directory not writable %s" % full_new_job_dir
! pass
return job_dir_rel
@extern
--- 5972,5978 ----
assert False, f"Error: could not create job directory {full_new_job_dir} due to {exc}"
# now we know path exists and is directory, now check writable:
if not os.access(full_new_job_dir, os.W_OK):
! assert False, "Error: new job directory not writable %s" % full_new_job_dir
return job_dir_rel
@extern
It now works as expected.
Conclusion
Basically wherever CryoSPARC does an os.access(path, os.W_OK) check was causing issues.
It seems the underlying issue is that CryoSPARC is relying on Python’s os.access check without considering that some may be using NFS with ACLs for granting access.
From Python’s os.access documentation:
“Note: I/O operations may fail even when access() indicates that they would succeed, particularly for operations on
network filesystems which may have permissions semantics beyond the usual POSIX permission-bit model.”
Would it be possible to not do the os.access checks and instead use the EAFP style to avoid these odd issues?
I tried to include everything but if something is unclear or confusing please let me know.