Permission denied during download

Install, Configure and Update
1

I have a user reporting similar errors and can conform that map images are not being generated. In addition, output files regarding map images are not able to be downloaded. User states this has stopped working sometime in the past couple of weeks. Due to an OS upgrade in that timeframe, I upgraded CryoSparc from 4.6.0 → 4.6.2, but the problem still exists. My logs are showing file permission errors:

# grep -i error ./run/command_vis.log
[2025-01-21 18:49:35,236] ERROR in app: Exception on /P3/J60.volume.map [GET]
PermissionError: [Errno 13] Permission denied: '/data/..../J60/J60_003_volume_map.cs'
    raise DatasetLoadError(f"Could not load dataset from file {file}") from err
cryosparc_tools.cryosparc.errors.DatasetLoadError: Could not load dataset from file /data/..../J60/J60_003_volume_map.cs

but the file(s) are owned by cryosparc and have not changed in months.

# ls -l /data/..../J60/J60_003_volume_map.cs
-rw-rw-r-- 1 cryosparc cryosparc 243 Oct  9 15:58 /data/..../J60/J60_003_volume_map.cs
# file /data/..../J60/J60_003_volume_map.cs
/data/..../J60/J60_003_volume_map.cs: NumPy array, version 1.0, header length 182

Note that this appears to be affecting ALL existing jobs/projects.

2

Welcome to the forum @shampton .

Please can you post the outputs of these commands:

ps -eo user:12,pid,ppid,start,command | grep -e cryosparc_ -e mongo
# in next command, replace /path/to/J60 with actual path
grep "$(df /path/to/J60 | tail -n 1 | awk '{print $NF}') " /proc/mounts
cryosparcm log supervisord | tail -n 40
3

Thank you for helping me look into this. Everything seems to be normal.

First command:

cryosparc    1499875       1 11:38:24 python /usr/local/cryosparc/v4/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /usr/local/cryosparc/v4/cryosparc_master/supervisord.conf
cryosparc    1500010 1499875 11:38:29 mongod --auth --dbpath /usr/local/cryosparc/v4/cryosparc_database --port 41001 --oplogSize 64 --replSet meteor --wiredTigerCacheSizeGB 4 --bind_ip_all
cryosparc    1500115 1499875 11:38:33 python /usr/local/cryosparc/v4/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:41002 cryosparc_command.command_core:start() -c /usr/local/cryosparc/v4/cryosparc_master/gunicorn.conf.py
cryosparc    1500116 1500115 11:38:33 python /usr/local/cryosparc/v4/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:41002 cryosparc_command.command_core:start() -c /usr/local/cryosparc/v4/cryosparc_master/gunicorn.conf.py
cryosparc    1500147 1499875 11:38:39 python /usr/local/cryosparc/v4/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_vis:app -n command_vis -b 0.0.0.0:41003 -c /usr/local/cryosparc/v4/cryosparc_master/gunicorn.conf.py
cryosparc    1500161 1500147 11:38:40 python /usr/local/cryosparc/v4/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_vis:app -n command_vis -b 0.0.0.0:41003 -c /usr/local/cryosparc/v4/cryosparc_master/gunicorn.conf.py
cryosparc    1500191 1499875 11:38:41 python /usr/local/cryosparc/v4/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_rtp:start() -n command_rtp -b 0.0.0.0:41005 -c /usr/local/cryosparc/v4/cryosparc_master/gunicorn.conf.py
cryosparc    1500192 1500191 11:38:41 python /usr/local/cryosparc/v4/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_rtp:start() -n command_rtp -b 0.0.0.0:41005 -c /usr/local/cryosparc/v4/cryosparc_master/gunicorn.conf.py
cryosparc    1500254 1499875 11:38:47 /usr/local/cryosparc/v4/cryosparc_master/cryosparc_app/nodejs/bin/node ./bundle/main.js

Second command:
hostname:/data /data nfs rw,nosuid,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.32.90.20,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.32.90.20 0 0

Third command:

2025-01-22 11:38:21,298 WARN received SIGTERM indicating exit request
2025-01-22 11:38:21,299 INFO waiting for app, app_api, command_core, command_rtp, command_vis, database to die
2025-01-22 11:38:21,314 WARN exited: app (terminated by SIGTERM; not expected)
2025-01-22 11:38:21,318 WARN exited: app_api (terminated by SIGTERM; not expected)
2025-01-22 11:38:22,104 INFO stopped: database (exit status 0)
2025-01-22 11:38:22,353 INFO exited: command_rtp (exit status 0; expected)
2025-01-22 11:38:22,766 INFO exited: command_core (exit status 0; expected)
2025-01-22 11:38:23,662 INFO stopped: command_vis (exit status 0)
2025-01-22 11:38:24,411 INFO RPC interface 'supervisor' initialized
2025-01-22 11:38:24,411 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-01-22 11:38:24,413 INFO daemonizing the supervisord process
2025-01-22 11:38:24,414 INFO supervisord started with pid 1499875
2025-01-22 11:38:29,760 INFO spawned: 'database' with pid 1500010
2025-01-22 11:38:31,717 INFO success: database entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-22 11:38:33,537 INFO spawned: 'command_core' with pid 1500115
2025-01-22 11:38:39,387 INFO success: command_core entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2025-01-22 11:38:40,104 INFO spawned: 'command_vis' with pid 1500147
2025-01-22 11:38:41,332 INFO success: command_vis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-22 11:38:41,502 INFO spawned: 'command_rtp' with pid 1500191
2025-01-22 11:38:42,730 INFO success: command_rtp entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-22 11:38:45,683 INFO spawned: 'app' with pid 1500236
2025-01-22 11:38:47,400 INFO success: app entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-22 11:38:47,583 INFO spawned: 'app_api' with pid 1500254
2025-01-22 11:38:48,762 INFO success: app_api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
4

Was the ls command run under the root account? What is the output when Linux user cryosparc runs

id
ls -l /abs/path/to/J60/J60_003_volume_map.cs
5

I was running ‘ls’ as root, but when I tried to run as the cryosparc user I got an error which led me to investigate and I realized that the top level /data folder didn’t have the correct permissions. I think my problem is solved. Thank you for your help and I apologize for dragging you into my PEBCAK.

1 Like