Add environment variable for log directory

bsobol · March 11, 2024, 10:27pm

Dear cryoSPARC Devs,

I’m an admin of cluster setup of cryoSPARC where each user (or research group) has its own instance of cryoSPARC (instance == database and configuration with license key), but all instances are started from a single read-only install directory. We are redirecting logs of each instance to the users home directory by applying our own patches to a few cryoSPARC source files.

In practice, we change references to $CRYOSPARC_ROOT_DIR/run to $CRYOSPARC_LOG_DIR/run where $CRYOSPARC_LOG_DIR is defined in config.sh.

It’s a very easy to implement (our patches change five files). Would you consider adding such feature (CRYOSPARC_LOG_DIR environment variable)?

wtempel · March 14, 2024, 7:31pm

Thanks @bsobol for providing details about your setup as this may be helpful for other users as well. Sharing a single installation by multiple CryoSPARC instances (and system users) is not something we have implemented official support for yet but we certainly see the utility, and the changes you describe may be a clean way to achieve this. We may make similar changes in a future release.

Could you please provide the specific patch that you apply to the files showing the lines changed?

Also, we are aware, in part due to posts by CryoSPARC’s awesome users and local admins on this forum, of related approached that may be helpful:

CryoSPARC in a container, for example with docker (search, link)
infrastructure-as-code, for example with ansible

bsobol · March 15, 2024, 9:25am

@wtempel sure, here are our patches regarding multi-instance setup for version 4.4.1. We probably missed some ROOT_DIR references that should be changed, but as we don’t have any issues with the current setup, we probably have most cases covered.

We have one more patch to allow import/include in Jinja templates of batch scripts from certain directories.
Everything else is handled via config scripts (global and instance-specific) and cluster submission script templates.

--- cryosparc_master/bin/cryosparcm.orig
+++ cryosparc_master/bin/cryosparcm
@@ -98,7 +98,7 @@

 # more global constants that are set at start time, derived from above
 root_dir_hash=$(echo -n $CRYOSPARC_ROOT_DIR | md5sum | awk '{print $1}')
-export CRYOSPARC_SUPERVISOR_SOCK_FILE=${CRYOSPARC_SUPERVISOR_SOCK_FILE:-"/tmp/cryosparc-supervisor-$root_dir_hash.sock"}
+export CRYOSPARC_SUPERVISOR_SOCK_FILE=${CRYOSPARC_SUPERVISOR_SOCK_FILE:-"~/.cryosparc/cs-svisor-$root_dir_hash.sock"}
 export CRYOSPARC_HTTP_APP_PORT=$CRYOSPARC_BASE_PORT
 export CRYOSPARC_MONGO_PORT=$[$CRYOSPARC_BASE_PORT+1]
 export CRYOSPARC_COMMAND_CORE_PORT=$[$CRYOSPARC_BASE_PORT+2]
@@ -476,7 +476,7 @@
     ;;
 # --------------------------------------------------------------------------------------------
     log)
-    cd "$CRYOSPARC_ROOT_DIR"
+    cd "$CRYOSPARC_LOG_DIR"
     shift
     log_service="$1"
     log_file_path="run/$log_service.log"
@@ -495,7 +495,7 @@
     ;;
 # --------------------------------------------------------------------------------------------
     filterlog)
-    cd "$CRYOSPARC_ROOT_DIR"
+    cd "$CRYOSPARC_LOG_DIR"
     shift

     # setup initial vars
@@ -627,8 +627,8 @@
     ;;
 # --------------------------------------------------------------------------------------------
     snaplogs)
-    snap_output=${CRYOSPARC_ROOT_DIR}/run/cryosparc_logs_$(date +%Y%m%d%H%M%S).tgz
-    tar zcvf $snap_output ${CRYOSPARC_ROOT_DIR}/run/*.log ${CRYOSPARC_ROOT_DIR}/config.sh
+    snap_output=${CRYOSPARC_LOG_DIR}/run/cryosparc_logs_$(date +%Y%m%d%H%M%S).tgz
+    tar zcvf $snap_output ${CRYOSPARC_LOG_DIR}/run/*.log ${CRYOSPARC_ROOT_DIR}/config.sh
     if [ $? -eq 0 ];then
         echo "Created log archive in $snap_output."
         exit 0
@@ -1502,7 +1502,7 @@
     fi

     # Mirror all standard output into update.log
-    update_log_path="${CRYOSPARC_ROOT_DIR}/run/update.log"
+    update_log_path="${CRYOSPARC_LOG_DIR}/run/update.log"
     echo "" >> "${update_log_path}"
     echo "Update at $(date +"%Y-%m-%dT%H:%M:%S%:z")" >> "${update_log_path}"
     exec &> >(tee -a "${update_log_path}")
@@ -1933,7 +1933,7 @@
     {
         if ! command python -c "from cryosparc_compute import database_management; database_management.check_mongo()"; then
             echo "[$(date -Is)] Error checking database. Most recent database log lines:" >&2
-            tail "$CRYOSPARC_ROOT_DIR/run/database.log"
+            tail "$CRYOSPARC_LOG_DIR/run/database.log"
             exit 1
         fi
     }; exit
@@ -1945,7 +1945,7 @@
         mongo_uri=$(python -c "from cryosparc_compute import database_management; print(database_management.get_mongo_uri('meteor', admin=True))")
         if ! command mongo "${mongo_uri}" --eval "rs.reconfig({\"_id\":\"meteor\",\"members\":[{\"_id\":0,\"host\":\"${db_address}\"}]},{\"force\":true})"; then
             echo "[$(date -Is)] Error fixing database port. Most recent database log lines:" >&2
-            tail "$CRYOSPARC_ROOT_DIR/run/database.log"
+            tail "$CRYOSPARC_LOG_DIR/run/database.log"
             exit 1
         fi
     }; exit
@@ -1953,9 +1953,9 @@
 # --------------------------------------------------------------------------------------------
     configuredb)
     {
-        if ! command python -c "from cryosparc_compute import database_management; database_management.configure_mongo(logfile='${CRYOSPARC_ROOT_DIR}/run/database.log')"; then
+        if ! command python -c "from cryosparc_compute import database_management; database_management.configure_mongo(logfile='${CRYOSPARC_LOG_DIR}/run/database.log')"; then
             echo "[$(date -Is)] Error configuring database. Most recent database log lines:" >&2
-            tail "$CRYOSPARC_ROOT_DIR/run/database.log"
+            tail "$CRYOSPARC_LOG_DIR/run/database.log"
             exit 1
         fi
     }; exit

--- cryosparc_master/supervisord.conf.orig
+++ cryosparc_master/supervisord.conf
@@ -5,15 +5,15 @@
 file=%(ENV_CRYOSPARC_SUPERVISOR_SOCK_FILE)s   ; (the path to the socket file)
 
 [supervisord]
-logfile=%(ENV_CRYOSPARC_ROOT_DIR)s/run/supervisord.log
+logfile=%(ENV_CRYOSPARC_LOG_DIR)s/run/supervisord.log
 logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
 logfile_backups=10           ; (num of main logfile rotation backups;default 10)
 loglevel=info                ; (log level;default info; others: debug,warn,trace)
-pidfile=%(ENV_CRYOSPARC_ROOT_DIR)s/run/supervisord.pid
+pidfile=%(ENV_CRYOSPARC_LOG_DIR)s/run/supervisord.pid
 nodaemon=false               ; (start in foreground if true;default false)
 minfds=1024                  ; (min. avail startup file descriptors;default 1024)
 minprocs=200                 ; (min. avail process descriptors;default 200)
-childlogdir=%(ENV_CRYOSPARC_ROOT_DIR)s/run/            ; ('AUTO' child log dir, default $TEMP)
+childlogdir=%(ENV_CRYOSPARC_LOG_DIR)s/run/            ; ('AUTO' child log dir, default $TEMP)
 umask = 002

 ; the below section must remain in the config file for RPC
@@ -31,7 +31,7 @@
 autostart=false
 autorestart=false
 redirect_stderr=true
-stdout_logfile=%(ENV_CRYOSPARC_ROOT_DIR)s/run/database.log
+stdout_logfile=%(ENV_CRYOSPARC_LOG_DIR)s/run/database.log

 [program:command_core]
 command=python -c "import cryosparc_command.command_core as serv; serv.start(port=%(ENV_CRYOSPARC_COMMAND_CORE_PORT)s)"
@@ -41,7 +41,7 @@
 startsecs=5
 startretries=2
 redirect_stderr=true
-stdout_logfile=%(ENV_CRYOSPARC_ROOT_DIR)s/run/command_core.log
+stdout_logfile=%(ENV_CRYOSPARC_LOG_DIR)s/run/command_core.log
 stopasgroup=true
 stopsignal=QUIT

@@ -51,7 +51,7 @@
 autostart=false
 autorestart=true
 redirect_stderr=true
-stdout_logfile=%(ENV_CRYOSPARC_ROOT_DIR)s/run/command_vis.log
+stdout_logfile=%(ENV_CRYOSPARC_LOG_DIR)s/run/command_vis.log
 stopasgroup=true
 stopsignal=QUIT

@@ -61,7 +61,7 @@
 autostart=false
 autorestart=true
 redirect_stderr=true
-stdout_logfile=%(ENV_CRYOSPARC_ROOT_DIR)s/run/command_rtp.log
+stdout_logfile=%(ENV_CRYOSPARC_LOG_DIR)s/run/command_rtp.log
 stopasgroup=true
 stopsignal=QUIT

@@ -71,7 +71,7 @@
 autostart=false
 autorestart=false
 redirect_stderr=true
-stdout_logfile=%(ENV_CRYOSPARC_ROOT_DIR)s/run/app.log
+stdout_logfile=%(ENV_CRYOSPARC_LOG_DIR)s/run/app.log
 environment=MONGO_URL="%(ENV_CRYOSPARC_APP_MONGO_URI)s",MONGO_OPLOG_URL="%(ENV_CRYOSPARC_APP_MONGO_OPLOG_URI)s",NODE_ENV="production",PORT="%(ENV_CRYOSPARC_HTTP_APP_PORT)s"

 [program:app_api]
@@ -80,7 +80,7 @@
 autostart=false
 autorestart=true
 redirect_stderr=true
-stdout_logfile=%(ENV_CRYOSPARC_ROOT_DIR)s/run/app_api.log
+stdout_logfile=%(ENV_CRYOSPARC_LOG_DIR)s/run/app_api.log
 environment=MONGO_URL="%(ENV_CRYOSPARC_APP_MONGO_URI)s",MONGO_OPLOG_URL="%(ENV_CRYOSPARC_APP_MONGO_OPLOG_URI)s",PORT="%(ENV_CRYOSPARC_HTTP_LIVEAPP_LEGACY_PORT)s",HTTP_FORWARDED_COUNT="1",ROOT_URL="http://%(ENV_CRYOSPARC_MASTER_HOSTNAME)s:%(ENV_CRYOSPARC_HTTP_LIVEAPP_LEGACY_PORT)s",NODE_OPTIONS="--max-old-space-size=8192",METEOR_SETTINGS='{"public":{"cryosparc_live":"prod"}}'

 [program:app_api_dev]
@@ -89,5 +89,5 @@
 autostart=false
 autorestart=false
 redirect_stderr=true
-stdout_logfile=%(ENV_CRYOSPARC_ROOT_DIR)s/run/app_api.log
+stdout_logfile=%(ENV_CRYOSPARC_LOG_DIR)s/run/app_api.log
 environment=MONGO_URL="%(ENV_CRYOSPARC_APP_MONGO_URI)s",MONGO_OPLOG_URL="%(ENV_CRYOSPARC_APP_MONGO_OPLOG_URI)s",NODE_OPTIONS="--max-old-space-size=8192",METEOR_OFFLINE_CATALOG="1"

--- cryosparc_master/cryosparc_compute/error_reporting.py.orig
+++ cryosparc_master/cryosparc_compute/error_reporting.py
@@ -197,7 +197,7 @@
     assert service_name in SERVICES, "invalid service name"
     # read file lines
     num_log_lines = 0
-    log_file_name = os.path.join(os.environ["CRYOSPARC_ROOT_DIR"], "run/{service_name}.log".format(service_name=service_name))
+    log_file_name = os.path.join(os.environ["CRYOSPARC_LOG_DIR"], "run/{service_name}.log".format(service_name=service_name))
     today = datetime.datetime.now()
     if date is not None:
         requested_date = datetime.datetime.strptime(date, "%Y-%m-%d")
@@ -265,7 +265,7 @@
         requested_date = today - datetime.timedelta(days=days)
     with open(
         os.path.join(
-            os.environ["CRYOSPARC_ROOT_DIR"],
+            os.environ["CRYOSPARC_LOG_DIR"],
             "run/{service_name}.log".format(service_name=service_name),
         ),
         "r",
@@ -307,7 +307,7 @@
     if output_folder:
         report_filepath = os.path.join(output_folder, report_filename)
     else:
-        report_filepath = os.path.join(os.environ["CRYOSPARC_ROOT_DIR"], report_filename)
+        report_filepath = os.path.join(os.environ["CRYOSPARC_LOG_DIR"], report_filename)
     save_error_report_to_disk(report_filepath, report_bytes_io)


@@ -330,7 +330,7 @@
         # the following logs are not filter supported
         for service in ["app", "app_api", "update"]:
             try:
-                log_filepath = os.path.join(os.environ["CRYOSPARC_ROOT_DIR"], "run/{service}.log".format(service=service))
+                log_filepath = os.path.join(os.environ["CRYOSPARC_LOG_DIR"], "run/{service}.log".format(service=service))
                 zip_file.write(log_filepath, "{service}_log.txt".format(service=service))
             except FileNotFoundError:
                 print("log file not found for service: " + service)

And same changes as above in the corresponding error_reporting.py in cryosparc_worker directory

wtempel · March 18, 2024, 8:18pm

@bsobol Thanks for posting your patches. They will be helpful when we get to work on this functionality.