We are a large CryoSPARC installation with approximately 25-30 users sharing an SSD cache of around 50 TB. Currently, each job takes a long time to check whether the required particles are already in the cache, as it has to scan the entire 50 TB cache.
To address this issue, we would like to configure a dedicated cache directory for each CryoSPARC user, rather than for each Unix user.
In the worker configuration, we currently have the following options:
Is there a way to access the CryoSPARC username associated with each job on workers config.sh connected directly to the master, similar to how we access it through the {{ cryosparc_username }} variable in SLURM script?
Ideally, we would like to dynamically set CRYOSPARC_SSD_PATH based on the CryoSPARC username so that each user has a dedicated cache directory, regardless of the Unix account under which the worker process runs.
Something like this where ${cryosparc_username} reflects the name from CS.
cache.py looks for CRYOSPARC_USER_SSD_TAG and replaces it with the user extracted from email iD. I have tested it and seems to work fine.
Diff cache.py cache.py(modifed )
1097a1098,1122
# Per-cryoSPARC-user SSD cache space: if the resolved cache path contains
# the user tag (default "__cryosparc_user__"), replace the tag with the
# email local part of the cryoSPARC user (not unix user) who created this
# job, e.g. "/fs/cs_scratch/__cryosparc_user__" -> "/fs/cs_scratch/rajan".
# If the tag is absent or the user cannot be resolved, the path is used
# as is (standard behavior, no tag substitution).
if cs_user_tag := os.environ.get("CRYOSPARC_USER_SSD_TAG", "__cryosparc_user__"):
if cs_user_tag in ssd_cache_path:
try:
from core import jobs as core_jobs # local import to avoid cycles
_, job_username = core_jobs.get_job_creator_and_username(rc.get_job())
_csuser = job_username.split("@", 1)[0]
ssd_cache_path = ssd_cache_path.replace(cs_user_tag, _csuser)
rc.log(
f"SSD cache: per-cryoSPARC-user cache for {job_username} at {ssd_cache_path}",
flag="cache",
)
except Exception:
core.logger.warning(
"Could not resolve cryoSPARC user for per-user SSD cache; "
"using static path "
f"{ssd_cache_path} (tag {cs_user_tag!r} not substituted)"
)
One thing to note is that when you connect it expects the SSD to be present. so make a dummy dir in the scratch space.
@Rajan You may want do keep in mind that user’s modifications in CryoSPARC code may be lost during a software upgrade.
Seconding @bsobol’s suggestion, you may want to reconsider whether the workers that are currently “directly” connected could instead be placed under the management of a workload manager, like SLURM. This approach would enable user-specific cache paths without the need to modify CryoSPARC code.