Job/project forced removal for a deleted project directory

Troubleshooting
1

I need to delete a job from a deleted project. All job related data have been deleted at the filesystem level: the project was created on incorrect filesystem location and then was deleted after job submission.

I tried the cli command:

cryosparcm cli "set_job_status('P1', 'J2', 'killed')"
Encountered error for method "set_job_status" with params ('P1', 'J2', 'killed'):
ServerError: validation error: lock file for P1 not found at /home_local/cryosparc/username/P1/cs.lock
Traceback (most recent call last):
  File "/home_local/cryosparc/cryosparc2_master/cryosparc_command/commandcommon.py", line 139, in wrapper
    res = func(*args, **kwargs)
  File "/home_local/cryosparc/cryosparc2_master/cryosparc_command/commandcommon.py", line 191, in wrapper
    lockfile_path_abs), f"validation error: lock file for {project_uid} not found at {lockfile_path_abs}"
AssertionError: validation error: lock file for P1 not found at /home_local/cryosparc/username/P1/cs.lock

Creating cs.lock file is obviously insufficient to remove it as well.

Is there any low level database command to solve this ?
Thank you
JC

2

Hi @jcducom,

You can try:

cryosparcm icli
# to delete the job
db.jobs.update_one({'uid': 'J2', 'project_uid': 'P1'}, {'$set': {'deleted': True}})
# and if you would like to delete the project too
db.projects.update_one({'uid':'P75'}, {'$set': {'deleted':True}})

Hope this helps!

1 Like
3

Thank you for the quick reply. It worked! Thank you
JC

1 Like
4

Hi,

how do we delete a session?
In the cli reference, I don’t find a delete_session tho.

Regards,

5

@qitsweauca Please describe your circumstances in more detail:

  • Have project data related to the session been removed?
  • What have you tried?
  • How have your attempts failed?
6

Hi @wtempel,

Sorry about not yet provide sufficient info, and thank you for following up.

  • yes the project data related to the live session has been accidentally remove from the file system, and in ‘cryosparcm log command_core’, it continuously reports the following JSONRPC ERROR at dump_workspaces repeatedly.

JSONRPC ERROR at dump_workspaces
2023-10-06 09:54:04,510 COMMAND.COMMON wrapper ERROR | Traceback (most recent call last):
2023-10-06 09:54:04,510 COMMAND.COMMON wrapper ERROR | File “/apps/cryosparc_master/cryosparc_command/commandcommon.py”, line 200, in wrapper
2023-10-06 09:54:04,510 COMMAND.COMMON wrapper ERROR | res = func(*args, **kwargs)
2023-10-06 09:54:04,510 COMMAND.COMMON wrapper ERROR | File “/apps/cryosparc_master/cryosparc_command/commandcommon.py”, line 262, in wrapper
2023-10-06 09:54:04,510 COMMAND.COMMON wrapper ERROR | assert not project[‘archived’], f"validation error: project {project_uid} is archived"
2023-10-06 09:54:04,510 COMMAND.COMMON wrapper ERROR | AssertionError: validation error: project P38 is archived

  • I tried serveral attempts to manually remove the project from the mongodb database via cryosparc icli, but currently there have been no success… including update_one with ‘deleted’: True.
    I also tried to take over the project again within cryosparc icli: cli.take_over_project(project_uid=‘P38’, force=True) and then cli.delete_project(project_uid=‘P38’), but the log command_core still continuously do the polling error of JSONRPC.

  • If I search through db.workspaces.find with running:

list(db.workspaces.find({‘session_uid’: {‘$ne’: None}, ‘status’: {‘$nin’: [‘paused’, ‘completed’]}}, {‘project_uid’: 1, ‘session_uid’: 1, ‘title’: 1, ‘status’: 1, ‘deleted’: True}))

The response from the mongodb is still:

[{‘_id’: ObjectId(‘641b90853415c6e11b23f9c1’),
‘project_uid’: ‘P38’,
‘session_uid’: ‘S5’,
‘title’: ‘TestwithSample’,
‘deleted’: False,
‘status’: ‘running’}]

Are there any ways to stop the continuous polling and clean the database glitch?

Thank you!

7

@qitsweauca Caution: We are not sure about the current state of your CryoSPARC instance and therefore unsure about the ultimate effect the following suggestions may have. Please backup your database and ensure there are no legitimate Live sessions running on your CryoSPARC instance if you wish to proceed (at your own risk):

# cryosparcm icli
import datetime
try:
    rtp.pause_session('P38', 'S5')
except:
    db.workspaces.update_one({'project_uid' : 'P38', 'session_uid' : 'S5'}, { '$set' : {'status' : 'paused'}, '$push' : {'paused_at' : datetime.datetime.utcnow()}})

(see Logs are being flooded - #3 by stephan)
Then, identify and “correct” inconsistent CryoSPARC sessions as described in No jobs showing up after recreating database - #6 by nfrasser.
[Edited 2023-10-11: try rtp.pause_session() first; ensure no running Live sessions]

1 Like
8

@wtempel
Thank you so much!! It successfully helps stopping the polling session.