Refused connection when cryosparc is running

[wtempel]

Thanks @Ana for posting the information.

Jul 11 12:51:32 sn4622119118 systemd-oomd[1771]: Killed /user.slice/user-1001.slice/user@1001.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-1f03abf9-d45b-4872-8707-eded864790df.scope due to memory pressure for /user.slice/user-1001.slice/user@1001.service being 71.69% > 50.00% for > 20s with reclaim activity
Jul 11 12:51:32 sn4622119118 systemd[34133]: vte-spawn-1f03abf9-d45b-4872-8707-eded864790df.scope: systemd-oomd killed 184 process(es) in this unit.
Jul 11 13:25:10 sn4622119118 systemd-oomd[1771]: Killed /user.slice/user-1001.slice/user@1001.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-ac9ed476-5ea3-426b-a9b0-03e39ecb7b79.scope due to memory pressure for /user.slice/user-1001.slice/user@1001.service being 78.28% > 50.00% for > 20s with reclaim activity
Jul 11 13:25:10 sn4622119118 systemd[34133]: vte-spawn-ac9ed476-5ea3-426b-a9b0-03e39ecb7b79.scope: systemd-oomd killed 154 process(es) in this unit.
Jul 11 13:39:52 sn4622119118 systemd-oomd[1771]: Killed /user.slice/user-1001.slice/user@1001.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-1a967f77-46b7-4d25-8d88-8c31a0f98cda.scope due to memory pressure for /user.slice/user-1001.slice/user@1001.service being 73.26% > 50.00% for > 20s with reclaim activity
Jul 11 13:39:52 sn4622119118 systemd[34133]: vte-spawn-1a967f77-46b7-4d25-8d88-8c31a0f98cda.scope: systemd-oomd killed 71 process(es) in this unit.
[..]
Jul 11 15:53:23 sn4622119118 systemd-oomd[1966]: Killed /user.slice/user-1001.slice/user@1001.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-74e968b6-b1af-4207-aa00-506291ef058e.scope due to memory pressure for /user.slice/user-1001.slice/user@1001.service being 78.39% > 50.00% for > 20s with reclaim activity
Jul 11 15:53:23 sn4622119118 systemd[2883]: vte-spawn-74e968b6-b1af-4207-aa00-506291ef058e.scope: systemd-oomd killed 59 process(es) in this unit.
Jul 11 16:43:51 sn4622119118 systemd-oomd[1966]: Killed /user.slice/user-1001.slice/user@1001.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-b7606e22-0cc2-4019-b48b-9e768c56a254.scope due to memory pressure for /user.slice/user-1001.slice/user@1001.service being 75.31% > 50.00% for > 20s with reclaim activity
Jul 11 16:43:51 sn4622119118 systemd[2883]: vte-spawn-b7606e22-0cc2-4019-b48b-9e768c56a254.scope: systemd-oomd killed 23 process(es) in this unit.
Jul 12 16:06:18 sn4622119118 systemd-oomd[1966]: Killed /user.slice/user-1001.slice/user@1001.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-55cfc509-e689-44dd-8eb8-3509459f29ab.scope due to memory pressure for /user.slice/user-1001.slice/user@1001.service being 54.87% > 50.00% for > 20s with reclaim activity
Jul 12 16:06:18 sn4622119118 systemd[2883]: vte-spawn-55cfc509-e689-44dd-8eb8-3509459f29ab.scope: systemd-oomd killed 198 process(es) in this unit.
Jul 12 16:19:15 sn4622119118 systemd-oomd[1966]: Killed /user.slice/user-1001.slice/user@1001.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-a54a40c7-8154-44d4-8ac0-3775e7fe3fb2.scope due to memory pressure for /user.slice/user-1001.slice/user@1001.service being 80.66% > 50.00% for > 20s with reclaim activity
Jul 12 16:19:15 sn4622119118 systemd[2883]: vte-spawn-a54a40c7-8154-44d4-8ac0-3775e7fe3fb2.scope: systemd-oomd killed 113 process(es) in this unit.
Jul 12 17:11:27 sn4622119118 systemd-oomd[1966]: Killed /user.slice/user-1001.slice/user@1001.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-4cebc8d9-ea3a-4f7f-9b9f-d5ecd0cd2299.scope due to memory pressure for /user.slice/user-1001.slice/user@1001.service being 63.57% > 50.00% for > 20s with reclaim activity
Jul 12 17:11:27 sn4622119118 systemd[2883]: vte-spawn-4cebc8d9-ea3a-4f7f-9b9f-d5ecd0cd2299.scope: systemd-oomd killed 124 process(es) in this unit.
[..]
Jul 12 18:19:44 sn4622119118 systemd-oomd[1930]: Killed /user.slice/user-1001.slice/user@1001.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-f67df135-9fab-4b6e-9c3b-7c6b3502bd45.scope due to memory pressure for /user.slice/user-1001.slice/user@1001.service being 53.98% > 50.00% for > 20s with reclaim activity
Jul 12 18:19:44 sn4622119118 systemd[2781]: vte-spawn-f67df135-9fab-4b6e-9c3b-7c6b3502bd45.scope: systemd-oomd killed 69 process(es) in this unit.
[..]
Jul 12 19:33:23 sn4622119118 systemd-oomd[1928]: Killed /user.slice/user-1001.slice/user@1001.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-2ce4f9ab-9129-4267-b060-ea0626d0a2e5.scope due to memory pressure for /user.slice/user-1001.slice/user@1001.service being 84.29% > 50.00% for > 20s with reclaim activity
Jul 12 19:33:23 sn4622119118 systemd[2836]: vte-spawn-2ce4f9ab-9129-4267-b060-ea0626d0a2e5.scope: systemd-oomd killed 146 process(es) in this unit.
[..]
Jul 15 10:38:01 sn4622119118 systemd-oomd[1927]: Killed /user.slice/user-1001.slice/user@1001.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-2ab401dc-78bc-46e1-adec-fbaccf435459.scope due to memory pressure for /user.slice/user-1001.slice/user@1001.service being 77.04% > 50.00% for > 20s with reclaim activity
Jul 15 10:38:01 sn4622119118 systemd[2835]: vte-spawn-2ab401dc-78bc-46e1-adec-fbaccf435459.scope: systemd-oomd killed 145 process(es) in this unit.
Jul 15 17:10:23 sn4622119118 systemd-oomd[1927]: Killed /user.slice/user-1001.slice/user@1001.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-c29da3e7-5fa7-4ea5-af5c-a814cfa080f4.scope due to memory pressure for /user.slice/user-1001.slice/user@1001.service being 68.23% > 50.00% for > 20s with reclaim activity
Jul 15 17:10:23 sn4622119118 systemd[2835]: vte-spawn-c29da3e7-5fa7-4ea5-af5c-a814cfa080f4.scope: systemd-oomd killed 94 process(es) in this unit.

Do these timestamps in any way correlate with events you described?

@Ana

This usually happens when I run either 2D classification or homogeneous refinement job, I haven’t noticed it happening with extraction.
Terminal gets closed and cryosparc gets disconnected, a loading sign appears on the screen and when I refresh the page, it says “unable to connect”.

[Ana]

Yes, these are the days and approximate times I’ve been facing this problem.

[wtempel]

@Ana

these are the days and approximate times I’ve been facing this problem.

Based on the information provided:

@Ana

usually happens when I run either 2D classification or homogeneous refinement job, I haven’t noticed it happening with extraction.

I hypothesize that certain combinations of workloads trigger the (configurable?) conditions

[quote="wtempel, post:23, topic:12832"]
`> 50.00% for > 20s with reclaim activity`
[/quote]
for systemd-oomd to `kill` processes, including some CryoSPARC processes in a way that prevents those processes from "cleaning up after themselves". This _could_ explain the presence of an orphaned

/tmp/cryosparc-supervisor-206773da3c7c06e952eddaffaea9188d.sock

file. Before removing the file, please confirm that the corresponding supervisord process is in fact no longer running (related discussion).
A google.com search indicates that some consider systemd-oomd as aggressive and suggest a variety of interventions. May I suggest a consultation with your system administrator about a potential reconfiguration, and the avoidance of workload patterns that exhaust available system RAM.