5.0.6: 2d classification crash

Troubleshooting
1

Hi

One user has problems with his 2d classification jobs. They start and then crash immediately.

Here an example of a log:

================= CRYOSPARC =================
Project P165 Job J1065
Master cryosparcmaster. Port 39000
===========================================================================
MAIN PROCESS PID 2443129
2026-07-10 12:25:53,210 core                 monitor          INFO   | MONITOR PROCESS PID 2443137
2026-07-10 12:25:53,210 core                 monitor          INFO   | ========= monitor process now waiting for main process
2026-07-10 12:25:53,210 core                 heartbeat        INFO   | ========= Updating heartbeat
================= CRYOSPARC =================
Project P165 Job J1065
Master cryosparcmaster. Port 39000
===========================================================================
MAIN PROCESS PID 2443129
========= updating job startup information at 2026-07-10 12:25:53.789202
========= now starting main process at 2026-07-10 12:25:54.198424
2026-07-10 12:25:54,351 core                 run              INFO   | Running job J1065 of type class_2D_new
2026-07-10 12:25:54,351 core                 run              INFO   | Running job on hostname gpu01.
2026-07-10 12:25:54,351 core                 run              INFO   | Allocated Resources: lane='RTX2080' lane_type='node' hostname='gpu01.' target=SchedulerTarget(cache_path='/disks/cryosparc-cache/cache', cache_reserve_mb=1024, cache_quota_mb=None, lane='RTX2080', name='gpu01.', title='Worker node gpu01.', desc=None, hostname='gpu01.', worker_bin_path='/opt/cryosparc/cryosparc_worker/bin/cryosparcw', config=Node(type='node', ssh_str='cryosparcuser@@gpu01.', resource_slots=ResourceSlots(CPU=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], GPU=[0, 1, 2, 3], RAM=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]), resource_fixed=FixedResourceSlots(SSD=True), monitor_port=None, gpus=[Gpu(id=0, name='NVIDIA GeForce RTX 2080 Ti', mem=11347623936), Gpu(id=1, name='NVIDIA GeForce RTX 2080 Ti', mem=11347623936), Gpu(id=2, name='NVIDIA GeForce RTX 2080 Ti', mem=11347623936), Gpu(id=3, name='NVIDIA GeForce RTX 2080 Ti', mem=11347623936)])) slots=ResourceSlots(CPU=[0, 1, 2, 3], GPU=[3], RAM=[9, 10, 11]) fixed=FixedResourceSlots(SSD=True) licenses_acquired=1
Transparent hugepages setting: always madvise [never]

2026-07-10 12:25:59,147 core.jobs            load_output      WARNIN | Added 2 missing field(s) to Job P165-J852 (Select 2D Classes) output particles_selected result pick_stats dset
2026-07-10 12:25:59,147 core.jobs            load_output      WARNIN | Added 1 missing field(s) to Job P165-J852 (Select 2D Classes) output particles_selected result location dset
2026-07-10 12:25:59,660 core.jobs            load_output      WARNIN | Added 2 missing field(s) to Job P165-J853 (Select 2D Classes) output particles_selected result pick_stats dset
2026-07-10 12:25:59,660 core.jobs            load_output      WARNIN | Added 1 missing field(s) to Job P165-J853 (Select 2D Classes) output particles_selected result location dset
2026-07-10 12:25:59,985 core.jobs            load_output      WARNIN | Added 2 missing field(s) to Job P165-J854 (Select 2D Classes) output particles_selected result pick_stats dset
2026-07-10 12:25:59,986 core.jobs            load_output      WARNIN | Added 1 missing field(s) to Job P165-J854 (Select 2D Classes) output particles_selected result location dset
2026-07-10 12:26:01,177 core                 run_with_executo INFO   | Resolving 19918 source path(s) for caching
2026-07-10 12:26:03,827 core                 heartbeat        INFO   | ========= Updating heartbeat
2026-07-10 12:26:05,841 core                 run_with_executo INFO   | Resolved 19918 sources in 4.66 seconds
2026-07-10 12:26:05,886 core                 lock_job_run     INFO   | Lock ssd_cache acquired by P165-J1065-1783679152
2026-07-10 12:26:05,888 core                 allocate         INFO   | Cache allocation start. Active run IDs: P115-J1723-1783333655, P122-J3423-1783452687, P165-J1058-1783672002, P165-J1061-1783672022, P174-J2572-1783677176, P174-J2574-1783668225, P165-J1065-1783679152, P174-J2575-1783678273, P174-J2576-1783678711, P174-J2577-1783678713, P174-J2578-1783678715, P174-J2579-1783678717, P174-J2580-1783678731, P174-J2581-1783678733, P174-J2582-1783678735, P174-J2583-1783678737, P174-J2584-1783678739, P174-J2585-1783678741, P174-J2586-1783678743, P174-J2587-1783678745, P174-J2588-1783678746, P288-J16-1783679109, P57-J717-1782838320, P57-J746-1782888729, P57-J747-1782888850, P57-J750-1782894019, P57-J764-1782927539, P57-J803-1782964426, P57-J804-1782964480, P57-J840-1782975332, P57-J841-1782975351, P57-J842-1782975373, P57-J845-1782975443, P57-J849-1782979708, P57-J851-1782979685
2026-07-10 12:26:06,718 core                 refresh          INFO   | Refreshed cache drive in 0.83 seconds
2026-07-10 12:26:06,750 core                 allocate         INFO   | Deleted 0 cached files, encountered 0 errors
2026-07-10 12:26:06,750 core                 allocate         INFO   | Allocated 0 stub cache files; creating links
2026-07-10 12:26:07,362 core                 allocate         INFO   | Cache allocation complete
2026-07-10 12:26:07,364 core                 unlock_job_run   INFO   | Releasing lock ssd_cache from P165-J1065-1783679152
2026-07-10 12:26:07,365 core                 run_with_executo INFO   | Cache allocation ran in 1.48 seconds
2026-07-10 12:26:07,365 core                 run_with_executo INFO   | Found 19918 SSD hit(s)
2026-07-10 12:26:07,365 core                 run_with_executo INFO   | Requested files successfully cached to SSD
2026-07-10 12:26:08,243 core                 run_with_executo INFO   | SSD cache complete
2026-07-10 12:26:13,835 core                 heartbeat        INFO   | ========= Updating heartbeat
WARNING: io_uring support disabled (supported by kernel, but initialization failed), I/O performance may degrade
Received SIGSEGV (addr=0000000000000000)
/opt/cryosparc/cryosparc_worker/core/ioengine/core.so(traceback_signal_handler+0x113)[0x7fcb19e1c873]
/lib64/libc.so.6(+0x3fc60)[0x7fcb3403fc60]
/lib64/libc.so.6(+0x15d88b)[0x7fcb3415d88b]
/opt/cryosparc/cryosparc_worker/core/ioengine/core.so(get_npy_dtype_from_str+0x18)[0x7fcb19e4e118]
/opt/cryosparc/cryosparc_worker/core/ioengine/core.so(wrap_array_from_iocmd+0x60)[0x7fcb19e4e350]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x21bc08)[0x556fc0639c08]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(_PyObject_MakeTpCall+0x2cc)[0x556fc061d17c]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x112b23)[0x556fc0530b23]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(PyObject_VectorcallMethod+0x87)[0x556fc0673597]
/opt/cryosparc/cryosparc_worker/compute/jobs/class2D/run.cpython-312-x86_64-linux-gnu.so(+0x71bc4)[0x7fcb2178abc4]
/opt/cryosparc/cryosparc_worker/compute/jobs/class2D/run.cpython-312-x86_64-linux-gnu.so(+0xc6300)[0x7fcb217df300]
/opt/cryosparc/cryosparc_worker/cli/run.cpython-312-x86_64-linux-gnu.so(+0x10b5d)[0x7fcb32f20b5d]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(PyObject_Vectorcall+0x2e)[0x556fc062fdde]
/opt/cryosparc/cryosparc_worker/cli/run.cpython-312-x86_64-linux-gnu.so(+0x20ef4)[0x7fcb32f30ef4]
/opt/cryosparc/cryosparc_worker/cli/run.cpython-312-x86_64-linux-gnu.so(+0x28250)[0x7fcb32f38250]
/opt/cryosparc/cryosparc_worker/cli/main.cpython-312-x86_64-linux-gnu.so(+0x91a7)[0x7fcb335801a7]
/opt/cryosparc/cryosparc_worker/cli/main.cpython-312-x86_64-linux-gnu.so(+0xbd02)[0x7fcb33582d02]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(PyObject_Vectorcall+0x2e)[0x556fc062fdde]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x112b23)[0x556fc0530b23]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x24523c)[0x556fc066323c]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x244d9e)[0x556fc0662d9e]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(_PyObject_Call+0x12b)[0x556fc064b11b]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x1136f0)[0x556fc05316f0]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x24523c)[0x556fc066323c]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x244d9e)[0x556fc0662d9e]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(_PyObject_Call+0x12b)[0x556fc064b11b]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x1136f0)[0x556fc05316f0]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(_PyObject_FastCallDictTstate+0x292)[0x556fc061fc12]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(_PyObject_Call_Prepend+0x69)[0x556fc0648639]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x2f70cb)[0x556fc07150cb]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(_PyObject_Call+0xb5)[0x556fc064b0a5]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x1136f0)[0x556fc05316f0]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(_PyObject_FastCallDictTstate+0x292)[0x556fc061fc12]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(_PyObject_Call_Prepend+0x69)[0x556fc0648639]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x2f70cb)[0x556fc07150cb]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(_PyObject_MakeTpCall+0x2cc)[0x556fc061d17c]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x112b23)[0x556fc0530b23]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(PyEval_EvalCode+0xa1)[0x556fc06ce1a1]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x2ea8ca)[0x556fc07088ca]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x2e5585)[0x556fc0703585]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x2e2620)[0x556fc0700620]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(_PyRun_SimpleFileObject+0x1ce)[0x556fc07002be]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(_PyRun_AnyFileObject+0x44)[0x556fc06fffe4]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(Py_RunMain+0x3a2)[0x556fc06fceb2]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(Py_BytesMain+0x37)[0x556fc06b8247]
/lib64/libc.so.6(+0x2a610)[0x7fcb3402a610]
/lib64/libc.so.6(__libc_start_main+0x80)[0x7fcb3402a6c0]
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/bin/python(+0x29a0ed)[0x556fc06b80ed]
rax 0000000098d00000  rbx 0000000000000000  rcx f3ffffefffffff00  rdx 00007fcb19e63680  
rsi 00007fcb19e5298d  rdi 0000000000000000  rbp 00007ffe06cb3200  rsp 00007ffe06cb31e8  
r8  0000000000000000  r9  0000000000000005  r10 0000000000000001  r11 0000000000000000  
r12 0000556fc0aa6910  r13 00007fcb21a37380  r14 0000000000000000  r15 00007fcb34aa4ce0  
c3 c4 e2 6a f7 c0 85 c0 0f 84 bb fe ff ff f3 0f bc c0 48 01 d0 c5 f8 77 c3 66 2e 0f 1f
84 00 00 00 00 00 66 90 f3 0f 1e fa c4 41 01 ef ff 89 f8 09 f0 c1 e0 14 3d 00 00 00 f8
0f 87 25 03 00 00
-->   c5 fe 6f 07 c5 fd 74 0e c5 85 74 d0 c5 ed df c9 c5 fd d7 c9 ff c1 74 5d f3 0f bc
c9 0f b6 04 0f 0f b6 0c 0e 29 c8 c5 f8 77 c3 f3 0f bc c9 0f b6 44 0f 20 0f b6 4c 0e 20
29 c8 c5 f8 77 c3 0f 1f

Any idea what that could trigger?

2

@biocit Please can you post the output of the command

cryosparcm cli "api.jobs.find_one('P165', 'J1065').instance_information"
3

Sure

{"platform_node": "gpu01", "platform_release": "5.14.0-687.15.1.el9_8.x86_64", "platform_version": "#1 SMP PREEMPT_DYNAMIC Thu Jun 11 16:33:25 UTC 2026", "platform_architecture": "x86_64", "cpu_model": "AMD EPYC 7742 64-Core Processor", "physical_cores": 64, "max_cpu_freq": 2250.0, "total_memory": "125.24GB", "available_memory": "108.08GB", "used_memory": "8.95GB", "ofd_soft_limit":
1024, "ofd_hard_limit": 524288, "driver_version": "13.3", "toolkit_version": "12.8", "CUDA_version": "12.8", "nvrtc_version": "12.8", "gpu_info": [{"id": 3, "name": "NVIDIA GeForce RTX 2080 Ti",
"mem": 11811160064, "bus_id": "0000:c1:00", "compute_mode": "Default", "persistence_mode": "Enabled", "power_limit": 250.0, "sw_power_limit": "Not Active", "hw_power_limit": "Not Active", "max_pcie_link_gen": 3, "current_pcie_link_gen": 1, "temperature": 25, "gpu_utilization": 0, "memory_utilization": 0, "driver_version": "610.43.02"}], "version": "v5.0.6"}