Hi,
I am trying to run RBMC on dataset that was initially processed in cryoSPARC Live and then refined through 2D and 3D classification. When I connect the refined particles and volume, together with the micrographs from a Live Export Job, I get an assertion error before the first movie. This is the detailed log for a failed job:
================= CRYOSPARCW ======= 2025-01-08 20:11:29.571546 =========
Project P94 Job J113
Master neptune.rcac.purdue.edu Port 49002
===========================================================================
MAIN PROCESS PID 32505
========= now starting main process at 2025-01-08 20:11:29.573547
motioncorrection.run_reference_motion cryosparc_compute.jobs.jobregister
MONITOR PROCESS PID 32515
========= monitor process now waiting for main process
========= sending heartbeat at 2025-01-08 20:11:33.370339
========= sending heartbeat at 2025-01-08 20:11:43.393599
***************************************************************
Transparent hugepages setting: always [madvise] never
Running job on hostname %s eris.rcac.purdue.edu
Allocated Resources : {'fixed': {'SSD': False}, 'hostname': 'eris.rcac.purdue.edu', 'lane': 'default', 'lane_type': 'node', 'license': True, 'licenses_acquired': 1, 'slots': {'CPU': [0, 1, 2, 3, 4, 5], 'GPU': [0], 'RAM': [0, 1, 2, 3, 8, 9, 10, 11, 12, 13, 14, 15]}, 'target': {'cache_path': '/fast-scratch', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 47622258688, 'name': 'NVIDIA A40'}, {'id': 1, 'mem': 47622258688, 'name': 'NVIDIA A40'}, {'id': 2, 'mem': 47622258688, 'name': 'NVIDIA A40'}, {'id': 3, 'mem': 47622258688, 'name': 'NVIDIA A40'}], 'hostname': 'eris.rcac.purdue.edu', 'lane': 'default', 'monitor_port': None, 'name': 'eris.rcac.purdue.edu', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], 'GPU': [0, 1, 2, 3], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]}, 'ssh_str': 'cryosparc-bemf@eris.rcac.purdue.edu', 'title': 'Worker node eris.rcac.purdue.edu', 'type': 'node', 'worker_bin_path': '/net/bemf/home/cryosparc-bemf/cryosparc-software/cryosparc2_worker/bin/cryosparcw'}}
FSC No-Mask... 0.143 at 148.950 radwn. 0.5 at 134.860 radwn. Took 4.319s.
FSC With Mask... 0.143 at 159.790 radwn. 0.5 at 149.409 radwn. Took 3.976s.
========= sending heartbeat at 2025-01-08 20:11:53.420307
========= sending heartbeat at 2025-01-08 20:12:03.444729
<string>:1: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
gpufft: creating new cufft plan (plan id 0 pid 32505)
gpu_id 0
ndims 3
dims 340 340 340
inembed 340 340 342
istride 1
idist 39535200
onembed 340 340 171
ostride 1
odist 19767600
batch 1
type R2C
wkspc automatic
Python traceback:
15714653990875575659 (0.049 GiB) -> cache in memory
10864037008023383388 (0.061 GiB) -> cache in memory
10606556534305398933 (0.049 GiB) -> cache in memory
18120478019534469339 (0.037 GiB) -> cache in memory
11073667489598353515 (0.049 GiB) -> cache in memory
10605571126398767723 (0.037 GiB) -> cache in memory
8625133709161805875 (0.024 GiB) -> cache in memory
5552689113636772835 (0.037 GiB) -> cache in memory
1267616596408572724 (0.061 GiB) -> cache in memory
7173874108459549634 (0.049 GiB) -> cache in memory
9392110103329650643 (0.098 GiB) -> cache in memory
8924516076198615890 (0.110 GiB) -> cache in memory
13882869968172647916 (0.134 GiB) -> cache in memory
2131900678677539096 (0.086 GiB) -> cache in memory
17798052215450126310 (0.073 GiB) -> cache in memory
12960270285096631634 (0.049 GiB) -> cache in memory
13552294199752650081 (0.134 GiB) -> cache in memory
9254782200252413070 (0.049 GiB) -> cache in memory
17985484274706761024 (0.049 GiB) -> cache in memory
7195980343085490559 (0.049 GiB) -> cache in memory
1484973523086071607 (0.086 GiB) -> cache in memory
18375290378040571032 (0.086 GiB) -> cache in memory
7854052367247519750 (0.073 GiB) -> cache in memory
12765000161083389904 (0.024 GiB) -> cache in memory
12253591094144565815 (0.049 GiB) -> cache in memory
10001328021126403270 (0.086 GiB) -> cache in memory
6751292344767161235 (0.049 GiB) -> cache in memory
7104051315813147439 (0.073 GiB) -> cache in memory
18400681246918286308 (0.061 GiB) -> cache in memory
16662359859890591801 (0.049 GiB) -> cache in memory
9461952417508905312 (0.098 GiB) -> cache in memory
11000816826172932337 (0.061 GiB) -> cache in memory
15493997870054648833 (0.024 GiB) -> cache in memory
15905436819047007891 (0.073 GiB) -> cache in memory
17780527014207317428 (0.024 GiB) -> cache in memory
12626770713388605487 (0.073 GiB) -> cache in memory
1350337953512542439 (0.122 GiB) -> cache in memory
10767646983914555610 (0.049 GiB) -> cache in memory
17353943957146161304 (0.110 GiB) -> cache in memory
7327901223680126781 (0.024 GiB) -> cache in memory
12754720621754460846 (0.061 GiB) -> cache in memory
1715631833348303851 (0.037 GiB) -> cache in memory
931077635068691098 (0.086 GiB) -> cache in memory
3135253529085551407 (0.073 GiB) -> cache in memory
234826528240029380 (0.049 GiB) -> cache in memory
2412950061585977731 (0.196 GiB) -> cache in memory
5442713648293714801 (0.049 GiB) -> cache in memory
9000915314853726698 (0.024 GiB) -> cache in memory
5286853903278291069 (0.098 GiB) -> cache in memory
8116513821265157072 (0.086 GiB) -> cache in memory
1901553088900389447 (0.024 GiB) -> cache in memory
16373321803803019808 (0.024 GiB) -> cache in memory
7529292124825302355 (0.037 GiB) -> cache in memory
10850838504384546540 (0.086 GiB) -> cache in memory
6318606728364832329 (0.037 GiB) -> cache in memory
15253867493749049051 (0.073 GiB) -> cache in memory
3023897382776363054 (0.024 GiB) -> cache in memory
2169083458071746829 (0.024 GiB) -> cache in memory
14921707849964423298 (0.098 GiB) -> cache in memory
11170450053891024138 (0.110 GiB) -> cache in memory
3343820236152430389 (0.024 GiB) -> cache in memory
16665450894541897366 (0.037 GiB) -> cache in memory
17569767467694230951 (0.049 GiB) -> cache in memory
14343663652404646314 (0.073 GiB) -> cache in memory
592315399227406426 (0.073 GiB) -> cache in memory
4526138443294034674 (0.024 GiB) -> cache in memory
12377487130199457460 (0.073 GiB) -> cache in memory
11513514496130099571 (0.049 GiB) -> cache in memory
6234799202976672058 (0.110 GiB) -> cache in memory
9074099556959108739 (0.024 GiB) -> cache in memory
13841960316125592421 (0.061 GiB) -> cache in memory
12343541842579970290 (0.037 GiB) -> cache in memory
16218676514268370900 (0.086 GiB) -> cache in memory
6285078204829094169 (0.049 GiB) -> cache in memory
3448485034342925488 (0.037 GiB) -> cache in memory
6136166274735476510 (0.049 GiB) -> cache in memory
14214445383765203072 (0.098 GiB) -> cache in memory
3540692389258762581 (0.098 GiB) -> cache in memory
7540804820815744209 (0.037 GiB) -> cache in memory
643330704206797800 (0.073 GiB) -> cache in memory
17476318301664950058 (0.086 GiB) -> cache in memory
6451780909474665762 (0.086 GiB) -> cache in memory
5046748331712060765 (0.024 GiB) -> cache in memory
7660964750566977394 (0.061 GiB) -> cache in memory
9520029389423350705 (0.110 GiB) -> cache in memory
1445781617708338537 (0.098 GiB) -> cache in memory
16139158145418379807 (0.086 GiB) -> cache in memory
14973497472553896449 (0.049 GiB) -> cache in memory
18222643165875583985 (0.037 GiB) -> cache in memory
10398612855050862296 (0.073 GiB) -> cache in memory
13717284295094427042 (0.061 GiB) -> cache in memory
17439483794145683252 (0.073 GiB) -> cache in memory
2909225269076407835 (0.049 GiB) -> cache in memory
2730599087510237478 (0.134 GiB) -> cache in memory
5210472661803402963 (0.049 GiB) -> cache in memory
8158639584767507184 (0.049 GiB) -> cache in memory
9150470190289511376 (0.049 GiB) -> cache in memory
14254403751857171119 (0.049 GiB) -> cache in memory
2484740894449160996 (0.086 GiB) -> cache in memory
10967947260295911591 (0.024 GiB) -> cache in memory
11271456270633530078 (0.049 GiB) -> cache in memory
17273944722559384878 (0.037 GiB) -> cache in memory
226776090773416270 (0.024 GiB) -> cache in memory
6528981407862367407 (0.061 GiB) -> cache in memory
1276568572145366229 (0.024 GiB) -> cache in memory
6198750815331246067 (0.037 GiB) -> cache in memory
9564495679304804873 (0.037 GiB) -> cache in memory
8786541953412554599 (0.037 GiB) -> cache in memory
10232334686804479580 (0.110 GiB) -> cache in memory
10195817194588267960 (0.122 GiB) -> cache in memory
1295303135759593453 (0.061 GiB) -> cache in memory
17757434826562528109 (0.098 GiB) -> cache in memory
17967651932866827412 (0.024 GiB) -> cache in memory
5008404667885597308 (0.024 GiB) -> cache in memory
2053043315548221879 (0.061 GiB) -> cache in memory
6828851979296027487 (0.073 GiB) -> cache in memory
1001233933254296694 (0.086 GiB) -> cache in memory
11647876126361843866 (0.037 GiB) -> cache in memory
10586866615356330434 (0.049 GiB) -> cache in memory
10931159252049120983 (0.073 GiB) -> cache in memory
172280845272985879 (0.049 GiB) -> cache in memory
2387950967612048221 (0.086 GiB) -> cache in memory
7103492019104292055 (0.037 GiB) -> cache in memory
16778859101224323813 (0.061 GiB) -> cache in memory
5198585033676027260 (0.086 GiB) -> cache in memory
5508467302791467142 (0.061 GiB) -> cache in memory
7008895762545611531 (0.098 GiB) -> cache in memory
11676147676940202471 (0.049 GiB) -> cache in memory
9800055970180948279 (0.037 GiB) -> cache in memory
1986484355677113778 (0.024 GiB) -> cache in memory
14684739885712072853 (0.024 GiB) -> cache in memory
6885193754165848779 (0.049 GiB) -> cache in memory
9293924223139844724 (0.024 GiB) -> cache in memory
9593324559469437207 (0.024 GiB) -> cache in memory
6277096285784097486 (0.049 GiB) -> cache in memory
14579032368619361645 (0.049 GiB) -> cache in memory
14897306808406985297 (0.037 GiB) -> cache in memory
3875233978773612078 (0.098 GiB) -> cache in memory
7831231554676907683 (0.037 GiB) -> cache in memory
3314652202107979506 (0.024 GiB) -> cache in memory
13668550961659506106 (0.159 GiB) -> cache in memory
2464727846905574633 (0.037 GiB) -> cache in memory
17189260311725988627 (0.024 GiB) -> cache in memory
13603839260978501496 (0.037 GiB) -> cache in memory
17659543522489275380 (0.122 GiB) -> cache in memory
1052603682312854703 (0.049 GiB) -> cache in memory
1973550165793974350 (0.086 GiB) -> cache in memory
10905599001245453415 (0.049 GiB) -> cache in memory
3773015732988320889 (0.037 GiB) -> cache in memory
4866759708248161953 (0.098 GiB) -> cache in memory
12282510073591128728 (0.061 GiB) -> cache in memory
437518629504945874 (0.061 GiB) -> cache in memory
5117754629713480809 (0.037 GiB) -> cache in memory
9966753264071334630 (0.049 GiB) -> cache in memory
7259762801241214997 (0.037 GiB) -> cache in memory
558731743913098531 (0.134 GiB) -> cache in memory
9696606822897554808 (0.061 GiB) -> cache in memory
9450457116076086453 (0.098 GiB) -> cache in memory
16210190140466225397 (0.049 GiB) -> cache in memory
11916497934559735372 (0.024 GiB) -> cache in memory
9467528016569619868 (0.098 GiB) -> cache in memory
2993503970683802027 (0.037 GiB) -> cache in memory
10948113617181747969 (0.061 GiB) -> cache in memory
781807923825921346 (0.024 GiB) -> cache in memory
13837222412157397543 (0.134 GiB) -> cache in memory
6317541362233033145 (0.098 GiB) -> cache in memory
13630639576925438727 (0.061 GiB) -> cache in memory
15094071000089936618 (0.037 GiB) -> cache in memory
5085152446132979021 (0.061 GiB) -> cache in memory
11741572513122811722 (0.024 GiB) -> cache in memory
16941253661462331870 (0.037 GiB) -> cache in memory
6410458872590131142 (0.024 GiB) -> cache in memory
13208218096087244812 (0.049 GiB) -> cache in memory
12201097002876381277 (0.024 GiB) -> cache in memory
2473036182575579607 (0.110 GiB) -> cache in memory
5751307957807223093 (0.037 GiB) -> cache in memory
1999569456866927847 (0.037 GiB) -> cache in memory
10231319286347091079 (0.098 GiB) -> cache in memory
2804043015072596658 (0.061 GiB) -> cache in memory
13067014861704053367 (0.086 GiB) -> cache in memory
14214063549899658930 (0.049 GiB) -> cache in memory
3505330486295527434 (0.049 GiB) -> cache in memory
963239243267765022 (0.049 GiB) -> cache in memory
4749656791283306138 (0.073 GiB) -> cache in memory
6822831811542037892 (0.061 GiB) -> cache in memory
17181818276358638042 (0.037 GiB) -> cache in memory
598231551170728076 (0.134 GiB) -> cache in memory
9080208449031785229 (0.024 GiB) -> cache in memory
1554068282539095344 (0.110 GiB) -> cache in memory
5285150084612905998 (0.024 GiB) -> cache in memory
16556955313518397255 (0.061 GiB) -> cache in memory
17890262785570703762 (0.037 GiB) -> cache in memory
15416726700149114061 (0.024 GiB) -> cache in memory
2676959520495439832 (0.073 GiB) -> cache in memory
1107417310952754468 (0.049 GiB) -> cache in memory
6735867641201037220 (0.049 GiB) -> cache in memory
15834504582153208571 (0.110 GiB) -> cache in memory
9108668753366080799 (0.098 GiB) -> cache in memory
14286855708859548733 (0.049 GiB) -> cache in memory
8660076498138362089 (0.049 GiB) -> cache in memory
8714437203130944951 (0.061 GiB) -> cache in memory
15679327404879310621 (0.024 GiB) -> cache in memory
10624823754000044351 (0.073 GiB) -> cache in memory
10404792009263690709 (0.110 GiB) -> cache in memory
16925390332812076883 (0.037 GiB) -> cache in memory
10705203682398954673 (0.049 GiB) -> cache in memory
17921562857170977555 (0.024 GiB) -> cache in memory
4948682153618666575 (0.037 GiB) -> cache in memory
3244137855999122797 (0.024 GiB) -> cache in memory
11271070335686945026 (0.086 GiB) -> cache in memory
6309519700484464561 (0.122 GiB) -> cache in memory
12733325572465468028 (0.061 GiB) -> cache in memory
10749044096693316057 (0.037 GiB) -> cache in memory
3018530929379704932 (0.049 GiB) -> cache in memory
7493791122422022575 (0.061 GiB) -> cache in memory
16461020008314510987 (0.024 GiB) -> cache in memory
482887464612195314 (0.098 GiB) -> cache in memory
7153767462448833097 (0.049 GiB) -> cache in memory
16982625421022038968 (0.134 GiB) -> cache in memory
4366531804967673732 (0.049 GiB) -> cache in memory
4861137383515485440 (0.049 GiB) -> cache in memory
2048805501083043269 (0.061 GiB) -> cache in memory
15139186591863477523 (0.086 GiB) -> cache in memory
17132550563344926442 (0.037 GiB) -> cache in memory
1801405611850047782 (0.134 GiB) -> cache in memory
15033980900519932590 (0.024 GiB) -> cache in memory
4726544752325458453 (0.037 GiB) -> cache in memory
10049943256219347793 (0.098 GiB) -> cache in memory
8253081422782388477 (0.024 GiB) -> cache in memory
12679658741853204940 (0.037 GiB) -> cache in memory
4500457600842736285 (0.037 GiB) -> cache in memory
4208770400987952100 (0.098 GiB) -> cache in memory
18429776197861862248 (0.061 GiB) -> cache in memory
2194023647833113513 (0.086 GiB) -> cache in memory
17738922197321991848 (0.024 GiB) -> cache in memory
17716974514768481141 (0.073 GiB) -> cache in memory
9171761853233470145 (0.037 GiB) -> cache in memory
10608368486307040432 (0.110 GiB) -> cache in memory
16991262453024009434 (0.049 GiB) -> cache in memory
3743232945621049946 (0.049 GiB) -> cache in memory
3340962448156008394 (0.061 GiB) -> cache in memory
438675994500384680 (0.098 GiB) -> cache in memory
18170022810151428680 (0.073 GiB) -> cache in memory
316310205512751859 (0.061 GiB) -> cache in memory
6196174431843661499 (0.024 GiB) -> cache in memory
5565891906968975881 (0.049 GiB) -> cache in memory
567194861746076199 (0.049 GiB) -> cache in memory
12645955682756910465 (0.110 GiB) -> cache in memory
13158065322993079915 (0.049 GiB) -> cache in memory
1375023995855176522 (0.024 GiB) -> cache in memory
10727129648016088301 (0.024 GiB) -> cache in memory
7553986261869870281 (0.086 GiB) -> cache in memory
7289849139539520766 (0.073 GiB) -> cache in memory
10964011082914585485 (0.037 GiB) -> cache in memory
12471441044785324685 (0.049 GiB) -> cache in memory
10498826689034845547 (0.110 GiB) -> cache in memory
4280062266859535384 (0.159 GiB) -> cache in memory
13484485082468055411 (0.073 GiB) -> cache in memory
14522784652241654328 (0.037 GiB) -> cache in memory
2622965413111138770 (0.061 GiB) -> cache in memory
7698100992537926052 (0.110 GiB) -> cache in memory
1457293893703299661 (0.159 GiB) -> cache in memory
15991562173429986199 (0.049 GiB) -> cache in memory
12693325164658696217 (0.086 GiB) -> cache in memory
11193858643252154568 (0.134 GiB) -> cache in memory
12158748661323737102 (0.134 GiB) -> cache in memory
5700041102543838584 (0.073 GiB) -> cache in memory
3800288664257937182 (0.061 GiB) -> cache in memory
14614847183728001151 (0.024 GiB) -> cache in memory
8615677510340457070 (0.073 GiB) -> cache in memory
18271602286015728172 (0.073 GiB) -> cache in memory
1951098391588247526 (0.086 GiB) -> cache in memory
1233626519470154979 (0.049 GiB) -> cache in memory
1720503050742200806 (0.086 GiB) -> cache in memory
10530290742827128916 (0.073 GiB) -> cache in memory
14902887966877017560 (0.037 GiB) -> cache in memory
6170816249429259650 (0.086 GiB) -> cache in memory
2628493653940172219 (0.049 GiB) -> cache in memory
17783331459564809639 (0.122 GiB) -> cache in memory
2567092520321232378 (0.024 GiB) -> cache in memory
9334002645591700301 (0.049 GiB) -> cache in memory
13827229482143404466 (0.024 GiB) -> cache in memory
11853265194960566479 (0.086 GiB) -> cache in memory
10892380298793088721 (0.049 GiB) -> cache in memory
17854251487848365036 (0.024 GiB) -> cache in memory
4791229382338290822 (0.024 GiB) -> cache in memory
2520745131388912033 (0.049 GiB) -> cache in memory
8918964147623399936 (0.037 GiB) -> cache in memory
HOST ALLOCATION FUNCTION: using numba.cuda.pinned_array
SEARCH RANGES:
zs:
6.2146
thetas:
-2.0420
-2.1468
-2.2515
-2.3562
-2.4609
-2.5656
-2.6704
r start:
0.1000
r end:
10.0000
r step:
0.6600
**** handle exception rc
Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/run.py", line 129, in cryosparc_master.cryosparc_compute.run.main
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py", line 374, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/refmotion.py", line 964, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.refmotion.mainfn_param_optimize
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/refmotion.py", line 982, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.refmotion.mainfn_param_optimize
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/refmotion.py", line 649, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.refmotion.slice_vol
File "cryosparc_master/cryosparc_compute/engine/newengine.py", line 381, in cryosparc_master.cryosparc_compute.engine.newengine.EngineThread.setup_comparison
AssertionError
set status to failed
========= main process now complete at 2025-01-08 20:12:13.467975.
========= monitor process now complete at 2025-01-08 20:12:13.473512.The box size of the particles is 640px, but the same error occurs when I downsample to 340px. I tried four different machines with different configurations of A40 and RTX 2080 Ti, without any success. Other RBMC jobs in different projects that followed the same workflow run without any issues on the same instance, so any suggestions to what might be going wrong here would be greatly appreciated.