@Panoskre is using the cryosparc instance that i operate.
Topaz: 0.2.5a
Cryosparc Version: 4.6.2
================= CRYOSPARCW ======= 2025-02-13 13:28:21.972709 =========
Project P100 Job J1023
Master cryosparcmaster Port 39002
===========================================================================
MAIN PROCESS PID 2384876
========= now starting main process at 2025-02-13 13:28:21.973416
topaz.run_topaz cryosparc_compute.jobs.jobregister
MONITOR PROCESS PID 2384878
========= monitor process now waiting for main process
========= sending heartbeat at 2025-02-13 13:28:23.000400
========= sending heartbeat at 2025-02-13 13:28:33.013967
========= sending heartbeat at 2025-02-13 13:28:43.027943
***************************************************************
Transparent hugepages setting: always madvise [never]
Running job on hostname %s gpu02
**** handle exception rc
Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/run.py", line 129, in cryosparc_master.cryosparc_compute.run.main
File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 332, in run_topaz_wrapper_train
utils.run_process(split_command)
File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 99, in run_process
assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})"
AssertionError: Subprocess exited with status 1 (/programs/x86_64-linux/topaz/0.2.5a_cu11.3_py36/bin/topaz train_test_split --number 109 --seed 1420656222 --image-dir /mnt/xxx/data04/xxxx/cryosparc/CS-xxxx-cftr-vesicles/J566 /mnt/xxx/data04/xxxx/cryosparc/CS-xxx-cftr-vesicles/J1023/topaz…)
set status to failed
========= main process now complete at 2025-02-13 13:28:53.042279.
========= monitor process now complete at 2025-02-13 13:28:53.046268.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "006231731896023480250_FoilHole_27059231_Data_27066474_26_20250128_225111_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "001904820508562434347_FoilHole_27054202_Data_27065497_16_20250128_173455_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "005613296379436869938_FoilHole_27061933_Data_27065500_33_20250129_022033_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "004390728773744558122_FoilHole_27042162_Data_27065500_12_20250128_060528_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "017992005504552028817_FoilHole_27032796_Data_27065497_35_20250127_230817_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "006152693300718219366_FoilHole_28124267_Data_27066474_25_20250129_065353_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "002424809494787319258_FoilHole_27060774_Data_27065497_22_20250129_005032_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "009997577164604924640_FoilHole_27037549_Data_27065497_12_20250128_023151_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "000170584651708938755_FoilHole_27030159_Data_27066474_10_20250127_205906_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "006280494022743254927_FoilHole_27058042_Data_27066474_3_20250128_211804_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "013654983446909551547_FoilHole_27027190_Data_27065500_22_20250127_181135_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "016886265913036348510_FoilHole_27048767_Data_27065497_21_20250128_123328_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "005226442435868559764_FoilHole_27060829_Data_27065497_1_20250129_005729_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] WARNING: no micrograph found matching image name "003369386825400160251_FoilHole_27058044_Data_27065500_10_20250128_212755_fractions_patch_aligned_doseweighted". Skipping it.
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] Traceback (most recent call last):
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] File "/programs/x86_64-linux/topaz/0.2.5a_cu11.3_py36/bin/topaz", line 8, in <module>
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] sys.exit(main())
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] File "/programs/x86_64-linux/topaz/0.2.5a_cu11.3_py36/topaz_extlib/envs/topaz-py3.6/lib/python3.6/site-packages/topaz/main.py", line 148, in main
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] args.func(args)
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] File "/programs/x86_64-linux/topaz/0.2.5a_cu11.3_py36/topaz_extlib/envs/topaz-py3.6/lib/python3.6/site-packages/topaz/commands/train_test_split.py", line 128, in main
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] image_list_train = pd.DataFrame({'image_name': image_names_train, 'path': paths_train})
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] File "/programs/x86_64-linux/topaz/0.2.5a_cu11.3_py36/topaz_extlib/envs/topaz-py3.6/lib/python3.6/site-packages/pandas/core/frame.py", line 468, in __init__
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] mgr = init_dict(data, index, columns, dtype=dtype)
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] File "/programs/x86_64-linux/topaz/0.2.5a_cu11.3_py36/topaz_extlib/envs/topaz-py3.6/lib/python3.6/site-packages/pandas/core/internals/construction.py", line 283, in init_dict
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] File "/programs/x86_64-linux/topaz/0.2.5a_cu11.3_py36/topaz_extlib/envs/topaz-py3.6/lib/python3.6/site-packages/pandas/core/internals/construction.py", line 78, in arrays_to_mgr
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] index = extract_index(arrays)
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] File "/programs/x86_64-linux/topaz/0.2.5a_cu11.3_py36/topaz_extlib/envs/topaz-py3.6/lib/python3.6/site-packages/pandas/core/internals/construction.py", line 397, in extract_index
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] raise ValueError("arrays must all be same length")
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] ValueError: arrays must all be same length
[Thu, 13 Feb 2025 12:28:48 GMT] [CPU RAM used: 280 MB] Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/run.py", line 129, in cryosparc_master.cryosparc_compute.run.main
File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 332, in run_topaz_wrapper_train
utils.run_process(split_command)
File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 99, in run_process
assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})"
AssertionError: Subprocess exited with status 1 (/programs/x86_64-linux/topaz/0.2.5a_cu11.3_py36/bin/topaz train_test_split --number 109 --seed 1420656222 --image-dir /mnt/xxx/data04/xxxx/cryosparc/CS-xxxx-cftr-vesicles/J566 /mnt/xx/data04/xxxx/cryosparc/CS-xxx-cftr-vesicles/J1023/topaz…)
[Thu, 13 Feb 2025 12:30:29 GMT] License is valid.
[Thu, 13 Feb 2025 12:30:29 GMT] Launching job on lane RTX4090 target gpu02 ...
[Thu, 13 Feb 2025 12:30:29 GMT] Running job on remote worker node hostname gpu02
This is a possible error:
[CPU: 268.0 MB Avail: 222.27 GB]
# splitting 548 micrographs with 563 labeled particles into 439 train and 109 test micrographs
[CPU: 268.0 MB Avail: 222.26 GB]
Traceback (most recent call last):
[CPU: 268.0 MB Avail: 222.26 GB]
File "/programs/x86_64-linux/topaz/0.2.5a/bin/topaz", line 33, in <module>
[CPU: 268.0 MB Avail: 222.26 GB]
sys.exit(load_entry_point('topaz-em==0.2.5a0', 'console_scripts', 'topaz')())
[CPU: 268.0 MB Avail: 222.25 GB]
File "/programs/x86_64-linux/topaz/0.2.5a/lib/python3.9/site-packages/topaz/main.py", line 148, in main
[CPU: 268.0 MB Avail: 222.24 GB]
args.func(args)
[CPU: 268.0 MB Avail: 222.15 GB]
File "/programs/x86_64-linux/topaz/0.2.5a/lib/python3.9/site-packages/topaz/commands/train_test_split.py", line 108, in main
[CPU: 268.0 MB Avail: 222.23 GB]
targets_train = pd.concat(groups_train, 0)
[CPU: 268.0 MB Avail: 222.22 GB]
TypeError: concat() takes 1 positional argument but 2 were given