Hello,
We are running V4.1.1 with 230110 patch. Ab-initio give the following message after running ~90h:
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 93, in cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/abinit/run.py”, line 422, in cryosparc_compute.jobs.abinit.run.run_homo_abinit
File “/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py”, line 1082, in rename_prefix
return self.rename_fields(field_map, copy=copy)
File “/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py”, line 1044, in rename_fields
result = type(self)([(f if f == “uid” else fm(f), self[f]) for f in self])
File “/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/particles.py”, line 61, in init
super().init(allocate, row_class=Particle)
File “/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py”, line 638, in init
self.add_fields([entry[0] for entry in populate])
File “/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py”, line 931, in add_fields
assert self._data.addcol_scalar(
AssertionError: Could not add (‘uid’, ‘<u8’) with dtype uint64
What should I do?
Huan
The text in the job’s job.log file:
Number of input particles: 12, 612,949.
Number of ab-initio classes: 4
the box size: 256. Fourier crop to 64.
BTW, the system is Ubuntu with four RTX 3090 cards.
@HuanBao Please can you post
- the lines from
job.log file as text
- the output of
md5sum /spshared/apps/cryosparc4/cryosparc_worker/cryosparc_tools/cryosparc/core.cpython-38-x86_64-linux-gnu.so
Thanks.
Here is the text from job.log:
========= sending heartbeat
========= sending heartbeat
uid: invalid handle 8, wrong generation counter (given 0, expected 32768) (errno 1: Operation not permitted)
add column: invalid handle 8, wrong generation counter (given 0, expected 32768) (errno 1: Operation not permitted)
**** handle exception rc
/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/jobs/runcommon.py:2057: RuntimeWarning: divide by zero encountered in float_scalars
run_old(*args, **kw)
/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/jobs/runcommon.py:2057: RuntimeWarning: divide by zero encountered in double_scalars
run_old(*args, **kw)
/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/jobs/runcommon.py:2057: RuntimeWarning: invalid value encountered in float_scalars
run_old(*args, **kw)
/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/jobs/runcommon.py:2057: RuntimeWarning: invalid value encountered in double_scalars
run_old(*args, **kw)
/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/jobs/runcommon.py:2057: RuntimeWarning: divide by zero encountered in float_scalars
run_old(*args, **kw)
/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/jobs/runcommon.py:2057: RuntimeWarning: divide by zero encountered in double_scalars
run_old(*args, **kw)
/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/jobs/runcommon.py:2057: RuntimeWarning: invalid value encountered in float_scalars
run_old(*args, **kw)
/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/jobs/runcommon.py:2057: RuntimeWarning: invalid value encountered in double_scalars
run_old(*args, **kw)
/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/plotutil.py:1040: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
fig = plt.figure(figsize=figsize)
/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/util/logsumexp.py:40: RuntimeWarning: divide by zero encountered in log
return n.log(wa * n.exp(a - vmax) + wb * n.exp(b - vmax) ) + vmax
/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/plotutil.py:1061: RuntimeWarning: invalid value encountered in arcsin
viewdirs_elevation = n.arcsin( viewdirs[:, 2])
Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/run.py", line 93, in cryosparc_compute.run.main
File "cryosparc_master/cryosparc_compute/jobs/abinit/run.py", line 422, in cryosparc_compute.jobs.abinit.run.run_homo_abinit
File "/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py", line 1082, in rename_prefix
return self.rename_fields(field_map, copy=copy)
File "/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py", line 1044, in rename_fields
result = type(self)([(f if f == "uid" else fm(f), self[f]) for f in self])
File "/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_compute/particles.py", line 61, in __init__
super().__init__(allocate, row_class=Particle)
File "/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py", line 638, in __init__
self.add_fields([entry[0] for entry in populate])
File "/spshared/apps/cryosparc4/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py", line 931, in add_fields
assert self._data.addcol_scalar(
AssertionError: Could not add ('uid', '<u8') with dtype uint64
set status to failed
And the output of
“md5sum /spshared/apps/cryosparc4/cryosparc_worker/cryosparc_tools/cryosparc/core.cpython-38-x86_64-linux-gnu.so”
is:
0d43d952285cb0e51ba1271c57f7e784 /spshared/apps/cryosparc4/cryosparc_worker/cryosparc_tools/cryosparc/core.cpython-38-x86_64-linux-gnu.so
@HuanBao If you are still experiencing this issue, please can you update to v4.2.1 and test if the issue has been fixed for you?
