diff options
| author | dante <dante19031999@gmail.com> | 2021-04-25 12:20:58 +0200 |
|---|---|---|
| committer | Mike Kaganski <mike.kaganski@collabora.com> | 2021-04-28 16:31:45 +0200 |
| commit | fd4df675dfe1012448285134082f61a0c03a7d15 (patch) | |
| tree | a8833550db046a6c69bf19a72ddad861e53a9fc9 /sc/qa/uitest/calc_tests6 | |
| parent | bc7c37f2f67ace27196a46c62c2d568501e9f796 (diff) | |
tdf#137679 Use Kahan summation for scmatrix operations
May also want implement Kahan sum in there:
sc/source/core/tool/arraysumSSE2.cxx
sc/source/core/inc/arraysumfunctor.hxx
Under some conditions the sum of a pointer type vector
may be perforemed by arraysumfunctor on NumericCellAccumulator.
arraysumSSE2 implements part of it.
This code has been left unmodified.
Why the test has been modified:
The error was: 238.89000000000001 != 238.89
|
17 th digit
IEEE 754 double-precision has 53 log10(2) ≈ 15.955 digits.
So it's just noise.
Change-Id: I6f84826bf3875be4f444f5eb61854bc1f95769bb
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/114627
Reviewed-by: Mike Kaganski <mike.kaganski@collabora.com>
Tested-by: Jenkins
Diffstat (limited to 'sc/qa/uitest/calc_tests6')
| -rw-r--r-- | sc/qa/uitest/calc_tests6/tdf118638.py | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/sc/qa/uitest/calc_tests6/tdf118638.py b/sc/qa/uitest/calc_tests6/tdf118638.py index 2e1b704a82d9..cc5c97f34a30 100644 --- a/sc/qa/uitest/calc_tests6/tdf118638.py +++ b/sc/qa/uitest/calc_tests6/tdf118638.py @@ -47,8 +47,8 @@ class Subtotals(UITestCase): self.assertEqual(get_cell_by_position(document, 0, 0, 15).getString(), "5408 Sum") self.assertEqual(get_cell_by_position(document, 0, 0, 16).getString(), "Grand Sum") - self.assertEqual(get_cell_by_position(document, 0, 1, 15).getValue(), 238.89) - self.assertEqual(get_cell_by_position(document, 0, 1, 16).getValue(), 238.89) + self.assertEqual(round(get_cell_by_position(document, 0, 1, 15).getValue(),12), 238.89) + self.assertEqual(round(get_cell_by_position(document, 0, 1, 16).getValue(),12), 238.89) self.assertEqual(get_cell_by_position(document, 0, 1, 15).getString(), "$238.89") self.assertEqual(get_cell_by_position(document, 0, 1, 16).getString(), "$238.89") |