xmlsecurity: fix testInsertCertificate_PEM_ODT with "dbm:" NSS DB

CentOS 7 system NSS defaults to legacy "dbm:" DB.

  test_desktop_lib.cxx:2830:Assertion
  Test name: DesktopLOKTest::testInsertCertificate_PEM_ODT
  equality assertion failed
  - Expected: 1
  - Actual  : 2

The problem is that getPrivateKey() doesn't work:

warn:xmlsecurity.xmlsec:624712:624712:xmlsecurity/source/xmlsec/nss/securityenvironment_nssimpl.cxx:824: Can't get the private key from the certificate.

In this function, there is a check for trust flags, and the CERTDB_USER
flag is not set, which causes the failure.

The certificate was inserted here and the trust flags were set; this
does write something to cert8.db and it's not clear why it doesn't work
(if this call is omitted with the "sql:" backend, the test fails with
NOTVALIDATED = 4 - as expected).

Oddly enough, while PK11_FindPrivateKeyFromCert() fails, there's another
function PK11_FindKeyByDERCert() that does appear to work, so call it as
a fallback.

Change-Id: I9821966a086574374f4f6df0ac5db2f7376fe742
Reviewed-on: https://gerrit.libreoffice.org/c/core/+/133576
Tested-by: Jenkins
Reviewed-by: Michael Stahl <michael.stahl@allotropia.de>

1 files changed, 7 insertions, 0 deletions

diff --git a/xmlsecurity/source/xmlsec/nss/x509certificate_nssimpl.cxx b/xmlsecurity/source/xmlsec/nss/x509certificate_nssimpl.cxx
index b75d73bb0a89..7e28cbc615bd 100644
--- a/xmlsecurity/source/xmlsec/nss/x509certificate_nssimpl.cxx
+++ b/xmlsecurity/source/xmlsec/nss/x509certificate_nssimpl.cxx
@@ -320,6 +320,13 @@ SECKEYPrivateKey* X509Certificate_NssImpl::getPrivateKey()
         SECKEYPrivateKey* pPrivateKey = PK11_FindPrivateKeyFromCert(m_pCert->slot, m_pCert, nullptr);
         if (pPrivateKey)
             return pPrivateKey;
+        pPrivateKey = PK11_FindKeyByDERCert(m_pCert->slot, m_pCert, nullptr);
+        if (pPrivateKey)
+        {
+            SAL_INFO("xmlsecurity.xmlsec", "fallback from PK11_FindPrivateKeyFromCert to PK11_FindKeyByDERCert needed");
+            return pPrivateKey;
+        }
+        SAL_WARN("xmlsecurity.xmlsec", "X509Certificate_NssImpl::getPrivateKey() cannot find private key");
     }
     return nullptr;
 }