MDEV-38248 Add MTR to verify TLS with chain of trust

[tonychen2001]

Description

Adding a test to explicitly verify that TLS connections support certificate chain of trust.

The contents of the commit are largely taken from the MySQL project: mysql/mysql-server@969afef

Credits to [email protected]

PR quality check

• I checked the CODING_STANDARDS.md file and my PR conforms to this where appropriate.
• For any trivial modifications to the PR, I am ok with the reviewer making the changes themselves.

Copyright

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.

[vaintroub]

You exclude your test from Windows . Can you explain why? I think certificate chains have to work also with schannel, and also on Windows.

[tonychen2001]

You exclude your test from Windows . Can you explain why? I think certificate chains have to work also with schannel, and also on Windows.

I see. Then this is probably a false assumption I made.

I had handled the different error messages due to different TLS library versions. But on Windows, the MTR was failing with a different signature. It seems like the server just won't start up.

Then I noticed other SSL related tests were being skipped on Windows. E.g:

main.ssl_crl 'file'                      w3 [ skipped ]  Needs OpenSSL
main.ssl_crl 'path'                      w19 [ skipped ]  Needs OpenSSL

I don't have convenient access to a Windows environment and the error message isn't very helpful here. How can I debug this?

I updated the test to not skip Windows and now the error message is:

2025-12-03 17:34:15 0 [ERROR] SSL error: Unable to get certificate
2025-12-03 17:34:15 0 [ERROR] Aborting

[vaintroub]

You exclude your test from Windows . Can you explain why? I think certificate chains have to work also with schannel, and also on Windows.

I see. Then this is probably a false assumption I made.

main.ssl_crl 'file'                      w3 [ skipped ]  Needs OpenSSL
main.ssl_crl 'path'                      w19 [ skipped ]  Needs OpenSSL

"Needs OpenSSL" is not the same as "Need not Windows".

I don't have convenient access to a Windows environment and the error message isn't very helpful here. How can I debug this?
I updated the test to not skip Windows and now the error message is:

2025-12-03 17:34:15 0 [ERROR] SSL error: Unable to get certificate
2025-12-03 17:34:15 0 [ERROR] Aborting

Ok , you build on Linux with -DWITH_SSL=bundled passed to cmake, and run the test under debugger. That will build with WolfSSL.
Appveyor builds on Windows with OpenSSL btw, while our tar.gz packages on Linux are built with WolfSSL.

[tonychen2001]

"Needs OpenSSL" is not the same as "Need not Windows".

Understood. I tried including include/have_openssl.inc, but the test still fails because this check for OpenSSL requires the database to be up which on Windows, it refuses to come up with these certificates.

Ok , you build on Linux with -DWITH_SSL=bundled passed to cmake, and run the test under debugger. That will build with WolfSSL. Appveyor builds on Windows with OpenSSL btw, while our tar.gz packages on Linux are built with WolfSSL.

I'm building from source and I haven't passed in any particular compiler flag regarding SSL. When I checked my build, I do see that it's built with OpenSSL by checking select @@have_openssl;. So the issue seems to be related to the configuration on Windows rather than OpenSSL?

My OS:

$ openssl --version
OpenSSL 3.5.2 5 Aug 2025 (Library: OpenSSL 3.5.2 5 Aug 2025)
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux forky/sid"

Build configuration I'm using:

cmake -S mariadb-server/ -B build/ -G Ninja --fresh        -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache        -DPLUGIN_COLUMNSTORE=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_S3=NO        -DPLUGIN_MROONGA=NO -DPLUGIN_CONNECT=NO -DPLUGIN_TOKUDB=NO        -DPLUGIN_PERFSCHEMA=NO -DWITH_WSREP=OFF -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILE_FLAGS='-Og -g3' -DCMAKE_C_COMPILE_FLAGS='-Og -g3'

[vaintroub]

You do not need to pass any compiler flags to the build. You need to pass a cmake flag I mentioned. In order to reproduce the error in your comfortable Linux setting, this is what you asked about originally, right? In this case sudo apt-get install gnutls-dev (do not ask why WITH_SSL=bundled means use gnutls on client, I do not know) and, importantly cmake . -DWITH_SSL=bundled. This is how our Linux tar.gz packages are built, and this is intentional, and not a configuration error. If you do not see an error in pull request on Linux now, this is only because of limited testing our CI offers for the pull requests.

I'd say, let try reproduce error in your convenient dev. environment, then figure out what to do.

We unfortunately have 4 different libraries that do SSL, it is either openssl or WolfSSL on server, and either OpenSSL or GnuTLS, or Schannel on client. It is different from MySQL, but this is how it is, and we need to know which combinations work, and which do not work, and maybe report errors to the library authors if something does not work,

[gkodinov]

I've played with it a bit. FWIW, wolfSSL's SSL_CTX_use_certificate_chain_file() fails to process server.cachain from the .opt file and returns an error.

[gkodinov]

Thank you for picking up the fact that serial 0 is a non-valid serial for wolfSSL and a valid one for openssl.
Maybe consider enabling -DWOLFSSL_ASN_ALLOW_0_SERIAL in extra/wolfssl/CMakeLists.txt to achieve better compatibility with openSSL (that seems to be perfectly happy with serial id 0) ?

[gkodinov]

any specific reason why you have disabled this test?

[tonychen2001]

This was just for testing. It should be enabled.

For an update, there were 2 issues,

1. As you mentioned above, WolfSSL does not allow the certificate serial number to be 0. This conforms to RFC 5820 Section 4.1.2.2.

Fixing this allowed the server to start up with the given server certificates.

2. Client certificate validation was failing because the intermediate CAs are missing the extension keyUsage = keyCertSign. Another case where OpenSSL has more relaxed checks but WolfSSL conforms to RFC 5820 Section 4.2.1.3 which requires keyCertSign usage to be enabled for intermediate CAs.

After fixing the above, the test passes when building with -DWITH_SSL=bundled.

Though it still fails on the Windows build. I assume schannel has it's own differences. Although unlike before, the server does start up but it's now failing when the client is making a secure connection.

[gkodinov]

I've got the same on my local box: it's working both with openSSL and wolfSSL. I'll give the windows failure a try today and report back on my findings.

[gkodinov]

FWIW, the schannel code only reads a single certificate (see https://github.com/mariadb-corporation/mariadb-connector-c/blob/46880b003653a000e9588bd73c8b1dd65088c686/libmariadb/secure/schannel_certs.c#L807).

This needs mending.

Probably something along the lines of https://learn.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-certaddencodedcertificatetostore once https://learn.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-cryptqueryobject succeeds and the store is created by it.

[gkodinov]

then, of course, there's the GnuTLS to mend.