This commit includes:
- use subprocess.check_output to report error and capture return
value
- add comment as a reminder for option --list-test-case
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
uniform_title is used to print identical format of $TITLE between
--list-test-case and run_client. In such way, no matter how $TITLE
is developed, --list-test-case will in the same format of test case
description as stored in OUTCOME.CSV.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
If the environment variable MBEDTLS_TEST_OUTCOME_FILE is set,
the test outcome file records each test case in a single line
with the format of
PLATFORM;CONFIGURATION;compat;TEST CASE DESCRIPTION;RESULT;[CAUSE]
- CONFIGURATION comes from MBEDTLS_TEST_CONFIGURATION to record
configuration of each test case
- PLATFORM is either set by users or calculated from test
platform
- RESULT is one of PASS, FAIL or SKIP. If test case fails,
srv_out/cli_out follows as FAILURE CAUSE.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
Test case description is printed by different block of code. This
causes code maintenance harder since we need to maintain two parts
of code with same functionality. print_test_title is used to
control test case description in compat.sh
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
The option --list-test-case lists all potential test cases without
executing them. The test case description is identical with $TITLE
during test case execution.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
The mechanism of detecting unsupported ciphersuites
for OpenSSL client doesn't work on a modern OpenSSL.
At least, it fails on Travis CI which is installed
with OpenSSL 1.1.1f. So we need to skip ECDH cipher-
suites for O->m.
Signed-off-by: Pengyu Lv <pengyu.lv@arm.com>
This commit add support to detect if openssl used for testing
supports static ECDH key exchange. Skip the ciphersutes if
openssl doesn't support them.
Signed-off-by: Pengyu Lv <pengyu.lv@arm.com>
Since PSK cipher suites do not allow client certificate verification,
PSK test cases should be executed under VERIFY=NO. SUB_VERIFIES is
used to constrain verification option for PSK tests.
With aforementioned change, the latter check of
$VERIFY=YES && $TYPE!=PSK is redundant so it's removed.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
With the introduction of PSK_TESTS,
- Either `compat.sh -V NO` or `compat.sh -V YES` runs the PSK tests
- `compat.sh` or `compat.sh -V "NO YES"` runs PSK tests only once
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
There is no need to provide CA file in PSK. Thus VERIFY is
meaningless for PSK. This change omits the arguments passed to
the client and server for $VERIFY=YES.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
Don't use external programs for string manipulation that the shell can do.
This makes the script a little faster (~10% when testing PSK).
For this commit, I only looked at code run in the innermost loop.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Python has a high startup cost, so go back to invoking it only once per
server start, rather than once per client start. This is a measurable
performance improvement (running time ~*0.5 with PSK, less dramatic with
asymmetric crypto).
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
We can detect PSK based on $TYPE. This allows more flexibility in how cipher
suites are spelled.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The argument passed to translate_ciphers.py is calculated from $1 in
run_client instead of passed as third argument.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
Move translation of cipher suite names after filter_ciphersuites
so that filter is based on standard cipher suite names.
Furthermore, an additional flag is passed to run_client to
determine the type of translation of cipher suite names.
Therefore, client receives cipher suite names based on
its naming convention but the reporting output is still
the standard cipher suite names.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
Since there is a plan to report and filter all cipher suite names
consistently, cipher suite names in compat.sh are changed to the
standard naming convention.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
Under Ubuntu-22.04, wait command prints out Terminated message.
Therefore server process is handled with identical ways like other
processes in compat.sh. In addition, PROCESS_ID is renamed as
SRV_PID to improve code readability.
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
Under Ubuntu-22.04, wait command prints out Terminated message
if the process has been killed by kill command. This messes up
the output in compat.sh
Signed-off-by: Yanray Wang <yanray.wang@arm.com>
These variables were both uses to select the default version of OpenSSL
to use for tests:
- when running compat.sh or ssl-opt.sh directly, OPENSSL_CMD was used;
- when running all.sh, OPENSSL was used.
This caused surprising situations if you had one but not the other set
in your environment. For example I used to have OPENSSL_CMD set but not
OPENSSL, so ssl-opt.sh was failing in some all.sh components but passing
when I ran it manually in the same configuration and build, a rather
unpleasant experience.
The natural name would be OPENSSL, and that's what set in the Docker
images used by the CI. However back in the 1.3.x days, that name was
already used in library/Makefile, so it was preferable to pick a
different one, hence OPENSSL_CMD. However the build system has not been
using this name since at least Mbed TLS 2.0.0, so it's now free for use
again (as demonstrated by the fact that it's been set in the CI without
causing any trouble).
So, unify things and use OPENSSL everywhere. Just leave an error message
for the benefit of developers which might have OPENSSL_CMD, not OPENSSL,
set in their environment from the old days.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
OpenSSL may be configured to support features such as cipher suites or
protocol versions that are disabled by default. Enable them all: we're
testing, we don't care about enabling insecure stuff. This is not needed
with the builds of OpenSSL that we're currently using on the Jenkins CI, but
it's needed with more recent versions such as typically found on developer
machines, and with future CI additions.
The syntax to do that was only introduced in OpenSSL 1.1.0; fortunately we
don't need to do anything special with earlier versions.
With OpenSSL 1.1.1f on Ubuntu 20.04, this allows SHA-1 in certificates,
which is still needed for a few test cases in ssl-opt.sh. Curiously, this is
also needed for the cipher suite TLS-DHE-PSK-WITH-ARIA-128-GCM-SHA256 (and
no other, including other DHE-PSK or ARIA cipher suites).
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
- comments about (3)DES were inconsistent (one saying "keep 3DES" and
the other "exclude it")
- we don't have any DES or 3DES ciphersuite anyway, so no need to
exclude them or comment about it
- while at it, fix comment about ARIA: it's in the default config now
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
The ciphers were originally separated by the version conditions, and the spacing
was retained in case it was required later. It has been decided it would be better
to have an unbroken list in alphabetically order to quickly locate ciphers when
necessary
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
Remove ciphersuites that are no longer supported.
running ./test/compat.sh -e '^$' -p mbedtls no longer skips any tests
Fixes#4818
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
As anything below (D)TLS 1.2 is no longer supported, checks for the
minor version will always be true, and have therefore been removed
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
Instead add TLS-RSA-WITH-NULL-SHA256 to list of common ciphersuites.
It therefore has to be removed from GnuTLS as it could then duplicate.
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
test_translate_ciphers_names.py
- Combined m, o and g ciphers all into one a single list of tuples to
avoid needing to rely on indexes
test_translate_ciphers_format.sh
- Removed redundant test
- Added return errors
compat.sh
- Improved how translate_ciphers.py is called
translate_ciphers.py
- Improve regex and translation to be more intutive and efficient
- change how arguments are taken and handelled to be more reliable
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
If translate_ciphers.py is used incorrectly in compat.sh, an error
check function - check_translation - is called to evaluate and inform
the user of the error that has occured.
Added an output that informs the users an error has taken place in
translate_ciphers.py incase the error response is an empty string.
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
