The previous check also warned when on tests that were already skipped
in the reference config, which are not really a problem. The purpose of
this "uselessly ignored" check is to make sure that the ignore list
(together with the config common to driver and reference in all.sh)
always correct reflects what works or doesn't in driver-only builds. For
this it's enough to warn when a test is ignored but passing.
The previous, stricter check, was causing issues like:
Error: uselessly ignored: test_suite_pkcs12;PBE Encrypt, pad = 8 (PKCS7 padding disabled)
Error: uselessly ignored: test_suite_pkcs12;PBE Decrypt, (Invalid padding & PKCS7 padding disabled)
Error: uselessly ignored: test_suite_pkcs5;PBES2 Decrypt (Invalid padding & PKCS7 padding disabled)
These are skipped in the reference config because is has PKCS7 padding
enabled, and that's OK.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
This may come in handy when ignoring patterns in test_suite_cipher for
example which is split in several .data files where we'll want to ignore
the same patterns.
Currently none of the entries had a '.' in the test suite name, so this
doesn't change anything for existing entries.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
Change from iterating on all available tests to iterating on tests with
an outcome: initially we were iterating on available tests but
immediately ignoring those that were always skipped. That last part
played poorly with the new error (we want to know if the test ignored
due to a pattern was skipped in the reference component), but when
removing it, we were left with iterating on all available tests then
skipping those that don't have outcomes entries: this is equivalent to
iterating on tests with an outcome entry, which is more readable.
Also, give an error if the outcome file contains no passing test from
the reference component: probably means we're using the wrong outcome
file.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
- remove unnecessary disabled items (most of them were already
disabled automatically once MBEDTLS_SSL_TLS_C was disabled)
- improve dependencies' comments, especially the last one which
list items depending on PSA_CRYPTO_C
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
Don't chdir when invoking a test suite executable with an explicit .datax
file. The point of the chdir is to automatically find the .datax file (and
the relative location of the data_files directory) in typical cases. This
conflicts with the expectation that passing a relative path to a .datax file
will work.
(This is what I had originally intended, and what is documented in the
comment, but I forgot to add the argc check in the initial commit.)
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This also adds a new task in analyze_outcomes.py for checking
the accelaration coverage against the reference element.
Signed-off-by: Valerio Setti <valerio.setti@nordicsemi.no>
