yuzu-emu/mbedtls
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

analyze_outcome: only warn on ignored tests that pass

Browse source 
The previous check also warned when on tests that were already skipped
in the reference config, which are not really a problem. The purpose of
this "uselessly ignored" check is to make sure that the ignore list
(together with the config common to driver and reference in all.sh)
always correct reflects what works or doesn't in driver-only builds. For
this it's enough to warn when a test is ignored but passing.

The previous, stricter check, was causing issues like:

Error: uselessly ignored: test_suite_pkcs12;PBE Encrypt, pad = 8 (PKCS7 padding disabled)
Error: uselessly ignored: test_suite_pkcs12;PBE Decrypt, (Invalid padding & PKCS7 padding disabled)
Error: uselessly ignored: test_suite_pkcs5;PBES2 Decrypt (Invalid padding & PKCS7 padding disabled)

These are skipped in the reference config because is has PKCS7 padding
enabled, and that's OK.

Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
This commit is contained in:
Manuel Pégourié-Gonnard 2023-10-30 10:21:22 +01:00
parent cd84a290a9
commit c51c411cc1
1 changed files with 4 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
tests/scripts/analyze_outcomes.py
View file

@ -145,11 +145,9 @@ def analyze_driver_vs_reference(results: Results, outcomes,
if component_ref in entry:
reference_test_passed = True
seen_reference_passing = True
if(reference_test_passed and not driver_test_passed):
if not ignored:
if reference_test_passed and not driver_test_passed and not ignored:
results.error("PASS -> SKIP/FAIL: {}", key)
else:
if ignored:
if ignored and driver_test_passed:
results.error("uselessly ignored: {}", key)
if not seen_reference_passing: