Commit graph

17 commits

Author SHA1 Message Date
Przemek Stekiel
85c54ea361 Allow providing space sepatated tasks
Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
2022-11-17 11:50:23 +01:00
Przemek Stekiel
542d932352 Fix handling of default value for task argument
Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
2022-11-17 09:43:34 +01:00
Przemek Stekiel
6419ab5299 Reduce number of skipped suites (after making configs more similar)
Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
2022-11-16 14:56:49 +01:00
Przemek Stekiel
d3068af2a8 Optimize code (tasks list initialization, task verification)
Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
2022-11-14 16:15:19 +01:00
Przemek Stekiel
733c76e08a Fix style issues pointed by pylint
Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
2022-11-14 08:33:21 +01:00
Przemek Stekiel
992de3c562 Make TASK parameter positional and allow more than one task
Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
2022-11-09 14:24:49 +01:00
Przemek Stekiel
be279c7bcc Make a list from ignored tests in TASKS
Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
2022-11-09 12:17:08 +01:00
Przemek Stekiel
51f30ff6e6 Make separate components for ref and driver in TASKS
Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
2022-11-09 12:07:29 +01:00
Przemek Stekiel
6856f4c70d Fix typos and comments
Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
2022-11-09 10:50:29 +01:00
Przemek Stekiel
4d13c833da analyze_outcomes.py: remove components and ignore parameters
Use a dictionary to specify optional parameters for each task.
If the task is not specified then all tasks are executed.

Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
2022-10-26 16:12:01 +02:00
Przemek Stekiel
c86dedfdc1 Fix code style
Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
2022-10-24 09:16:04 +02:00
Przemek Stekiel
58bbc23ca3 Use coverage analyze as default task
Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
2022-10-24 08:10:10 +02:00
Przemek Stekiel
4e95590ae7 analyze_outcomes.py: Add test coverage regresion analyze for driver only builds
Signed-off-by: Przemek Stekiel <przemyslaw.stekiel@mobica.com>
2022-10-21 13:43:13 +02:00
Gilles Peskine
686c292e8a Move collect_available_test_cases to check_test_cases.py
No behavior change.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2022-01-07 18:34:12 +01:00
Gilles Peskine
3d863f2631 Document the fields of TestCasesOutcomes
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2020-06-26 18:31:26 +02:00
Gilles Peskine
8d3c70a279 Check test case coverage
Check that every available test case in the test suites and ssl-opt.sh
has been executed at least once.

For the time being, only report a warning, because our coverage is
incomplete. Once we've updated all.sh to have full coverage, this
warning should become an error.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2020-06-26 18:29:34 +02:00
Gilles Peskine
15c2cbfed5 New script for test outcome analysis
This is a new script designed to analyze test outcomes collected
during a whole CI run.

This commit introduces the script, the code to read the outcome file,
and a very simple framework to report errors. It does not perform any
actual analysis yet.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
2020-06-26 18:29:34 +02:00