From aee05bbe70dfdc1a468c0a2788fef62888a04266 Mon Sep 17 00:00:00 2001 From: Azim Khan Date: Mon, 2 Jul 2018 16:01:04 +0100 Subject: [PATCH] Add test suite framework summary --- tests/scripts/generate_test_code.py | 151 ++++++++++++++++++++++++++-- 1 file changed, 140 insertions(+), 11 deletions(-) diff --git a/tests/scripts/generate_test_code.py b/tests/scripts/generate_test_code.py index c4c11fc39..a9ec566e6 100755 --- a/tests/scripts/generate_test_code.py +++ b/tests/scripts/generate_test_code.py @@ -19,18 +19,147 @@ # This file is part of mbed TLS (https://tls.mbed.org) """ -This script dynamically generates test suite code for Mbed TLS, by -taking following input files. +This script is a key part of Mbed TLS test suites framework. For +understanding the script it is important to understand the +framework. This doc string contains a summary of the framework +and explains the function of this script. + +Mbed TLS test suites: +===================== +Scope: +------ +The test suites focus on unit testing the crypto primitives and also +include x509 parser tests. Tests can be added to test any MBED TLS +module. However, the framework is not capable of testing SSL +protocol, since that requires full stack execution and that is best +tested as part of the system test. + +Test case definition: +--------------------- +Tests are defined in a test_suite_[.].data +file. A test definition contains: + test name + optional build macro dependencies + test function + test parameters + +Test dependencies are build macros that can be specified to indicate +the build config in which the test is valid. For example if a test +depends on a feature that is only enabled by defining a macro. Then +that macro should be specified as a dependency of the test. + +Test function is the function that implements the test steps. This +function is specified for different tests that perform same steps +with different parameters. + +Test parameters are specified in string form separated by ':'. +Parameters can be of type string, binary data specified as hex +string and integer constants specified as integer, macro or +as an expression. Following is an example test definition: + +X509 CRL Unsupported critical extension (issuingDistributionPoint) +depends_on:MBEDTLS_PEM_PARSE_C:MBEDTLS_RSA_C:MBEDTLS_SHA256_C +mbedtls_x509_crl_parse:"data_files/crl-idp.pem":MBEDTLS_ERR_X509_INVALID_EXTENSIONS + MBEDTLS_ERR_ASN1_UNEXPECTED_TAG + +Test functions: +--------------- +Test functions are coded in C in test_suite_.function files. +Functions file is itself not compilable and contains special +format patterns to specify test suite dependencies, start and end +of functions and function dependencies. Check any existing functions +file for example. + +Execution: +---------- +Tests are executed in 3 steps: +- Generating test_suite_[.].c file + for each corresponding .data file. +- Building each source file into executables. +- Running each executable and printing report. + +Generating C test source requires more than just the test functions. +Following extras are required: +- Process main() +- Reading .data file and dispatching test cases. +- Platform specific test case execution +- Dependency checking +- Integer expression evaluation +- Test function dispatch + +Build dependencies and integer expressions (in the test parameters) +are specified as strings in the .data file. Their run time value is +not known at the generation stage. Hence, they need to be translated +into run time evaluations. This script generates the run time checks +for dependencies and integer expressions. + +Similarly, function names have to be translated into function calls. +This script also generates code for function dispatch. + +The extra code mentioned here is either generated by this script +or it comes from the input files: helpers file, platform file and +the template file. + +Helper file: +------------ +Helpers file contains common helper/utility functions and data. + +Platform file: +-------------- +Platform file contains platform specific setup code and test case +dispatch code. For example, host_test.function reads test data +file from host's file system and dispatches tests. +In case of on-target target_test.function tests are not dispatched +on target. Target code is kept minimum and only test functions are +dispatched. Test case dispatch is done on the host using tools like +Greentea. + +Template file: +--------- +Template file for example main_test.function is a template C file in +which generated code and code from input files is substituted to +generate a compilable C file. It also contains skeleton functions for +dependency checks, expression evaluation and function dispatch. These +functions are populated with checks and return codes by this script. + +Template file contains "replacement" fields that are formatted +strings processed by Python str.format() method. + +This script: +============ +Core function of this script is to fill the template file with +code that is generated or read from helpers and platform files. + +This script replaces following fields in the template and generates +the test source file: + +{test_common_helpers} <-- All common code from helpers.function + is substituted here. +{functions_code} <-- Test functions are substituted here + from the input test_suit_xyz.function + file. C preprocessor checks are generated + for the build dependencies specified + in the input file. This script also + generates wrappers for the test + functions with code to expand the + string parameters read from the data + file. +{expression_code} <-- This script enumerates the + expressions in the .data file and + generates code to handle enumerated + expression Ids and return the values. +{dep_check_code} <-- This script enumerates all + build dependencies and generate + code to handle enumerated build + dependency Id and return status: if + the dependency is defined or not. +{dispatch_code} <-- This script enumerates the functions + specified in the input test data file + and generates the initializer for the + function table in the template + file. +{platform_code} <-- Platform specific setup and test + dispatch code. -test_suite_xyz.function - Test suite functions file contains test - functions. -test_suite_xyz.data - Contains test case vectors. -main_test.function - Template to substitute generated test - functions, dispatch code, dependency - checking code etc. -platform .function - Platform specific initialization and - platform code. -helpers.function - Common/reusable data and functions. """