Add test suite framework summary

tests/scripts/generate_test_code.py

@@ -19,18 +19,147 @@
 # This file is part of mbed TLS (https://tls.mbed.org)
 
 """
-This script dynamically generates test suite code for Mbed TLS, by
+This script is a key part of Mbed TLS test suites framework. For
-taking following input files.
+understanding the script it is important to understand the
+framework. This doc string contains a summary of the framework
+and explains the function of this script.
+
+Mbed TLS test suites:
+=====================
+Scope:
+------
+The test suites focus on unit testing the crypto primitives and also
+include x509 parser tests. Tests can be added to test any MBED TLS
+module. However, the framework is not capable of testing SSL
+protocol, since that requires full stack execution and that is best
+tested as part of the system test.
+
+Test case definition:
+---------------------
+Tests are defined in a test_suite_<module>[.<optional sub module>].data
+file. A test definition contains:
+ test name
+ optional build macro dependencies
+ test function
+ test parameters
+
+Test dependencies are build macros that can be specified to indicate
+the build config in which the test is valid. For example if a test
+depends on a feature that is only enabled by defining a macro. Then
+that macro should be specified as a dependency of the test.
+
+Test function is the function that implements the test steps. This
+function is specified for different tests that perform same steps
+with different parameters.
+
+Test parameters are specified in string form separated by ':'.
+Parameters can be of type string, binary data specified as hex
+string and integer constants specified as integer, macro or
+as an expression. Following is an example test definition:
+
+X509 CRL Unsupported critical extension (issuingDistributionPoint)
+depends_on:MBEDTLS_PEM_PARSE_C:MBEDTLS_RSA_C:MBEDTLS_SHA256_C
+mbedtls_x509_crl_parse:"data_files/crl-idp.pem":MBEDTLS_ERR_X509_INVALID_EXTENSIONS + MBEDTLS_ERR_ASN1_UNEXPECTED_TAG
+
+Test functions:
+---------------
+Test functions are coded in C in test_suite_<module>.function files.
+Functions file is itself not compilable and contains special
+format patterns to specify test suite dependencies, start and end
+of functions and function dependencies. Check any existing functions
+file for example.
+
+Execution:
+----------
+Tests are executed in 3 steps:
+- Generating test_suite_<module>[.<optional sub module>].c file
+  for each corresponding .data file.
+- Building each source file into executables.
+- Running each executable and printing report.
+
+Generating C test source requires more than just the test functions.
+Following extras are required:
+- Process main()
+- Reading .data file and dispatching test cases.
+- Platform specific test case execution
+- Dependency checking
+- Integer expression evaluation
+- Test function dispatch
+
+Build dependencies and integer expressions (in the test parameters)
+are specified as strings in the .data file. Their run time value is
+not known at the generation stage. Hence, they need to be translated
+into run time evaluations. This script generates the run time checks
+for dependencies and integer expressions.
+
+Similarly, function names have to be translated into function calls.
+This script also generates code for function dispatch.
+
+The extra code mentioned here is either generated by this script
+or it comes from the input files: helpers file, platform file and
+the template file.
+
+Helper file:
+------------
+Helpers file contains common helper/utility functions and data.
+
+Platform file:
+--------------
+Platform file contains platform specific setup code and test case
+dispatch code. For example, host_test.function reads test data
+file from host's file system and dispatches tests.
+In case of on-target target_test.function tests are not dispatched
+on target. Target code is kept minimum and only test functions are
+dispatched. Test case dispatch is done on the host using tools like
+Greentea.
+
+Template file:
+---------
+Template file for example main_test.function is a template C file in
+which generated code and code from input files is substituted to
+generate a compilable C file. It also contains skeleton functions for
+dependency checks, expression evaluation and function dispatch. These
+functions are populated with checks and return codes by this script.
+
+Template file contains "replacement" fields that are formatted
+strings processed by Python str.format() method.
+
+This script:
+============
+Core function of this script is to fill the template file with
+code that is generated or read from helpers and platform files.
+
+This script replaces following fields in the template and generates
+the test source file:
+
+{test_common_helpers}       <-- All common code from helpers.function
+                                is substituted here.
+{functions_code}            <-- Test functions are substituted here
+                                from the input test_suit_xyz.function
+                                file. C preprocessor checks are generated
+                                for the build dependencies specified
+                                in the input file. This script also
+                                generates wrappers for the test
+                                functions with code to expand the
+                                string parameters read from the data
+                                file.
+{expression_code}           <-- This script enumerates the
+                                expressions in the .data file and
+                                generates code to handle enumerated
+                                expression Ids and return the values.
+{dep_check_code}            <-- This script enumerates all
+                                build dependencies and generate
+                                code to handle enumerated build
+                                dependency Id and return status: if
+                                the dependency is defined or not.
+{dispatch_code}             <-- This script enumerates the functions
+                                specified in the input test data file
+                                and generates the initializer for the
+                                function table in the template
+                                file.
+{platform_code}             <-- Platform specific setup and test
+                                dispatch code.
 
-test_suite_xyz.function     -   Test suite functions file contains test
-                                functions.
-test_suite_xyz.data         -   Contains test case vectors.
-main_test.function          -   Template to substitute generated test
-                                functions, dispatch code, dependency
-                                checking code etc.
-platform .function          -   Platform specific initialization and
-                                platform code.
-helpers.function -              Common/reusable data and functions.
 """