Use Release mode (-O2) for component_test_full_cmake_clang which runs SSL
tests.
To have some coverage with Check mode (which enables more compiler warnings
but compiles with -Os), change a few other builds that only run unit tests
at most to Check mode.
Don't add any new builds, to keep the total build volume down. We don't need
extensive coverage of all combinations, just a reasonable set.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
SSL testing benefits from faster executables, so use -O2 rather than -O1.
Some builds use -O1, but that's intended for jobs that only run unit tests,
where the build takes longer than the tests.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Don't default to unbridled -j, which causes a load spike and isn't really
faster.
"Number of CPUs" is implemented here as a reasonable compromise between
portability, correctness and simplicity. This is just a default that can be
overridden by setting MAKEFLAGS in the environment.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
When not using DEBUG_C, but using the DTLS CID feature -
a null pointer was accessed in ssl_tls.c.
Signed-off-by: Andrzej Kurek <andrzej.kurek@arm.com>
Don't try to enumerate excluded files. List included files, and remove names
from the list if they match an excluded-file pattern.
This resolves the problem that the script could get into an infinite loop
due to the use of recursive globbing. Unfortunately, Python's recursive
globs follows symbolic links to directories, which leads to an infinite loop
if a symbolic link points to an ancestor of the directory that contains it.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
To facilitate maintenance and to make it easier to reproduce all.sh builds
manually, remove the long, repeated list of -D options from
component_test_psa_crypto_config_basic and component_test_psa_crypto_drivers
and put it in a header file instead.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
When building with make, `make test` runs `run-test-suites.pl` which has a
verbose mode that reports the failing test cases, but it didn't provide a
way to enable this verbose mode. With the present commit, you can run `make
test TEST_FLAGS=-v` to use verbose mode.
Base the default for verbose mode on the same environment variable that
`make test` uses when building with CMake: default off, but enabled if
`CTEST_OUTPUT_ON_FAILURE` is true. In particular, verbose mode will now be
on when building from `all.sh`.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The list was trimmed previously according to code coverage, however
this did not really evalute all test cases, e.g in the case of re.sub
or m_cipher.replace. These lines are executed no matter what, so code
coverage is not suitable.
I have gone through each step in the translate functions and made sure
there is at least one ciphersuite per step
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
Having a list of every ciphersuite suggests that it should be maintained with
any new ciphersuites that are added in the future. This in turn almost defeats
the purpose of having translation functions to begin with
Instead, the unit test now only test a much smaller subset of ciphersuite names
that exercise each stage in the OpenSSL and GnuTLS translate functions.
In the future, if a new cipersuite is added that requires an extra stage in
translation, then that cipher can be added to the test suite, otherwise it
should not be necessary.
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
As test_translate_ciphers_format.sh was made as a testing ground before
utilising translate_ciphers.py in compat.sh, once it was translated to
python code - as a unit test, it became redundant.
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
Rather than having the tests seperated into different files, they were integrated
into translate_ciphers.py and can be run from root using:
`python -m unittest tests/scripts/translate_ciphers.py`
test_translate_ciphers_format.sh was originally made as a testing ground before
having the translation tool being implmented into compat.sh. Translating it to
python code makes it redundant and therefore it will be removed.
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
test_translate_ciphers_names.py
- Combined m, o and g ciphers all into one a single list of tuples to
avoid needing to rely on indexes
test_translate_ciphers_format.sh
- Removed redundant test
- Added return errors
compat.sh
- Improved how translate_ciphers.py is called
translate_ciphers.py
- Improve regex and translation to be more intutive and efficient
- change how arguments are taken and handelled to be more reliable
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
To run test_translate_ciphers_names.py and _format.sh in the CI, include
it in all.sh component_check_generate_test_code.
Rename check_generate_test_code to check_test_helpers
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
After improving coding style, pylint suggeted using enumerate but
zip is more appropriate to avoid indexing
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
As per check-python-files.sh, added string documentation for
files and functions.
Modified for loops to use enumerate rather than range(len(
although as the same iteration index is used for multiple
lists it does not seem quite appropriate
Signed-off-by: Joe Subbiani <joe.subbiani@arm.com>
- Problem() is a parent abstract class that should only be used for
subclassing.
- With the help of ABC, implement abstract methods that force
subclasses to implement quiet and verbose outputs.
- The repeated logic of "if self.quiet" is consolidated in Problem.
Signed-off-by: Yuto Takano <yuto.takano@arm.com>
This reverts a previous change where line_no was removed and put into
a triple tuple. It was discovered that re.Match.span() conveniently
returns (start, end), so separating line_no again makes the code cleaner.
The legibility of the code heavily outweighs the issues pointed out by
Pylint (hence disabled).
Signed-off-by: Yuto Takano <yuto.takano@arm.com>
The try/catch was used to catch Exceptions and exit with code 1,
a legacy from check_names.py which uses the pattern to exit with
code 2. But code 1 is the default for the Python runtime anyway,
so it is redundant and can be removed.
Signed-off-by: Yuto Takano <yuto.takano@arm.com>
- Instead of os.path.join, use glob patterns (supports Windows too)
- Instead of creating the lists beforehand (which adds messiness), pass glob
expessions to functions and let them memoise it.
- Add support for excluding based on glob patterns, which isn't used now but
could come in handy.
Signed-off-by: Yuto Takano <yuto.takano@arm.com>
This is the equivalent of `list-identifiers.sh --internal`, which is
useful for generating an exclusion file for ABI/API checking.
Signed-off-by: Yuto Takano <yuto.takano@arm.com>
glob is more flexible and simplifies the function arguments drastically.
It is also much more intuitive to extend in the future when the filepaths
need to be extended or changed.
setup_logger had to be called as the first thing after instantiation, so
this commit simplify makes it automatic.
Several clarification comments are added too.
Signed-off-by: Yuto Takano <yuto.takano@arm.com>
This is necessary to import check_names from other scripts, which
will inevitably happen in the next few commits to implement the equivalent
of `list-identifiers.sh --internal`.
Signed-off-by: Yuto Takano <yuto.takano@arm.com>
- Add documentation to all classes and functions that were
not self-explanatory.
- Fix the parsing of identifiers, so it now behaves identically
to the original shell script. Detects the same amount of identifiers.
- Fix macro parsing so MBEDTLS_PSA_ACCEL didn't error out
- Reformat output to be comprehensible
Signed-off-by: Yuto Takano <yuto.takano@arm.com>
Otherwise $COMMAND_LINE_COMPONENTS would try to expand wildcard patterns
based on files in the current directory.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Was getting errors like:
In file included from /usr/include/limits.h:25:0,
from /usr/lib/gcc-cross/arm-linux-gnueabi/5/include-fixed/limits.h:168,
from /usr/lib/gcc-cross/arm-linux-gnueabi/5/include-fixed/syslimits.h:7,
from /usr/lib/gcc-cross/arm-linux-gnueabi/5/include-fixed/limits.h:34,
from ../include/mbedtls/check_config.h:30,
from ../include/mbedtls/build_info.h:81,
from common.h:26,
from asn1write.c:20:
/usr/include/features.h:367:25: fatal error: sys/cdefs.h: No such file or directory
There are two packages to choose from: armhf or armel. Since the comment
in all.sh says we're trying to be close to Debian's "armel"
architecture, choose that, and fix a comment that was mentioning
gnueabihf for no apparent reason.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
Currently it can't be mandatory, since we can't install the required toolchain
on Jenkins right away.
Also, while at it, remove `SHELL='sh -x'` from the other arm5vte component; it
was a leftover from debugging.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
Have simpler patterns related to 'test' (the central objective being to keep
going if 'make test' or 'tests/...' fails, but not if 'make tests' fails).
Add 'cd' as a can't-keep-going command.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Count invocations from 1 to n instead of n to 1.
Explain how changing the loop variable would cause an error if the function
was not executed in a subshell.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The previous implementation was hard to understand and could in principle
fail to notice if there was a test case failure and the writing of the
line "Note: $TOTAL_FAIL failures." failed. KISS.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Save the "Test Report Summary" to a file. This can help both CI scripts and
human readers who want the summary after the fact without having to copy the
console output.
Take care to exit with a nonzero status if there is a failure while
generating the test report summary.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
all.sh restores **/Makefile from git in case the version in the worktree was
from doing a cmake in-tree build. Instead of doing this unconditionally, do
it only if the toplevel Makefile seems to have been automatically
generated (by cmake or otherwise, e.g. by mbedtls-prepare-build). This way
all.sh no longer silently wipes changes made to Makefile but not committed yet.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Back up the config files at the beginning of all.sh, rather than before each
component. In particular, create the backup before running cleanup for the
first time. This fixes#3139 (all.sh using a config.h.bak from a previous
job), and makes all.sh more robust against accidentally using a modified
config.h midway through because a component messed with the backup.
Use a different extension (*.all.bak rather than *.bak) for the backups.
This is necessary to ensure that auxiliary scripts such as depends*.pl that
make their own backup don't remove all.sh's backup, which the code from this
commit does not support.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Generate programs/test/cpp_dummy_build.cpp dynamically instead of
maintaining it manually. This removes the need to update it when the list of
headers changes.
Include all the headers unconditionally except for the ones that cannot be
included directly.
Support this dynamic generation both with make and with cmake.
Adapt all.sh accordingly. Remove the redundant C build from
component_build_default_make_gcc_and_cxx (it was also done in
component_test_default_out_of_box), leaving a component_test_make_cxx. Also
run the C++ program, because why not. Do this in the full configuration
which may catch a bit more problems in headers.
Fixes#2570 for good.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
redirect_out was no longer used and redirect_err was only used to
quiet dd. Change the dd invocation to only print diagnostics on
error (on platforms where this is possible).
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This is necessary to actually keep going and finish the
component-specific cleanup in component_test_cmake_out_of_source if
ssl-opt.err is non-empty.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
In all.sh, when an explicit list of components is specified, error out
if one of the components is not known or not supported. Patterns that
happen to match zero components are still effectively ignored.
Fix#2783
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
In the `not` function, in keep-going mode, arrange to report the
failing command (rather than `"$@"`).
Note that the `!` keyword should not be used, because failures with
`!` are not reported properly.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Remove the obsolete functions record_status and if_build_succeeded.
They didn't affect error detection, but they made error reporting
worse since $BASH_COMMAND would be the unexpanded "$@".
Keep the function definitions for the sake of pull requests using them
that may still be in flight.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Remove ssl-opt.err even if it's empty.
Call cat unconditionally: it'll have no visible effect if the file is
empty.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This commit completely rewrites keep-going mode. Instead of relying
solely on "set -e", which has some subtle limitations (such as being
off anywhere inside a conditional), use an ERR trap to record errors.
Run each component in a subshell. This way a component can set
environment variables, change the current directory, etc., without
affecting other components.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This will let us use bash features that are not found in some other sh
implementations, such as DEBUG and ERR traps, "set -o pipefail", etc.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Restore the optimization done in
HEAD^{/Speed up the generation of storage format test cases}
which was lost during refactoring made when adding support for
implicit usage flags.
There are still more than one call to the C compiler, but the extra
calls are only for some key usage test cases.
This is an internal refactoring. This commit does not change the
output of generate_psa_tests.py
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
