The primary purpose is to use it to run all.sh -k -q in the pre-push hook, but
this can be useful in any circumstance where you're not interested in the full
output from each component and just want a short summary of which components
were run (and if any failed).
Note that only stdout from components is suppressed, stderr is preserved so
that errors are reported. This means components should avoid printing to
stderr in normal usage (ie in the absence of errors).
Currently all the `check_*` components obey this convention except:
- check_generate_test_code: unittest prints progress to stderr
- check_test_cases: lots of non-fatal warnings printed to stderr
These components will be fixed in follow-up commits.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
People who prefer to rely on HMAC_DRBG (for example because they use it for
deterministic ECDSA and don't want a second DRBG for code size reasons) should
be able to build and run the tests suites without CTR_DRBG.
Ideally we should make sure the level of testing (SSL) is the same regardless
of which DRBG modules is enabled, but that's a more significant piece of work.
For now, just ensure everything builds and `make test` passes.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
Enable branch coverage output in basic_build_test.sh. This
includes enabling branch coverage output to the lcov make target,
which is disabled by default.
Signed-off-by: Dan Handley <dan.handley@arm.com>
We're only interested in files that are committed and pushed to be
included in Mbed TLS, not in any other files that may be lying around.
So ask git for the list of file names.
This script is primarily intended to run on the CI, and there it runs
on a fresh Git checkout plus potentially some other checkouts or
leftovers from a previous part of the CI job. It should also run
reasonably well on developer machines, where there may be various
additional files. In both cases, git is available.
Ad hoc directory exclusions are no longer needed.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Have an explicit list of exemptions for specific checks rather than
whitelisting files to check. Some checks, such as permissions, should
apply to all files.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This is supposed to be for GCC (or a compiler with a compatible
command line interface) targeting arm-none-eabi, so name it
accordingly.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Make it possible to use a compiler that isn't in $PATH, or that's
installed with a different name, or even a compiler for a different
target such as arm-linux-gnueabi.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Almost everything the selftest program does is in the test suites. But
just in case run the selftest program itself once in the full
configuration, and once in the default configuration with ASan, in
addition to running it out of box.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Use "no_deprecated" both in the name of the configuration and in the
name of all.sh components, rather than a mixture of "no_deprecated"
and "non_deprecated".
Make all.sh component names more consistent.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
build_deprecated combined the testing of deprecated features, and
testing of the build without deprecated features. Also, it violated the
component naming convention by being called build_xxx but running tests.
Replace it by:
* test_default_no_deprecated: check that you can remove deprecated
features from the default build.
* test_full_no_deprecated: check that the library builds when deprecated
features are disabled (and incidentally that the tests run).
* test_no_deprecated_warning: check that there are no warnings when
deprecated features are disabled and MBEDTLS_DEPRECATED_WARNING
is enabled.
* test_deprecated: test the deprecated features.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
It's deprecated, but not otherwise counter-indicated for the full
config: it doesn't conflict with anything and enabling it doesn't make
testing harder (especially since it defaults off in compat.sh).
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Ensure that there is a build with -pedantic in the full config, not
just in "exotic" configurations.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
It started out as be experimental, but it is now robust enough not to
break the rest, so there's no reason to leave it out.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The intended logic around MBEDTLS_xxx_ALT is to exclude them from full
because they require the alternative implementation of one or more
library functions, except that MBEDTLS_PLATFORM_xxx_ALT are different:
they're alternative implementations of a platform function and they
have a built-in default, so they should be included in full. Document
this.
Fix a bug whereby MBEDTLS_PLATFORM_xxx_ALT didn't catch symbols where
xxx contains an underscore. As a consequence,
MBEDTLS_PLATFORM_GMTIME_R_ALT and MBEDTLS_PLATFORM_NV_SEED_ALT are now
enabled in the full config. Explicitly exclude
MBEDTLS_PLATFORM_SETUP_TEARDOWN_ALT because it behaves like the
non-platform ones, requiring an extra build-time dependency.
Explicitly exclude MBEDTLS_PLATFORM_NV_SEED_ALT from baremetal
because it requires MBEDTLS_ENTROPY_NV_SEED, and likewise explicitly
unset it from builds that unset MBEDTLS_ENTROPY_NV_SEED.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Simplify the code in minor ways. Each of this changes fixes a warning
from Pylint 2.4 that doesn't appear with Pylint 1.7.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Python 2 is no longer supported upstream. Actively drop compatibility
with Python 2.
Removing the inheritance of a class on object pacifies recent versions
of Pylint (useless-object-inheritance).
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The "full" configuration excludes some deprecated or experimental
features. Enable the ones that have tests, don't have extra
requirements and don't turn off some other feature.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The seedfile needs to have the size of the entropy accumulator, which
is 64 bytes (512 bits) since the entropy accumulator uses SHA-512 and
the seed size needs to be the same as the hash output (or larger).
We used to enable MBEDTLS_ENTROPY_FORCE_SHA256 in the full config, so
the entropy accumulator was 256 bits (32 bytes), and therefore a
32-byte seedfile worked. But we no longer turn on this option in the
full config, so the 32-byte seedfile no longer works.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Check Windows files for some issues, including permissions. Omit the
checks related to special characters (whitespace, line endings,
encoding) as appropriate.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
To test a file name exactly, prepend a / to the base name.
files_to_check actually checks suffixes, not file names, so rename it
to extensions_to_check.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
check_python_files was optional in all.sh because we used to have CI
machines where pylint wasn't available. But this had the downside that
check_python_files kept breaking because it wasn't checked in the CI.
Now our CI has pylint and check_python_files should not be optional.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
On some systems, such as Ubuntu up to 19.04, `pylint` is for Python 2
and `pylint3` is for Python 3, so we should not use `pylint` even if
it's available.
Use the Python module instead of the trivial shell wrapper. This way
we can make sure to use the correct Python version.
Fix#3111
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Merge the latest state of the target branch (mbedtls/development) into the
pull request to merge mbed-crypto into mbedtls.
Conflicts:
* ChangeLog: add/add conflict. Resolve by using the usual section order.
Look for any other invocation of test scripts that was removed: look for a change that removes a line (`^-.*…`) containing one of the names of a test script (without its path because it may be used with a relative path). Look for `ssl-opt.sh` and `compat.sh`, as well any file in `tests/scripts` that only exists in tls.
```
git diff 'HEAD^{/^Merge}~1' HEAD --diff-filter=M -- . ':!library/error.c' ':!library/version_features.c' ':!programs/test/query_config.c' ':!visualc' ':!*.pdf' ':!*.der' | grep -E "^-.*($(comm -23 <(git ls-tree -r --name-only 'HEAD^{/^Merge}~1') <(git ls-tree -r --name-only $(git merge-base upstream-crypto/development 'HEAD^{/^Merge}^2')) | sed -n 's!^tests/scripts/!!p' | sed 's/\./\\./g' | tr '\n' '|')ssl-opt\.sh|compat\.sh)" | grep -v '^---'
```
This only turns up changes in `basic-in-docker.sh`.
The merge of mbed-crypto removed some tls coverage. Restore it. Also
remove references to the `crypto` subdirectory brought by the mbedtls
side of the merge. In more detail:
* `tests/scripts/all.sh`:
* `fuzz` in comments (×2): restore it.
* `CTEST_OUTPUT_ON_FAILURE=1`: don't remove it.
* `cd crypto` for `make clean`: don't restore it.
* `cleanup`: do restore `programs/fuzz/Makefile`. Don't go into `crypto`. Keep only one copy of the calls to `rm` in `cmake_subproject`.
* Comment legacy options: don't remove it.
* `crypto/Makefile` and `pre_check_seedfile`: don't restore either. See below regarding the lack of need for `pre_check_seedfile`.
* blank line in `pre_print_configuration`: restore it.
* blank line before `#### Build and test`: restore it.
* SSL tests in `component_test_full_cmake_gcc_asan` and zlib components: restore it.
* `component_test_no_pem_no_fs` (×2): the merge placed two copies in different locations. Reconcile them: unset PSA storage like in crypto, and call `ssl-opt.sh` like in tls. Put the merged version at the tls location.
* `component_test_everest`: do add it at the tls location.
* `component_test_small_mbedtls_ssl_dtls_max_buffering`: restore the tls value.
* `component_test_new_ecdh_context`…: move `component_test_new_ecdh_context` before `component_test_everest` and add a calls to `compat.sh` and `ssl-opt.sh` like in `component_test_everest`. Remove the redundant crypto-only `component_test_everest`. Don't remove `component_test_psa_collect_statuses`.
* `component_test_full_cmake_clang`: don't remove `clang` in the `msg` call. Don't remove the call to `test_psa_constant_names.py`.
* `component_test_full_make_gcc_o0`: remove it. It's subsumed by `component_test_gcc_opt`.
* `component_build_deprecated`: don't remove anything.
* `component_test_memory_buffer_allocator`: restore `ssl-opt.sh`.
* `component_test_when_no_ciphersuites_have_mac`: restore it.
* `component_test_platform_calloc_macro`: don't restore `unset MBEDTLS_MEMORY_BUFFER_ALLOC_C` which is now redundant. Don't restore explicit flags instead of `$ASAN_CFLAGS`.
* `component_test_aes_fewer_tables`…: don't remove it.
* `component_test_m32_o1`: restore SSL testing.
* `component_test_m32_everest`: restore SSL testing.
* `component_test_min_mpi_window_size`…: don't remove it.
* `component_test_valgrind`: do restore the tls version of the comment.
* `run_component`: don't remove the seedfile creation. This is better than `pre_check_seedfile` (see below).
* `pre_check_seedfile`: don't restore it. `pre_check_seedfile` (from tls) creates a seedfile once and for all. This is not good enough if a component fails in such a way as to leave a broken seedfile, or if a component leaves a seedfile with a size that's wrong for the next component to run. Instead (from crypto), `run_component` creates a sufficiently large seedfile before each component.
Merge `unremove-non-crypto` into `mbedtls/development`. The branch
`unremove-non-crypto` was obtained by starting from `mbed-crypto/development`,
then reverting many commits that removed X.509 and TLS functionality when Mbed
Crypto forked from Mbed TLS (the “unremoval”), then make a few tweaks to
facilitate the merge.
The unremoval step restored old versions of some tls files. If a file doesn't
exist in mbed-crypto, check out the mbedtls version, regardless of what
happened during the unremoval of tls files in the crypto tree. Also
unconditionally take the mbedtls version of a few files where the
modifications are completely project-specific and are not relevant in
mbed-crypto:
* `.github/issue_template.md`: completely different. We may want to reconcile
them independently as a follow-up.
* `.travis.yml`: would only be reverted to an earlier tls version.
* `README.md`: completely different. We may want to reconcile them
independently as a follow-up.
* `doxygen/input/doc_mainpage.h`: the changes in crypto were minimal and not
relevant except as a stopgap as mbed-crypto did not have its own product
versioning in the Doxygen documentation.
* `tests/.jenkins/Jenkinsfile`: completely different.
* `tests/data_files/Makefile`: there were no changes in mbed-crypto,
but the unremoval step restored an old version.
Shell script for everything to do after the merge apart from the conflict
resolution:
```
tls_files=($(comm -23 <(git ls-tree -r --name-only HEAD) <(git ls-tree -r --name-only $(git merge-base upstream-crypto/development MERGE_HEAD))))
tls_files+=($tls_files .github/issue_template.md .travis.yml README.md doxygen/input/doc_mainpage.h tests/.jenkins/Jenkinsfile tests/data_files/Makefile)
git checkout --theirs HEAD -- $tls_files
git add -- $tls_files
```
Resolve the remaining conflicts:
* `library/CMakeLists.txt`:
* Keep the TLS definition of `src_crypto`
* `USE_SHARED_MBEDTLS_LIBRARY`: keep all three libraries, with both
`include` and `crypto/include` in `target_include_directories`, all with
version `2.21.0`.
* `programs/Makefile`:
* Reconcile the APPS lists (add/add from a differently-formatted common
ancestor): insert the `psa/*` from crypto into the tls list.
* Keep the `fuzz` target defined only in tls version.
* Keep the recipe (only in tls version) cleaning `ssl_pthread_server`
stuff for the `clean` target.
* `scripts/config.py`:
* `include_in_full`: add/add conflict. Keep both.
* `tests/scripts/all.sh`:
* `component_test_no_use_psa_crypto_full_cmake_asan`: partially old
version in crypto. Take the tls version.
* `component_test_malloc_0_null` and more: take
`component_test_malloc_0_null` from crypto (with `config.py` rather than
`config.pl`, and with `$ASAN_FLAGS` rather than an explicit list), but
add the call to `ssl-opt.sh` from tls. Take the other components from
crypto.
With this commit, building and running the unit tests with both `make ` and
`cmake` work in the default configuration on Linux. Other platforms, build
systems and configurations are likely not to work, and there is some
regression in test coverage.
There is some loss of functionality because the unremoval step restored older
versions of tls content. This commit contains the latest tls version of
tls-only files, but some changes from the tls side in files that existed on
both sides have regressed. Most problematic changes are hunks that remove some
tls-specific feature and contain either a C preprocessor symbol identifying a
tls-specific module or option, or the name of a tls-specific file. Hunks
that remove a tls-specific preprocessor symbol can be identified with the
regular expression `^-.*MBEDTLS_(ERR_)?(PKCS11|X509|NET|SSL)_`.
Subsequent commits will revert a few parts of the patch from this merge commit
in order to restore the tls functionality that it removes, ensure that the
test coverage includes what was covered in either branch, and fix test
failures.
This reverts commit 9b90f2e294.
Conflicts:
* tests/scripts/all.sh: do the same changes, dancing around the new
outcome file feature and components added in the same places.
Make sure that the components that are getting added back are at the
same locations as where they are now in mbedtls.
This reverts commit 9afb2e9921.
Conflicts:
* include/CMakeLists.txt
* "Make config.h available" comment: there has been a change
adjacent to where it was removed. Just re-add what was removed.
* tests/CMakeLists.txt:
* compat.sh: there has been a change immediately before where it was
removed. Just re-add what was removed.
This reverts commit 1ad37309e4.
Conflicts:
* tests/scripts/test-ref-configs.pl:
* config-ccm-psk-tls1_2.h: there has been an addition of
"config-symmetric-only.h" at the place where the configurations
that are added back were removed. Keep this configuration.
This reverts commit 1c66e48670.
Conflicts:
* include/mbedtls/check_config.h:
* MBEDTLS_SSL_PROTO_SSL3: there has been an addition (of
MBEDTLS_SHA512_NO_SHA384) at the place where it was removed. Re-add it
after (alphabetical order).
* MBEDTLS_ENABLE_WEAK_CIPHERSUITES: there has been an addition (of
MBEDTLS_CTR_DRBG_USE_128_BIT_KEY) at the place where it was removed.
Re-add it after (alphabetical order).
* MBEDTLS_SSL_ALL_ALERT_MESSAGES: there has been an addition (of
MBEDTLS_SHA512_SMALLER) at the place where it was removed. Re-add it
after (alphabetical order).
* include/mbedtls/config.h:
* MBEDTLS_ENABLE_WEAK_CIPHERSUITES: there has been an addition (of
MBEDTLS_CTR_DRBG_USE_128_BIT_KEY) at the place where it was removed.
Re-add it after (alphabetical order).
* MBEDTLS_SSL_ALL_ALERT_MESSAGES: there has been an addition (of
MBEDTLS_SHA512_SMALLER) at the place where it was removed. Re-add it
after (alphabetical order).
* library/version_features.c: re-generate by running
scripts/generate_features.pl.
* programs/test/query_config.c: re-generate by running
scripts/generate_query_config.pl.
* scripts/config.pl: this file has been replaced by config.py. Port
the reversed changes to config.py:
* Revert removing three symbols from the list of symbols to
exclude from full.
* Revert removing one symbol (MBEDTLS_NET_C) from the list of symbols
to exclude from baremetal.
* scripts/footprint.sh:
* Re-add the line to unset MBEDTLS_NET_C, but with config.py instead of
config.pl.
* tests/scripts/all.sh:
* component_test_no_platform: re-add the line to unset MBEDTLS_NET_C, but
with config.py instead of config.pl.
* component_build_arm_none_eabi_gcc,
component_build_arm_none_eabi_gcc_no_udbl_division,
component_build_arm_none_eabi_gcc_no_64bit_multiplication,
component_build_armcc: these components now use the baremetal
configuration, so they do not need to turn off MBEDTLS_NET_C explicitly.
This reverts commit bb1f701212.
* include/mbedtls/check_config.h:
* MBEDTLS_X509_RSASSA_PSS_SUPPORT: there has been an addition (of
MBEDTLS_SHA512_NO_SHA384) at the place where it was removed.
Re-add it before MBEDTLS_SHA512_NO_SHA384 to keep it grouped
with MBEDTLS_RSA_C.
Conflicts:
* scripts/config.pl: this file has been replaced by config.py. Port
the reversed changes to config.py:
* Revert removing three symbols from the list of symbols to
exclude from full.
Pylint when installed as a distro package can be installed as pylint3, whilst as
a PEP egg, it can be installed as pylint.
This commit changes the scripts to first use pylint if installed, and optionally
look for pylint3 if not installed. This is to allow a preference for the PEP
version over the distro version, assuming the PEP one is more likely to be
the correct one.
Signed-off-by: Simon Butcher <simon.butcher@arm.com>
The .pl version is now a compat wrapper around the .py script. Better call the
.py script directly.
Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
Exercise the feature alone, with record splitting and DTLS connection ID.
Signed-off-by: Andrzej Kurek <andrzej.kurek@arm.com>
Signed-off-by: Darryl Green <darryl.green@arm.com>
Exercise the library functions with calloc returning NULL for a size
of 0. Make this a separate job with UBSan (and ASan) to detect
places where we try to dereference the result of calloc(0) or to do
things like
buf = calloc(size, 1);
if (buf == NULL && size != 0) return INSUFFICIENT_MEMORY;
memcpy(buf, source, size);
which has undefined behavior when buf is NULL at the memcpy call even
if size is 0.
This is needed because other test components jobs either use the system
malloc which returns non-NULL on Linux and FreeBSD, or the
memory_buffer_alloc malloc which returns NULL but does not give as
useful feedback with ASan (because the whole heap is a single C
object).
As the SSL programs, like ssl_client2 and ssl_server2, are dependent on
SSL and therefore about to be removed, the only consumer of query_config
is the query_compile_time_config test. As such, it makes sense to move
query_config to be next to what uses it.
Goals:
* Build with common compilers with common options, so that we don't
miss a (potentially useful) warning only triggered with certain
build options.
* A previous commit removed -O0 test jobs, leaving only the one with
-m32. We have inline assembly that is disabled with -O0, falling
back to generic C code. This commit restores a test that runs the
generic C code on a 64-bit platform.
For each of the crypto-only presets, run the build and check that the
resulting libmbedx509 and libmbedtls are empty.
Don't bother testing, because for each crypto-only preset, another
component builds that plus the x509 and tls parts and tests
everything.
We already have a specific component in all.sh for testing SSLv3, we don't
need to also test it in components that aren't specifically about it.
Previously config.py full enabled SSLv3, but it no longer does since it is
deprecated.
All key types now have an encoding on 32 bits where the bottom 16 bits
are zero. Change to using 16 bits only.
Keep 32 bits for key types in storage, but move the significant
half-word from the top to the bottom.
Likewise, change EC curve and DH group families from 32 bits out of
which the top 8 and bottom 16 bits are zero, to 8 bits only.
Reorder psa_core_key_attributes_t to avoid padding.
Define constants for ECC curve families and DH group families. These
constants have 0x0000 in the lower 16 bits of the key type.
Support these constants in the implementation and in the PSA metadata
tests.
Switch the slot management and secure element driver HAL tests to the
new curve encodings. This requires SE driver code to become slightly
more clever when figuring out the bit-size of an imported EC key since
it now needs to take the data size into account.
Switch some documentation to the new encodings.
Remove the macro PSA_ECC_CURVE_BITS which can no longer be implemented.
This is useful to inspect what the script does manually, in particular
to check that expected values do get tested. --keep-c provides the
same information but in a way that's harder to access.
Filter out non-ASCII characters in automatically processed headers.
Do this in a way that minimizes the code change: keep manipulating
strings, but strip off non-ASCII characters when reading lines, which
should only remove characters in comments that we don't parse anyway.
When MBEDTLS_TEST_DEPRECATED is defined, run some additional tests to
validate deprecated PSA macros. We don't need to test deprecated
features extensively, but we should at least ensure that they don't
break the build.
Add some code to component_build_deprecated in all.sh to run these
tests with MBEDTLS_DEPRECATED_WARNING enabled. The tests are also
executed when MBEDTLS_DEPRECATED_WARNING and
MBEDTLS_DEPRECATED_REMOVED are both disabled.
Move backward compatibility aliases to a separate header. Reserve
crypto_extra.h for implementation-specific extensions that we intend
to keep supporting.
This is better documentation for users. New users should simply ignore
backward compatibility aliases, and old users can look at
crypto_compat.h to see what is deprecated without bothering about new
features appearing in crypto_extra.h.
This facilitates maintenance because scripts such as
generate_psa_constants that want to ignore backward compability
aliases can simply exclude crypto_compat.h from their parsing.
When gathering test cases from test_suite_psa_crypto_metadata, look up
the test function explicitly. This way test_psa_constant_names will
error out if we add a new test function that needs coverage here.
This change highlights an omission in the previous version:
asymmetric_signature_wildcard was silently ignored as a source of
algorithm expressions to test. Fix that.
Key agreement algorithms were excluded back when they were constructed
with a macro conveying the key agreement itself taking the KDF as an
argument, because that was hard to support. Now the encoding has
changed and key agreement algorithms are constructed with
PSA_ALG_KEY_AGREEMENT taking two arguments, one that identifies the
raw key agreement and one that identifies the KDF. This is easy to
process, so add support.
Insist that test cases must only use macro names that are declared in
a header. This may catch errors such as not parsing the intended
files.
Make this check easily overridden in a derived class.
The size of the seedfile used by the entropy module when
MBEDTLS_ENTROPY_NV_SEED is enabled is 32 byte when
MBEDTLS_ENTROPY_FORCE_SHA256 is enabled or MBEDTLS_SHA512_C is
disabled, and 64 bytes otherwise. A larger seedfile is ok on
entry (the code just grabs the first N bytes), but a smaller seedfile
is not ok. Therefore, if you run a component with a 32-byte seedfile
and then a component with a 64-byte seedfile, the second component
fails in the unit tests (up to test_suite_entropy which erases the
seedfile and creates a fresh one).
This is ok up to now because we only enable MBEDTLS_ENTROPY_NV_SEED
together with MBEDTLS_ENTROPY_FORCE_SHA256. But it prevents enabling
MBEDTLS_ENTROPY_NV_SEED without MBEDTLS_ENTROPY_FORCE_SHA256.
To fix this, unconditionally create a seedfile before each component.
Enabling MBEDTLS_MEMORY_BUFFER_ALLOC_C module together with
MBEDTLS_PLATFORM_MEMORY causes the library to use its own malloc
replacement. This makes memory management analyzers such as ASan
largely ineffective. We now test MBEDTLS_MEMORY_BUFFER_ALLOC_C
separately. Disable it in the "full" config.
This mirrors a change that was made in Mbed TLS on config.pl and had
not been ported to Mbed Crypto yet.
With this commit, config.py is aligned in Mbed Crypto and Mbed TLS.
First deal with deleted files.
* Files deleted by us: keep them deleted.
* Files deleted by them, whether modified by us or not: keep our version.
```
git rm $(git status -s | sed -n 's/^DU //p')
git reset -- $(git status -s | sed -n 's/^D //p')
git checkout -- $(git status -s | sed -n 's/^ D //p')
git add -- $(git status -s | sed -n 's/^UD //p')
```
Individual files with conflicts:
* `3rdparty/everest/library/Hacl_Curve25519_joined.c`: spurious conflict because git mistakenly identified this file as a rename. Keep our version.
* `README.md`: conflict due to their change in a paragraph that doesn't exist in our version. Keep our version of this paragraph.
* `docs/architecture/Makefile`: near-identical additions. Adapt the definition of `all_markdown` and include the clean target.
* `doxygen/input/docs_mainpage.h`: conflict in the version number. Keep our version number.
* `include/mbedtls/config.h`: two delete/modify conflicts. Keep the removed chunks out.
* `library/CMakeLists.txt`: discard all their changes as they are not relevant.
* `library/Makefile`:
* Discard the added chunk about the crypto submodule starting with `INCLUDING_FROM_MBEDTLS:=1`.
* delete/modify: keep the removed chunk out.
* library build: This is almost delete/modify. Their changes are mostly not applicable. Do keep the `libmbedcrypto.$(DLEXT): | libmbedcrypto.a` order dependency.
* `.c.o`: `-o` was added on both sides but in a different place. Change to their place.
* `library/error.c`: to be regenerated.
* `library/version_features.c`: to be regenerated.
* `programs/Makefile`: Most of the changes are not relevant. The one relevant change is in the `clean` target for Windows; adapt it by removing `/S` from our version.
* `programs/test/query_config.c`: to be regenerated.
* `scripts/config.py`: added in parallel on both sides. Keep our version.
* `scripts/footprint.sh`: parallel changes. Keep our version.
* `scripts/generate_visualc_files.pl`: one delete/modify conflict. Keep the removed chunks out.
* `tests/Makefile`: discard all of their changes.
* `tests/scripts/all.sh`:
* `pre_initialize_variables` add `append_outcome`: add it.
* `pre_initialize_variables` add `ASAN_CFLAGS`: already there, keep our version.
* `pre_parse_command_line` add `--no-append-outcome`: add it.
* `pre_parse_command_line` add `--outcome-file`: add it.
* `pre_print_configuration`: add `MBEDTLS_TEST_OUTCOME_FILE`.
* Several changes in SSL-specific components: keep our version without them.
* Several changes where `config.pl` was changed to `config.py` and there was an adjacent difference: keep our version.
* Changes regarding the inclusion of `MBEDTLS_MEMORY_xxx`: ignore them here, they will be normalized in a subsequent commit.
* `component_test_full_cmake_gcc_asan`: add it without the TLS tests.
* `component_test_no_use_psa_crypto_full_cmake_asan`: keep the fixed `msg`, discard other changes.
* `component_test_memory_buffer_allocator_backtrace`, `component_test_memory_buffer_allocator`: add them without the TLS tests.
* `component_test_m32_everest`: added in parallel on both sides. Keep our version.
* `tests/scripts/check-names.sh`, `tests/scripts/list-enum-consts.pl`, `tests/scripts/list-identifiers.sh`, ``tests/scripts/list-macros.sh`: discard all of their changes.
* `tests/scripts/test-ref-configs.pl`: the change in the conflict is not relevant, so keep our version there.
* `visualc/VS2010/*.vcxproj`: to be regenerated.
Regenerate files:
```
scripts/generate_visualc_files.pl
git add visualc/VS2010/*.vcxproj
scripts/generate_errors.pl
git add library/error.c
scripts/generate_features.pl
git add library/version_features.c
scripts/generate_query_config.pl
git add programs/test/query_config.c
```
Rejected changes in non-conflicting files:
* `CMakeLists.txt`: discard their addition which has already been side-ported.
* `doxygen/mbedtls.doxyfile`: keep the version number change. Discard the changes related to `../crypto` paths.
Keep the following changes after examination:
* `.travis.yml`: all of their changes are relevant.
* `include/mbedtls/error.h`: do keep their changes. Even though Crypto doesn't use TLS errors, it must not encroach on TLS's allocated numbers.
* `tests/scripts/check-test-cases.py`: keep the code dealing with `ssl-opt.sh`. It works correctly when the file is not present.
The tests were not covering get for a symbol with a value. No symbol
has an uncommented value in the default config.h. (Actually there's
_CRT_SECURE_NO_DEPRECATE, but that's a bit of a hack that this script
is not expected to handle, so don't use it).
Add tests of "get FOO" after "set FOO" and "set FOO value", so that we
have coverage for "get FOO" when "FOO" has a value.
We currently test setting a symbol with a value even if it didn't
originally had one and vice versa. So there's no need to have separate
lists of symbols to test with. Just test everything we want to test
with each symbol.
Normally a valueless symbol remains valueless and a symbol with a
value keeps having one. But just in case a symbol does get changed
from valueless to having a value, make sure there's a space between
the symbol and the value. And if a symbol gets changed from having a
value to valueless, strip trailing whitespace.
Add corresponding tests.
Also fix the case of a valueless symbol added with the set method,
which would have resulted in attempting to use None as a string. This
only happened with the Python API, not with the command line API.
Run config.py with various options and store the results in files.
This script also supports the now-removed config.pl.
This is a framework to run non-regression tests on config.py: run it
with the old version, run it with the new version, and compare the
output.
This is deliberately not a functional test suite so that we don't need
to maintain a set of known outputs. When something changes in
config.py (or config.h), run the script before, run it after, and
check manually whether any differences in the output are acceptable.
git grep -Fl /config.pl | xargs sed -i -e 's!/config\.pl!/config.py!g'
Also:
* Change one comment in include/mbedtls/check_config.h.
* Change PERL to PYTHON in CMakeLists.txt.
MBEDTLS_PK_SIGNATURE_MAX_SIZE is tested in Mbed Crypto. Its effect on
Mbed TLS is also tested via the X.509 tests. The case of
MBEDTLS_MPI_MAX_SIZE < MBEDTLS_ECDSA_MAX_LEN, for which this component
was added as a regression test, is covered by config-suite-b.h which
is tested via test-ref-configs.pl.
Enabling memory_buffer_alloc is slow and makes ASan ineffective. We
have a patch pending to remove it from the full config. In the
meantime, disable it explicitly.
When running 'make test' with GNU make, if a test suite program
displays "PASSED", this was automatically counted as a pass. This
would in particular count as passing:
* A test suite with the substring "PASSED" in a test description.
* A test suite where all the test cases succeeded, but the final
cleanup failed, in particular if a sanitizer reported a memory leak.
Use the test executable's return status instead to determine whether
the test suite passed. It's always 0 on PASSED unless the executable's
cleanup code fails, and it's never 0 on any failure.
FixARMmbed/mbed-crypto#303
Some sanitizers default to displaying an error message and recovering.
This could result in a test being recorded as passing despite a
complaint from the sanitizer. Turn off sanitizer recovery to avoid
this risk.
When running 'make test' with GNU make, if a test suite program
displays "PASSED", this was automatically counted as a pass. This
would in particular count as passing:
* A test suite with the substring "PASSED" in a test description.
* A test suite where all the test cases succeeded, but the final
cleanup failed, in particular if a sanitizer reported a memory leak.
Use the test executable's return status instead to determine whether
the test suite passed. It's always 0 on PASSED unless the executable's
cleanup code fails, and it's never 0 on any failure.
FixARMmbed/mbed-crypto#303
Some sanitizers default to displaying an error message and recovering.
This could result in a test being recorded as passing despite a
complaint from the sanitizer. Turn off sanitizer recovery to avoid
this risk.
config-symmetric-only.h enables MBEDTLS_ENTROPY_NV_SEED so it needs a
seedfile. Create it in test-ref-configs.pl so that the script works on
its own, even if it is not invoked by all.sh.
Add a reference configuration with most symmetric cryptographic
algorithms enabled, but without any asymmetric cryptography. This
checks that we don't have spurious unexpected dependencies on
asymmetric-only modules such as bignum.
Keep HAVE_ASM disabled because it's platform-specific.
Keep HAVEGE disabled because it's untested and not recommended.
Keep MEMORY_BUFFER_ALLOC out because it isn't related to cryptography
at all and it makes memory sanitizers ineffective.
Keep THREADING disabled because it requires special build options.
* origin/pr/2843: (26 commits)
Make hyperlink a hyperlink in every markdown flavor
Update the crypto submodule to be the same as development
Document test case descriptions
Restore MBEDTLS_TEST_OUTCOME_FILE after test_default_out_of_box
ssl-opt.sh: Fix some test case descriptions
Reject non-ASCII characters in test case descriptions
Process input files as binary
Factor description-checking code into a common function
Fix cosmetic error in warnings
Fix regex matching run_test calls in ssl-opt.sh
all.sh: run check-test-cases.py
Better information messages for quick checks
Fix configuration short name in key-exchanges.pl
Make test case descriptions unique
New test script check-test-cases.py
Document the test outcome file
Create infrastructure for architecture documents in Markdown
all.sh --outcome-file creates an outcome file
Set meaningful test configuration names when running tests
ssl-opt: remove semicolons from test case descriptions
...
Since components run in the main process, unsetting
MBEDTLS_TEST_OUTCOME_FILE unset it in subsequent components as well.
To avoid this, save and restore the value.
(Making each component run in a subshell would be a better solution,
but it would be a much bigger change.)
This is testing with $kex, not without $kex, so use $kex, not "-$kex".
In test-ref-configs.pl, use $conf rather than "$conf". This is purely
a matter of Perl coding style.
This script checks test case descriptions in test_suite_*.data and
ssl-opt.sh.
It reports the following issues:
* Error: forbidden character in a test case description.
* Error: Duplicate test description.
* Warning: Test description is too long.
Exercise the library functions with calloc returning NULL for a size
of 0. Make this a separate job with UBSan (and ASan) to detect
places where we try to dereference the result of calloc(0) or to do
things like
buf = calloc(size, 1);
if (buf == NULL && size != 0) return INSUFFICIENT_MEMORY;
memcpy(buf, source, size);
which has undefined behavior when buf is NULL at the memcpy call even
if size is 0.
This is needed because other test components jobs either use the system
malloc which returns non-NULL on Linux and FreeBSD, or the
memory_buffer_alloc malloc which returns NULL but does not give as
useful feedback with ASan (because the whole heap is a single C
object).
The tests were not covering get for a symbol with a value. No symbol
has an uncommented value in the default config.h. (Actually there's
_CRT_SECURE_NO_DEPRECATE, but that's a bit of a hack that this script
is not expected to handle, so don't use it).
Add tests of "get FOO" after "set FOO" and "set FOO value", so that we
have coverage for "get FOO" when "FOO" has a value.
We currently test setting a symbol with a value even if it didn't
originally had one and vice versa. So there's no need to have separate
lists of symbols to test with. Just test everything we want to test
with each symbol.
Set MBEDTLS_TEST_PLATFORM and MBEDTLS_TEST_CONFIGURATION to meaningful
values in all.sh. These environment variables are used when
writing an outcome file, which happens if MBEDTLS_TEST_OUTCOME_FILE is
also set.
When running one of the try-multiple-configuration scripts, set
MBEDTLS_TEST_CONFIGURATION to a value that uniquely describes the
configuration.
Normally a valueless symbol remains valueless and a symbol with a
value keeps having one. But just in case a symbol does get changed
from valueless to having a value, make sure there's a space between
the symbol and the value. And if a symbol gets changed from having a
value to valueless, strip trailing whitespace.
Add corresponding tests.
Also fix the case of a valueless symbol added with the set method,
which would have resulted in attempting to use None as a string. This
only happened with the Python API, not with the command line API.
Run config.py with various options and store the results in files.
This script also supports the now-removed config.pl.
This is a framework to run non-regression tests on config.py: run it
with the old version, run it with the new version, and compare the
output.
This is deliberately not a functional test suite so that we don't need
to maintain a set of known outputs. When something changes in
config.py (or config.h), run the script before, run it after, and
check manually whether any differences in the output are acceptable.
git grep -Fl /config.pl | xargs sed -i -e 's!/config\.pl!/config.py!g'
Also:
* Change one comment in include/mbedtls/check_config.h.
* Change PERL to PYTHON in CMakeLists.txt.
* origin/pr/2469:
Enable MBEDTLS_MEMORY_DEBUG in memory buffer alloc test in all.sh
Remove unnecessary memory buffer alloc unsets
Disable DTLS proxy tests for MEMORY_BUFFER_ALLOC test
all.sh: restructure memory allocator tests
Add missing dependency in memory buffer alloc set in all.sh
Don't set MBEDTLS_MEMORY_DEBUG through `scripts/config.pl full`
Add cfg dep MBEDTLS_MEMORY_DEBUG->MBEDTLS_MEMORY_BUFFER_ALLOC_C
Fix memory leak in CSR test suite on failure
Fix a memory leak in x509write test suite
Add all.sh run with full config and ASan enabled
Add all.sh run with MBEDTLS_MEMORY_BUFFER_ALLOC_C enabled
Update documentation of exceptions for `config.pl full`
Adapt all.sh to removal of buffer allocator from full config
Disable memory buffer allocator in full config
Check dependencies of MBEDTLS_MEMORY_BACKTRACE in check_config.h
Add infrastructure to run unit tests and collect the return values for
every PSA API function that returns psa_status_t.
./tests/scripts/psa_collect_statuses.py >statuses.txt
With the removal of MBEDTLS_MEMORY_BUFFER_ALLOC_C from the
full config, there are no tests for it remaining in all.sh.
This commit adds a build as well as runs of `make test` and
`ssl-opt.sh` with MBEDTLS_MEMORY_BUFFER_ALLOC_C enabled to all.sh.
Previously, numerous all.sh tests manually disabled the buffer allocator
or memory backtracting after setting a full config as the starting point.
With the removal of MBEDTLS_MEMORY_BACKTRACE and MBEDTLS_MEMORY_BUFFER_ALLOC_C
from full configs, this is no longer necessary.
* origin/development:
Fix copypasta in msg
When not using PSA crypto, disable it
Disable MEMORY_BUFFER_ALLOC with ASan
Remove config.pl calls with no effect
ssl-opt.sh: wait for proxy to start before running the script further
Adapt ChangeLog
Fix mpi_bigendian_to_host() on bigendian systems
* origin/pr/2771:
Fix copypasta in msg
When not using PSA crypto, disable it
Disable MEMORY_BUFFER_ALLOC with ASan
Remove config.pl calls with no effect
* origin/development: (42 commits)
Handle deleting non-existant files on Windows
Update submodule
Use 3rdparty headers from the submodule
Add Everest components to all.sh
3rdparty: Add config checks for Everest
Fix macros in benchmark.c
Update generated files
3rdparty: Fix inclusion order of CMakeLists.txt
Fix trailing whitespace
ECDH: Fix inclusion of platform.h for proper use of MBEDTLS_ERR_PLATFORM_FEATURE_UNSUPPORTED
ECDH: Fix use of ECDH API in full handshake benchmark
ECDH: Removed unnecessary calls to mbedtls_ecp_group_load in ECDH benchmark
ECDH: Fix Everest x25519 make_public
Fix file permissions
3rdparty: Rename THIRDPARTY_OBJECTS
3rdparty: Update description of MBEDTLS_ECDH_VARIANT_EVEREST_ENABLED
3rdparty: Fix Makefile coding conventions
ECDSA: Refactor return value checks for mbedtls_ecdsa_can_do
Add a changelog entry for Everest ECDH (X25519)
Document that curve lists can include partially-supported curves
...
* origin/development: (114 commits)
Don't redefine calloc and free
Add changelog entry to record checking
Fix compiler warning
Add debug messages
Remove duplicate entries from ChangeLog
Fix parameter name in doxygen
Add missing guards for mac usage
Improve reability and debugability of large if
Fix a typo in a comment
Fix MSVC warning
Fix compile error in reduced configurations
Avoid duplication of session format header
Implement config-checking header to context s11n
Provide serialisation API only if it's enabled
Fix compiler warning: comparing signed to unsigned
Actually reset the context on save as advertised
Re-use buffer allocated by handshake_init()
Enable serialisation tests in ssl-opt.sh
Change requirements for setting timer callback
Add setting of forced fields when deserializing
...
The size of the ticket used in this test dropped from 192 to 143 bytes, so
move all sizes used in this test down 50 bytes. Also, we now need to adapt the
server response size as the default size would otherwise collide with the new
mtu value.
Essentially adds the Everest .h and .c files to the various variables. This
should be generalized at some point, but there is no infrastructure for this
yet.
* crypto/pr/212: (337 commits)
Make TODO comments consistent
Fix PSA tests
Fix psa_generate_random for >1024 bytes
Add tests to generate more random than MBEDTLS_CTR_DRBG_MAX_REQUEST
Fix double free in psa_generate_key when psa_generate_random fails
Fix copypasta in test data
Avoid a lowercase letter in a macro name
Correct some comments
Fix PSA init/deinit in mbedtls_xxx tests when using PSA
Make psa_calculate_key_bits return psa_key_bits_t
Adjust secure element code to the new ITS interface
More refactoring: consolidate attribute validation
Fix policy validity check on key creation.
Add test function for import with a bad policy
Test key creation with an invalid type (0 and nonzero)
Remove "allocated" flag from key slots
Take advantage of psa_core_key_attributes_t internally #2
Store the key size in the slot in memory
Take advantage of psa_core_key_attributes_t internally: key loading
Switch storage functions over to psa_core_key_attributes_t
...
* development:
Update crypto to a repo with latest crypto
Update Mbed Crypto
tls: Remove duplicate psa_util.h include
Remove unused cryptography test files
Remove crypto C files
Remove files sourced from Mbed Crypto
config: Fix Doxygen link to MBEDTLS_PARAM_FAILED
Use mbedtls-based path for includes
check-names: Consider crypto-sourced header files
Resolve conflicts by performing the following actions:
- Reject changes to ChangeLog, as Mbed Crypto doesn't have one
- Reject changes to tests/compat.sh, as Mbed Crypto doesn't have it
- Reject changes to programs/fuzz/onefile.c, as Mbed Crypto doesn't have
it
- Resolve minor whitespace differences in library/ecdsa.c by taking the
version from Mbed TLS upstream.
* origin/development:
Honor MBEDTLS_CONFIG_FILE in fuzz tests
Test that a shared library build produces a dynamically linked executable
Test that the shared library build with CMake works
Add a test of MBEDTLS_CONFIG_FILE
Exclude DTLS 1.2 only with older OpenSSL
Document the rationale for the armel build
Switch armel build to -Os
Add a build on ARMv5TE in ARM mode
Add changelog entry for ARM assembly fix
bn_mul.h: require at least ARMv6 to enable the ARM DSP code
Adapt ChangeLog
ECP restart: Don't calculate address of sub ctx if ctx is NULL
Without any -O option, the default is -O0, and then the assembly code
is not used, so this would not be a non-regression test for the
assembly code that doesn't build.
Conflict resolution:
* `scripts/config.pl`:
Take the exclusion of `MBEDTLS_PSA_CRYPTO_SE_C` from the API branch.
Take the removal of `MBEDTLS_PSA_CRYPTO_STORAGE_ITS_C` (obsolete) from
the development branch.
* `tests/scripts/all.sh`:
Multiple instances of factoring a sequence of `config.pl` calls into
a mere `config.pl baremetal` in the development branch, and a change in
the composition of `baremetal` in the API branch. In each case, take the
version from development.
* `tests/suites/test_suite_psa_crypto_slot_management.function`:
A function became non-static in development and disappeared in the API
branch. Keep the version from the API branch. Functions need to be
non-static if they're defined but unused in some configurations,
which is not the case for any function in this file at the moment.
* `tests/suites/test_suite_psa_crypto.function`:
Consecutive changes in the two branches, reconciled.
Resolve conflicts by performing the following operations:
- Reject changes related to building a crypto submodule, since Mbed
Crypto is the crypto submodule.
- Reject X.509, NET, and SSL changes.
- Reject changes to README, as Mbed Crypto is a different project from
Mbed TLS, with a different README.
- Avoid adding mention of ssl-opt.sh in a comment near some modified
code in include/CMakeLists.txt (around where ENABLE_TESTING as added).
- Align config.pl in Mbed TLS with config.pl in Mbed Crypto where PSA
options are concerned, to make future merging easier. There is no
reason for the two to be different in this regard, now that Mbed TLS
always depends on Mbed Crypto. Remaining differences are only the
PSA_CRYPTO_KEY_FILE_ID_ENCODES_OWNER option and the absence of X.509,
NET, and SSL related options in Mbed Crypto's config.pl.
- Align config.h in Mbed Crypto with Mbed TLS's copy, with a few notable
exceptions:
- Leave CMAC on by default.
- Leave storage on by default (including ITS emulation).
- Avoid documenting the PSA Crypto API as is in beta stage in
documentation for MBEDTLS_PSA_CRYPTO_C.
The only remaining differences are a lack of X.509, NET, and SSL
options in Mbed Crypto's config.h, as well as an additional
Mbed-Crypto-specific PSA_CRYPTO_KEY_FILE_ID_ENCODES_OWNER option.
Documentation for the check params feature and related macros is also
updated to match Mbed TLS's description.
- Reject tests/data_files/Makefile changes to generate DER versions of
CRTs and keys, as none of those are used by Mbed Crypto tests.
- Add the "no PEM and no filesystem" test to all.sh, without ssl-opt.sh
run, as Mbed Crypto doesn't have ssl-opt.sh. Also remove use of PSA
Crypto storage and ITS emulation, since those depend on filesystem
support.
- Reject addition of test when no ciphersuites have MAC to all.sh, as
the option being tested, MBEDTLS_SSL_SOME_MODES_USE_MAC, is not
present in Mbed Crypto.
- Use baremetal config in all.sh, as Mbed Crypto's baremetal
configuration does exclude the net module (as it doesn't exist in Mbed
Crypto)
- Reject cmake_subproject_build changes, continuing to link only
libmbedcrypto.
- Reject changes to visualc and associated templates. Mbed Crypto
doesn't need additional logic to handle submodule-sourced headers.
- Avoid adding fuzzers from Mbed TLS. The only relevant fuzzers are the
privkey and pubkey fuzzers, but non-trivial work would be required to
integrate those into Mbed Crypto (more than is comfortable in a merge
commit).
- Reject addition of Docker wrappers for compat.sh and ssl-opt.sh, as
those are not present in Mbed Crypto.
- Remove calls to SSL-related scripts from basic-in-docker.sh
Fix test errors by performing the following:
- Avoid using a link that Doxygen can't seem to resolve in Mbed Crypto,
but can resolve in Mbed TLS. In documentation for
MBEDTLS_CHECK_PARAMS, don't attempt to link to MBEDTLS_PARAM_FAILED.
* origin/development: (339 commits)
Do not build fuzz on windows
No booleans and import config
Removing space before opening parenthesis
Style corrections
Syntax fix
Fixes warnings from MSVC
Add a linker flag to enable gcov in basic-build-test.sh
Update crypto submodule to a revision with the HAVEGE header changes
Test with MBEDTLS_ECP_RESTARTABLE
Allow TODO in code
Use the docstring in the command line help
Split _abi_compliance_command into smaller functions
Record the commits that were compared
Document how to build the typical argument for -s
Allow running /somewhere/else/path/to/abi_check.py
tests: Limit each log to 10 GiB
Warn if VLAs are used
Remove redundant compiler flag
Consistently spell -Wextra
Fix parsing issue when int parameter is in base 16
...
* origin/development: (36 commits)
Do not build fuzz on windows
No booleans and import config
Removing space before opening parenthesis
Style corrections
Syntax fix
Fixes warnings from MSVC
Add a linker flag to enable gcov in basic-build-test.sh
tests: Limit each log to 10 GiB
Fix parsing issue when int parameter is in base 16
checks MBEDTLS_PEM_PARSE_C
Restore programs/fuzz/Makefile after in-tree cmake
Move fuzz directory to programs
Documentation for corpus generation
Restore tests/fuzz/Makefile after in-tree cmake
Adding ifdefs to avoid warnings for unused globals
Adds LDFLAGS fsanitize=address
Refactor receive_uint32()
Refactor get_byte function
Make the script portable to both pythons
Update the test encoding to support python3
...
* origin/pr/2660:
Fix parsing issue when int parameter is in base 16
Refactor receive_uint32()
Refactor get_byte function
Make the script portable to both pythons
Update the test encoding to support python3
update the test script
* origin/pr/1622: (29 commits)
Do not build fuzz on windows
No booleans and import config
Removing space before opening parenthesis
Style corrections
Syntax fix
Fixes warnings from MSVC
Add a linker flag to enable gcov in basic-build-test.sh
checks MBEDTLS_PEM_PARSE_C
Restore programs/fuzz/Makefile after in-tree cmake
Move fuzz directory to programs
Documentation for corpus generation
Restore tests/fuzz/Makefile after in-tree cmake
Adding ifdefs to avoid warnings for unused globals
Adds LDFLAGS fsanitize=address
Ignore compiled object files and executables
Also clean the fuzz subdirectory
copyediting README.md
Protecting client/server fuzz targts with ifdefs
Makefile support 1
Fuzz README and direct compilation
...
* origin/development: (33 commits)
Test with MBEDTLS_ECP_RESTARTABLE
Allow TODO in code
Use the docstring in the command line help
Split _abi_compliance_command into smaller functions
Record the commits that were compared
Document how to build the typical argument for -s
Allow running /somewhere/else/path/to/abi_check.py
Warn if VLAs are used
Remove redundant compiler flag
Consistently spell -Wextra
Update Mbed Crypto to contain mbed-crypto#152
Improve compatibility with firewalled networks
Dockerfile: apt -> apt-get
Change Docker container to bionic
Clean up file prologue comments
Add docker-based test scripts
ChangeLog: Add ChangeLog entry for #2681
Allow declarations after statements
CMake: Add a subdirectory build regression test
README: Enable builds as a CMake subproject
...
* origin/pr/2706:
Update Mbed Crypto to contain mbed-crypto#152
CMake: Add a subdirectory build regression test
README: Enable builds as a CMake subproject
ChangeLog: Enable builds as a CMake subproject
Remove use of CMAKE_SOURCE_DIR
We accidentally disabled testing with MBEDTLS_ECP_RESTARTABLE. Re-enable
testing with restartable ECP when MBEDTLS_USE_PSA_CRYPTO is not set.
Fixes 971dea3745 ("Enable USE_PSA_CRYPTO with config.pl full")
Fix error `ValueError: invalid literal for int() with base 10:` that
is caused when a parameter is given in base 16. Use relevant base
when calling `int()` function.
* Allow specifying MBEDTLS_DOCKER_REGISTRY for organizations that have
a mirroring proxy of Docker Hub
* Specify "--network host" during build to ensure use of the host's
DNS resolution.
Commit 117b8a4516 requires version 6+
of i686-w64-mingw32-gcc to run the mingw builds, but Ubuntu Xenial (16.04)
supplies 5.3.1. Change the Docker container to Ubuntu Bionic (18.04) to
pick up a version that will run the tests.
Enable running tests under Docker. This makes it easier to spin up an
environment with all dependencies (especially the multiple versions of
openssl and gnutls needed).
* tests/docker/xenial/Dockerfile: Definition for the docker image,
including local builds for openssl and gnutls.
* tests/scripts/docker_env.sh: New helper script that creates the Docker
image and has a function to run a command in the Docker container.
* tests/docker/all-in-docker.sh: Wrapper for all.sh under Docker.
* tests/docker/basic-in-docker.sh: Script that runs the same commands as
.travis.yml, in Docker.
* tests/ssl-opt-in-docker.sh: Wrapper to run ssl-opt.sh in Docker.
* tests/compat-in-docker.sh: Wrapper to run compat.sh in Docker.
* tests/make-in-docker.sh: Wrapper to run make in Docker.
Change-Id: Ie092b1deed24c24c3859754535589523ce1d0a58
Secure element support has its own source file, and in addition
requires many hooks in other files. This is a nontrivial amount of
code, so make it optional (but default on).
Since Python3 handles encoding differently than Python2,
a change in the way the data is encoded and sent to the target is needed.
1. Change the test data to be sent as hex string
2. Convert the characters to binary bytes.
This is done because the mbed tools translate the encoding differently
(mbed-greentea, and mbed-htrunner)
* restricted/pr/573:
Remove redundant config.pl call
Add a test for signing content with a long ECDSA key
Add documentation notes about the required size of the signature buffers
Add missing MBEDTLS_ECP_C dependencies in check_config.h
Change size of preallocated buffer for pk_sign() calls
* origin/pr/2697:
Update crypto submodule
Add all.sh component that exercises invalid_param checks
Remove mbedtls_param_failed from programs
Make it easier to define MBEDTLS_PARAM_FAILED as assert
Make test suites compatible with #include <assert.h>
Pass -m32 to the linker as well
Don't systematically rebuild programs
With the change to the full config, there were no longer any tests
that exercise invalid-parameter behavior. The test suite exercises
invalid-parameter behavior by calling TEST_INVALID_PARAM and friends,
relying on the test suite's mbedtls_check_param function. This
function is only enabled if MBEDTLS_CHECK_PARAMS is defined but not
MBEDTLS_CHECK_PARAMS_ASSERT.
Add a component to all.sh that enables MBEDTLS_CHECK_PARAMS but
disables MBEDTLS_CHECK_PARAMS_ASSERT and doesn't define
MBEDTLS_PARAM_FAILED. This way, the xxx_invalid_param() tests do run.
Since sample programs don't provide a mbedtls_check_param function,
this component doesn't build the sample programs.
* origin/pr/2260:
Update crypto submodule
Remove heading spaces in tests/data_files/Makefile
Re-generate library/certs.c from script
Add new line at the end of test-ca2.key.enc
Use strict syntax to annotate origin of test data in certs.c
Add run to all.sh exercising !MBEDTLS_PEM_PARSE_C + !MBEDTLS_FS_IO
Allow DHM self test to run without MBEDTLS_PEM_PARSE_C
ssl-opt.sh: Auto-skip tests that use files if MBEDTLS_FS_IO unset
Document origin of hardcoded certificates in library/certs.c
Adapt ChangeLog
Rename server1.der to server1.crt.der
Add DER encoded files to git tree
Add build instructions to generate DER versions of CRTs and keys
Document "none" value for ca_path/ca_file in ssl_client2/ssl_server2
ssl_server2: Skip CA setup if `ca_path` or `ca_file` argument "none"
ssl_client2: Skip CA setup if `ca_path` or `ca_file` argument "none"
Correct white spaces in ssl_server2 and ssl_client2
Adapt ssl_client2 to parse DER encoded test CRTs if PEM is disabled
Adapt ssl_server2 to parse DER encoded test CRTs if PEM is disabled
Due to the way the current PK API works, it may have not been clear
for the library clients, how big output buffers they should pass
to the signing functions. Depending on the key type they depend on
MPI or EC specific compile-time constants.
Inside the library, there were places, where it was assumed that
the MPI size will always be enough, even for ECDSA signatures.
However, for very small sizes of the MBEDTLS_MPI_MAX_SIZE and
sufficiently large key, the EC signature could exceed the MPI size
and cause a stack overflow.
This test establishes both conditions -- small MPI size and the use
of a long ECDSA key -- and attempts to sign an arbitrary file.
This can cause a stack overvlow if the signature buffers are not
big enough, therefore the test is performed for an ASan build.
For unit tests and sample programs, CFLAGS=-m32 is enough to get a
32-bit build, because these programs are all compiled directly
from *.c to the executable in one shot. But with makefile rules that
first build object files and then link them, LDFLAGS=-m32 is also
needed.
When testing a configuration where no ciphersuites have MAC, via
component_test_when_no_ciphersuites_have_mac(), perform a targeted test
of only encrypt-then-MAC tests within ssl-opt.sh.
When MBEDTLS_SSL_ENCRYPT_THEN_MAC is enabled, but not
MBEDTLS_SSL_SOME_MODES_USE_MAC, mbedtls_ssl_derive_keys() and
build_transforms() will attempt to use a non-existent `encrypt_then_mac`
field in the ssl_transform.
Compile [ 93.7%]: ssl_tls.c
[Error] ssl_tls.c@865,14: 'mbedtls_ssl_transform {aka struct mbedtls_ssl_transform}' ha
s no member named 'encrypt_then_mac'
[ERROR] ./mbed-os/features/mbedtls/src/ssl_tls.c: In function 'mbedtls_ssl_derive_keys'
:
./mbed-os/features/mbedtls/src/ssl_tls.c:865:14: error: 'mbedtls_ssl_transform {aka str
uct mbedtls_ssl_transform}' has no member named 'encrypt_then_mac'
transform->encrypt_then_mac = session->encrypt_then_mac;
^~
Change mbedtls_ssl_derive_keys() and build_transforms() to only access
`encrypt_then_mac` if `encrypt_then_mac` is actually present.
Add a regression test to detect when we have regressions with
configurations that do not include any MAC ciphersuites.
Fixes d56ed2491b ("Reduce size of `ssl_transform` if no MAC ciphersuite is enabled")
Use PEP 257 indented docstring style, mostly: always with """, with the
terminating """ on a separate line if the docstring is more than one
line, and with all lines indented to the opening """.
This commit does not change the text to keep the first paragraph single-line.
When running make with parallelization, running both "clean" and "lib"
with a single make invocation can lead to each target building in
parallel. It's bad if lib is partially done building something, and then
clean deletes what was built. This can lead to errors later on in the
lib target.
$ make -j9 clean lib
CC aes.c
CC aesni.c
CC arc4.c
CC aria.c
CC asn1parse.c
CC ./library/error.c
CC ./library/version.c
CC ./library/version_features.c
AR libmbedcrypto.a
ar: aes.o: No such file or directory
Makefile:120: recipe for target 'libmbedcrypto.a' failed
make[2]: *** [libmbedcrypto.a] Error 1
Makefile:152: recipe for target 'libmbedcrypto.a' failed
make[1]: *** [libmbedcrypto.a] Error 2
Makefile:19: recipe for target 'lib' failed
make: *** [lib] Error 2
make: *** Waiting for unfinished jobs....
To avoid this sort of trouble, always invoke clean by itself without
other targets throughout the library. Don't run clean in parallel with
other rules. The only place where clean was run in parallel with other
targets was in list-symbols.sh.
* Rename internal methods and fields to start with an underscore.
* Rename global constants to uppercase.
* Change methods that don't use self to be class methods or static
methods as appropriate.
No behavior change in this commit.
Conflicts:
* library/ssl_cli.c, library/ssl_tls.c:
Removed on the development branch. Keep them removed.
* include/psa/crypto_extra.h, library/psa_crypto_storage.c,
tests/suites/test_suite_psa_crypto.data,
tests/suites/test_suite_psa_crypto.function,
tests/suites/test_suite_psa_crypto_persistent_key.data,
tests/suites/test_suite_psa_crypto_slot_management.data,
tests/suites/test_suite_psa_crypto_slot_management.function:
Modified on the development branch only to implement the enrollment
algorithm, which has been reimplemented on the API branch.
Keep the API branch.
* origin/pr/2403: (24 commits)
crypto: Update to Mbed Crypto 8907b019e7
Create seedfile before running tests
crypto: Update to Mbed Crypto 81f9539037
ssl_cli.c : add explicit casting to unsigned char
Generating visualc files - let Mbed TLS take precedence over crypto
Add a link to the seedfile for out-of-tree cmake builds
Adjust visual studio file generation to always use the crypto submodule
all.sh: unparallelize mingw tests
all.sh - disable parallelization for shared target tests
config.pl: disable PSA_ITS_FILE and PSA_CRYPTO_STORAGE for baremetal
all.sh: unset crypto storage define in a psa full config cmake asan test
all.sh: unset FS_IO-dependent defines for tests that do not have it
curves.pl - change test script to not depend on the implementation
Export the submodule flag to sub-cmakes
Disable MBEDTLS_ECP_RESTARTABLE in full config
Export the submodule flag to sub-makes
Force the usage of crypto submodule
Fix crypto submodule usage in Makefile
Documentation rewording
Typo fixes in documentation
...
With MBEDTLS_USE_PSA_CRYPTO and MBEDTLS_ENTROPY_NV_SEED enabled, the
tests need a seedfile. Since test_suite_entropy is no longer there to
create it, and MBEDTLS_USE_PSA_CRYPTO is now enabled in the full
config, create tests/seedfile explicitly in basic-build-test.sh.
Sometimes, when building the shared target with crypto submodule, one could
get an "No rule to make target '../crypto/library/libmbedcrypto.so'" error.
This is due to two reasons - building in parallel and (probably) an
incomplete/incorrect list of dependencies for certain targets. The proposed
solution is to disable parallel builds with crypto submodule for now.
An issue has been raised here: https://github.com/ARMmbed/mbedtls/issues/2634
Sometimes, when building the shared target with crypto submodule, one could
get an "No rule to make target '../crypto/library/libmbedcrypto.so'" error.
This is due to two reasons - building in parallel and (probably) an
incomplete/incorrect list of dependencies for certain targets. The proposed
solution is to disable parallel builds with crypto submodule for now.
An issue has been raised here: https://github.com/ARMmbed/mbedtls/issues/2634
Currently the top-level makefile deploys commands to both Mbed TLS and
the submodule. Running make in the "tests" directory builds only the TLS tests.
The top level CMake on the other hand does not have the "tests" target defined,
so it also cannot be used, hence the raw "make".
Previously it was disabled as too experimental, which no longer holds. Also,
this option introduces new APIs, so it's not only about an internal
alternative (as the comment in config.pl used to state) - people who request a
full config should get all of the available APIs.
Adapt all.sh: now all builds with full config will also test this option, and
builds with the default config will test without it. Just to be sure, let's
have a build with full config minus this option.
Update documentation of MBEDTLS_USE_PSA_CRYPTO to reflect the status of the
new APIs it enables in Mbed TLS and why they're still opt-in.
Also enable it in scripts/config.pl full, as well as two storage options that
were only blacklisted from full config because they depended on
MBEDTLS_PSA_CRYPTO_C.
Adapt tests in all.sh:
- tests with submodule enabled (default) no longer need to enable it
explicitly, and no longer need runtime tests, as those are now handled by
all other test cases in this script
- tests with submodule disabled (old default) now need to disable it
explicitly, and execute some runtime tests, as those are no longer tested
anywhere else in this script
Adapt documentation in Readme: remove the section "building with submodule"
and replace it with a new section before the other building sections.
Purposefully don't document how to build not from the submodule, as that
option is going away soon.
Parametrize finite-field Diffie-Hellman key types with a DH group
identifier, in the same way elliptic curve keys are parametrized with
an EC curve identifier.
Define the DH groups from the TLS registry (these are the groups from
RFC 7919).
Replicate the macro definitions and the metadata tests from elliptic
curve identifiers to DH group identifiers.
Define PSA_DH_GROUP_CUSTOM as an implementation-specific extension for
which domain parameters are used to specify the group.
We've observed that sometimes check-names.sh exits unexpectedly with
status 2 and no error message. The failure is not reproducible. This
commits makes the script print a trace if it exits unexpectedly.
Resolve merge conflicts by performing the following actions:
- Reject changes to deleted files.
- Reject changes to generate_errors.pl and generate_visualc_files.pl.
Don't add an 'include-crypto' option which would attempt to use the
non-existent crypto submodule.
- list-identifiers.sh had the `--internal` option added to it, which
lists identifiers only in internal headers. Add PSA-specific internal
headers to list-identifiers.sh.
* origin/development: (40 commits)
Document the scripts behaviour further
Use check_output instead of Popen
all.sh: Require i686-w64-mingw32-gcc version >= 6
generate_visualc_files.pl: add mbedtls source shadowing by crypto
generate_errors.pl: refactor and simplify the code
Start unused variable with underscore
Correct documentation
generate_errors.pl: typo fix
revert changes to generate_features.pl and generate_query_config.pl
Check that the report directory is a directory
Use namespaces instead of full classes
Fix pylint issues
Don't put abi dumps in subfolders
Add verbose switch to silence all output except the final report
Fetch the remote crypto branch, rather than cloning it
Prefix internal functions with underscore
Add RepoVersion class to make handling of many arguments easier
Reduce indentation levels
Improve documentation
Use optional arguments for setting repositories
...
Remove TLS and NET options from config files and scripts.
Note that this fails check-names.sh because options that TLS and NET
files use are no longer present in config.h.
Merge Mbed TLS at f790a6cbee into Mbed Crypto.
Resolve conflicts by performing the following:
- Reject changes to README.md
- Don't add crypto as a submodule
- Remove test/ssl_cert_test from programs/Makefile
- Add cipher.nist_kw test to tests/CMakeLists.txt
- Reject removal of crypto-specific all.sh tests
- Reject update to SSL-specific portion of component_test_valgrind
in all.sh
- Reject addition of ssl-opt.sh testing to component_test_m32_o1 in
all.sh
* tls/development: (87 commits)
Call mbedtls_cipher_free() to reset a cipher context
Don't call mbedtls_cipher_setkey twice
Update crypto submodule
Minor fixes in get certificate policies oid test
Add certificate policy oid x509 extension
cpp_dummy_build: Add missing header psa_util.h
Clarify comment mangled by an earlier refactoring
Add an "out-of-box" component
Run ssl-opt.sh on 32-bit runtime
Don't use debug level 1 for informational messages
Skip uncritical unsupported extensions
Give credit to OSS-Fuzz for #2404
all.sh: remove component_test_new_ecdh_context
Remove crypto-only related components from all.sh
Remove ssl_cert_test sample app
Make CRT callback tests more robust
Rename constant in client2.c
Document and test flags in x509_verify
Fix style issues and a typo
Fix a rebase error
...
Run ssl-opt.sh on x86_32 with ASan. This may detect bugs that only
show up on 32-bit platforms, for example due to size_t overflow.
For this component, turn off some memory management features that are
not useful, potentially slow, and may reduce ASan's effectiveness at
catching buffer overflows.
