From b1320f3319c1ce704deb3f1713b182b4d97fbe31 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Sat, 5 Oct 2019 11:27:39 +0200 Subject: [PATCH 01/19] Write a change log from mbedcrypto-2.0.0 to the merge of #75 Get started on writing a change log file for Mbed Crypto. I went through pull requests merged since the tag mbedcrypto-2.0.0 and up to #75, i.e. commit 9ab7c07f1f370636fcaa8bc02e6f45035fab1596. --- ChangeLog.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 ChangeLog.md diff --git a/ChangeLog.md b/ChangeLog.md new file mode 100644 index 000000000..3e17237f2 --- /dev/null +++ b/ChangeLog.md @@ -0,0 +1,28 @@ +# Mbed Crypto change history + +## Unreleased changes + +### Interface changes + +* In the PSA API, forbid zero-length keys. To pass a zero-length input to a key derivation function, use a buffer instead (this is now always possible). + +### New features + +* Key derivation inputs in the PSA API can now either come from a key object or from a buffer regardless of the step type. + +### Bug fixes + +* Fix a buffer overflow in the PSA HMAC code when using a long key with an unsupported algorithm. Fixes #254. +* Fix `mbedtls_asn1_get_int` to support any number of leading zeros. +* Fix `mbedtls_asn1_get_bitstring_null` to correctly parse bitstrings of at most 2 bytes. + +### Performance improvements + +* Remove a useless call to mbedtls_ecp_group_free(). Contributed by Alexander Krizhanovsky in #210. +* Speed up PBKDF2 by caching the digest calculation. Contributed by Jack Lloyd and Fortanix Inc in #277. + +### Other changes + +* Remove the technical possibility to define custom md_info structures, which was exposed only in an internal header. + +## Mbed Crypto 2.0.0 From 40b3f411ecb16522beb073e1131f8e727be29ea4 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Sun, 13 Oct 2019 21:44:25 +0200 Subject: [PATCH 02/19] Changelog merging script assemble_changelog.py reads changelog entries from ChangeLog.d/*.md and merges them into ChangeLog.md. The changelog entries are merged into the first version in ChangeLog.md. The script does not yet support creating a new version in ChangeLog.md. The changelog entries are merged in alphabetical order of the file names. Future versions of this script are likely to adopt a different order that reflects the git history of the entries. --- ChangeLog.d/README | 21 +++ scripts/assemble_changelog.py | 241 ++++++++++++++++++++++++++++++++++ 2 files changed, 262 insertions(+) create mode 100644 ChangeLog.d/README create mode 100755 scripts/assemble_changelog.py diff --git a/ChangeLog.d/README b/ChangeLog.d/README new file mode 100644 index 000000000..2f9f049f9 --- /dev/null +++ b/ChangeLog.d/README @@ -0,0 +1,21 @@ +This directory contains changelog entries that have not yet been merged +to the changelog file (../ChangeLog.md). + +A changelog entry file must have the extension *.md and must have the +following format: + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +### Section title + +* Change descritpion. +* Another change description. + +### Another section title + +* Yet another change description. +* Yet again another change description. + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +See STANDARD_SECTIONS in ../scripts/assemble_changelog.py for +recognized section titles. diff --git a/scripts/assemble_changelog.py b/scripts/assemble_changelog.py new file mode 100755 index 000000000..91db79350 --- /dev/null +++ b/scripts/assemble_changelog.py @@ -0,0 +1,241 @@ +#!/usr/bin/env python3 + +"""Assemble Mbed Crypto change log entries into the change log file. +""" + +# Copyright (C) 2019, Arm Limited, All Rights Reserved +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# This file is part of Mbed Crypto (https://tls.mbed.org) + +import argparse +import glob +import os +import re +import sys + +class InputFormatError(Exception): + def __init__(self, filename, line_number, message, *args, **kwargs): + self.filename = filename + self.line_number = line_number + self.message = message.format(*args, **kwargs) + def __str__(self): + return '{}:{}: {}'.format(self.filename, self.line_number, self.message) + +STANDARD_SECTIONS = ( + b'Interface changes', + b'Default behavior changes', + b'Requirement changes', + b'New deprecations', + b'Removals', + b'New features', + b'Security', + b'Bug fixes', + b'Performance improvements', + b'Other changes', +) + +class ChangeLog: + """An Mbed Crypto changelog. + + A changelog is a file in Markdown format. Each level 2 section title + starts a version, and versions are sorted in reverse chronological + order. Lines with a level 2 section title must start with '##'. + + Within a version, there are multiple sections, each devoted to a kind + of change: bug fix, feature request, etc. Section titles should match + entries in STANDARD_SECTIONS exactly. + + Within each section, each separate change should be on a line starting + with a '*' bullet. There may be blank lines surrounding titles, but + there should not be any blank line inside a section. + """ + + _title_re = re.compile(br'#*') + def title_level(self, line): + """Determine whether the line is a title. + + Return (level, content) where level is the Markdown section level + (1 for '#', 2 for '##', etc.) and content is the section title + without leading or trailing whitespace. For a non-title line, + the level is 0. + """ + level = re.match(self._title_re, line).end() + return level, line[level:].strip() + + def add_sections(self, *sections): + """Add the specified section titles to the list of known sections. + + Sections will be printed back out in the order they were added. + """ + for section in sections: + if section not in self.section_content: + self.section_list.append(section) + self.section_content[section] = [] + + def __init__(self, input_stream): + """Create a changelog object. + + Read lines from input_stream, which is typically a file opened + for reading. + """ + level_2_seen = 0 + current_section = None + self.header = [] + self.section_list = [] + self.section_content = {} + self.add_sections(*STANDARD_SECTIONS) + self.trailer = [] + for line in input_stream: + level, content = self.title_level(line) + if level == 2: + level_2_seen += 1 + if level_2_seen <= 1: + self.header.append(line) + else: + self.trailer.append(line) + elif level == 3 and level_2_seen == 1: + current_section = content + self.add_sections(current_section) + elif level_2_seen == 1 and current_section != None: + if line.strip(): + self.section_content[current_section].append(line) + elif level_2_seen <= 1: + self.header.append(line) + else: + self.trailer.append(line) + + def add_file(self, input_stream): + """Add changelog entries from a file. + + Read lines from input_stream, which is typically a file opened + for reading. These lines must contain a series of level 3 + Markdown sections with recognized titles. The corresponding + content is injected into the respective sections in the changelog. + The section titles must be either one of the hard-coded values + in assemble_changelog.py or already present in ChangeLog.md. + """ + filename = input_stream.name + current_section = None + for line_number, line in enumerate(input_stream, 1): + if not line.strip(): + continue + level, content = self.title_level(line) + if level == 3: + current_section = content + if current_section not in self.section_content: + raise InputFormatError(filename, line_number, + 'Section {} is not recognized', + str(current_section)[1:]) + elif level == 0: + if current_section is None: + raise InputFormatError(filename, line_number, + 'Missing section title at the beginning of the file') + self.section_content[current_section].append(line) + else: + raise InputFormatError(filename, line_number, + 'Only level 3 headers (###) are permitted') + + def write(self, filename): + """Write the changelog to the specified file. + """ + with open(filename, 'wb') as out: + for line in self.header: + out.write(line) + for section in self.section_list: + lines = self.section_content[section] + while lines and not lines[0].strip(): + del lines[0] + while lines and not lines[-1].strip(): + del lines[-1] + if not lines: + continue + out.write(b'### ' + section + b'\n\n') + for line in lines: + out.write(line) + out.write(b'\n') + for line in self.trailer: + out.write(line) + +def finish_output(files_to_remove, changelog, output_file): + """Write the changelog to the output file. + + Remove the specified input files. + """ + if os.path.exists(output_file) and not os.path.isfile(output_file): + # The output is a non-regular file (e.g. pipe). Write to it directly. + output_temp = output_file + else: + # The output is a regular file. Write to a temporary file, + # then move it into place atomically. + output_temp = output_file + '.tmp' + changelog.write(output_temp) + for filename in files_to_remove: + sys.stderr.write('Removing ' + filename + '\n') + #os.remove(filename) + if output_temp != output_file: + os.rename(output_temp, output_file) + +def merge_entries(options): + """Merge changelog entries into the changelog file. + + Read the changelog file from options.input. + Read entries to merge from the directory options.dir. + Write the new changelog to options.output. + Remove the merged entries if options.keep_entries is false. + """ + with open(options.input, 'rb') as input_file: + changelog = ChangeLog(input_file) + files_to_merge = glob.glob(os.path.join(options.dir, '*.md')) + if not files_to_merge: + sys.stderr.write('There are no pending changelog entries.\n') + return + for filename in files_to_merge: + with open(filename, 'rb') as input_file: + changelog.add_file(input_file) + files_to_remove = [] if options.keep_entries else files_to_merge + finish_output(files_to_remove, changelog, options.output) + +def set_defaults(options): + """Add default values for missing options.""" + output_file = getattr(options, 'output', None) + if output_file is None: + options.output = options.input + if getattr(options, 'keep_entries', None) is None: + options.keep_entries = (output_file is not None) + +def main(): + """Command line entry point.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--dir', '-d', metavar='DIR', + default='ChangeLog.d', + help='Directory to read entries from (default: ChangeLog.d)') + parser.add_argument('--input', '-i', metavar='FILE', + default='ChangeLog.md', + help='Existing changelog file to read from and augment (default: ChangeLog.md)') + parser.add_argument('--keep-entries', + action='store_true', dest='keep_entries', default=None, + help='Keep the files containing entries (default: remove them if --output/-o is not specified)') + parser.add_argument('--no-keep-entries', + action='store_false', dest='keep_entries', + help='Remove the files containing entries after they are merged (default: remove them if --output/-o is not specified)') + parser.add_argument('--output', '-o', metavar='FILE', + help='Output changelog file (default: overwrite the input)') + options = parser.parse_args() + set_defaults(options) + merge_entries(options) + +if __name__ == '__main__': + main() From f296cdb2aba1af147f80cedb275d7907ce3beb0e Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Wed, 22 Jan 2020 12:43:20 +0100 Subject: [PATCH 03/19] Fix formatting --- ChangeLog.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ChangeLog.md b/ChangeLog.md index 3e17237f2..06efdc7a5 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -18,11 +18,11 @@ ### Performance improvements -* Remove a useless call to mbedtls_ecp_group_free(). Contributed by Alexander Krizhanovsky in #210. +* Remove a useless call to `mbedtls_ecp_group_free()`. Contributed by Alexander Krizhanovsky in #210. * Speed up PBKDF2 by caching the digest calculation. Contributed by Jack Lloyd and Fortanix Inc in #277. ### Other changes -* Remove the technical possibility to define custom md_info structures, which was exposed only in an internal header. +* Remove the technical possibility to define custom `mbedtls_md_info` structures, which was exposed only in an internal header. ## Mbed Crypto 2.0.0 From 974232f045781a17b494f6993b2b02e150476653 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Wed, 22 Jan 2020 12:43:29 +0100 Subject: [PATCH 04/19] Minor documentation improvements --- scripts/assemble_changelog.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/scripts/assemble_changelog.py b/scripts/assemble_changelog.py index 91db79350..be83fd18c 100755 --- a/scripts/assemble_changelog.py +++ b/scripts/assemble_changelog.py @@ -88,8 +88,9 @@ class ChangeLog: def __init__(self, input_stream): """Create a changelog object. - Read lines from input_stream, which is typically a file opened - for reading. + Populate the changelog object from the content of the file + input_stream. This is typically a file opened for reading, but + can be any generator returning the lines to read. """ level_2_seen = 0 current_section = None @@ -125,7 +126,9 @@ class ChangeLog: Markdown sections with recognized titles. The corresponding content is injected into the respective sections in the changelog. The section titles must be either one of the hard-coded values - in assemble_changelog.py or already present in ChangeLog.md. + in STANDARD_SECTIONS in assemble_changelog.py or already present + in ChangeLog.md. Section titles must match byte-for-byte except that + leading or trailing whitespace is ignored. """ filename = input_stream.name current_section = None From 5e39c9e94f62d924ec94f8887baa088f05930b07 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Wed, 22 Jan 2020 14:55:37 +0100 Subject: [PATCH 05/19] Actually remove files Minor rework of how files are removed. Actually do remove the files (earlier I accidentally committed a debug version with removal commented out). --- scripts/assemble_changelog.py | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/scripts/assemble_changelog.py b/scripts/assemble_changelog.py index be83fd18c..1de57bbbb 100755 --- a/scripts/assemble_changelog.py +++ b/scripts/assemble_changelog.py @@ -172,10 +172,9 @@ class ChangeLog: for line in self.trailer: out.write(line) -def finish_output(files_to_remove, changelog, output_file): +def finish_output(changelog, output_file): """Write the changelog to the output file. - Remove the specified input files. """ if os.path.exists(output_file) and not os.path.isfile(output_file): # The output is a non-regular file (e.g. pipe). Write to it directly. @@ -185,12 +184,13 @@ def finish_output(files_to_remove, changelog, output_file): # then move it into place atomically. output_temp = output_file + '.tmp' changelog.write(output_temp) - for filename in files_to_remove: - sys.stderr.write('Removing ' + filename + '\n') - #os.remove(filename) if output_temp != output_file: os.rename(output_temp, output_file) +def remove_merged_entries(files_to_remove): + for filename in files_to_remove: + os.remove(filename) + def merge_entries(options): """Merge changelog entries into the changelog file. @@ -208,8 +208,9 @@ def merge_entries(options): for filename in files_to_merge: with open(filename, 'rb') as input_file: changelog.add_file(input_file) - files_to_remove = [] if options.keep_entries else files_to_merge - finish_output(files_to_remove, changelog, options.output) + finish_output(changelog, options.output) + if not options.keep_entries: + remove_merged_entries(files_to_merge) def set_defaults(options): """Add default values for missing options.""" From 8c4a84c5ded7bb19a5a607591b38971ed79b0a11 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Wed, 22 Jan 2020 15:40:39 +0100 Subject: [PATCH 06/19] Split read_main_file out of the ChangeLog constructor Keep the constructor code simple. No behavior change. --- scripts/assemble_changelog.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/scripts/assemble_changelog.py b/scripts/assemble_changelog.py index 1de57bbbb..1e59bb0bc 100755 --- a/scripts/assemble_changelog.py +++ b/scripts/assemble_changelog.py @@ -92,13 +92,22 @@ class ChangeLog: input_stream. This is typically a file opened for reading, but can be any generator returning the lines to read. """ - level_2_seen = 0 - current_section = None self.header = [] self.section_list = [] self.section_content = {} self.add_sections(*STANDARD_SECTIONS) self.trailer = [] + self.read_main_file(input_stream) + + def read_main_file(self, input_stream): + """Populate the changelog object from the content of the file. + + This method is only intended to be called as part of the constructor + of the class and may not act sensibly on an object that is already + partially populated. + """ + level_2_seen = 0 + current_section = None for line in input_stream: level, content = self.title_level(line) if level == 2: From 566407d6f68e906946ff80648c2bde9024bda192 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Wed, 22 Jan 2020 15:55:36 +0100 Subject: [PATCH 07/19] Simpler definition of a custom exception class --- scripts/assemble_changelog.py | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/scripts/assemble_changelog.py b/scripts/assemble_changelog.py index 1e59bb0bc..bba2c39c4 100755 --- a/scripts/assemble_changelog.py +++ b/scripts/assemble_changelog.py @@ -28,11 +28,9 @@ import sys class InputFormatError(Exception): def __init__(self, filename, line_number, message, *args, **kwargs): - self.filename = filename - self.line_number = line_number - self.message = message.format(*args, **kwargs) - def __str__(self): - return '{}:{}: {}'.format(self.filename, self.line_number, self.message) + message = '{}:{}: {}'.format(filename, line_number, + message.format(*args, **kwargs)) + super().__init__(message) STANDARD_SECTIONS = ( b'Interface changes', From 6e91009cfe0bf578ee6afb0cba09f188a238f4af Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Wed, 22 Jan 2020 15:58:18 +0100 Subject: [PATCH 08/19] Split strings on some very long lines --- scripts/assemble_changelog.py | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/scripts/assemble_changelog.py b/scripts/assemble_changelog.py index bba2c39c4..e6ebe6f5c 100755 --- a/scripts/assemble_changelog.py +++ b/scripts/assemble_changelog.py @@ -232,18 +232,23 @@ def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument('--dir', '-d', metavar='DIR', default='ChangeLog.d', - help='Directory to read entries from (default: ChangeLog.d)') + help='Directory to read entries from' + ' (default: ChangeLog.d)') parser.add_argument('--input', '-i', metavar='FILE', default='ChangeLog.md', - help='Existing changelog file to read from and augment (default: ChangeLog.md)') + help='Existing changelog file to read from and augment' + ' (default: ChangeLog.md)') parser.add_argument('--keep-entries', action='store_true', dest='keep_entries', default=None, - help='Keep the files containing entries (default: remove them if --output/-o is not specified)') + help='Keep the files containing entries' + ' (default: remove them if --output/-o is not specified)') parser.add_argument('--no-keep-entries', action='store_false', dest='keep_entries', - help='Remove the files containing entries after they are merged (default: remove them if --output/-o is not specified)') + help='Remove the files containing entries after they are merged' + ' (default: remove them if --output/-o is not specified)') parser.add_argument('--output', '-o', metavar='FILE', - help='Output changelog file (default: overwrite the input)') + help='Output changelog file' + ' (default: overwrite the input)') options = parser.parse_args() set_defaults(options) merge_entries(options) From 2b242495e12ffb9010c177fa2d62fc01854e70b8 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Wed, 22 Jan 2020 15:41:50 +0100 Subject: [PATCH 09/19] Add a sanity check on the output Check that no line from any of the input files was lost. This is not perfect for several reasons. It doesn't check that the content goes to the desired location. It doesn't check that sections are created as necessary. It doesn't support whitespace normalization that the parsing code does. But it's a good start. --- scripts/assemble_changelog.py | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/scripts/assemble_changelog.py b/scripts/assemble_changelog.py index e6ebe6f5c..b9a647f2a 100755 --- a/scripts/assemble_changelog.py +++ b/scripts/assemble_changelog.py @@ -32,6 +32,11 @@ class InputFormatError(Exception): message.format(*args, **kwargs)) super().__init__(message) +class LostContent(Exception): + def __init__(self, filename, line): + message = ('Lost content from {}: "{}"'.format(filename, line)) + super().__init__(message) + STANDARD_SECTIONS = ( b'Interface changes', b'Default behavior changes', @@ -179,9 +184,30 @@ class ChangeLog: for line in self.trailer: out.write(line) -def finish_output(changelog, output_file): +def check_output(generated_output_file, main_input_file, merged_files): + """Make sanity checks on the generated output. + + The intent of these sanity checks is to have reasonable confidence + that no content has been lost. + + The sanity check is that every line that is present in an input file + is also present in an output file. This is not perfect but good enough + for now. + """ + generated_output = set(open(generated_output_file, 'rb')) + for line in open(main_input_file, 'rb'): + if line not in generated_output: + raise LostContent('original file', line) + for merged_file in merged_files: + for line in open(merged_file, 'rb'): + if line not in generated_output: + raise LostContent(merged_file, line) + +def finish_output(changelog, output_file, input_file, merged_files): """Write the changelog to the output file. + The input file and the list of merged files are used only for sanity + checks on the output. """ if os.path.exists(output_file) and not os.path.isfile(output_file): # The output is a non-regular file (e.g. pipe). Write to it directly. @@ -191,6 +217,7 @@ def finish_output(changelog, output_file): # then move it into place atomically. output_temp = output_file + '.tmp' changelog.write(output_temp) + check_output(output_temp, input_file, merged_files) if output_temp != output_file: os.rename(output_temp, output_file) @@ -215,7 +242,7 @@ def merge_entries(options): for filename in files_to_merge: with open(filename, 'rb') as input_file: changelog.add_file(input_file) - finish_output(changelog, options.output) + finish_output(changelog, options.output, options.input, files_to_merge) if not options.keep_entries: remove_merged_entries(files_to_merge) From c26479c1af343aa74d2605bfbf5d37622014144a Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Wed, 22 Jan 2020 17:56:28 +0100 Subject: [PATCH 10/19] Update ChangeLog up to mbedcrypto-3.0.0d0 --- ChangeLog.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/ChangeLog.md b/ChangeLog.md index 06efdc7a5..fd537da26 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -5,24 +5,40 @@ ### Interface changes * In the PSA API, forbid zero-length keys. To pass a zero-length input to a key derivation function, use a buffer instead (this is now always possible). +* Rename `psa_asymmetric_sign()` to `psa_sign_hash()` and `psa_asymmetric_verify()` to `psa_verify_hash()`. + +### Default behavior changes + +* The initial seeding of a CTR\_DRBG instance makes a second call to the entropy function to obtain entropy for a nonce if the entropy size is less than 3/2 times the key size. In case you want to disable the extra call to grab entropy, you can call `mbedtls_ctr_drbg_set_nonce_len()` to force the nonce length to 0. ### New features * Key derivation inputs in the PSA API can now either come from a key object or from a buffer regardless of the step type. +* The CTR_DRBG module can grab a nonce from the entropy source during the initial seeding. The default nonce length is chosen based on the key size to achieve the security strength defined by NIST SP 800-90A. You can change it with `mbedtls_ctr_drbg_set_nonce_len()`. +* Add ENUMERATED tag support to the ASN.1 module. Contributed by msopiha-linaro in #307. + +### Security + +* Enforce that `mbedtls_entropy_func()` gathers a total of `MBEDTLS_ENTROPY_BLOCK_SIZE` bytes or more from strong sources. In the default configuration, on a platform with a single entropy source, the entropy module formerly only grabbed 32 bytes, which is good enough for security if the source is genuinely strong, but less than the expected 64 bytes (size of the entropy accumulator). ### Bug fixes * Fix a buffer overflow in the PSA HMAC code when using a long key with an unsupported algorithm. Fixes #254. -* Fix `mbedtls_asn1_get_int` to support any number of leading zeros. +* Fix `mbedtls_asn1_get_int` to support any number of leading zeros. Credit to OSS-Fuzz for finding a bug in an intermediate version of the fix. * Fix `mbedtls_asn1_get_bitstring_null` to correctly parse bitstrings of at most 2 bytes. +* `mbedtls_ctr_drbg_set_entropy_len()` and `mbedtls_hmac_drbg_set_entropy_len()` now work if you call them before `mbedtls_ctr_drbg_seed()` or `mbedtls_hmac_drbg_seed()`. +* Fix some false-positive uninitialized variable warnings. Fix contributed by apple-ihack-geek in ARMmbed/mbedtls#2663. ### Performance improvements * Remove a useless call to `mbedtls_ecp_group_free()`. Contributed by Alexander Krizhanovsky in #210. * Speed up PBKDF2 by caching the digest calculation. Contributed by Jack Lloyd and Fortanix Inc in #277. +* Small performance improvement of `mbedtls_mpi_div_mpi()`. Contributed by Alexander Krizhanovsky in #308. ### Other changes * Remove the technical possibility to define custom `mbedtls_md_info` structures, which was exposed only in an internal header. +* `psa_close_key(0)` and `psa_destroy_key(0)` now succeed (doing nothing, as before). +* Variables containing error codes are now initialized to an error code rather than success, so that coding mistakes or memory corruption tends to cause functions to return this error code rather than a success. There are no known instances where this changes the behavior of the library: this is merely a robustness improvement. #323 ## Mbed Crypto 2.0.0 From d8b6c77388f2c6756aa16bfe3932d956b4d1d2d1 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Tue, 28 Jan 2020 18:57:47 +0100 Subject: [PATCH 11/19] Use OrderedDict instead of reinventing it --- scripts/assemble_changelog.py | 22 ++++++---------------- 1 file changed, 6 insertions(+), 16 deletions(-) diff --git a/scripts/assemble_changelog.py b/scripts/assemble_changelog.py index b9a647f2a..3d82c1127 100755 --- a/scripts/assemble_changelog.py +++ b/scripts/assemble_changelog.py @@ -21,6 +21,7 @@ # This file is part of Mbed Crypto (https://tls.mbed.org) import argparse +from collections import OrderedDict import glob import os import re @@ -78,16 +79,6 @@ class ChangeLog: level = re.match(self._title_re, line).end() return level, line[level:].strip() - def add_sections(self, *sections): - """Add the specified section titles to the list of known sections. - - Sections will be printed back out in the order they were added. - """ - for section in sections: - if section not in self.section_content: - self.section_list.append(section) - self.section_content[section] = [] - def __init__(self, input_stream): """Create a changelog object. @@ -96,9 +87,9 @@ class ChangeLog: can be any generator returning the lines to read. """ self.header = [] - self.section_list = [] - self.section_content = {} - self.add_sections(*STANDARD_SECTIONS) + self.section_content = OrderedDict() + for section in STANDARD_SECTIONS: + self.section_content[section] = [] self.trailer = [] self.read_main_file(input_stream) @@ -121,7 +112,7 @@ class ChangeLog: self.trailer.append(line) elif level == 3 and level_2_seen == 1: current_section = content - self.add_sections(current_section) + self.section_content.setdefault(content, []) elif level_2_seen == 1 and current_section != None: if line.strip(): self.section_content[current_section].append(line) @@ -169,8 +160,7 @@ class ChangeLog: with open(filename, 'wb') as out: for line in self.header: out.write(line) - for section in self.section_list: - lines = self.section_content[section] + for section, lines in self.section_content.items(): while lines and not lines[0].strip(): del lines[0] while lines and not lines[-1].strip(): From 974349d40e69b4e93102abebc79dd54e7beeb4bc Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Tue, 28 Jan 2020 19:00:59 +0100 Subject: [PATCH 12/19] Style: follow PEP8 --- scripts/assemble_changelog.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/assemble_changelog.py b/scripts/assemble_changelog.py index 3d82c1127..616978e32 100755 --- a/scripts/assemble_changelog.py +++ b/scripts/assemble_changelog.py @@ -113,7 +113,7 @@ class ChangeLog: elif level == 3 and level_2_seen == 1: current_section = content self.section_content.setdefault(content, []) - elif level_2_seen == 1 and current_section != None: + elif level_2_seen == 1 and current_section is not None: if line.strip(): self.section_content[current_section].append(line) elif level_2_seen <= 1: From 37d670a1e17f86b3411fdf15c643b4e1caf7a503 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Tue, 28 Jan 2020 19:14:15 +0100 Subject: [PATCH 13/19] Document read_main_file and simplify the logic a little --- scripts/assemble_changelog.py | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/scripts/assemble_changelog.py b/scripts/assemble_changelog.py index 616978e32..1d4755708 100755 --- a/scripts/assemble_changelog.py +++ b/scripts/assemble_changelog.py @@ -86,10 +86,15 @@ class ChangeLog: input_stream. This is typically a file opened for reading, but can be any generator returning the lines to read. """ + # Content before the level-2 section where the new entries are to be + # added. self.header = [] + # Content of the level-3 sections of where the new entries are to + # be added. self.section_content = OrderedDict() for section in STANDARD_SECTIONS: self.section_content[section] = [] + # Content of level-2 sections for already-released versions. self.trailer = [] self.read_main_file(input_stream) @@ -100,21 +105,23 @@ class ChangeLog: of the class and may not act sensibly on an object that is already partially populated. """ + # Parse the first level-2 section. Everything before the first + # level-3 section title ("###...") following the first level-2 + # section title ("##...") is passed through as the header + # and everything after the second level-2 section title is passed + # through as the trailer. Inside the first level-2 section, + # split out the level-3 sections. level_2_seen = 0 current_section = None for line in input_stream: level, content = self.title_level(line) if level == 2: level_2_seen += 1 - if level_2_seen <= 1: - self.header.append(line) - else: - self.trailer.append(line) elif level == 3 and level_2_seen == 1: current_section = content self.section_content.setdefault(content, []) - elif level_2_seen == 1 and current_section is not None: - if line.strip(): + if level_2_seen == 1 and current_section is not None: + if level != 3 and line.strip(): self.section_content[current_section].append(line) elif level_2_seen <= 1: self.header.append(line) From da14e8225e48250375c2f9aa07d195f7f5e9a5b8 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Tue, 28 Jan 2020 19:21:11 +0100 Subject: [PATCH 14/19] Remove useless blank line removal in ChangeLog.write The parsing functions eliminate blank lines, so there shouldn't be any at this stage. --- scripts/assemble_changelog.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/scripts/assemble_changelog.py b/scripts/assemble_changelog.py index 1d4755708..a5a114c67 100755 --- a/scripts/assemble_changelog.py +++ b/scripts/assemble_changelog.py @@ -168,10 +168,6 @@ class ChangeLog: for line in self.header: out.write(line) for section, lines in self.section_content.items(): - while lines and not lines[0].strip(): - del lines[0] - while lines and not lines[-1].strip(): - del lines[-1] if not lines: continue out.write(b'### ' + section + b'\n\n') From a26079613a0fb75ba616657f9e15a73912a056ef Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Tue, 28 Jan 2020 19:58:17 +0100 Subject: [PATCH 15/19] Create a new level-2 section if needed Automatically create a level-2 section for unreleased changes if needed. --- scripts/assemble_changelog.py | 39 ++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/scripts/assemble_changelog.py b/scripts/assemble_changelog.py index a5a114c67..04a5d6c39 100755 --- a/scripts/assemble_changelog.py +++ b/scripts/assemble_changelog.py @@ -1,6 +1,10 @@ #!/usr/bin/env python3 """Assemble Mbed Crypto change log entries into the change log file. + +Add changelog entries to the first level-2 section. +Create a new level-2 section for unreleased changes if needed. +Remove the input files unless --keep-entries is specified. """ # Copyright (C) 2019, Arm Limited, All Rights Reserved @@ -79,6 +83,28 @@ class ChangeLog: level = re.match(self._title_re, line).end() return level, line[level:].strip() + # Only accept dotted version numbers (e.g. "3.1", not "3"). + # Refuse ".x" in a version number: this indicates a version that is + # not yet released. + _version_number_re = re.compile(br'[0-9]\.[0-9][0-9.]+([^.]|\.[^0-9x])') + + def section_is_released_version(self, title): + """Whether this section is for a released version. + + True if the given level-2 section title indicates that this section + contains released changes, otherwise False. + """ + # Assume that a released version has a numerical version number + # that follows a particular pattern. These criteria may be revised + # as needed in future versions of this script. + version_number = re.search(self._version_number_re, title) + return bool(version_number) + + def unreleased_version_title(self): + """The title to use if creating a new section for an unreleased version.""" + # pylint: disable=no-self-use; this method may be overridden + return b'Unreleased changes' + def __init__(self, input_stream): """Create a changelog object. @@ -105,18 +131,29 @@ class ChangeLog: of the class and may not act sensibly on an object that is already partially populated. """ - # Parse the first level-2 section. Everything before the first + # Parse the first level-2 section, containing changelog entries + # for unreleased changes. + # If we'll be expanding this section, everything before the first # level-3 section title ("###...") following the first level-2 # section title ("##...") is passed through as the header # and everything after the second level-2 section title is passed # through as the trailer. Inside the first level-2 section, # split out the level-3 sections. + # If we'll be creating a new version, the header is everything + # before the point where we want to add the level-2 section + # for this version, and the trailer is what follows. level_2_seen = 0 current_section = None for line in input_stream: level, content = self.title_level(line) if level == 2: level_2_seen += 1 + if level_2_seen == 1: + if self.section_is_released_version(content): + self.header.append(b'## ' + + self.unreleased_version_title() + + b'\n\n') + level_2_seen = 2 elif level == 3 and level_2_seen == 1: current_section = content self.section_content.setdefault(content, []) From afc9db8bb71bd76dce45f5246f9ea5b52d59a700 Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Thu, 30 Jan 2020 11:38:01 +0100 Subject: [PATCH 16/19] Fix version number recognition heuristics The regexp was wrong, for example it matched "2.20x" but failed to match "3.1". Some test cases: >>> def f(title): ... version_number = re.search(_version_number_re, title) ... if version_number: ... return not re.search(_incomplete_version_number_re, ... version_number.group(0)) ... else: ... return False ... >>> [(s, f(s.encode('ascii'))) for s in ['foo', 'foo 3', 'foo 3.', 'foo 3.1', 'foo 3.14', 'foo 3.2.1', 'foo 3.2.1alpha', 'foo 3.1.a', 'foo 3.a', 'foo 3.x.1']] [('foo', False), ('foo 3', False), ('foo 3.', False), ('foo 3.1', True), ('foo 3.14', True), ('foo 3.2.1', True), ('foo 3.2.1alpha', True), ('foo 3.1.a', False), ('foo 3.a', False), ('foo 3.x.1', False)] --- scripts/assemble_changelog.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/scripts/assemble_changelog.py b/scripts/assemble_changelog.py index 04a5d6c39..a3f720167 100755 --- a/scripts/assemble_changelog.py +++ b/scripts/assemble_changelog.py @@ -84,9 +84,10 @@ class ChangeLog: return level, line[level:].strip() # Only accept dotted version numbers (e.g. "3.1", not "3"). - # Refuse ".x" in a version number: this indicates a version that is - # not yet released. - _version_number_re = re.compile(br'[0-9]\.[0-9][0-9.]+([^.]|\.[^0-9x])') + # Refuse ".x" in a version number where x is a letter: this indicates + # a version that is not yet released. Something like "3.1a" is accepted. + _version_number_re = re.compile(br'[0-9]+\.[0-9A-Za-z.]+') + _incomplete_version_number_re = re.compile(br'.*\.[A-Za-z]') def section_is_released_version(self, title): """Whether this section is for a released version. @@ -98,7 +99,11 @@ class ChangeLog: # that follows a particular pattern. These criteria may be revised # as needed in future versions of this script. version_number = re.search(self._version_number_re, title) - return bool(version_number) + if version_number: + return not re.search(self._incomplete_version_number_re, + version_number.group(0)) + else: + return False def unreleased_version_title(self): """The title to use if creating a new section for an unreleased version.""" From 49f8e3157c5571bbaee62cf4fb12f8c7154f7d29 Mon Sep 17 00:00:00 2001 From: Simon Butcher Date: Tue, 3 Mar 2020 15:51:50 +0000 Subject: [PATCH 17/19] Fix warning with gcc7/maximum allocation exceeded Adds an additional boundary check to the test parameter bytes_arg in generate_random() in test_suite_psa_crypto.function. This is to check against a possible underflow, and to avoid a warning, "argument ... exceeds maximum object size", (-Werror=alloc-size-larger-than) that GCC 7 generates. Signed-off-by: Simon Butcher --- tests/suites/test_suite_psa_crypto.function | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/suites/test_suite_psa_crypto.function b/tests/suites/test_suite_psa_crypto.function index b6e6e5a97..bc95f6fb0 100644 --- a/tests/suites/test_suite_psa_crypto.function +++ b/tests/suites/test_suite_psa_crypto.function @@ -5297,6 +5297,8 @@ void generate_random( int bytes_arg ) size_t i; unsigned run; + TEST_ASSERT( bytes_arg >= 0 ); + ASSERT_ALLOC( output, bytes + sizeof( trail ) ); ASSERT_ALLOC( changed, bytes ); memcpy( output + bytes, trail, sizeof( trail ) ); From cdcb187eb2764fdc3a6458c035f818d1020c05e2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Manuel=20P=C3=A9gouri=C3=A9-Gonnard?= Date: Tue, 24 Mar 2020 11:50:07 +0100 Subject: [PATCH 18/19] Have issue and PR templates redirect to Mbed TLS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Manuel Pégourié-Gonnard --- .github/issue_template.md | 48 +++++--------------------------- .github/pull_request_template.md | 10 +++++++ 2 files changed, 17 insertions(+), 41 deletions(-) create mode 100644 .github/pull_request_template.md diff --git a/.github/issue_template.md b/.github/issue_template.md index 7d4f1e840..a3faf2e1c 100644 --- a/.github/issue_template.md +++ b/.github/issue_template.md @@ -1,44 +1,10 @@ - - -### Description - - - - -### Issue request type - - - [ ] Question - [ ] Enhancement - [ ] Bug +Please open your issue in the mbedtls repository: +https://github.com/ARMmbed/mbedtls/issues/new +Thanks for your interest in Mbed Crypto! diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 000000000..9ece01afb --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,10 @@ +We are no longer accepting new pull requests in this repository +=============================================================== + +Mbed Crypto is now maintained as part of Mbed TLS, see +https://github.com/ARMmbed/mbedtls/issues/3064 + +Please open your pull request in the mbedtls repository: +https://github.com/ARMmbed/mbedtls/pulls + +Thanks for your interest in Mbed Crypto! From 854770b84e4fdbf479827cf26bbbbdc409350a7c Mon Sep 17 00:00:00 2001 From: Gilles Peskine Date: Wed, 25 Mar 2020 17:13:19 +0100 Subject: [PATCH 19/19] Temporarily remove ChangeLog.md All the content that is currently in ChangeLog.md is the result of merging Mbed Crypto, and that content is already present in ChangeLog. Since we aren't using ChangeLog.md yet, remove it for now. Signed-off-by: Gilles Peskine --- ChangeLog.md | 44 -------------------------------------------- 1 file changed, 44 deletions(-) delete mode 100644 ChangeLog.md diff --git a/ChangeLog.md b/ChangeLog.md deleted file mode 100644 index fd537da26..000000000 --- a/ChangeLog.md +++ /dev/null @@ -1,44 +0,0 @@ -# Mbed Crypto change history - -## Unreleased changes - -### Interface changes - -* In the PSA API, forbid zero-length keys. To pass a zero-length input to a key derivation function, use a buffer instead (this is now always possible). -* Rename `psa_asymmetric_sign()` to `psa_sign_hash()` and `psa_asymmetric_verify()` to `psa_verify_hash()`. - -### Default behavior changes - -* The initial seeding of a CTR\_DRBG instance makes a second call to the entropy function to obtain entropy for a nonce if the entropy size is less than 3/2 times the key size. In case you want to disable the extra call to grab entropy, you can call `mbedtls_ctr_drbg_set_nonce_len()` to force the nonce length to 0. - -### New features - -* Key derivation inputs in the PSA API can now either come from a key object or from a buffer regardless of the step type. -* The CTR_DRBG module can grab a nonce from the entropy source during the initial seeding. The default nonce length is chosen based on the key size to achieve the security strength defined by NIST SP 800-90A. You can change it with `mbedtls_ctr_drbg_set_nonce_len()`. -* Add ENUMERATED tag support to the ASN.1 module. Contributed by msopiha-linaro in #307. - -### Security - -* Enforce that `mbedtls_entropy_func()` gathers a total of `MBEDTLS_ENTROPY_BLOCK_SIZE` bytes or more from strong sources. In the default configuration, on a platform with a single entropy source, the entropy module formerly only grabbed 32 bytes, which is good enough for security if the source is genuinely strong, but less than the expected 64 bytes (size of the entropy accumulator). - -### Bug fixes - -* Fix a buffer overflow in the PSA HMAC code when using a long key with an unsupported algorithm. Fixes #254. -* Fix `mbedtls_asn1_get_int` to support any number of leading zeros. Credit to OSS-Fuzz for finding a bug in an intermediate version of the fix. -* Fix `mbedtls_asn1_get_bitstring_null` to correctly parse bitstrings of at most 2 bytes. -* `mbedtls_ctr_drbg_set_entropy_len()` and `mbedtls_hmac_drbg_set_entropy_len()` now work if you call them before `mbedtls_ctr_drbg_seed()` or `mbedtls_hmac_drbg_seed()`. -* Fix some false-positive uninitialized variable warnings. Fix contributed by apple-ihack-geek in ARMmbed/mbedtls#2663. - -### Performance improvements - -* Remove a useless call to `mbedtls_ecp_group_free()`. Contributed by Alexander Krizhanovsky in #210. -* Speed up PBKDF2 by caching the digest calculation. Contributed by Jack Lloyd and Fortanix Inc in #277. -* Small performance improvement of `mbedtls_mpi_div_mpi()`. Contributed by Alexander Krizhanovsky in #308. - -### Other changes - -* Remove the technical possibility to define custom `mbedtls_md_info` structures, which was exposed only in an internal header. -* `psa_close_key(0)` and `psa_destroy_key(0)` now succeed (doing nothing, as before). -* Variables containing error codes are now initialized to an error code rather than success, so that coding mistakes or memory corruption tends to cause functions to return this error code rather than a success. There are no known instances where this changes the behavior of the library: this is merely a robustness improvement. #323 - -## Mbed Crypto 2.0.0