Merge tag 'v1.8.5' into parse-bytecode

New port to Zephyr, upip runs on baremetal, and reduction in code size

This release adds a new port of MicroPython to run on top of the Zephyr
real-time operating system.  As part of this there is now basic support for
using mbedTLS as the ussl module.  The release also brings initial support
for running the package manager upip on baremetal systems with low heap
memory (such as esp8266), through a Python module interface.

Work has been done in this release to clean up the core, removing redundant
and/or unreachable code, and factoring existing duplicated code patterns.
This brings a reduction of 828 bytes in code size to the bare-arm port, and
1368 bytes to the minimal port.  There is also improved coverage through
the addition of new tests for corner cases.

The "micropython" module sees the addition of the "const" identity function
which should be used as "from micropython import const" in scripts that
want to use the MicroPython constant optimisations in the compile stage.
There is also the addition of the "opt_level" function to change the
parser/compiler optimisation level at runtime.

The behaviour of "sys.exit" (and "raise SystemExit") on baremetal is now
changed: this function no longer does a soft-reset of the board, rather it
just stops the running script and drops to the REPL.  In order to do an
actual soft reset the "machine.soft_reset" function has been added (to the
stmhal port only, for the time being).

Following CPython, KeyError exceptions for dictionary lookups now have the
failed key stored as the argument of the exception instance, accessible as
exc.args[0].  The "ujson.load" function has also been added, to load JSON
data from an arbitrary stream.

The I2C support in the machine module now has clock stretching, the
addition of the "addrsize" parameter in memory transfer methods, and I2C
scanning now uses dummy writes instead of dummy reads to make the scanning
more reliable.

The CMSIS library has been upgrade to v4.30, and the boards section of
the stmhal port has been refactored to use a common.ld script.  The stmhal
port now has a full implementation of the machine.SPI class, with support
for both hardware SPI peripherals and fast software SPI.  The USB HID
driver in stmhal has added support to receive USB HID messages from the
host.

py core:
- asmthumb: flush D-cache, and invalidate I-cache on STM32F7
- makeqstrdefs.py: use python 2.6 syntax for set creation
- objnone: remove unnecessary handling of MP_UNARY_OP_BOOL
- move frozen modules rules from esp8266 port for reuse across ports
- combine 3 comprehension emit functions (list/dict/set) into 1
- combine 3 comprehension opcodes (list/dict/set) into 1
- vstr: remove vstr.had_error flag and inline basic vstr functions
- objnone: use mp_generic_unary_op instead of custom one
- showbc: make printf's go to the platform print stream
- remove 'name' member from mp_obj_module_t struct
- builtinimport: fix nanbox build after change to better handle -m modules
- stream: remove unnecessary check for NULL return from vstr_extend
- py.mk: suppress some compiler warnings when building berkeley-db
- shrink mp_arg_t struct by using reduced-size integer members
- update opcode format table because 3 opcodes were removed, 1 added
- parse: only replace constants that are standalone identifiers
- py.mk: add support for building modussl_mbedtls
- only store the exception instance on Py stack in bytecode try block
- vm: use MP_OBJ_FROM_PTR to cast a type to an object
- modmicropython: add micropython.const, alias for identity function
- objstr: remove unreachable function used only for terse error msgs
- emitbc: remove/refactor unreachable code, to improve coverage
- objfun: remove unnecessary check for viper fun with 5 or more args
- objfun: use if instead of switch to check return value of VM execute
- objset: use mp_check_self() to check args of set/frozenset methods
- objset: ensure that use of frozenset.update raises an exception
- compile: fix async-for/async-with to work with simpler exc on stack
- scope: use lookup-table to determine a scope's simple name
- scope: shrink scope_t struct by 1 machine word
- scope: factor common code to find locals and close over them
- compile: fix typo when checking for parse-node kind
- argcheck: simplify if-chain so that the last one is the default
- objbool: defer bool's unary op implementation to small int
- objbool: make a slight simplification of bool constructor
- modstruct: remove unreachable code, and add comment about CPy diff
- add mp_raise_OSError(errno) helper function
- objstringio: add readinto() method
- modmicropython: add micropython.opt_level([value]) function
- compile: remove unreachable code
- mpz: in divmod, replace check for rhs!=0 with assert
- mpz: use assert to verify mpz does not have a fixed digit buffer
- factor duplicated function to calculate size of formatted int
- objint: use size_t for arguments that measure bytes/sizes
- compile: remove debugging code for compiler dispatch
- lexer: remove unnecessary code, and unreachable code
- vstr: combine vstr_new_size with vstr_new since they are rarely used
- objdict: fix optimisation for allocating result in fromkeys
- objdict: actually provide the key that failed in KeyError exception
- use mp_raise_msg helper function where appropriate
- add the module specified by -m to sys.modules as '__main__'

extmod:
- modussl_mbedtls: initial implementation of mbedTLS ussl module
- uctypes: allow full 32-bit address range
- modubinascii: fix crc32() function on 32-bit platforms
- modussl_mbedtls: implement key= and cert= args to wrap_socket()
- modussl_mbedtls: use 2-component include paths
- machine_i2c: add clock stretching support
- modussl_mbedtls: add server_hostname param for wrap_socket()
- uzlib: add tinfgzip.c (gzip header parsing) from upstream
- moduzlib: DecompIO: Add support for gzip-formatted streams
- uzlib/: update uzlib to v2.0.3
- vfs_fat: add fat_vfs_statvfs(), reused from stmhal
- machine_i2c: add support for the addrsize parameter in mem xfers
- machine_spi: simplify SPI xfer function to only take one buf len
- machine_spi: factor out software SPI code from esp8266 to extmod
- machine_spi: use delay_half, not baudrate, for internal timing
- machine_spi: add optional support for fast software SPI
- vfs_fat: use mp_raise_OSError helper function
- modlwip: use mp_raise_OSError helper function
- use mp_raise_OSError helper function
- vfs_fat_file: use MP_Exxx errno constants
- uzlib: update to upstream v2.1
- machine_i2c: use writes not reads in i2c.scan()
- vfs_fat: add file and directory checks for remove and rmdir
- modujson: implement ujson.load() to load JSON from a stream
- modujson: fix nanbox build
- utime_mphal: factor out implementations in terms of mp_hal_* for reuse
- utime_mphal: sleep_us/ms(): Don't wait on negative argument
- modussl_mbedtls: add dummy setblocking() method

lib:
- interrupt_char: factor out typical Ctrl+C handling from esp8266 port
- cmsis: move CMSIS headers to lib/
- cmsis: remove CMSIS-DSP headers, they are unused
- cmsis: upgrade CMSIS-CORE to V4.30
- utils/pyexec: don't treat SystemExit as "forced exit"
- utils/pyexec: allow behaviour of SystemExit to be configurable

drivers:
- dht: use mp_raise_OSError helper function

tools:
- update upip to 0.8, fixes IPv6 support
- upgrade upip to 1.0, fully self-hosted release (without fallbacks), and
  uses stream gzip decompression (step towards support for limited-heap
  baremetal systems)
- upgrade upip to 1.1.3, initial support for running on a baremetal,
  low-heap systems (like esp8266), using Python module interface

tests:
- pyb: add test for ExtInt when doing swint while disabled
- pyb: update exp file for previously updated extint test
- extmod/urandom: add urandom tests for error cases
- basics: add errno1 test, to check basics of uerrno module
- extmod: add test for machine.time_pulse_us()
- struct1: test "l" specifier to improve coverage
- array1: add tests for "l", "L" array types to improve coverage
- get cmdline verbose tests running again
- run-tests: add --via-mpy option to run test from precompiled code
- uzlib_decompio_gz: test for DecompIO with gzip bitstream
- basics: add test case for overflowing Py stack in try-finally
- micropython: add tests for const names being replaced in parser
- cmdline/cmd_showbc: fix test now that 1 value is stored on stack
- extmod/vfs_fat_ramdisk: add test for VFS.statvfs()
- float: add test for parsing a float from an empty string
- basics: add test for set.difference_update with arg being itself
- basics: add further tests for nonlocal scoping and closures
- import: add test for compiling "import a.b as c"
- basics: add test constructing a set from a non-trivial expression
- basics: add test for printing OSError when errno is unknown
- run-tests: disable cmdline/cmd_showbc test on Windows
- extmod/btree1: checks for put, seq, string print and unsupported binary op
- fix expected output of verbose cmdline test
- extmod/uzlib: test adaptive huffman tree for tinflate coverage
- improve coverage of struct with test for non-compliant behaviour
- io/write_ext: add description comment
- io/bytesio_ext: add test for readinto()
- micropython: add test for micropython.opt_level() function
- improve test coverage of py/compile.c
- extmod/vfs_fat: test coverage for remove() and rmdir()
- extmod: add test for ujson.load()
- extmod/vfs_fat: replace asserts with prints and expected outputs
- micropython: add tests for heap_lock, and emergency exceptions
- cmdline: improve coverage test for printing bytecode
- improve coverage of array, range, dict, slice, exc, unicode
- add test to print full KeyError exc from failed dict lookup
- run-tests: enable extmod/machine1.py on pyboard

unix port:
- fix build for when MICROPY_PY_SOCKET=0
- modjni: implement subscription for object arrays
- modjni: add array() top-level function to create Java array
- modjni: array(): Support creation of object arrays
- enable btree module for coverage build
- use mp_raise_OSError helper function
- use common RAISE_ERRNO macro from mphalport.h

windows port:
- enable MICROPY_PY_UERRNO

qemu-arm port:
- enable lots of extmods and enable tests for them

stmhal port:
- lcd: de-assert chip select after completing SPI transmission
- {accel,lcd}: use GPIO_{set,clear}_pin
- extint: force 0 to 1 transition on swint()
- boards: add pllvalues.py script to compute PLL values for sysclk
- boards: for OLIMEX_E407, enable UART1 and fix I2C1 mapping
- use attribute to avoid inlining
- put common definitions from linker files to common.ld
- remove STM32CubeF2 HAL files, they are unused/unsupported
- modmachine: fix clearing of reset-cause flags
- add virtual com port support for STM32L476DISC
- remove CMSIS STM32F2xx files, they are unused/unsupported
- spi: simplify spi_transfer function to take only one buf len arg
- mphalport: implement mp_hal_pin_{input,output,write}
- spi: make machine.SPI class conform to correct API
- mphalport: fix mp_hal_pin_write to use correct pin_mask
- spi: use software SPI if no periph id given, even if pins given
- spi: enable use of fast software SPI
- fix linker map for STM32L476 chips
- usbdev: add OUT endpoint to HID interface
- usb: add support to receive USB HID messages from host
- usb: use correct ClassData structure for HID receive
- usb: use real packet size (not maximum) in HID receive
- fix ESPRUINO_PICO by adding ld scripts with correct flash size
- mphalport: change pin obj type to const pointer, to avoid casts
- moduos: implement total-number-of-blocks field in statvfs
- disable network and usocket for ESPRUINO_PICO
- enable machine.time_pulse_us() function
- use mp_raise_OSError helper function
- pybstdio: use size_t instead of mp_uint_t
- modutime: refactor to use extmod/utime_mphal.c
- implement machine.soft_reset()
- enable str.center(), str.[r]partition() and builtin compile()

cc3200 port:
- add ssl_version argument to ssl.wrap_socket()

esp8266 port:
- Makefile: rename SCRIPTDIR to FROZEN_DIR for consistency with FROZEN_MPY_DIR
- ets_alt_task: ets_post: Should return 0 on success, !0 - failure
- esp_mphal: add tentative change to mp_hal_stdin_rx_chr() to wait IRQ
- extend system microsecond counter to 64-bits; use in ticks_ms
- add uos.statvfs() to get filesystem status
- moduos: move stat/statvfs funcs to sit within #if VFS guard
- modmachine: idle(): Return number of CPU cycles spent idling
- main: put /lib before / in sys.path
- modpybrtc.c: implement machine.RTC.alarm_left()
- make PY_UHASHLIB_SHA1 config depend on PY_USSL and SSL_AXTLS
- add FLASH_MODE,FLASH_SIZE options for make deploy target
- use mp_raise_OSError helper function
- make neopixel support configurable
- mpconfigport: enable MICROPY_PY_BUILTINS_SLICE_ATTRS
- enable sys.{stdin,stdout,stderr}.buffer for raw serial access
- enable importing of precompiled .mpy files
- enable micropython.alloc_emergency_exception_buf()

zephyr port:
- initial Zephyr RTOS port, MicroPython part
- initial Zephyr RTOS port, Zephyr part
- add zephyr_getchar module to handle console input
- switch to microkernel, required for network to work in background
- automatically derive ARCH
- support extra make targets
- Makefile: automatically derive target-specific CFLAGS
- use recently added "make outputexports" Zephyr target
- add README
- enable stack checking and micropython.mem_info()
- enable frozen modules support
- main: execute main.py frozen module on boot, if available
- zephyr_getchar: add support for Ctrl+C handling
- add Ctrl+C handling
- implement the help() function
- add copyright blurbs

README:
- remove issue-stats badges, the service is no longer available
- mention _thread module availability in select ports

docs:
- library/pyb.SPI: init(): describe "bits" argument
- library/machine: update description of disable/enable IRQ funcs
- uos: add uos.statvfs() documentation
- wipy: correct deep sleep current figure
- wipy: small doc fixes
- reference: add constrained.rst doc

travis:
- abandon mingw32 in favour of mingw-w64
- run feature and coverage test for precompiled mpy files

examples:
- network/http_client*: use \r\n line-endings in request

1 files changed, 75 insertions, 6 deletions

diff --git a/stmhal/usbdev/class/src/usbd_cdc_msc_hid.c b/stmhal/usbdev/class/src/usbd_cdc_msc_hid.c
index d50934ada..d13657023 100644
--- a/stmhal/usbdev/class/src/usbd_cdc_msc_hid.c
+++ b/stmhal/usbdev/class/src/usbd_cdc_msc_hid.c
@@ -27,10 +27,10 @@
 #include "usbd_ioreq.h"
 #include "usbd_cdc_msc_hid.h"
 
-#define MAX_TEMPLATE_CONFIG_DESC_SIZE (100) // should be maximum of all template config desc's
+#define MAX_TEMPLATE_CONFIG_DESC_SIZE (107) // should be maximum of all template config desc's
 #define CDC_TEMPLATE_CONFIG_DESC_SIZE (67)
 #define CDC_MSC_TEMPLATE_CONFIG_DESC_SIZE (98)
-#define CDC_HID_TEMPLATE_CONFIG_DESC_SIZE (100)
+#define CDC_HID_TEMPLATE_CONFIG_DESC_SIZE (107)
 #define CDC_HID_TEMPLATE_HID_DESC_OFFSET (9)
 #define HID_DESC_OFFSET_SUBCLASS (6)
 #define HID_DESC_OFFSET_PROTOCOL (7)
@@ -39,6 +39,9 @@
 #define HID_DESC_OFFSET_MAX_PACKET_LO (22)
 #define HID_DESC_OFFSET_MAX_PACKET_HI (23)
 #define HID_DESC_OFFSET_POLLING_INTERVAL (24)
+#define HID_DESC_OFFSET_MAX_PACKET_OUT_LO (29)
+#define HID_DESC_OFFSET_MAX_PACKET_OUT_HI (30)
+#define HID_DESC_OFFSET_POLLING_INTERVAL_OUT (31)
 #define HID_SUBDESC_LEN (9)
 
 #define CDC_IFACE_NUM_ALONE (0)
@@ -48,7 +51,9 @@
 #define HID_IFACE_NUM_WITH_CDC (0)
 #define HID_IFACE_NUM_WITH_MSC (1)
 #define HID_IN_EP_WITH_CDC (0x81)
+#define HID_OUT_EP_WITH_CDC (0x01)
 #define HID_IN_EP_WITH_MSC (0x83)
+#define HID_OUT_EP_WITH_MSC (0x03)
 
 #define USB_DESC_TYPE_ASSOCIATION (0x0b)
 
@@ -77,11 +82,14 @@ typedef struct {
     uint32_t             IdleState;
     uint32_t             AltSetting;
     HID_StateTypeDef     state;
+    uint8_t              *RxBuffer;
+    uint32_t             RxLength;
 } USBD_HID_HandleTypeDef;
 
 static uint8_t usbd_mode;
 static uint8_t cdc_iface_num;
 static uint8_t hid_in_ep;
+static uint8_t hid_out_ep;
 static uint8_t hid_iface_num;
 static uint8_t usbd_config_desc_size;
 static uint8_t *hid_desc;
@@ -89,6 +97,7 @@ static const uint8_t *hid_report_desc;
 
 static USBD_CDC_ItfTypeDef *CDC_fops;
 static USBD_StorageTypeDef *MSC_fops;
+static USBD_HID_ItfTypeDef *HID_fops;
 
 static USBD_CDC_HandleTypeDef CDC_ClassData;
 static USBD_MSC_BOT_HandleTypeDef MSC_BOT_ClassData;
@@ -274,7 +283,7 @@ static const uint8_t cdc_hid_template_config_desc[CDC_HID_TEMPLATE_CONFIG_DESC_S
     USB_DESC_TYPE_INTERFACE, // bDescriptorType: interface descriptor
     HID_IFACE_NUM_WITH_CDC, // bInterfaceNumber: Number of Interface
     0x00,   // bAlternateSetting: Alternate setting
-    0x01,   // bNumEndpoints
+    0x02,   // bNumEndpoints
     0x03,   // bInterfaceClass: HID Class
     0x01,   // bInterfaceSubClass: 0=no sub class, 1=boot
     0x02,   // nInterfaceProtocol: 0=none, 1=keyboard, 2=mouse
@@ -300,6 +309,15 @@ static const uint8_t cdc_hid_template_config_desc[CDC_HID_TEMPLATE_CONFIG_DESC_S
     HIBYTE(USBD_HID_MOUSE_MAX_PACKET),
     0x08,                           // bInterval: Polling interval
 
+    // Endpoint OUT descriptor
+    0x07,                           // bLength: Endpoint descriptor length
+    USB_DESC_TYPE_ENDPOINT,         // bDescriptorType: Endpoint descriptor type
+    HID_OUT_EP_WITH_CDC,             // bEndpointAddress: OUT
+    0x03,                           // bmAttributes: Interrupt endpoint type
+    LOBYTE(USBD_HID_MOUSE_MAX_PACKET), // wMaxPacketSize
+    HIBYTE(USBD_HID_MOUSE_MAX_PACKET),
+    0x08,                           // bInterval: Polling interval
+
     //==========================================================================
     // Interface Association for CDC VCP
     0x08,   // bLength: 8 bytes
@@ -581,6 +599,7 @@ int USBD_SelectMode(uint32_t mode, USBD_HID_ModeInfoTypeDef *hid_info) {
             memcpy(usbd_config_desc, cdc_hid_template_config_desc, sizeof(cdc_hid_template_config_desc));
             cdc_iface_num = CDC_IFACE_NUM_WITH_HID;
             hid_in_ep = HID_IN_EP_WITH_CDC;
+            hid_out_ep = HID_OUT_EP_WITH_CDC;
             hid_iface_num = HID_IFACE_NUM_WITH_CDC;
             hid_desc = usbd_config_desc + CDC_HID_TEMPLATE_HID_DESC_OFFSET;
             break;
@@ -595,6 +614,7 @@ int USBD_SelectMode(uint32_t mode, USBD_HID_ModeInfoTypeDef *hid_info) {
             // not implemented
         case USBD_MODE_MSC_HID:
             hid_in_ep = HID_IN_EP_WITH_MSC;
+            hid_out_ep = HID_OUT_EP_WITH_MSC;
             hid_iface_num = HID_IFACE_NUM_WITH_MSC;
             break;
             */
@@ -612,6 +632,9 @@ int USBD_SelectMode(uint32_t mode, USBD_HID_ModeInfoTypeDef *hid_info) {
         hid_desc[HID_DESC_OFFSET_MAX_PACKET_LO] = hid_info->max_packet_len;
         hid_desc[HID_DESC_OFFSET_MAX_PACKET_HI] = 0;
         hid_desc[HID_DESC_OFFSET_POLLING_INTERVAL] = hid_info->polling_interval;
+        hid_desc[HID_DESC_OFFSET_MAX_PACKET_OUT_LO] = hid_info->max_packet_len;
+        hid_desc[HID_DESC_OFFSET_MAX_PACKET_OUT_HI] = 0;
+        hid_desc[HID_DESC_OFFSET_POLLING_INTERVAL_OUT] = hid_info->polling_interval;
         hid_report_desc = hid_info->report_desc;
     }
 
@@ -681,16 +704,28 @@ static uint8_t USBD_CDC_MSC_HID_Init(USBD_HandleTypeDef *pdev, uint8_t cfgidx) {
     if (usbd_mode & USBD_MODE_HID) {
         // HID component
 
-        // get max packet length from descriptor
-        uint16_t mps =
+        // get max packet lengths from descriptor
+        uint16_t mps_in =
             hid_desc[HID_DESC_OFFSET_MAX_PACKET_LO]
             | (hid_desc[HID_DESC_OFFSET_MAX_PACKET_HI] << 8);
+        uint16_t mps_out =
+            hid_desc[HID_DESC_OFFSET_MAX_PACKET_OUT_LO]
+            | (hid_desc[HID_DESC_OFFSET_MAX_PACKET_OUT_HI] << 8);
 
         // Open EP IN
         USBD_LL_OpenEP(pdev,
                        hid_in_ep,
                        USBD_EP_TYPE_INTR,
-                       mps);
+                       mps_in);
+
+        // Open EP OUT
+        USBD_LL_OpenEP(pdev,
+                       hid_out_ep,
+                       USBD_EP_TYPE_INTR,
+                       mps_out);
+
+        // Prepare Out endpoint to receive next packet
+        USBD_LL_PrepareReceive(pdev, hid_out_ep, HID_ClassData.RxBuffer, mps_out);
 
         HID_ClassData.state = HID_IDLE;
     }
@@ -730,6 +765,7 @@ static uint8_t USBD_CDC_MSC_HID_DeInit(USBD_HandleTypeDef *pdev, uint8_t cfgidx)
 
         // close endpoints
         USBD_LL_CloseEP(pdev, hid_in_ep);
+        USBD_LL_CloseEP(pdev, hid_out_ep);
     }
 
     return 0;
@@ -931,6 +967,9 @@ static uint8_t USBD_CDC_MSC_HID_DataOut(USBD_HandleTypeDef *pdev, uint8_t epnum)
     } else if ((usbd_mode & USBD_MODE_MSC) && epnum == (MSC_OUT_EP & 0x7f)) {
         MSC_BOT_DataOut(pdev, epnum);
         return USBD_OK;
+    } else if ((usbd_mode & USBD_MODE_HID) && epnum == (hid_out_ep & 0x7f)) {
+        HID_ClassData.RxLength = USBD_LL_GetRxDataSize(pdev, epnum);
+        HID_fops->Receive(HID_ClassData.RxBuffer, HID_ClassData.RxLength);
     }
 
     return USBD_OK;
@@ -1008,6 +1047,36 @@ uint8_t USBD_MSC_RegisterStorage(USBD_HandleTypeDef *pdev, USBD_StorageTypeDef *
     }
 }
 
+uint8_t USBD_HID_RegisterInterface(USBD_HandleTypeDef *pdev, USBD_HID_ItfTypeDef *fops) {
+    if (fops == NULL) {
+        return USBD_FAIL;
+    } else {
+        HID_fops = fops;
+        return USBD_OK;
+    }
+}
+
+uint8_t USBD_HID_SetRxBuffer(USBD_HandleTypeDef *pdev, uint8_t *pbuff) {
+    HID_ClassData.RxBuffer = pbuff;
+    return USBD_OK;
+}
+
+// prepare OUT endpoint for reception
+uint8_t USBD_HID_ReceivePacket(USBD_HandleTypeDef *pdev) {
+    // Suspend or Resume USB Out process
+    if (pdev->dev_speed == USBD_SPEED_HIGH) {
+        return USBD_FAIL;
+    }
+
+    // Prepare Out endpoint to receive next packet
+    uint16_t mps_out =
+        hid_desc[HID_DESC_OFFSET_MAX_PACKET_OUT_LO]
+        | (hid_desc[HID_DESC_OFFSET_MAX_PACKET_OUT_HI] << 8);
+    USBD_LL_PrepareReceive(pdev, hid_out_ep, HID_ClassData.RxBuffer, mps_out);
+
+    return USBD_OK;
+}
+
 int USBD_HID_CanSendReport(USBD_HandleTypeDef *pdev) {
     return pdev->dev_state == USBD_STATE_CONFIGURED && HID_ClassData.state == HID_IDLE;
 }