summaryrefslogtreecommitdiff
path: root/tools
AgeCommit message (Collapse)Author
2025-09-19mpy-cross: Add RISC-V RV64IMC support in MPY files.Alessandro Gatti
MPY files can now hold data to be run on RV64IMC. This can be accomplished by passing the `-march=rv64imc` flag to mpy-cross. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-09-18tools/ci.sh: Make this script runnable as command.Jeff Epler
This makes it easier to run a sequence of ci steps locally. A help message is also provided. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-09-15tools: Remove unneeded future imports.Jeff Epler
Signed-off-by: Jeff Epler <jepler@unpythonic.net>
2025-09-15all: Remove Python 2.7 support.Jeff Epler
Python 2.7 has been EOL since January 2020. Ubuntu oldoldlts (Focal Fossa, 20.04) has Python 3.8. Debian oldoldstable (Buster, from 2019) has Python 3.7. RHEL 8 (from 2019) has Python 3.6. It's easier than ever to install a modern Python using uv. Given this, it seems like a fine idea to drop Python 2.7 support. Even though the build is not tested on Python as old as 3.3, I left comments stating that "3.3+" is the baseline Python version. However, it might make sense to bump this to e.g., 3.10, the oldest Python 3 version used during CI. Or, using uv or another method actually test on the oldest Python interpreter that is desirable to support (uv goes back to Python 3.7 easily; in October 2025, the oldest supported Python interpreter version will be 3.10) Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-09-10tools/mpremote: Don't apply Espressif DTR/RTS quirk to TinyUSB CDC dev.Angus Gratton
The DTR quirk workaround from dea949e86 is needed for the Espressif Serial/JTAG device, but not for TinyUSB - in fact DTR must be set for TinyUSB to correctly determine if the serial port is open (and leads to issues with lost bytes otherwise). See discussion in PR #17999. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-09-09esp32: Add IDF Component Lockfiles to git repo.Angus Gratton
This is recommended by Espressif, and it's the only way to ensure everyone builds the same set of component versions. The awkward part is that updating the ESP-IDF version will churn a line in each of these files (and possibly other changes). Adds a build-time check for lock file changes, which is either a warning or a hard error depending on the value of MICROPY_MAINTAINER_BUILD flag (introduced in previous commit). This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-09-09tools: Add an environment variable MICROPY_MAINTAINER_BUILD.Angus Gratton
This allows us to have some things which are fatal errors in CI or nightly builds, but warnings in normal developer builds. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-08-28tools/ci.sh: Skip unreliable thread/thread_gc1.py test in Github CI.Anson Mansfield
The `thread/thread_gc1.py` test is a constant source of spurious failures in Github CI. This commit adds it to the list of tests skipped when running on Github CI using either macos, qemu_riscv64, qemu_mips, or qemu_arm, to help reduce the overall false positive rate and improve the predictive value of the test fail indication. Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
2025-08-26tools/mpy-tool.py: Allow dumping MPY segments into their own files.Alessandro Gatti
This commit lets "tools/mpy-tool.py" extract MPY segments into their own files, one file per segment. A pair of new command line arguments were added, namely "-e"/"--extract" that takes a filename prefix to use as a base for the generated files' name, and "--extract-only" that - combined with "--extract" - allows selecting which kinds of segment should be dumped to the filesystem. So, for example, assuming there's a file called "module.mpy", running "./mpy-tool.py --extract segments module.mpy" would yield a series of files with names like "segments_0_module.py_QSTR_module.py.bin", "segments_1_module.py_META__module_.bin", "segments_2_module.py_QSTR_function.bin", etc. In short the file name format is "<base>_<count>_<sourcefile>_<segmentkind>_<segmentname>.bin", with <segmentkind> being META, QSTR, OBJ, or CODE. Source file names and segment names will only contain characters in the range "a-zA-Z0-9_-." to avoid having output file names with unexpected characters. The "--extract-only" option can accept one or more kinds, separated by commas and treated as case insensitive strings. The supported kinds match what is currently handled by the "MPYSegment" class in "tools/mpy-tool.py": "META", "QSTR", "OBJ", and "CODE". The absence of this command line option implies dumping every segment found. If "--extract" is passed along with "--merge", dumping is performed after the merge process takes place, in order to dump all possible segments that match the requested segment kinds. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-08-26tools/ci.sh: Extend Arm testing to include hardfp targets.Alessandro Gatti
This commit lets CI extend the testing scope of the QEMU Arm target, by letting it perform the usual battery of tests (interpreter and natmods) also on hardfp targets. The default board for Arm testing lacks hardware floating point support, so natmods weren't tested in that specific configuration. With the introduction of the "MPS_AN500" QEMU target, now this is made possible as said board emulates a Cortex-M7 machine with a single- and double-precision floating point unit. To reduce the impact on build times, the "ci_qemu_build_arm_thumb" CI step was split in two: "ci_qemu_build_arm_thumb_softfp" and "ci_qemu_build_arm_thumb_hardfp" - so hopefully those can run in parallel whenever possible. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-08-19py/misc: Add a way to detect sanitizer builds.Angus Gratton
Clang and gcc>=14 can use __has_feature() to detect if a sanitizer is enabled, but older GCC has no mechanism - need to set a macro explicitly for this to be recognised. Necessary for increasing some resource limits in sanitizer builds. Important not to use to avoid real issues! This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-08-19tools/ci: Add UBSan to longlong CI build.Angus Gratton
Also rewrite the sanitizer argument variables to not assume a variant. longlong variant currently fails in this config, due to a bug fixed in follow-up commit. Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-08-16zephyr: Upgrade to Zephyr v4.2.0.Maureen Helm
Updates the Zephyr port build instructions and CI to use the latest Zephyr release tag. Tested on max32690fthr and frdm_k64f. Signed-off-by: Maureen Helm <maureen.helm@analog.com>
2025-08-11tools/codeformat.py: Print filename + linenumber when dedenting fails.Daniël van de Giessen
Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
2025-08-11tools/codeformat.py: Iterate lines instead of modifying list in-place.Daniël van de Giessen
Co-authored-by: David Lechner <david@pybricks.com> Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
2025-08-11tools/mpremote: Fix encoding error in PyboardCommand.Jos Verlinde
This is a fix for utf-8 decoding errors that are thrown when non-utf-8 content is received. For instance during a reboot of an ESP8266 module. The fix is to handle conversion errors by replacing illegal characters. Note that these illegal characters most often occur during an MCU reboot sequence when the MCU is using baudrates different from 115200. Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
2025-08-02github/workflows: Add a CI job to build ESP32-C2 and ESP32-C6 boards.Damien George
So that all six supported SoCs are built by CI. Signed-off-by: Damien George <damien@micropython.org>
2025-08-02tools/mpremote: Locate config.py location across different host OSes.Jos Verlinde
Use `platformdirs.user_config_dir()` (see https://platformdirs.readthedocs.io/en/latest/api.html#user-config-directory) to provide portability across many different OSes and configuration styles. Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
2025-08-02tools/mpremote: Add platformdirs dependency to requirements.txt.Jos Verlinde
Needed to easily find the user configuration file. Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
2025-08-01tools/mpremote: Update ESPxxx detection for USB-CDC ports.Jos Verlinde
Detection of ESP-XX devices was based on just the names of the USB driver name, and did not account for the switch of the newer ESP-xx devices to USB-CDC. On Windows this caused unwanted/unneeded resets as the DTR/RTS signals are also used for automatic device reset over USB-CDC. See https://github.com/micropython/micropython/issues/9659#issuecomment-3124704572 This commit uses the Espressif registered VID 0x303A to detect USB-CDC ports, to enable the same DTR/RTS settings as used on the UART-USB connection. Also improved the robustness of the code using `getattr()`. Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
2025-08-01tools/mpremote: Fix errno.ENOTBLK attribute error on Windows.Jos Verlinde
Not all errors defined in stdlib errno are available on Windows. Specifically, errno.ENOTBLK is not. Fixes issue #17773. Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
2025-07-24tools/ci.sh: Test building all natmod examples with all ARM-M archs.Damien George
And run both armv6m and armv7m under qemu. Signed-off-by: Damien George <damien@micropython.org>
2025-07-24all: Go back to using default ruff quote style.Damien George
Commit dc2fcfcc5511a371ff684f7d7772e7a7b479246d seems to have accidentally changed the ruff quote style to "preserve", instead of keeping it at the default which is "double". Put it back to the default and update relevant .py files with this rule. Signed-off-by: Damien George <damien@micropython.org>
2025-07-23esp32: Update to use ESP-IDF v5.4.2.Damien George
This is a patch release of the IDF. Comparing with 5.4.1, firmware size is up by about 1.5k on ESP32 and 9k on ESP32-S3. But IRAM usage (of the IDF) is down by about 500 byte on ESP32 and DRAM usage is down by about 20k on ESP32 and 10k on ESP32-S3. Testing on ESP32, ESP32-S2, ESP32-S3 and ESP32-C3 shows no regressions, except in BLE MTU ordering (the MTU exchange event occuring before the connect event). Signed-off-by: Damien George <damien@micropython.org>
2025-07-23tools/pyboard.py: Add timeout argument to Pyboard.exec_/exec.Damien George
Signed-off-by: Damien George <damien@micropython.org>
2025-07-23tools/ci.sh: Change averaging to 1 for run-perfbench.py test.Damien George
The `run-perfbench.py` test is run as part of CI, but the actual performance results are not used. Rather, the test is just testing that all the performance tests run correctly. So there's no need to run with the default averaging of 8 (which runs each test 8 times and takes the average time for the performance result) which can take a lot of time for slower builds, eg unix sanitize, settrace and stackless builds. This commit changes the averaging to just 1. Signed-off-by: Damien George <damien@micropython.org>
2025-07-23github/workflows: Add new CI job to test unix port with GIL enabled.Damien George
Signed-off-by: Damien George <damien@micropython.org>
2025-07-23tools/ci.sh: Skip thread/stress_recurse.py on unix qemu test runs.Damien George
This test passes sometimes and fails other times. Eventually that should be fixed, but for now just skip this test. Signed-off-by: Damien George <damien@micropython.org>
2025-07-23tools/ci.sh: Skip thread/stress_heap.py test on macOS test run.Damien George
This test passes sometimes and fails other times. Eventually that should be fixed, but for now just skip this test. Signed-off-by: Damien George <damien@micropython.org>
2025-07-23tools/ci.sh: Increase timeout for stackless clang test runs.Damien George
Stackless mode makes `tests/thread/stress_aes.py` slow, around 75 seconds for this CI job (probably due to contention among the many threads for the GC lock, to allocate frames for function calls). So increase the timeout to allow this test to pass. Signed-off-by: Damien George <damien@micropython.org>
2025-07-23tools/ci.sh: Increase timeout for unix qemu test runs.Damien George
The qemu emulation introduces enough overhead that the `tests/thread/stress_aes.py` test overruns the default timeout. So increase it to allow this test to pass. Signed-off-by: Damien George <damien@micropython.org>
2025-07-20examples/natmod/btree: Fix build on RV32 with Picolibc.Alessandro Gatti
This commit fixes building the "btree" example natmod on RV32 when Picolibc is being used and uses thread-local storage for storing the errno variable. The fix is surprisingly simple: Picolibc allows overriding the function that will provide a pointer to the "errno" variable, and the btree natmod integration code already has all of this machinery set up as part of its library integration. Redirecting Picolibc to the already existing pointer provider function via a compile-time definition is enough to let the module compile and pass QEMU tests. This workaround will work on any Picolibc versions (Arm, RV32, Xtensa, etc.) even if TLS support was not enabled to begin with, and will effectively do nothing if the toolchain used will rely on Newlib to provide standard C library functions. Given that the btree module now builds and passes the relevant natmod tests, said module is now part of the QEMU port's natmod testing procedure, and CI now will build the btree module for RV32 as part to its checks. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-07-20tools/pyboard.py: Align execpty prefix.Yanfeng Liu
This aligns the prefix string in L285 to that in L284 though the two strings have equal length. Signed-off-by: Yanfeng Liu <yfliu2008@qq.com>
2025-07-18unix/variants: Add a 'longlong' variant to test 64-bit bigints in CI.Angus Gratton
Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-07-12tools/ci.sh: Always call `apt-get update` before `apt-get install`.Jeff Epler
There have been recent build failures in build_renesas_ra_board. It appears to be the case that a security update for this package was recently issued by Ubuntu for CVE-2025-4565 and the buggy version is no longer on package servers. However, it is still referred to by the cached apt metadata in the GitHub runners. Add `apt-get update` to fix this, and audit for other sites in `ci.sh` where it might also be necessary. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-07-08tools/mpremote: Support OSError's on targets without errno.Damien George
Targets without the `errno` module enabled will not render `OSError`s with the name of the error. Instead they just print the numeric error code. Add support for such targets by explicitly recognising certain error codes. Signed-off-by: Damien George <damien@micropython.org>
2025-07-06github/workflows: Use Python 3.11 for unix coverage testing.Jeff Epler
This removes the need for an explicit `sys_settrace_features.py.exp` file. This means that people testing locally will also need to install Python 3.11 in some way, such as with pyenv or uv, and use it during `make VARIANT=coverage test`, or they will get failures. When using Python from GitHub actions/setup-python, pip3 can't be wrapped by sudo, because this invokes the operating system python instead. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-07-06tools/ci.sh: Increase test timeout to 60s in coverage jobs.Jeff Epler
The additional overhead of the settrace profiler means that the `aes_stress.py` test was running too slowly on GitHub CI. Double the timeout to 60 seconds. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-07-06github/workflows: Remove the unix "settrace" CI job.Jeff Epler
This becomes redundant when the main coverage build includes settrace. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-07-04tools/mpremote: Fix disconnect handling on Windows and Linux.Andrew Leech
Changes in this commit: - Handle SerialException on Windows when device disconnects. - Print clean 'device disconnected' message instead of stack trace. - Fix terminal formatting issues on Linux after disconnect. - Return disconnected state after console cleanup to avoid terminal issues. This ensures proper disconnect messages on both platforms without showing confusing error traces to users. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
2025-07-01tools/ci.sh: Disable "stack use after return" in ASan build.Jeff Epler
This check, runtime-enabled by default in gcc 13 (and existing at least since gcc 12, but runtime-disabled) changes the stack layout in ways that are not compatible with assumptions spread across the core code (nlr, gc, and stack checking). Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-06-26tools/autobuild: Build alif boards as part of auto-build.Damien George
Signed-off-by: Damien George <damien@micropython.org>
2025-06-16tools/ci.sh: Add functions for sanitizer builds.Jeff Epler
Includes both undefined and address sanitizer configurations. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-06-16tools/mpremote: Improve df command to use new no-arg vfs.mount() query.Damien George
The existing `mpremote df` command is not very good, because it needs to assume that all directories in the root directory are mount points, and also doesn't correctly stat filesystems when the current directory is not the root. This leads to wrong results With the introduction of `vfs.mount()` to return a list of mounted filesystems and their path, a much better df can be implemented, as done in this commit. The new df will also fall back to using the old approach of listing the root directory if the no-arg `vfs.mount()` query is not supported. Signed-off-by: Damien George <damien@micropython.org>
2025-06-10py/dynruntime.mk: Enable single-precision float by default on armv6/7m.Damien George
Soft float now works on these ARM targets thanks to the parent commit. Signed-off-by: Damien George <damien@micropython.org>
2025-06-10tools/mpy_ld.py: Support R_ARM_ABS32 relocation in text.Damien George
Add support for R_ARM_ABS32 relocations in native .mpy files. These can be rewritten in the same way that data relocations are. Fixes issue #14430. Signed-off-by: Damien George <damien@micropython.org>
2025-06-10tools/ci.sh: Fix nanbox CI test runs.Alessandro Gatti
This commit fixes CI test runs for the `nanbox` target, which were broken by the unconditional native emitter code output changes in the test runner. The `nanbox` configuration does not enable native emitters of any kind, and with a full test run that includes executing emitted native code things would break when doing CI runs. This is worked around by introducing a common subset of tests that do not involve the native emitter, and a more comprehensive set of tests that include both non-emitter and emitter tests. The `nanbox` CI test run will stop at the first subset, whilst other configurations will run that and execute further tests. Function names have been kept the same for steps that involve native code, with the `nanbox` subset having another one. This should not trigger any breakage in existing CI configurations or external scripts. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-06-10tools/boardgen.py: Ensure board pin locals_dict has consistent order.Andrew Leech
`tools/boardgen.py` is used by the `make-pins.py` scripts in many ports to generate the pin definitions for the machine module. In #17391 it was found that this is currently generating the C structs for board pin definitions with inconsistent ordering (across different build runs), which makes it sometimes impossible to get a consistent binary file even for no change in source files. This commit fixes that by sorting the board pin names alphabetically. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
2025-06-10github/workflows: Split QEMU/Arm builds into separate entries.Alessandro Gatti
This commit takes the QEMU/Arm CI build and test step and splits it into three separate steps (bigendian, sabrelite, thumb), to allow them to run in parallel. Currently the QEMU/Arm CI build step would take up to 16 minutes, often being the last step blocking a full test run. With this commit, when the steps run in parallel the time it takes to complete the QEMU/Arm build and test procedure is cut in half - taking between 8 to 9 minutes depending on the CI runner load. The existing `ci_build_and_test_arm` function has been removed, in favour of having three separate functions - one per configuration. They are called `ci_build_and_test_arm_bigendian`, `ci_build_and_test_arm_sabrelite`, and `ci_build_and_test_arm_thumb`. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-06-05tools/ci.sh: Allow errors in code-size build to fail the CI.Damien George
It was possible for CI to pass even if the bare-arm port fails to build. This commit fixes that. Signed-off-by: Damien George <damien@micropython.org>