summaryrefslogtreecommitdiff
path: root/tools
AgeCommit message (Collapse)Author
6 daysgithub/workflows: Add a CI job to build ESP32-C2 and ESP32-C6 boards.Damien George
So that all six supported SoCs are built by CI. Signed-off-by: Damien George <damien@micropython.org>
6 daystools/mpremote: Locate config.py location across different host OSes.Jos Verlinde
Use `platformdirs.user_config_dir()` (see https://platformdirs.readthedocs.io/en/latest/api.html#user-config-directory) to provide portability across many different OSes and configuration styles. Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
6 daystools/mpremote: Add platformdirs dependency to requirements.txt.Jos Verlinde
Needed to easily find the user configuration file. Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
7 daystools/mpremote: Update ESPxxx detection for USB-CDC ports.Jos Verlinde
Detection of ESP-XX devices was based on just the names of the USB driver name, and did not account for the switch of the newer ESP-xx devices to USB-CDC. On Windows this caused unwanted/unneeded resets as the DTR/RTS signals are also used for automatic device reset over USB-CDC. See https://github.com/micropython/micropython/issues/9659#issuecomment-3124704572 This commit uses the Espressif registered VID 0x303A to detect USB-CDC ports, to enable the same DTR/RTS settings as used on the UART-USB connection. Also improved the robustness of the code using `getattr()`. Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
7 daystools/mpremote: Fix errno.ENOTBLK attribute error on Windows.Jos Verlinde
Not all errors defined in stdlib errno are available on Windows. Specifically, errno.ENOTBLK is not. Fixes issue #17773. Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
2025-07-24tools/ci.sh: Test building all natmod examples with all ARM-M archs.Damien George
And run both armv6m and armv7m under qemu. Signed-off-by: Damien George <damien@micropython.org>
2025-07-24all: Go back to using default ruff quote style.Damien George
Commit dc2fcfcc5511a371ff684f7d7772e7a7b479246d seems to have accidentally changed the ruff quote style to "preserve", instead of keeping it at the default which is "double". Put it back to the default and update relevant .py files with this rule. Signed-off-by: Damien George <damien@micropython.org>
2025-07-23esp32: Update to use ESP-IDF v5.4.2.Damien George
This is a patch release of the IDF. Comparing with 5.4.1, firmware size is up by about 1.5k on ESP32 and 9k on ESP32-S3. But IRAM usage (of the IDF) is down by about 500 byte on ESP32 and DRAM usage is down by about 20k on ESP32 and 10k on ESP32-S3. Testing on ESP32, ESP32-S2, ESP32-S3 and ESP32-C3 shows no regressions, except in BLE MTU ordering (the MTU exchange event occuring before the connect event). Signed-off-by: Damien George <damien@micropython.org>
2025-07-23tools/pyboard.py: Add timeout argument to Pyboard.exec_/exec.Damien George
Signed-off-by: Damien George <damien@micropython.org>
2025-07-23tools/ci.sh: Change averaging to 1 for run-perfbench.py test.Damien George
The `run-perfbench.py` test is run as part of CI, but the actual performance results are not used. Rather, the test is just testing that all the performance tests run correctly. So there's no need to run with the default averaging of 8 (which runs each test 8 times and takes the average time for the performance result) which can take a lot of time for slower builds, eg unix sanitize, settrace and stackless builds. This commit changes the averaging to just 1. Signed-off-by: Damien George <damien@micropython.org>
2025-07-23github/workflows: Add new CI job to test unix port with GIL enabled.Damien George
Signed-off-by: Damien George <damien@micropython.org>
2025-07-23tools/ci.sh: Skip thread/stress_recurse.py on unix qemu test runs.Damien George
This test passes sometimes and fails other times. Eventually that should be fixed, but for now just skip this test. Signed-off-by: Damien George <damien@micropython.org>
2025-07-23tools/ci.sh: Skip thread/stress_heap.py test on macOS test run.Damien George
This test passes sometimes and fails other times. Eventually that should be fixed, but for now just skip this test. Signed-off-by: Damien George <damien@micropython.org>
2025-07-23tools/ci.sh: Increase timeout for stackless clang test runs.Damien George
Stackless mode makes `tests/thread/stress_aes.py` slow, around 75 seconds for this CI job (probably due to contention among the many threads for the GC lock, to allocate frames for function calls). So increase the timeout to allow this test to pass. Signed-off-by: Damien George <damien@micropython.org>
2025-07-23tools/ci.sh: Increase timeout for unix qemu test runs.Damien George
The qemu emulation introduces enough overhead that the `tests/thread/stress_aes.py` test overruns the default timeout. So increase it to allow this test to pass. Signed-off-by: Damien George <damien@micropython.org>
2025-07-20examples/natmod/btree: Fix build on RV32 with Picolibc.Alessandro Gatti
This commit fixes building the "btree" example natmod on RV32 when Picolibc is being used and uses thread-local storage for storing the errno variable. The fix is surprisingly simple: Picolibc allows overriding the function that will provide a pointer to the "errno" variable, and the btree natmod integration code already has all of this machinery set up as part of its library integration. Redirecting Picolibc to the already existing pointer provider function via a compile-time definition is enough to let the module compile and pass QEMU tests. This workaround will work on any Picolibc versions (Arm, RV32, Xtensa, etc.) even if TLS support was not enabled to begin with, and will effectively do nothing if the toolchain used will rely on Newlib to provide standard C library functions. Given that the btree module now builds and passes the relevant natmod tests, said module is now part of the QEMU port's natmod testing procedure, and CI now will build the btree module for RV32 as part to its checks. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-07-20tools/pyboard.py: Align execpty prefix.Yanfeng Liu
This aligns the prefix string in L285 to that in L284 though the two strings have equal length. Signed-off-by: Yanfeng Liu <yfliu2008@qq.com>
2025-07-18unix/variants: Add a 'longlong' variant to test 64-bit bigints in CI.Angus Gratton
Signed-off-by: Angus Gratton <angus@redyak.com.au>
2025-07-12tools/ci.sh: Always call `apt-get update` before `apt-get install`.Jeff Epler
There have been recent build failures in build_renesas_ra_board. It appears to be the case that a security update for this package was recently issued by Ubuntu for CVE-2025-4565 and the buggy version is no longer on package servers. However, it is still referred to by the cached apt metadata in the GitHub runners. Add `apt-get update` to fix this, and audit for other sites in `ci.sh` where it might also be necessary. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-07-08tools/mpremote: Support OSError's on targets without errno.Damien George
Targets without the `errno` module enabled will not render `OSError`s with the name of the error. Instead they just print the numeric error code. Add support for such targets by explicitly recognising certain error codes. Signed-off-by: Damien George <damien@micropython.org>
2025-07-06github/workflows: Use Python 3.11 for unix coverage testing.Jeff Epler
This removes the need for an explicit `sys_settrace_features.py.exp` file. This means that people testing locally will also need to install Python 3.11 in some way, such as with pyenv or uv, and use it during `make VARIANT=coverage test`, or they will get failures. When using Python from GitHub actions/setup-python, pip3 can't be wrapped by sudo, because this invokes the operating system python instead. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-07-06tools/ci.sh: Increase test timeout to 60s in coverage jobs.Jeff Epler
The additional overhead of the settrace profiler means that the `aes_stress.py` test was running too slowly on GitHub CI. Double the timeout to 60 seconds. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-07-06github/workflows: Remove the unix "settrace" CI job.Jeff Epler
This becomes redundant when the main coverage build includes settrace. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-07-04tools/mpremote: Fix disconnect handling on Windows and Linux.Andrew Leech
Changes in this commit: - Handle SerialException on Windows when device disconnects. - Print clean 'device disconnected' message instead of stack trace. - Fix terminal formatting issues on Linux after disconnect. - Return disconnected state after console cleanup to avoid terminal issues. This ensures proper disconnect messages on both platforms without showing confusing error traces to users. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
2025-07-01tools/ci.sh: Disable "stack use after return" in ASan build.Jeff Epler
This check, runtime-enabled by default in gcc 13 (and existing at least since gcc 12, but runtime-disabled) changes the stack layout in ways that are not compatible with assumptions spread across the core code (nlr, gc, and stack checking). Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-06-26tools/autobuild: Build alif boards as part of auto-build.Damien George
Signed-off-by: Damien George <damien@micropython.org>
2025-06-16tools/ci.sh: Add functions for sanitizer builds.Jeff Epler
Includes both undefined and address sanitizer configurations. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-06-16tools/mpremote: Improve df command to use new no-arg vfs.mount() query.Damien George
The existing `mpremote df` command is not very good, because it needs to assume that all directories in the root directory are mount points, and also doesn't correctly stat filesystems when the current directory is not the root. This leads to wrong results With the introduction of `vfs.mount()` to return a list of mounted filesystems and their path, a much better df can be implemented, as done in this commit. The new df will also fall back to using the old approach of listing the root directory if the no-arg `vfs.mount()` query is not supported. Signed-off-by: Damien George <damien@micropython.org>
2025-06-10py/dynruntime.mk: Enable single-precision float by default on armv6/7m.Damien George
Soft float now works on these ARM targets thanks to the parent commit. Signed-off-by: Damien George <damien@micropython.org>
2025-06-10tools/mpy_ld.py: Support R_ARM_ABS32 relocation in text.Damien George
Add support for R_ARM_ABS32 relocations in native .mpy files. These can be rewritten in the same way that data relocations are. Fixes issue #14430. Signed-off-by: Damien George <damien@micropython.org>
2025-06-10tools/ci.sh: Fix nanbox CI test runs.Alessandro Gatti
This commit fixes CI test runs for the `nanbox` target, which were broken by the unconditional native emitter code output changes in the test runner. The `nanbox` configuration does not enable native emitters of any kind, and with a full test run that includes executing emitted native code things would break when doing CI runs. This is worked around by introducing a common subset of tests that do not involve the native emitter, and a more comprehensive set of tests that include both non-emitter and emitter tests. The `nanbox` CI test run will stop at the first subset, whilst other configurations will run that and execute further tests. Function names have been kept the same for steps that involve native code, with the `nanbox` subset having another one. This should not trigger any breakage in existing CI configurations or external scripts. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-06-10tools/boardgen.py: Ensure board pin locals_dict has consistent order.Andrew Leech
`tools/boardgen.py` is used by the `make-pins.py` scripts in many ports to generate the pin definitions for the machine module. In #17391 it was found that this is currently generating the C structs for board pin definitions with inconsistent ordering (across different build runs), which makes it sometimes impossible to get a consistent binary file even for no change in source files. This commit fixes that by sorting the board pin names alphabetically. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
2025-06-10github/workflows: Split QEMU/Arm builds into separate entries.Alessandro Gatti
This commit takes the QEMU/Arm CI build and test step and splits it into three separate steps (bigendian, sabrelite, thumb), to allow them to run in parallel. Currently the QEMU/Arm CI build step would take up to 16 minutes, often being the last step blocking a full test run. With this commit, when the steps run in parallel the time it takes to complete the QEMU/Arm build and test procedure is cut in half - taking between 8 to 9 minutes depending on the CI runner load. The existing `ci_build_and_test_arm` function has been removed, in favour of having three separate functions - one per configuration. They are called `ci_build_and_test_arm_bigendian`, `ci_build_and_test_arm_sabrelite`, and `ci_build_and_test_arm_thumb`. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-06-05tools/ci.sh: Allow errors in code-size build to fail the CI.Damien George
It was possible for CI to pass even if the bare-arm port fails to build. This commit fixes that. Signed-off-by: Damien George <damien@micropython.org>
2025-06-04tools/ci.sh: Clean the correct MPY files when batch compiling.Alessandro Gatti
This commit fixes a small yet harmless issue that occurs when invoking `ci_native_mpy_modules_build` on a persistent environment, as only X64 MPY files would be removed by the cleaning process. Now the correct architecture is passed at all times when cleaning before building a natmod for a particular architecture, forcing a full build of all files to better simulate the CI environment (where there's no state persisted between runs for this step). Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-06-04tools/ci.sh: Remove natmod build restrictions for Xtensa.Alessandro Gatti
This commit lets the CI pipeline build all natmods for the Xtensa target, now that ROM symbols can be used in the linking process. The restriction was put in place due to build failures on certain natmods for Xtensa, as ROM symbols would not be used, causing undefined symbol errors at build time. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-06-04tools/mpy_ld.py: Resolve fixed-address symbols if requested.Alessandro Gatti
This commit lets mpy_ld.py resolve symbols not only from the object files involved in the linking process, or from compiler-supplied static libraries, but also from a list of symbols referenced by an absolute address (usually provided by the system's ROM). This is needed for ESP8266 targets as some C stdlib functions are provided by the MCU's own ROM code to reduce the final code footprint, and therefore those functions' implementation was removed from the compiler's support libraries. This means that unless `LINK_RUNTIME` is set (which lets tooling look at more libraries to resolve symbols) the build process will fail as tooling is unaware of the ROM symbols' existence. With this change, fixed-address symbols can be exposed to the symbol resolution step when performing natmod linking. If there are symbols coming in from a fixed-address symbols list and internal code or external libraries, the fixed-address symbol address will take precedence in all cases. Although this is - in theory - also working for the whole range of ESP32 MCUs, testing is currently limited to Xtensa processors and the example natmods' makefiles only make use of this commit's changes for the ESP8266 target. Natmod builds can set the MPY_EXTERN_SYM_FILE variable pointing to a linkerscript file containing a series of symbols (weak or strong) at a fixed address; these symbols will then be used by the MicroPython linker when packaging the natmod. If a different natmod build method is used (eg. custom CMake scripts), `tools/mpy_ld.py` can now accept a command line parameter called `--externs` (or its short variant `-e`) that contains the path of a linkerscript file with the fixed-address symbols to use when performing the linking process. The linkerscript file parser can handle a very limited subset of binutils's linkerscript syntax, namely just block comments, strong symbols, and weak symbols. Each symbol must be in its own line for the parser to succeed, empty lines or comment blocks are skipped. For an example of what this parser was meant to handle, you can look at `ports/esp8266/boards/eagle.rom.addr.v6.ld` and follow its format. The natmod developer documentation is also updated to reflect the new command line argument accepted by `mpy_ld.py` and the use cases for the changes introduced by this commit. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
2025-06-02tools/pyboard.py: Add write_timeout and catch errors in enter_raw_repl.Damien George
If the USB serial device locks up, then writes to that device can hang forever. That can make the test runner `tests/run-tests.py` lock up, among other problems. This commit introduces a 5 second write-timeout, and catches any OSError's raised during `enter_raw_repl()`. Now, if a USB serial device locks up, `enter_raw_repl()` will eventually raise an exception. Signed-off-by: Damien George <damien@micropython.org>
2025-06-02tools/pyboard.py: Introduce timeout_overall for read_until().Damien George
This applies the mpremote commit 03fe9c55ea8a3ef8bede5547476f37d7fca20119 to pyboard.py. The `timeout_overall` is used in `enter_raw_repl()`. It prevents waiting forever for a serial device that does not respond to the Ctrl-C/Ctrl-D/etc commands and is constantly outputting data. Signed-off-by: Damien George <damien@micropython.org>
2025-06-02tools/pyboard.py: Avoid initial blocking read in read_until().Damien George
This applies the mpremote commit 0d46e45a1f72ee61a153c41aaaf6c63818ccffb0 to pyboard.py. If the target does not return any data then `read_until()` will block indefinitely. Fix this by making the initial read part of the general read look, which always checks `inWaiting() > 0` before reading from the serial device. Also added the UART timeout to the constructor. This is not currently used but may be used as an additional safeguard. Signed-off-by: Damien George <damien@micropython.org>
2025-05-18tools/mpremote/tests: Add tests for 'fs tree' command.Jos Verlinde
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
2025-05-18tools/mpremote: Add new 'fs tree' command.Jos Verlinde
Add `mpremote fs tree` command to show a tree of the device's files. It: - Shows a treeview from current path or specified path. - Uses the graph chars ("├── ", "└── ") (not configurable). - Has the options: -v/--verbose adds the serial device name to the top of the tree -s/--size add a size to the files -h/--human add a human readable size to the files Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
2025-05-16esp32: Update to use ESP-IDF v5.4.1.IhorNehrutsa
This version of the IDF uses about 1KB more IRAM and 1KB more DRAM on most boards, but 6.5KB more DRAM usage on the S3. It seems that's due to a lot of small increases in many components. Signed-off-by: Ihor Nehrutsa <Ihor.Nehrutsa@gmail.com>
2025-05-16tools/gen-cpydiff.py: Ensure every item has at least 2 TOC levels.Jeff Epler
Previously, the navigation ended up messy when the (long) description of the item became used as a 2nd level header, meaning that it was placed in the navigation. Check for this when generating cpydiff so that new cases don't sneak in unnoticed. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-05-16tools/gen-cpydiff.py: Fix RST heading generation.Jeff Epler
The heading character for the difference title was always "~", but items had been added which had just a single heading level. This made the generated table of contents confused about heading levels, because heading levels are not fixed in rst, but are inferred from the order they appear in the document. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-05-16tools/gen-cpydiff.py: Improve stdout vs stderr interleaving.Jeff Epler
In the syntax_space cpydiff, all the warnings were shown after the other output. This is because the output always showed all of stdout first and all of stdout second. By running Python in unbuffered mode and using `stderr=STDOUT`, the two streams are interleaved in exactly the order they're printed, so the SyntaxWarnings are interleaved with the other output. By using the `encoding=` argument of Popen, the need to explicitly convert to utf-8 is avoided. The encoding of the input also becomes utf-8 in this case, which all the test cases are (well, they're all ASCII, I think). As in `run-tests.py`, setting PYTHONIOENCODING ensures the Python interpreter's input and output are in utf-8, which is not always the case, especially on Windows systems. I spot-checked the generated doc pages and they all seemed to make sense still. Signed-off-by: Jeff Epler <jepler@gmail.com>
2025-05-15tools/verifygitlog.py: Allow long co-author and sign-off names.Daniël van de Giessen
Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
2025-05-15tools/verifygitlog.py: Disallow a leading slash in commit subject line.Damien George
Signed-off-by: Damien George <damien@micropython.org>
2025-05-12tools/verifygitlog.py: Apply stricter rules on git subject line.Damien George
There is a bit of ambiguity as to how the prefix of the git subject line should look like. Eg `py/vm: ...` vs `py/vm.c: ...` (whether the extension should be there or not). This commit makes the existing CI check of the git commit message stricter, by applying extra rules to the prefix, the bit before the : in the subject line. It now checks that the subject prefix: - doesn't start with unwanted bits: ., /, ports/ - doesn't have an extension: .c, .h, .cpp, .js, .rst or .md Full error messages are given when a rule does not pass. This helps to reduce maintainer burden by applying stricter rules, to keep the git commit history consistent. Signed-off-by: Damien George <damien@micropython.org>
2025-05-02tools/mpremote: For mip install, use hash to skip files that exist.Damien George
When using `mip install`, if a file that needs to be downloaded already exists locally, then the hash of that local file will be computed and if it matches the known hash of the remote file it will not be downloaded. Hashes in mip are guaranteed unique, so this change should never leave stale files on the filesystem. This behaviour follows that of the `mip` package in `micropython-lib`. Signed-off-by: Damien George <damien@micropython.org>