| Age | Commit message (Collapse) | Author |
|---|
|
Back in commit 8978102f3595ae321484a6be44c1dcf25c8909a9 (see PR #16111) the
`run-tests.py` script was changed to use an improved way of selecting the
test instance, eg:
$ ./run-tests.py -t a0
that would run on /dev/ttyACM0. This has been a very nice improvement,
makes it easier to specify the target.
This commit updates `run-multitests.py` to use the same scheme. It
previously used `-i` but now that's changed to `-t`.
Signed-off-by: Damien George <damien@micropython.org>
|
|
It's possible for a test to output non-ASCII characters (for example, due
to a hard fault or serial noise or memory corruption). Rather than crashing
the test runner, backslash escape those characters and treat them as
program output.
Refactors the string encoding step to a single helper to avoid copy-paste.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
|
|
Reuse the `create_test_report()` function from `run-tests.py` to generate a
`_result.json` file summarising the test run. If there's more than one
permutation of the test run, only the last result is saved.
Signed-off-by: Damien George <damien@micropython.org>
|
|
Includes adding some ESP8266 port output to the ignored output list for the
multitest runner.
This test passes on ESP8266 and various ESP32s (including talking to each
other). Without the fix in the parent commit, ESP32 AP will fail if the
station can report its channel (i.e. channel is wrong).
Testing with a CYW43 (RPI_PICO_W) currently fails but I have some fixes
to submit so it can pass as well.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
|
|
This matches the behaviour of run-tests.py, which sets cwd to the directory
containing the test script, which helps to isolate the filesystem.
It means that the SSL tests no longer need to know the name of their
containing directory to find the certificate files, and helps to run these
tests on bare-metal.
Signed-off-by: Damien George <damien@micropython.org>
|
|
Make tests run in an isolated environment (i.e. `import io` would
otherwise get the `tests/io` directory).
This work was funded through GitHub Sponsors.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
|
|
If a multitest calls `multitest.output_metric(...)` then that output will
be collected separately, not considered as part of the test verification
output, and instead be printed at the end. This is useful for tests that
want to output performance/timing metrics that may change from one run to
the next.
Signed-off-by: Damien George <damien@micropython.org>
|
|
The device-under-test should use `multitest.expect_reboot()` to indicate
that it will reboot.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
|
|
This follow the change made for Makefile-based projects in b2e82402.
|
|
Allows running from a different directory, etc.
This work was funded by Planet Innovation.
|
|
|
|
These were missed by 47c84286e8c8d9873e99f12711a683ecd6b9ca62
Signed-off-by: Damien George <damien@micropython.org>
|
|
This works if your network is pre-configured in boot.py as an object called
"nic". Without this, multitests expects to access the WLAN/LAN class which
isn't always correct.
Signed-off-by: Andrew Leech <andrew@alelec.net>
|
|
This allows running the test on boards with just a LAN interface.
Fixes issue #8681.
|
|
Signed-off-by: Damien George <damien@micropython.org>
|
|
BLE still functions correctly even though these messages are sometimes
printed by the IDF. Ignoring them allows the multi_bluetooth tests to pass
on an esp32 board.
Signed-off-by: Damien George <damien@micropython.org>
|
|
Test instances can now use the following methods to synchronise their
execution:
multitest.broadcast("sync message")
multitest.wait("sync message")
Signed-off-by: Damien George <damien@micropython.org>
|
|
Signed-off-by: Damien George <damien@micropython.org>
|
|
Signed-off-by: Damien George <damien@micropython.org>
|
|
It's now possible to specify a device serial port using shorcuts like:
$ ./run-multitests.py -i pyb:a0 -i pyb:u1 multi_bluetooth/*.py
Signed-off-by: Damien George <damien@micropython.org>
|
|
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
|
|
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
|
|
|
|
For example, to run the BLE multitests entirely with the unix port:
env MICROPY_MICROPYTHON=../ports/unix/micropython-dev ./run-multitests.py \
-i micropython,MICROPYBTUSB=01 \
-i micropython,MICROPYBTUSB=02:02 \
multi_bluetooth/ble_*.py
|
|
Updates the tests to use non-bitmask events, event renames, as well as some
of the new completion events to improve reliability of the tests.
|
|
To make it easier to understand the output when both -s and -t are used.
|
|
One can now use `-i micropython` and `-i cpython` to add instances using
the `MICROPYTHON` and `CPYTHON3` variables (which can be overridden by env
vars).
|
|
And use this new feature to filter out certain lines in the Bluetooth
multitests.
|
|
This adds the Python files in the tests/ directory to be formatted with
./tools/codeformat.py. The basics/ subdirectory is excluded for now so we
aren't changing too much at once.
In a few places `# fmt: off`/`# fmt: on` was used where the code had
special formatting for readability or where the test was actually testing
the specific formatting.
|
|
|
|
|
|
This commit adds a test runner and initial test scripts which run multiple
Python/MicroPython instances (eg executables, target boards) in parallel.
This is useful for testing, eg, network and Bluetooth functionality.
Each test file has a set of functions called instanceX(), where X ranges
from 0 up to the maximum number of instances that are needed, N-1. Then
run-multitests.py will execute this script on N separate instances (eg
micropython executables, or attached boards via pyboard.py) at the same
time, synchronising their start in the right order, possibly passing IP
address (or other address like bluetooth MAC) from the "server" instance to
the "client" instances so they can connect to each other. It then runs
them to completion, collects the output, and then tests against what
CPython gives (or what's in a provided .py.exp file).
The tests will be run using the standard unix executable for all instances
by default, eg:
$ ./run-multitests.py multi_net/*.py
Or they can be run with a board and unix executable via:
$ ./run-multitests.py --instance pyb:/dev/ttyACM0 --instance exec:micropython multi_net/*.py
|
