This commit updates the list of the tests that must be skipped when the
suite is executed on a target that does not have floating point
support.
Two more tests, namely `extmod/vfs_rom.py` and
`micropython/const_float.py` have been added to the list, since they
both rely on floating point support being there.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
Otherwise, a very deeply nested regular expression like
re.compile("(" * 65536)
can exhaust the host stack during the compile phase. This turns
that into a `RuntimeError: maximum recursion depth exceeded`
instead.
This crash was found via fuzzing.
Signed-off-by: Jeff Epler <jepler@unpythonic.net>
Implemented according to API docs in a parent comment.
Adds new multi_extmod/machine_can_* tests which pass when testing between
NUCLEO_G474RE, NUCLEO_H723ZG and PYBDV11.
This work was mostly funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Reclassify failures of tests listed in flaky_tests_to_ignore as "ignored"
instead of retrying them. Ignored tests still run and their output is
reported, but they don't affect the exit code. The ci.sh --exclude lists
for these tests are removed so they run normally.
Signed-off-by: Andrew Leech <andrew.leech@planet-innovation.com>
Includes corresponding .exp files because this feature is only available in
Python 3.14+.
Tests for `!a` conversion specifier and space after `!` are not included
because they are not supported by MicroPython.
Signed-off-by: Koudai Aono <koxudaxi@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
Test file paths which get passed to the run_tests function can be
absolute or relative and with or without leading slash in the latter
case, depending on the arguments to run-tests.py, but since that path
is used to:
- display which tests run
- record which tests ran in the results.json
- craft the filename for the .exp/.out file for failed tests
it is desirable to always use the same file path irregardless of
how the user passed the path.
In practice this means that all forms of running our own tests like:
>python ./run-tests.py -i extmod
>python ./run-tests.py -d extmod
>python ./run-tests.py -d ./extmod
>python ./run-tests.py -d ../tests/extmod
>python ./run-tests.py -d /full/path/to/tests/extmod
will now consistently all display the tests like
pass extmod/time_time_ns.py
FAIL extmod/some_failing_test.py
and produce output files like
results/extmod_some_failing_test.py.exp
results/extmod_some_failing_test.py.out
instead of displaying/using the exact path as passed.
For external tests, meaning not in the tests/ directory, we also want
to be consistent so there the choice was made to always use absolute
paths.
Signed-off-by: stijn <stijn@ignitron.net>
Scan the --test-dirs argument for the main tests directory being
passed and if so do the same thing as if running from within that
main test directory.
In practice this makes the following (which used to counterintuitively
try and fail to run the .py files in the tests/ directory itself)
>python micropython/tests/run-tests.py -d micropython/tests
do the same thing as
>cd micropython/tests
>python ./run-tests.py
which is logical and convenient.
Signed-off-by: stijn <stijn@ignitron.net>
Test file paths which get passed to the run_tests function can be
absolute or relative and with or without leading slash in the latter
case, depending on the arguments to run-tests.py, but the skip_tests
list with tests to skip only contains relative paths so using simple
string equality comparison easily leads to false negatives.
Compare the full absolute path instead such that it doesn't matter
anymore in which form the tests are passed. Note:
- use realpath to resolve symlinks plus make the comparison case
insensitive on windows
- the test_file passed to run_one_test is not altered by this commit,
such that when the user inputs relative paths the tests are also
still displayed with relative paths
- likewise the test_file_abspath is not modified because functions
like run_micropython rely on it having forward slashes
In practice this means that it used to be so that the only forms
of running tests for which the skip_tests lists actually worked were:
>python ./run-tests.py
>python ./run-tests.py -d extmod
whereas it now works consistently so also for these invocations,
which in the end all point to the exact same path:
>python ./run-tests.py -d ./extmod
>python ./run-tests.py -d ../tests/extmod
>python ./run-tests.py -d /full/path/to/tests/extmod
These examples used to not skip any of the tests in the extmod/
directory thereby leading to test failures.
Signed-off-by: stijn <stijn@ignitron.net>
This is convenient when trying to figure out the correct values
for --include/--exclude/--test-dirs/... arguments.
Signed-off-by: stijn <stijn@ignitron.net>
This is a follow up to 6436f8b93f that
catches more cases of a failed raw REPL.
If the target device is broken in a certain way then it can have a serial
write error instead of just not returning any data. In that case the error
raised by `pyboard.py` is "could not enter raw repl: Write timeout", which
is slightly different to "could not enter raw repl" (the latter is raised
when a serial read fails to return the correct data).
The patch here accounts for all cases of a failed raw REPL by using
`str.startswith()` instead of a string equality.
This can be tested on RPI_PICO_W and RPI_PICO2_W which currently crash in
the specific way needed to trigger the write timeout when running the
`tests/extmod/socket_badconstructor.py` test. In particular adding the
following code to the end of a test (eg `tests/extmod/random_extra.py`)
will trigger the issue:
import socket
try:
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, None)
except TypeError:
pass
With the fix here, the `run-tests.py` and `run-natmodtests.py` will abort
early if that code is added (prior to the fix they would continue to run
all tests and take a long time).
Signed-off-by: Damien George <damien@micropython.org>
Prior to this change, if a test crashed a board to the point that it was no
longer responsive but the serial connection was still alive and able to
read/write (eg due to an infinite loop with ctrl-C unavailable), it would
take (30 + 10 * number-of-remaining-tests) seconds to finish the test run.
That can be 10 minutes or more.
The change here aborts the test run if the target does not respond 4 times
to when trying to enter the raw REPL at the start of a test. Now, it only
takes a maximum of (30 + 4 * 5) = 50 seconds to detect a crashed state and
abort the entire test run.
Can test this by adding the following to a test:
import time, micropython
micropython.kbd_intr(-1)
while 1:
time.sleep(1)
For example, add the above to the end of `tests/extmod/random_extra.py` and
test via:
$ ./run-tests.py -t a0 extmod/[r-z]*.py
$ ./run-natmodtests.py -t a0 extmod/[r-z]*.py
Previously the `run-tests.py` run would take around 16 minutes. Now it
takes under a minute.
Signed-off-by: Damien George <damien@micropython.org>
The tests runners usually exit with error code 1 when something fails. But
it's useful to distinguish between "a test failed" and "the test runner
failed".
To that end, `run-tests.py` now returns error code 2 when unittest is not
installed but is needed. And `run-natmodtests.py` now returns error code 2
when it cannot automatically detect the target architecture.
Signed-off-by: Damien George <damien@micropython.org>
This commit converts `tests/extmod/machine_spi_rate.py` to use the target
wiring mechanism, moving the port-specefic definition of the SPI instances
that are being tested out of the `tests/extmod/machine_spi_rate.py` file
and into the target wiring code. As part of this, the test is changed to
always use the default pins for the given SPI instance.
This simplifies the test, makes it easier to add support for new
ports/boards, and allows reusing the `spi_standalone_args_list` definition
in other tests (if needed one day).
Signed-off-by: Damien George <damien@micropython.org>
The test runners have evolved over time and become more and more complex.
In particular `tests/run-tests.py` is rather large now. The test runners
also duplicate some functionality amongst themselves.
As a start to improving this situation, this commit factors out the helper
functions from `run-tests.py` into a new `test_utils.py` file, and uses
that new module in all test runners.
There should be no functional change here.
Signed-off-by: Damien George <damien@micropython.org>
This variant of `schedule.py` explicitly calls a zero sleep. The existing
variant is kept to ensure the scheduler is called between bytecodes.
Signed-off-by: Jeff Epler <jepler@unpythonic.net>
This test was ignored since 2020, which hid a new bug in the test for the
native emitter added in 2024. Hopefully other changes to the test will
make it more reliable.
Signed-off-by: Jeff Epler <jepler@unpythonic.net>
This commit fixes a test exercising complex numbers creation, working
around a deprecation introduced in CPython 3.14.
Creating complex numbers using a complex number as the real part in the
constructor is deprecated in CPython 3.14, but that construction method
is still supported by MicroPython and covered by tests. To work around
this, the specific constructor is extracted into its own test, providing
an expected output file recorded using CPython 3.11.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit performs the necessary changes to handle an additional RV32
CPU extension flag, for the Zcmp extension in this case.
The changes are not limited to RV32-only code, as other parts of the
tooling need to be modified for this: the testing framework has to be
made aware that an extra bit can be set in sys.implementation._mpy and
needs to know how it is called, and "mpy-cross" must be able to actually
set that flag bit in the first place via the appropriate command line
argument.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit extends the test runner to automatically discover inline
assembler tests for known RV32 extensions, and checks whether to add the
discovered tests to the enabled tests list.
Automatic discovery requires that inline assembler tests for RV32
extensions follow a specific pattern both for filenames and for the
tests' output in case of success. A valid RV32 extension test must
have:
* A code fragment that checks for support of the extension on the
running target in "/tests/feature_check", called
"inlineasm_rv32_<extensionname>.py" that should print the string
"rv32_<extensionname>" if the extension is supported
* A matching expected result file in "/tests/feature_check" called
"inlineasm_rv32_<extensionname>.py.exp" that must contain the string
"rv32_<extensionname>" (without quotes)
* A regular MicroPython test file in "/tests/inlineasm/rv32" called
"asm_ext_<extensionname>.py"
For example, to test the Zba extension, there must be a file called
"/tests/feature_check/inlineasm_rv32_zba.py" that should print the
string "rv32_zba" if the extension is supported, together with a file
called "/test/feature_check/inlineasm_rv32_zba.py.exp" that contains the
string "rv32_zba" in it, and finally there must be a regular MicroPython
test file called "/tests/inlineasm/rv32/asm_ext_zba.py".
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit marks the "stress/list_sort.py" test to be skipped when
running on ESP8266.
The test just takes too long without yielding to the OS whilst doing the
sort, causing the internal software watchdog to kick in and reboot the
board.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
Add a general normalize_newlines() function that handles newline variations
(\\r\\r\\n, \\r\\n) to \\n while preserving literal \\r characters that are
part of test content.
This provides a robust solution for cross-platform test compatibility,
particularly addressing PTY double-newline issues that can occur with some
terminal implementations.
The function is applied to all test output before comparison, eliminating
platform-specific newline issues.
Includes a unit test to verify the normalization behavior.
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
This commit lets "run-tests.py" use the encoded architecture flags
provided by the interpreter when invoking "mpy-cross".
If architecture flags are detected, they're mapped into the necessary
strings needed by "mpy-cross"'s "-march-flags" argument, so that tests
will always use all available extensions reported by the target.
Currently this is limited to the RV32 platform, as it is the only one
that is making use of this facility as of now. This also lets the QEMU
port remove forced arguments to "mpy-cross" when running the test suite.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit adds the currently supported architecture flags value as the
upper part of "sys.implementation._mpy".
This had the side effect of perturbing quite a bit of testing
infrastructure and invalidating documentation related to MPY files. To
make the test suite run successfully and keep the documentation in sync
the following changes have been made:
* The target info feature check file now isolates eventual architecture
flags and adds them as a separate field
* The test runner now picks up the new architecture flags field, reports
it to STDOUT if needed and stores it for future uses
* Relevant test files for MPY files import code had to be updated to
mask out the architecture flags bits in order to perform correctly
* MPY file format documentation was updated to show how to mask off and
properly display the architecture flags information.
This works out of the box if the flag bits can fit in a smallint value
once merged with the MPY file header value.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This moves the `port:<dev>` check for the test instance to the
`convert_device_shortcut_to_real_device()` helper function, which means
that `run-multitests.py` can use this specification.
Signed-off-by: Damien George <damien@micropython.org>
This makes sure that `open()` raises `OSError(ENOENT)` if the test script
attempts to open a file that does not exist.
For example, the test `tests/ports/esp32/check_err_str.py` attempts to do
this and this test currently fails when run with `--via-mpy` because the
injected filesystem lets a file by any name be opened.
The fix here tries to be as minimal as possible, so includes a size
optimisation for `stat()` to use a fixed 10-tuple. It also fixes the
`OSError` argument, which should be positive.
Signed-off-by: Damien George <damien@micropython.org>
The unix minimal variant cannot run these tests because:
- it doesn't support -v -v debug printing
- it doesn't have `open()`
Signed-off-by: Damien George <damien@micropython.org>
This skips some additional tests that use slice, and no longer skips
`basics/string_format_modulo.py` which doesn't actually use slice (but
`float/string_format_modulo.py` does).
Signed-off-by: Damien George <damien@micropython.org>
This commit adds support for Zba opcodes to the RV32 inline assembler.
Three new opcodes were added, SH1ADD, SH2ADD, and SH3ADD, which performs
a scaled addition (by 1, 2, or 3 bits respectively). At the moment only
qemu's VIRT_RV32 and rp2's RPI_PICO2/RPI_PICO2_W ports support these
opcodes (the latter only when using the RISCV variant).
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
All the existing `machine.UART` tests in extmod and extmod_hardware are
converted to use the new `target_wiring` scheme, which removes a lot of
duplicated board-specific settings.
All the existing boards that were supported by these UART tests now have
their own `target_wiring` file. Some configurations are board specific (eg
NUCLEO_WB55) and others are port specific.
Signed-off-by: Damien George <damien@micropython.org>
There are currently a few "hardware" tests that need external board
connections to be made for them to work, such as bridging a pair of pins.
For example, all tests in `tests/extmod_hardware` need some kind of
connection.
Along with the physical connections -- which are different for each board
-- there needs to be corresponding Python settings, eg which UART instance
to use and which pins for TX/RX.
These settings are currently hard-coded in each test file. That has a few
problems:
- settings are repeated, eg all the UART tests have pretty much the same
settings code duplicated across them
- changing the settings means changing many files
- adding a new board means adding a lot of code
- the tests get bigger and bigger with each new board that they support,
meaning they may not fit on targets with a small amount of RAM (that's
already the case with `tests/extmod_hardware/machine_pwm.py`)
- if you have a custom board you have to manually edit the test to add your
own settings
This commit aims to solve all the above problems by splitting the board-
specific settings out into separate files, one for each board (or port).
They are placed in the `tests/target_wiring/` directory. The
`run-tests.py` test runner then loads the appropriate configuration for the
target that is being tested, sends it to the board so it's available as
`import target_wiring` (without needing a filesystem), and then executes
the test. Note tht only tests that need `target_wiring` have it loaded.
This commit adds support for this `target_wiring` scheme.
Signed-off-by: Damien George <damien@micropython.org>
This commits lifts the unconditional restriction on inline assembler FPU
tests for the Qemu platform, and makes said restriction conditional to
the lack of an available floating point unit on the running platform.
The Qemu platform supported only emulated machines that could target up
to a Cortex-M3, so an ArmV7-M target that had no support for floating
point. With the addition of MPS2_AN500 to the list of emulated targets
the range was extended to cover up to Cortex-M7, so a floating point
unit may possibly be available and thus able to run the FPU inlineasm
tests.
For that, the test runner was changed to detect the running architecture
when checking the target capabilities; if the target reports its
MicroPython architecture to be either "armv7emsp" or "armv7emdp"
(providing single-precision and double-precision floating point unit
support respectively) then the FPU-only inline tests are not put into
the blocked tests list.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
There are currently a few tests that are excluded when using the native
emitter because they test printing of exception tracebacks, which includes
line numbers. And the native emitter doesn't store line numbers, so gets
these tests wrong.
But we'd still like to run these tests using the native emitter, because
they test useful things even if the line number info is not in the
traceback (eg that threads which crash print out their exception).
This commit adds support for native-specific .exp files, which are of the
form `<test>.py.native.exp`. If such an .exp file exists then it take
precedence over any normal `<test>.py.exp` file.
(Actually, the implementation here is general enough that it also supports
`<test>.py.bytecode.exp` as well, if bytecode ever needs a specific exp
file.)
Signed-off-by: Damien George <damien@micropython.org>
Instead of using a feature check. This is more consistent with how other
optional modules are skipped.
Signed-off-by: Damien George <damien@micropython.org>
Alessandro Gatti
Dan Halbert
Jeff Epler
Damien George
Angus Gratton
Andrew Leech
Koudai Aono
stijn
Ihor Nehrutsa