790 Commits

Author	SHA1	Message	Date
Damien George	17951cee87	py/dynruntime.mk: Enable single-precision float by default on armv6/7m. ... Soft float now works on these ARM targets thanks to the parent commit. Signed-off-by: Damien George <damien@micropython.org>	2025-06-10 13:43:03 +10:00
Damien George	718ff4fdd5	tools/mpy_ld.py: Support R_ARM_ABS32 relocation in text. ... Add support for R_ARM_ABS32 relocations in native .mpy files. These can be rewritten in the same way that data relocations are. Fixes issue #14430. Signed-off-by: Damien George <damien@micropython.org>	2025-06-10 13:40:11 +10:00
Alessandro Gatti	0da22b2c30	tools/ci.sh: Fix nanbox CI test runs. ... This commit fixes CI test runs for the `nanbox` target, which were broken by the unconditional native emitter code output changes in the test runner. The `nanbox` configuration does not enable native emitters of any kind, and with a full test run that includes executing emitted native code things would break when doing CI runs. This is worked around by introducing a common subset of tests that do not involve the native emitter, and a more comprehensive set of tests that include both non-emitter and emitter tests. The `nanbox` CI test run will stop at the first subset, whilst other configurations will run that and execute further tests. Function names have been kept the same for steps that involve native code, with the `nanbox` subset having another one. This should not trigger any breakage in existing CI configurations or external scripts. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2025-06-10 11:29:02 +10:00
Andrew Leech	95203ab88b	tools/boardgen.py: Ensure board pin locals_dict has consistent order. ... `tools/boardgen.py` is used by the `make-pins.py` scripts in many ports to generate the pin definitions for the machine module. In #17391 it was found that this is currently generating the C structs for board pin definitions with inconsistent ordering (across different build runs), which makes it sometimes impossible to get a consistent binary file even for no change in source files. This commit fixes that by sorting the board pin names alphabetically. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>	2025-06-10 11:09:14 +10:00
Alessandro Gatti	b8e56a17b1	github/workflows: Split QEMU/Arm builds into separate entries. ... This commit takes the QEMU/Arm CI build and test step and splits it into three separate steps (bigendian, sabrelite, thumb), to allow them to run in parallel. Currently the QEMU/Arm CI build step would take up to 16 minutes, often being the last step blocking a full test run. With this commit, when the steps run in parallel the time it takes to complete the QEMU/Arm build and test procedure is cut in half - taking between 8 to 9 minutes depending on the CI runner load. The existing `ci_build_and_test_arm` function has been removed, in favour of having three separate functions - one per configuration. They are called `ci_build_and_test_arm_bigendian`, `ci_build_and_test_arm_sabrelite`, and `ci_build_and_test_arm_thumb`. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2025-06-10 10:57:32 +10:00
Damien George	4c55b0879b	tools/ci.sh: Allow errors in code-size build to fail the CI. ... It was possible for CI to pass even if the bare-arm port fails to build. This commit fixes that. Signed-off-by: Damien George <damien@micropython.org>	2025-06-05 14:57:23 +10:00
Alessandro Gatti	193603dbac	tools/ci.sh: Clean the correct MPY files when batch compiling. ... This commit fixes a small yet harmless issue that occurs when invoking `ci_native_mpy_modules_build` on a persistent environment, as only X64 MPY files would be removed by the cleaning process. Now the correct architecture is passed at all times when cleaning before building a natmod for a particular architecture, forcing a full build of all files to better simulate the CI environment (where there's no state persisted between runs for this step). Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2025-06-04 22:35:40 +10:00
Alessandro Gatti	e8c92240e2	tools/ci.sh: Remove natmod build restrictions for Xtensa. ... This commit lets the CI pipeline build all natmods for the Xtensa target, now that ROM symbols can be used in the linking process. The restriction was put in place due to build failures on certain natmods for Xtensa, as ROM symbols would not be used, causing undefined symbol errors at build time. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2025-06-04 22:35:39 +10:00
Alessandro Gatti	bf2005de9e	tools/mpy_ld.py: Resolve fixed-address symbols if requested. ... This commit lets mpy_ld.py resolve symbols not only from the object files involved in the linking process, or from compiler-supplied static libraries, but also from a list of symbols referenced by an absolute address (usually provided by the system's ROM). This is needed for ESP8266 targets as some C stdlib functions are provided by the MCU's own ROM code to reduce the final code footprint, and therefore those functions' implementation was removed from the compiler's support libraries. This means that unless `LINK_RUNTIME` is set (which lets tooling look at more libraries to resolve symbols) the build process will fail as tooling is unaware of the ROM symbols' existence. With this change, fixed-address symbols can be exposed to the symbol resolution step when performing natmod linking. If there are symbols coming in from a fixed-address symbols list and internal code or external libraries, the fixed-address symbol address will take precedence in all cases. Although this is - in theory - also working for the whole range of ESP32 MCUs, testing is currently limited to Xtensa processors and the example natmods' makefiles only make use of this commit's changes for the ESP8266 target. Natmod builds can set the MPY_EXTERN_SYM_FILE variable pointing to a linkerscript file containing a series of symbols (weak or strong) at a fixed address; these symbols will then be used by the MicroPython linker when packaging the natmod. If a different natmod build method is used (eg. custom CMake scripts), `tools/mpy_ld.py` can now accept a command line parameter called `--externs` (or its short variant `-e`) that contains the path of a linkerscript file with the fixed-address symbols to use when performing the linking process. The linkerscript file parser can handle a very limited subset of binutils's linkerscript syntax, namely just block comments, strong symbols, and weak symbols. Each symbol must be in its own line for the parser to succeed, empty lines or comment blocks are skipped. For an example of what this parser was meant to handle, you can look at `ports/esp8266/boards/eagle.rom.addr.v6.ld` and follow its format. The natmod developer documentation is also updated to reflect the new command line argument accepted by `mpy_ld.py` and the use cases for the changes introduced by this commit. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2025-06-04 22:35:39 +10:00
Damien George	dea3035b88	tools/pyboard.py: Add write_timeout and catch errors in enter_raw_repl. ... If the USB serial device locks up, then writes to that device can hang forever. That can make the test runner `tests/run-tests.py` lock up, among other problems. This commit introduces a 5 second write-timeout, and catches any OSError's raised during `enter_raw_repl()`. Now, if a USB serial device locks up, `enter_raw_repl()` will eventually raise an exception. Signed-off-by: Damien George <damien@micropython.org>	2025-06-02 17:02:59 +10:00
Damien George	b1c947ab45	tools/pyboard.py: Introduce timeout_overall for read_until(). ... This applies the mpremote commit 03fe9c55ea to pyboard.py. The `timeout_overall` is used in `enter_raw_repl()`. It prevents waiting forever for a serial device that does not respond to the Ctrl-C/Ctrl-D/etc commands and is constantly outputting data. Signed-off-by: Damien George <damien@micropython.org>	2025-06-02 17:02:30 +10:00
Damien George	aaedd59b7c	tools/pyboard.py: Avoid initial blocking read in read_until(). ... This applies the mpremote commit 0d46e45a1f to pyboard.py. If the target does not return any data then `read_until()` will block indefinitely. Fix this by making the initial read part of the general read look, which always checks `inWaiting() > 0` before reading from the serial device. Also added the UART timeout to the constructor. This is not currently used but may be used as an additional safeguard. Signed-off-by: Damien George <damien@micropython.org>	2025-06-02 17:02:30 +10:00
Jos Verlinde	d945316432	tools/mpremote/tests: Add tests for 'fs tree' command. ... Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>	2025-05-18 00:20:43 +10:00
Jos Verlinde	1dfb5092fc	tools/mpremote: Add new 'fs tree' command. ... Add `mpremote fs tree` command to show a tree of the device's files. It: - Shows a treeview from current path or specified path. - Uses the graph chars ("├── ", "└── ") (not configurable). - Has the options: -v/--verbose adds the serial device name to the top of the tree -s/--size add a size to the files -h/--human add a human readable size to the files Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>	2025-05-18 00:19:14 +10:00
IhorNehrutsa	f48b981567	esp32: Update to use ESP-IDF v5.4.1. ... This version of the IDF uses about 1KB more IRAM and 1KB more DRAM on most boards, but 6.5KB more DRAM usage on the S3. It seems that's due to a lot of small increases in many components. Signed-off-by: Ihor Nehrutsa <Ihor.Nehrutsa@gmail.com>	2025-05-16 14:02:39 +10:00
Jeff Epler	6d74b4e3c1	tools/gen-cpydiff.py: Ensure every item has at least 2 TOC levels. ... Previously, the navigation ended up messy when the (long) description of the item became used as a 2nd level header, meaning that it was placed in the navigation. Check for this when generating cpydiff so that new cases don't sneak in unnoticed. Signed-off-by: Jeff Epler <jepler@gmail.com>	2025-05-16 11:52:03 +10:00
Jeff Epler	a19d3f742e	tools/gen-cpydiff.py: Fix RST heading generation. ... The heading character for the difference title was always "~", but items had been added which had just a single heading level. This made the generated table of contents confused about heading levels, because heading levels are not fixed in rst, but are inferred from the order they appear in the document. Signed-off-by: Jeff Epler <jepler@gmail.com>	2025-05-16 11:52:03 +10:00
Jeff Epler	605eda158d	tools/gen-cpydiff.py: Improve stdout vs stderr interleaving. ... In the syntax_space cpydiff, all the warnings were shown after the other output. This is because the output always showed all of stdout first and all of stdout second. By running Python in unbuffered mode and using `stderr=STDOUT`, the two streams are interleaved in exactly the order they're printed, so the SyntaxWarnings are interleaved with the other output. By using the `encoding=` argument of Popen, the need to explicitly convert to utf-8 is avoided. The encoding of the input also becomes utf-8 in this case, which all the test cases are (well, they're all ASCII, I think). As in `run-tests.py`, setting PYTHONIOENCODING ensures the Python interpreter's input and output are in utf-8, which is not always the case, especially on Windows systems. I spot-checked the generated doc pages and they all seemed to make sense still. Signed-off-by: Jeff Epler <jepler@gmail.com>	2025-05-16 11:52:03 +10:00
Daniël van de Giessen	4208970451	tools/verifygitlog.py: Allow long co-author and sign-off names. ... Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>	2025-05-15 13:03:11 +10:00
Damien George	51b821ce82	tools/verifygitlog.py: Disallow a leading slash in commit subject line. ... Signed-off-by: Damien George <damien@micropython.org>	2025-05-15 13:01:41 +10:00
Damien George	61eedbbd11	tools/verifygitlog.py: Apply stricter rules on git subject line. ... There is a bit of ambiguity as to how the prefix of the git subject line should look like. Eg `py/vm: ...` vs `py/vm.c: ...` (whether the extension should be there or not). This commit makes the existing CI check of the git commit message stricter, by applying extra rules to the prefix, the bit before the : in the subject line. It now checks that the subject prefix: - doesn't start with unwanted bits: ., /, ports/ - doesn't have an extension: .c, .h, .cpp, .js, .rst or .md Full error messages are given when a rule does not pass. This helps to reduce maintainer burden by applying stricter rules, to keep the git commit history consistent. Signed-off-by: Damien George <damien@micropython.org>	2025-05-12 13:47:03 +10:00
Damien George	e53f262a85	tools/mpremote: For mip install, use hash to skip files that exist. ... When using `mip install`, if a file that needs to be downloaded already exists locally, then the hash of that local file will be computed and if it matches the known hash of the remote file it will not be downloaded. Hashes in mip are guaranteed unique, so this change should never leave stale files on the filesystem. This behaviour follows that of the `mip` package in `micropython-lib`. Signed-off-by: Damien George <damien@micropython.org>	2025-05-02 11:44:32 +10:00
Damien George	4117a2d9b5	tools/ci.sh: Update URL for xtensa-lx106-elf-standalone.tar.gz. ... The https://github.com/jepler/esp-open-sdk repository has been removed, so use the file hosted at micropython.org (it's the same file). Signed-off-by: Damien George <damien@micropython.org>	2025-05-02 11:39:49 +10:00
Maureen Helm	3c8d1b13f5	zephyr: Upgrade to Zephyr v4.0.0. ... Updates the Zephyr port build instructions. The CI is updated to use Zephyr docker image 0.27.4, SDK 0.17.0 and the latest Zephyr release tag. Tested on max32690fthr and frdm_k64f. Signed-off-by: Maureen Helm <maureen.helm@analog.com> Signed-off-by: Detlev Zundel <dzu@member.fsf.org>	2025-04-29 23:00:37 +10:00
Jos Verlinde	6406afb1f3	tools/mpremote: Prevent deletion of /remote files via rm -r. ... Removes the risk of inadvertently deleting files on the host by preventing the deletion of files via `rm -r` on the `/remote` vfs mount point. Fixes issue #17147. Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>	2025-04-26 16:07:44 +10:00
Jos Verlinde	37fe3f66c3	tools/mpremote/tests: Add test for rm -r on /remote vfs. ... Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>	2025-04-26 16:07:40 +10:00
Anson Mansfield	dc46cf15c1	tools/mpremote: Fix possibly-missing EOPNOTSUPP errno name. ... Signed-off-by: Anson Mansfield <amansfield@mantaro.com>	2025-04-26 15:51:49 +10:00
Anson Mansfield	805fe083a3	tools/mpremote: Refactor error handling to apply generally to any errno. ... This rewrites the code that previously manually emitted and caught various OSError subclasses with equivalent code that uses the errno name dictionary to do this generically, and updates the exception handler in do_filesystem to catch them in a similarly-generic fashion using os.strerror to retrieve an appropriate error message text equivalent to the current messages. Note that in the CPython environments where mpremote runs, the call to the OSError constructor already returns an instance of the corresponding mapped exception subtype. Signed-off-by: Anson Mansfield <amansfield@mantaro.com>	2025-04-26 15:50:00 +10:00
Anson Mansfield	cee0419021	tools/mpremote/tests: Add tests for errno behavior. ... Signed-off-by: Anson Mansfield <amansfield@mantaro.com>	2025-04-26 15:49:48 +10:00
Damien George	dcca3ff602	tools/mpremote: Use zlib.compressobj instead of zlib.compress. ... Because the `wbits` parameter was only added to `zlib.compress` in CPython 3.11. Using `zlib.compressobj` makes the code compatible with much older CPython versions. Fixes issue #17140. Signed-off-by: Damien George <damien@micropython.org>	2025-04-24 22:33:05 +10:00
Angus Gratton	f5cb9eb974	top: Bump Ruff version to v0.11.6. ... Brings it into sync with a matching change to micropython-lib (which was much older). Includes one small automatic fix. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2025-04-24 21:42:21 +10:00
Jos Verlinde	ef8282c717	docs/reference/mpremote: Update docs for mpremote rm -r. ... Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>	2025-04-09 10:51:48 +10:00
Jos Verlinde	72d4c40941	tools/mpremote/tests: Add tests for mpremote rm -r. ... Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>	2025-04-09 10:51:45 +10:00
Jos Verlinde	1aa9b3d94b	tools/mpremote: Add recursive remove functionality to filesystem cmds. ... mpremote now supports `mpremote rm -r`. Addresses #9802 and #16845. Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>	2025-04-09 10:44:45 +10:00
iabdalkader	547207ddc8	github/workflows: Add Alif port to CI. ... Signed-off-by: iabdalkader <i.abdalkader@gmail.com>	2025-04-09 00:22:33 +10:00
iabdalkader	9e9be83fd6	tools/mpremote: Allow .img for ROMFS file and validate ROMFS image. ... Currently the tool allows writing an invalid ROMFS image, with a bad header or images smaller than minimum size, and only checks the image extension. This commit allows deploying a ROMFS with either a ".img" or ".romfs" extension (in the future support may be added for other extensions that have different semantics, eg a manifest), and validates the image header before writing. Signed-off-by: iabdalkader <i.abdalkader@gmail.com>	2025-04-08 00:27:55 +10:00
Angus Gratton	74a5bf94c1	tools/gen-cpydiff.py: Fail CPython diff generation if output matches. ... Previously this information was recorded in a "status" field of the result, but nothing ever parsed this result which led to non-differences not being removed. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2025-04-07 16:10:05 +10:00
Damien George	e34412f0f4	tools/ci.sh: Manually install picotool for rp2 builds. ... If picotool is not installed, it's fetched and built when compiling each rp2 board. And the "develop" branch of picotool is used instead of a release. Installing it manually using the "master" branch means the latest released version is used (instead of a possibly unstable development version), and also makes building each rp2 board a little faster. Signed-off-by: Damien George <damien@micropython.org>	2025-04-07 13:53:38 +10:00
Jon Nordby	3805e65ed3	tools/mpy_ld.py: Give better error for unsupported ARM absolute relocs. ... This is a known limitation, so better to give a clear warning than a catch-all AssertionError. Happens for example when trying to use soft-float on ARCH=armv6m Also give more details on the assertion for unknown relocations, such that one can see which symbol it affects etc, to aid in debugging. References issue #14430. Signed-off-by: Jon Nordby <jononor@gmail.com>	2025-04-03 16:37:38 +11:00
Damien George	2db0c0225f	tools/mpremote: Make mip install skip /rom*/lib directories. ... If a ROMFS is mounted then "/rom/lib" is usually in `sys.path` before the writable filesystem's "lib" entry. The ROMFS directory cannot be installed to, so skip it if found. Signed-off-by: Damien George <damien@micropython.org>	2025-03-27 17:10:00 +11:00
Alessandro Gatti	31a008c6e2	tools/ci.sh: Do not assume the Python interpreter is called "python". ... This commit removes the assumption made by the CI scripts that the system-provided python executable is simply named "python". The scripts will now look for a binary called "python3" first, and then fall back to "python" if that is not found. Whilst this is currently the case for the CI environment, there are no guarantees for this going forward. For example minimal CI environments set up by some developers, using the same base OS, have their python executable called "python3". Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2025-03-17 13:04:53 +11:00
Alessandro Gatti	1a7cbac879	tools/ci.sh: Build Xtensa natmods as part of the CI process. ... This commit expands the CI tests by checking whether the example native modules are able to be built for the Xtensa architecture. This was made possible by the changes to mpy_ld that allow symbol resolution across standard compiler-provided libraries. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2025-03-17 13:04:53 +11:00
Volodymyr Shymanskyy	51976110e2	tools/mpy_ld.py: Allow linking static libraries. ... This commit introduces an additional symbol resolution mechanism to the natmod linking process. This allows the build scripts to look for required symbols into selected libraries that are provided by the compiler installation (libgcc and libm at the moment). For example, using soft-float code in natmods, whilst technically possible, was not an easy process and required some additional work to pull it off. With this addition all the manual (and error-prone) operations have been automated and folded into `tools/mpy_ld.py`. Both newlib and picolibc toolchains are supported, albeit the latter may require a bit of extra configuration depending on the environment the build process runs on. Picolibc's soft-float functions aren't in libm - in fact the shipped libm is nothing but a stub - but they are inside libc. This is usually not a problem as these changes cater for that configuration quirk, but on certain compilers the include paths used to find libraries in may not be updated to take Picolibc's library directory into account. The bare metal RISC-V compiler shipped with the CI OS image (GCC 10.2.0 on Ubuntu 22.04LTS) happens to exhibit this very problem. To work around that for CI builds, the Picolibc libraries' path is hardcoded in the Makefile directives used by the linker, but this can be changed by setting the PICOLIBC_ROOT environment library when building natmods. Signed-off-by: Volodymyr Shymanskyy <vshymanskyi@gmail.com> Co-authored-by: Alessandro Gatti <a.gatti@frob.it>	2025-03-17 13:03:27 +11:00
Damien George	0c98c60b68	tools/mpremote: Add romfs query, build and deploy commands. ... These commands use the `vfs.rom_ioctl()` function to manage the ROM partitions on a device, and create and deploy ROMFS images. Signed-off-by: Damien George <damien@micropython.org>	2025-03-06 12:52:35 +11:00
Volodymyr Shymanskyy	6fba1e406b	tools/mpy-tool.py: Support calling main() from an external script. ... Signed-off-by: Volodymyr Shymanskyy <vshymanskyi@gmail.com> Signed-off-by: Angus Gratton <angus@redyak.com.au>	2025-03-05 16:02:08 +11:00
Volodymyr Shymanskyy	e1b2f2e078	tools/mpy-tool.py: Add support for self-hosting of mpy-tool. ... This allows running mpy-tool using MicroPython itself. An appropriate test is added to CI to make sure it continues to work. Signed-off-by: Volodymyr Shymanskyy <vshymanskyi@gmail.com> Signed-off-by: Angus Gratton <angus@redyak.com.au>	2025-03-05 16:01:12 +11:00
Damien George	71c7c03e41	tools/mpremote/tests: Add test for RemoteFile.readline. ... Signed-off-by: Damien George <damien@micropython.org>	2025-02-28 14:14:58 +11:00
Andrew Leech	86526e9c2b	tools/mpremote: Optimise readline support in mount. ... This significantly speeds up readline on files opened directly from an mpremote mount. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>	2025-02-28 14:14:39 +11:00
Christian Clauss	dc2fcfcc55	all: Upgrade to ruff v0.9.6. ... Signed-off-by: Christian Clauss <cclauss@me.com>	2025-02-25 13:02:49 +01:00
Angus Gratton	8b1ed4473d	github/workflows: Include the Python version in the ESP-IDF cache key. ... Changing runner OS can change Python version, and ESP-IDF installs are keyed on ESP-IDF and Python version together. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2025-02-25 15:13:13 +11:00