755 Commits

Author	SHA1	Message	Date
iabdalkader	9e9be83fd6	tools/mpremote: Allow .img for ROMFS file and validate ROMFS image. ... Currently the tool allows writing an invalid ROMFS image, with a bad header or images smaller than minimum size, and only checks the image extension. This commit allows deploying a ROMFS with either a ".img" or ".romfs" extension (in the future support may be added for other extensions that have different semantics, eg a manifest), and validates the image header before writing. Signed-off-by: iabdalkader <i.abdalkader@gmail.com>	2025-04-08 00:27:55 +10:00
Angus Gratton	74a5bf94c1	tools/gen-cpydiff.py: Fail CPython diff generation if output matches. ... Previously this information was recorded in a "status" field of the result, but nothing ever parsed this result which led to non-differences not being removed. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2025-04-07 16:10:05 +10:00
Damien George	e34412f0f4	tools/ci.sh: Manually install picotool for rp2 builds. ... If picotool is not installed, it's fetched and built when compiling each rp2 board. And the "develop" branch of picotool is used instead of a release. Installing it manually using the "master" branch means the latest released version is used (instead of a possibly unstable development version), and also makes building each rp2 board a little faster. Signed-off-by: Damien George <damien@micropython.org>	2025-04-07 13:53:38 +10:00
Jon Nordby	3805e65ed3	tools/mpy_ld.py: Give better error for unsupported ARM absolute relocs. ... This is a known limitation, so better to give a clear warning than a catch-all AssertionError. Happens for example when trying to use soft-float on ARCH=armv6m Also give more details on the assertion for unknown relocations, such that one can see which symbol it affects etc, to aid in debugging. References issue #14430. Signed-off-by: Jon Nordby <jononor@gmail.com>	2025-04-03 16:37:38 +11:00
Damien George	2db0c0225f	tools/mpremote: Make mip install skip /rom*/lib directories. ... If a ROMFS is mounted then "/rom/lib" is usually in `sys.path` before the writable filesystem's "lib" entry. The ROMFS directory cannot be installed to, so skip it if found. Signed-off-by: Damien George <damien@micropython.org>	2025-03-27 17:10:00 +11:00
Alessandro Gatti	31a008c6e2	tools/ci.sh: Do not assume the Python interpreter is called "python". ... This commit removes the assumption made by the CI scripts that the system-provided python executable is simply named "python". The scripts will now look for a binary called "python3" first, and then fall back to "python" if that is not found. Whilst this is currently the case for the CI environment, there are no guarantees for this going forward. For example minimal CI environments set up by some developers, using the same base OS, have their python executable called "python3". Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2025-03-17 13:04:53 +11:00
Alessandro Gatti	1a7cbac879	tools/ci.sh: Build Xtensa natmods as part of the CI process. ... This commit expands the CI tests by checking whether the example native modules are able to be built for the Xtensa architecture. This was made possible by the changes to mpy_ld that allow symbol resolution across standard compiler-provided libraries. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2025-03-17 13:04:53 +11:00
Volodymyr Shymanskyy	51976110e2	tools/mpy_ld.py: Allow linking static libraries. ... This commit introduces an additional symbol resolution mechanism to the natmod linking process. This allows the build scripts to look for required symbols into selected libraries that are provided by the compiler installation (libgcc and libm at the moment). For example, using soft-float code in natmods, whilst technically possible, was not an easy process and required some additional work to pull it off. With this addition all the manual (and error-prone) operations have been automated and folded into `tools/mpy_ld.py`. Both newlib and picolibc toolchains are supported, albeit the latter may require a bit of extra configuration depending on the environment the build process runs on. Picolibc's soft-float functions aren't in libm - in fact the shipped libm is nothing but a stub - but they are inside libc. This is usually not a problem as these changes cater for that configuration quirk, but on certain compilers the include paths used to find libraries in may not be updated to take Picolibc's library directory into account. The bare metal RISC-V compiler shipped with the CI OS image (GCC 10.2.0 on Ubuntu 22.04LTS) happens to exhibit this very problem. To work around that for CI builds, the Picolibc libraries' path is hardcoded in the Makefile directives used by the linker, but this can be changed by setting the PICOLIBC_ROOT environment library when building natmods. Signed-off-by: Volodymyr Shymanskyy <vshymanskyi@gmail.com> Co-authored-by: Alessandro Gatti <a.gatti@frob.it>	2025-03-17 13:03:27 +11:00
Damien George	0c98c60b68	tools/mpremote: Add romfs query, build and deploy commands. ... These commands use the `vfs.rom_ioctl()` function to manage the ROM partitions on a device, and create and deploy ROMFS images. Signed-off-by: Damien George <damien@micropython.org>	2025-03-06 12:52:35 +11:00
Volodymyr Shymanskyy	6fba1e406b	tools/mpy-tool.py: Support calling main() from an external script. ... Signed-off-by: Volodymyr Shymanskyy <vshymanskyi@gmail.com> Signed-off-by: Angus Gratton <angus@redyak.com.au>	2025-03-05 16:02:08 +11:00
Volodymyr Shymanskyy	e1b2f2e078	tools/mpy-tool.py: Add support for self-hosting of mpy-tool. ... This allows running mpy-tool using MicroPython itself. An appropriate test is added to CI to make sure it continues to work. Signed-off-by: Volodymyr Shymanskyy <vshymanskyi@gmail.com> Signed-off-by: Angus Gratton <angus@redyak.com.au>	2025-03-05 16:01:12 +11:00
Damien George	71c7c03e41	tools/mpremote/tests: Add test for RemoteFile.readline. ... Signed-off-by: Damien George <damien@micropython.org>	2025-02-28 14:14:58 +11:00
Andrew Leech	86526e9c2b	tools/mpremote: Optimise readline support in mount. ... This significantly speeds up readline on files opened directly from an mpremote mount. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>	2025-02-28 14:14:39 +11:00
Christian Clauss	dc2fcfcc55	all: Upgrade to ruff v0.9.6. ... Signed-off-by: Christian Clauss <cclauss@me.com>	2025-02-25 13:02:49 +01:00
Angus Gratton	8b1ed4473d	github/workflows: Include the Python version in the ESP-IDF cache key. ... Changing runner OS can change Python version, and ESP-IDF installs are keyed on ESP-IDF and Python version together. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2025-02-25 15:13:13 +11:00
Damien George	7e9d19e06b	github/workflows: Stop using ubuntu-20.04. ... For GitHub Actions, ubuntu-20.04 is deprecated and will be removed by 1st April 2025. See announcement at https://github.com/actions/runner-images/issues/11101 This commit changes actions that use ubuntu-20.04 to a newer image. Signed-off-by: Damien George <damien@micropython.org>	2025-02-25 15:11:43 +11:00
Glenn Moloney	2992e34956	tools/mpremote: Add support for relative urls in package.json files. ... URLs in `package.json` may now be specified relative to the base URL of the `package.json` file. Relative URLs wil work for `package.json` files installed from the web as well as local file paths. Docs: update `docs/reference/packages.rst` to add documentation for: - Installing packages from local filesystems (PR #12476); and - Using relative URLs in the `package.json` file (PR #12477); - Update the packaging example to encourage relative URLs as the default in `package.json`. Add `tools/mpremote/tests/test_mip_local_install.sh` to test the installation of a package from local files using relative URLs in the `package.json`. Signed-off-by: Glenn Moloney <glenn.moloney@gmail.com>	2025-02-24 13:29:40 +11:00
Angus Gratton	1034b17558	tools/ci.sh: Build the W5100S_EVB_PICO board with no threads. ... Serves as a build test for a config we don't otherwise support. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2025-02-20 00:57:45 +11:00
rufusclark	1a67d720c7	tools/pyboard.py: Make get_time use machine.RTC instead of pyb.RTC. ... The current code evaluates `pyb.RTC().datetime()` resulting in a remote side exception, as `pyb` is not defined on most ports (only stm32). The code should evaluate `machine.RTC().datetime()` and hence return the current time. Signed-off-by: rufusclark <50201718+rufusclark@users.noreply.github.com> Signed-off-by: Damien George <damien@micropython.org>	2025-02-12 12:53:46 +11:00
Glenn Moloney	11c9656fad	tools/mpremote: Support mip install from package.json on local fs. ... Add support for `mpremote mip install package.json` where `package.json` is a json file on the local filesystem. Without this, package json files can only be loaded from http, https, github or gitlab URLs. This is useful for testing `package.json` files for pacages in development and for constructing one's own `package.json` files for Python packages which are not yet available for installation using mip. Signed-off-by: Glenn Moloney <glenn.moloney@gmail.com>	2025-02-10 15:59:58 +11:00
iabdalkader	752c1672af	tools/ci.sh: Build MIMXRT1060_EVK with MSC enabled as part of mimxrt CI. ... Signed-off-by: iabdalkader <i.abdalkader@gmail.com>	2025-02-10 12:53:45 +11:00
robert-hh	e574f68820	mimxrt: Add support for a UF2 bootloader. ... Allowing to use e.g. the Adafruit bootloaders with MicroPython. The .uf2 file is created in addition to the .bin and .hex files allowing to use the latter ones without the bootloader for debugging and testing. Changes: - Set the location of the ISR Vector and .text segment to 0x6000C000 and 0x6000C400. - Reserve an area at the start of ITCM for a copy of the interrupt vector table and copy the table on reset to this place. - Extend `machine.bootloader()` by setting the magic number to enable the bootloader on reset. - Create a .uf2 file which skips the segments below 0x6000C000. The bootloader has to be installed as a preparation step using the board specific methods, but then the firmware's .uf2 file version can be installed using the bootloader. The bootloader can be invoked with: - double reset - calling machine.bootloader() - Using the touch1200 method Double reset is hard to achieve on MIMXRT boards, since there is no clean reset pin. Some MIMXRT boards provide it by switching the power. Some boards are excluded from the .uf2 build: - MIMXRT1050_EVK: The uf2 bootloader is built for the QSPI version of the board. MicroPython supports the Hyperflash version. - MIMXRT1176_EVK: No support for this board yet, but it should be possible. Signed-off-by: robert-hh <robert@hammelrath.com>	2025-02-10 11:28:39 +11:00
Alessandro Gatti	f594c6f66e	tools/ci.sh: Add natmod tests for QEMU/Arm. ... This commit adds the natmod tests for the MPS2_AN385 board running inside QEMU to the CI pipeline. Now natmod tests capabilities are equal between the Arm and RV32 platforms for the QEMU port. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2025-02-07 17:37:04 +11:00
Hans Maerki	03fe9c55ea	tools/mpremote: Introduce timeout_overall for read_until(). ... And use it in `enter_raw_repl()`. This prevents waiting forever for a serial device that does not respond to the Ctrl-C/Ctrl-D/etc commands and is constantly outputting data. Signed-off-by: Hans Maerki <buhtig.hans.maerki@ergoinfo.ch>	2025-01-29 10:45:08 +11:00
Hans Maerki	0d46e45a1f	tools/mpremote: Avoid initial blocking read in read_until(). ... If the target does not return any data then `read_until()` will block indefinitely. Fix this by making the initial read part of the general read look, which always checks `inWaiting() > 0` before reading from the serial device. Also added the UART timeout to the constructor. This is not currently used but may be used as an additional safeguard. Signed-off-by: Hans Maerki <buhtig.hans.maerki@ergoinfo.ch>	2025-01-27 10:20:40 +11:00
Angus Gratton	d89e71e6c0	tools/autobuild,esp32: Template the generation of esp32 port deploy.md. ... Allows two source files (ports/esp32/boards/deploy.md and deploy_nativeusb.md for boards with only native USB) for all esp32 installation steps, with templated chip name and flash offset inserted via string formatting. The new files add more text to explain the esptool.py port auto-detection, remove the unnecessary -z feature (already enabled by default), and add a bit of troubleshooting and port detection info. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2025-01-17 16:03:45 +11:00
Alessandro Gatti	0950f65ac4	tools/pyboard.py: Wait a bit before accessing the PTY serial port. ... Some PTY targets, namely `NETDUINO2` and `MICROBIT` under Qemu, take a bit more time to present a REPL than usual. The pyboard tool is a bit too impatient and would bail out before any of those targets had a chance to respond to the raw REPL request. Co-authored-by: Damien George <damien@micropython.org> Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2025-01-16 12:10:28 +11:00
Damien George	d533c9067a	tools/autobuild: Don't allow a board to change its ID. ... All board IDs are now the board directory name. Signed-off-by: Damien George <damien@micropython.org>	2025-01-15 16:48:10 +11:00
Damien George	495ce91cae	tools/ci.sh: Run test_full for qemu port CI. ... Signed-off-by: Damien George <damien@micropython.org>	2025-01-06 17:57:48 +11:00
Alessandro Gatti	4729a89504	tools/verifygitlog.py: Show invalid commit subjects in quotes. ... If a commit subject line has any trailing whitespace it won't match the repository validation rules, and the line will show up as part of the relevant error message. However, since there's no quotation marks around the offending text, the trailing whitespace may go unnoticed, and given that the commit message is then discarded when the commit operation is retried this can get fairly annoying. This commit simply modifies the error output for invalid subject lines to add quotation marks around the offending text, so trailing whitespace is much easier to see. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2024-12-23 12:55:20 +11:00
Alessandro Gatti	7ca6e5eb68	qemu: Add test_natmod target for RV32 and use as part of CI pipeline. ... This commit brings the natmod tests in the CI build process for the RV32 platform. Not all example natmods are tested at the moment, as `features` requires soft-float support, and `btree` needs thread-local storage support in `mpy_ld.py` when built with the CI's toolchain. Co-authored-by: Damien George <damien@micropython.org> Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2024-12-23 10:04:19 +11:00
Alessandro Gatti	6760e00817	tools/mpy_ld.py: Add native modules support for RV32 code. ... This commit adds support for RV32IMC native modules, as in embedding native code into a self-contained MPY module and and make its exported functions available to the MicroPython environment. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>	2024-12-23 10:02:20 +11:00
Angus Gratton	a9945fc528	ci: Add caching of ccache for Zephyr. ... Similar to the ESP32 builds, but needs additional step to pass the ccache directory through to the Zephyr container. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2024-12-18 15:11:01 +11:00
Angus Gratton	db4b095644	ci: Pull the Zephyr CI docker image from GitHub container reg. ... This image is 15GB so in theory this may be faster, although in testing the improvement is either non-existent or marginal. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2024-12-18 15:11:01 +11:00
Angus Gratton	92a5ea51b4	ci: Cache Zephyr workspace installation. ... Can save several minutes downloading the Zephyr docker image and/or cloning repo from GitHub. Cache keyed on the Zephyr version, which AFAIK is the only determinant for the workspace contents. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2024-12-18 15:11:01 +11:00
Damien George	3203e950fc	tools/boardgen.py: Provide macro defns for number of cpu/board pins. ... So a port can use them if needed to exclude the Pin.cpu/Pin.board objects. Signed-off-by: Damien George <damien@micropython.org>	2024-12-17 23:53:24 +11:00
Damien George	ee8d8b3448	tools/ci.sh: Re-enable vfs_posix tests on unix qemu MIPS CI. ... These work now that _FILE_OFFSET_BITS=64. Signed-off-by: Damien George <damien@micropython.org>	2024-12-16 11:58:43 +11:00
Angus Gratton	d49c1e836c	tools/ci.sh: Remove explicit macOS pkg-config install. ... Reasons to remove this: - GitHub's macOS runners install this package by default nowadays. - Brew renamed this package to 'pkgconf' so installing the old name on top of the new package name has started failing. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2024-11-30 11:24:36 +11:00
Damien George	4fd5b72a8b	tools/mpremote: Support trailing slash on dest for non-recursive copy. ... This fixes a regression in db59e55fe7: prior to that commit `mpremote` supported trailing slashes on the destination of a normal (non-recursive) copy. Add back support for that, with the semantics that a trailing slash requires the destination to be an existing directory. Also add a test for this. Signed-off-by: Damien George <damien@micropython.org>	2024-11-13 11:51:35 +11:00
Damien George	3b6024a699	tools/mpremote: Add test for forced copy. ... Signed-off-by: Damien George <damien@micropython.org>	2024-11-13 11:51:35 +11:00
Damien George	76e6c6345c	tools/mpremote: Make sure stdout and stderr output appear in order. ... mpremote error messages now go to stderr, so make sure stdout is flushed before printing them. Also update the test runner to capture error messages. Signed-off-by: Damien George <damien@micropython.org>	2024-11-13 11:51:35 +11:00
Damien George	8978102f35	tests/run-tests.py: Change --target/--device options to --test-instance. ... Previously to this commit, running the test suite on a bare-metal board required specifying the target (really platform) and device, eg: $ ./run-tests.py --target pyboard --device /dev/ttyACM1 That's quite a lot to type, and you also need to know what the target platform is, when a lot of the time you either don't care or it doesn't matter. This commit makes it easier to run the tests by replacing both of these options with a single `--test-instance` (`-t` for short) option. That option specifies the executable/port/device to test. Then the target platform is automatically detected. The `--test-instance` can be passed: - "unix" (the default) to use the unix version of MicroPython - "webassembly" to test the webassembly port - anything else is considered a port/device to pass to Pyboard There are also some shortcuts to specify a port/device, following `mpremote`: - a<n> is short for /dev/ttyACM<n> - u<n> is short for /dev/ttyUSB<n> - c<n> is short for COM<n> For example: $ ./run-tests.py -t a1 Note that the default test instance is "unix" and so this commit does not change the standard way to run tests on the unix port, by just doing `./run-tests.py`. As part of this change, the platform (and it's native architecture if it supports importing native .mpy files) is show at the start of the test run. Signed-off-by: Damien George <damien@micropython.org>	2024-11-04 12:47:47 +11:00
Angus Gratton	787c424cfc	tools/ci.sh: Fix reference commit for code size comparison. ... Previously the code size comparison was between the merge base (i.e. where the PR branched), and the generated merge commit into master. If the PR branch was older than current master, this meant the size comparison could incorrectly include changes already merged on master but missing from the PR branch. This commit changes it to compare the generated merge commit against current master, i.e. the size impact if this PR was to be merged. This commit also disables running the code size check on "push", it now only runs on pull_request events. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2024-10-31 23:19:55 +11:00
Angus Gratton	9591b0a53c	tools/ci.sh: Fix commit msg checking when PR branch HEAD behind master. ... Fixes the problem noted at https://github.com/micropython/micropython/pull/15547#issuecomment-2434479702 which is that, because default CI HEAD for a PR is a (generated) merge commit into the master branch's current HEAD, then if the PR branch isn't fully rebased then the commit check runs against commits from master as well! Also drops running this check on push, the pull_request event is triggered by default on open and update ("synchronized" event), which probably covers the cases where this check should run. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2024-10-31 23:16:25 +11:00
Glenn Moloney	9ea8d2a031	tools/mpremote: Fix UnboundLocalError in Transport.fs_writefile(). ... The variable `written` was being used before it was defined in the `fs_writefile()` method of the Transport class. This was causing an `UnboundLocalError` to be raised when the `progress_callback` was not provided. Fixes issue #16084. Signed-off-by: Glenn Moloney <glenn.moloney@gmail.com>	2024-10-28 12:23:10 +11:00
Damien George	f9cebe676e	tools/ci.sh: Add RPI_PICO2 to CI. ... Signed-off-by: Damien George <damien@micropython.org>	2024-10-15 12:21:10 +11:00
Damien George	d92694c3e8	tools: Only issue a single Ctrl-C when entering raw REPL. ... A long time ago when there was only the `stm` port, Ctrl-C would trigger a preemptive NLR jump to break out of running code. Then in commit 124df6f8d0 a more general approach to asynchronous `KeyboardInterrupt` exceptions was implemented, and `stmhal` supported both approaches, with the general (soft) interrupt taking priority. Then in commit bc1488a05f `pyboard.py` was updated with a corresponding change to make it issue a double Ctrl-C to break out of any existing code when entering the raw REPL (two Ctrl-C characters were sent in order to more reliably trigger the preemptive NLR jump). No other port has preemptive NLR jumps and so a double Ctrl-C doesn't really behave any differently to a single Ctrl-C: with USB CDC the double Ctrl-C would most likely be in the same USB packet and so processed in the same low-level USB callback, so it's just setting the keyboard interrupt flag twice in a row. The VM/runtime then just sees one keyboard interrupt and acts as though only one Ctrl-C was sent. This commit changes the double Ctrl-C to a single Ctrl-C in `pyboard.py` and `mpremote`. That keeps things as simple as they need to be. Signed-off-by: Damien George <damien@micropython.org>	2024-10-15 11:40:08 +11:00
Damien George	a25b6b9c65	tools/mpremote: Add option to force copy. ... This adds a -f/--force option to the "cp" command, which forces unconditional copies, in particular does not check the hash. Signed-off-by: Damien George <damien@micropython.org>	2024-10-09 16:39:06 +11:00
Jim Mussared	6461ffd9d1	tools/mpremote: Add initial regression tests for mpremote. ... These tests are specifically for the command-line interface and cover: - resume/soft-reset/connect/disconnect - mount - fs cp,touch,mkdir,cat,sha256sum,rm,rmdir - eval/exec/run This work was funded through GitHub Sponsors. Signed-off-by: Jim Mussared <jim.mussared@gmail.com> Signed-off-by: Damien George <damien@micropython.org>	2024-10-09 16:39:00 +11:00
Jim Mussared	dd6f78f014	tools/mpremote: Improve error output. ... Makes the filesystem command give standard error messages rather than just printing the exception from the device. Makes the distinction between CommandError and TransportError clearer. This work was funded through GitHub Sponsors. Signed-off-by: Jim Mussared <jim.mussared@gmail.com>	2024-10-09 16:38:08 +11:00