28 Commits

Author	SHA1	Message	Date
Damien George	0c7726a89c	all: Use enum instead of bool argument to mp_handle_pending. ... This should be a no-op because the enum and bool have the same numeric value. Signed-off-by: Damien George <damien@micropython.org>	2026-02-05 00:29:36 +11:00
Andrew Leech	a6864109db	shared/runtime/pyexec: Set PYEXEC_FORCED_EXIT flag for SystemExit. ... When `MICROPY_PYEXEC_ENABLE_EXIT_CODE_HANDLING` is enabled, `SystemExit` now sets the `PYEXEC_FORCED_EXIT` flag in addition to the exit code. This allows the REPL to properly detect and exit when SystemExit is raised, while still preserving the exit code in the lower bits. Fixes `repl_lock.py` test which expects REPL to exit on `SystemExit`. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>	2025-11-22 00:06:59 +11:00
Andrew Leech	5f815b8a2d	unix: Enable exit code handling for sys.exit(). ... Enable `MICROPY_PYEXEC_ENABLE_EXIT_CODE_HANDLING` to propagate `sys.exit()` exit codes properly. Update `convert_pyexec_result()` to handle return values where pyexec returns the exit code with `PYEXEC_FORCED_EXIT` flag set for `SystemExit`. Extract the exit code from the lower 8 bits when the flag is set, otherwise return as-is (0 for success, 1 for exception). Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>	2025-11-22 00:06:59 +11:00
Damien George	fd1ddc3f12	shared/runtime/pyexec: Call mp_hal_stdio_mode_orig/raw as appropriate. ... This ensures that ctrl-C works on the unix port when executing code at the REPL. Signed-off-by: Damien George <damien@micropython.org>	2025-11-22 00:06:59 +11:00
Andrew Leech	7b3a0fc6b7	shared/runtime/pyexec: Provide support for compile-only mode. ... When `MICROPY_PYEXEC_COMPILE_ONLY` is enabled and the global `mp_compile_only` is True, code is compiled but not executed. Also add comprehensive tests for compile-only functionality covering both successful compilation and syntax error detection. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>	2025-11-22 00:06:58 +11:00
Andrew Leech	3b2b8dd538	shared/runtime/pyexec: Set __file__ for file input when enabled. ... When `MICROPY_MODULE___FILE__` is enabled and parsing file input, set the global `__file__` variable to the source filename. This matches the behavior of the unix port and provides the current filename to the executing script. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>	2025-11-22 00:01:28 +11:00
Andrew Leech	5b3c928f53	unix/main: Use standard pyexec REPL for unix and windows ports. ... This improves REPL usage consistency across ports, by utilizing the pyexec code for the unix REPL. Only enabled when MICROPY_USE_READLINE == 1 (the default). Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>	2025-11-22 00:00:07 +11:00
John Smith	1e5da2642c	shared/runtime: Set exit code according to the SystemExit exception. ... Add abort setup code `nlr_set_abort` to the standard runtime executor. This makes the standard runtime respond to abort signal without any further modifications. - When aborted, the program exits with 137 exit code (configurable, same as posix sig abort), to differentiate from a normal shutdown. - When exited by exception/crash, the program will exit with exit code 1 (configurable). - When exited by exception KeyboardInterrupt, the program will exit with exit code 130 (configurable, same as posix sig int). - When exited with a exit code (from Python environment), this code is propagated. When a different object is passed, exit code is set to 1 and the value printed, to be consistent with Python docs: https://python.readthedocs.io/en/latest/library/exceptions.html#SystemExit Signed-off-by: John Smith <jsmith@jsmith.cz>	2025-11-04 16:47:19 +11:00
Andrew Leech	6b0e1c2701	shared/runtime/pyexec: Remove legacy USB IRQ enable code. ... It's very STM32 USB stack specific and doesn't generalise well to other ports. So remove it. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>	2025-10-31 10:51:07 +11:00
Jeff Epler	4db2cf09e0	shared/runtime/pyexec: Unconditionally reset lock depth. ... Saves code size for the same functionality. Signed-off-by: Jeff Epler <jepler@gmail.com>	2025-09-02 13:41:20 +10:00
iabdalkader	f187c77da8	shared/runtime/pyexec: Add helper function to execute a vstr. ... Add `pyexec_vstr()` to execute Python code from a vstr source. Signed-off-by: iabdalkader <i.abdalkader@gmail.com>	2025-03-14 17:10:35 +11:00
Damien George	69c25ea865	shared/runtime/pyexec: Make a raised SystemExit always do a forced exit. ... The current situation with SystemExit and soft reset is the following: - `sys.exit()` follows CPython and just raises `SystemExit`. - On the unix port, raising `SystemExit` quits the application/MicroPython, whether at the REPL or in code (this follows CPython behaviour). - On bare-metal ports, raising `SystemExit` at the REPL does nothing, raising it in code will stop the code and drop into the REPL. - `machine.soft_reset()` raises `SystemExit` but with a special flag set, and bare-metal targets check this flag when it propagates to the top-level and do a soft reset when they receive it. The original idea here was that a bare-metal target can't "quit" like the unix port can, and so dropping to the REPL was considered the same as "quit". But this bare-metal behaviour is arguably inconsistent with unix, and "quit" should mean terminate everything, including REPL access. This commit changes the behaviour to the following, which is more consistent: - Raising `SystemExit` on a bare-metal port will do a soft reset (unless the exception is caught by the application). - `machine.soft_reset()` is now equivalent to `sys.exit()`. - unix port behaviour remains unchanged. Tested running the test suite on an stm32 board and everything still passes, in particular tests that skip by raising `SystemExit` still correctly skip. Signed-off-by: Damien George <damien@micropython.org>	2024-07-20 12:13:14 +10:00
Angus Gratton	decf8e6a8b	all: Remove the "STATIC" macro and just use "static" instead. ... The STATIC macro was introduced a very long time ago in commit d5df6cd44a. The original reason for this was to have the option to define it to nothing so that all static functions become global functions and therefore visible to certain debug tools, so one could do function size comparison and other things. This STATIC feature is rarely (if ever) used. And with the use of LTO and heavy inline optimisation, analysing the size of individual functions when they are not static is not a good representation of the size of code when fully optimised. So the macro does not have much use and it's simpler to just remove it. Then you know exactly what it's doing. For example, newcomers don't have to learn what the STATIC macro is and why it exists. Reading the code is also less "loud" with a lowercase static. One other minor point in favour of removing it, is that it stops bugs with `STATIC inline`, which should always be `static inline`. Methodology for this commit was: 1) git ls-files | egrep '\.[ch]$' | \ xargs sed -Ei "s/(^| )STATIC($| )/\1static\2/" 2) Do some manual cleanup in the diff by searching for the word STATIC in comments and changing those back. 3) "git-grep STATIC docs/", manually fixed those cases. 4) "rg -t python STATIC", manually fixed codegen lines that used STATIC. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>	2024-03-07 14:20:42 +11:00
Damien George	e2ff00e811	py/emitglue: Introduce mp_proto_fun_t as a more general mp_raw_code_t. ... Allows bytecode itself to be used instead of an mp_raw_code_t in the simple and common cases of a bytecode function without any children. This can be used to further reduce frozen code size, and has the potential to optimise other areas like importing. Signed-off-by: Damien George <damien@micropython.org>	2024-02-16 14:17:01 +11:00
Jim Mussared	5015779a6f	py/builtinevex: Handle invalid filenames for execfile. ... If a non-string buffer was passed to execfile, then it would be passed as a non-null-terminated char* to mp_lexer_new_from_file. This changes mp_lexer_new_from_file to take a qstr instead (as in almost all cases a qstr will be created from this input anyway to set the `__file__` attribute on the module). This now makes execfile require a string (not generic buffer) argument, which is probably a good fix to make anyway. Fixes issue #12522. This work was funded through GitHub Sponsors. Signed-off-by: Jim Mussared <jim.mussared@gmail.com>	2023-10-12 15:17:59 +11:00
Damien George	b1229efbd1	all: Fix spelling mistakes based on codespell check. ... Signed-off-by: Damien George <damien@micropython.org>	2023-04-27 18:03:06 +10:00
David Grayson	c046b23ea2	shared/runtime/pyexec: Don't allow Ctrl+C to interrupt frozen boot code. ... Helps prevent the filesystem from getting formatted by mistake, among other things. For example, on a Pico board, entering Ctrl+D and Ctrl+C fast many times will eventually wipe the filesystem (without warning or notice). Further rationale: Ctrl+C is used a lot by automation scripts (eg mpremote) and UI's (eg Mu, Thonny) to get the board into a known state. If the board is not responding for a short time then it's not possible to know if it's just a slow start up (eg in _boot.py), or an infinite loop in the main application. The former should not be interrupted, but the latter should. The only way to distinguish these two cases would be to wait "long enough", and if there's nothing on the serial after "long enough" then assume it's running the application and Ctrl+C should break out of it. But defining "long enough" is impossible for all the different boards and their possible behaviour. The solution in this commit is to make it so that frozen start-up code cannot be interrupted by Ctrl+C. That code then effectively acts like normal C start-up code, which also cannot be interrupted. Note: on the stm32 port this was never seen as an issue because all start-up code is in C. But now other ports start to put more things in _boot.py and so this problem crops up. Signed-off-by: David Grayson <davidegrayson@gmail.com>	2023-04-05 10:38:50 +10:00
Damien George	b3c8ab37ec	py/gc: Make gc_dump_info/gc_dump_alloc_table take a printer as argument. ... So that callers can redirect the output if needed. Signed-off-by: Damien George <damien@micropython.org>	2023-03-10 10:58:10 +11:00
Damien George	1978b838b7	shared/runtime: Use mp_printf consistently, instead of printf. ... Signed-off-by: Damien George <damien@micropython.org>	2023-01-24 16:57:26 +11:00
David Lechner	68f46342aa	shared/runtime/pyexec: Use MP_REGISTER_ROOT_POINTER(). ... This uses MP_REGISTER_ROOT_POINTER() to register repl_line instead of using a conditional inside of mp_state_vm_t. Signed-off-by: David Lechner <david@pybricks.com>	2022-07-18 13:52:01 +10:00
Damien George	402df833fe	py/modsys: Introduce sys.implementation._machine constant. ... This contains a string useful for identifying the underlying machine. This string is kept consistent with the second part of the REPL banner via the new config option MICROPY_BANNER_MACHINE. This makes os.uname() more or less redundant, as all the information in os.uname() is now available in the sys module. Signed-off-by: Damien George <damien@micropython.org>	2022-04-28 17:23:03 +10:00
Damien George	40047823bc	py/modsys: Append MicroPython git version and build date to sys.version. ... This commit adds the git hash and build date to sys.version. This is allowed according to CPython docs, and is what PyPy does. The docs state: A string containing the version number of the Python interpreter plus additional information on the build number and compiler used. Eg on CPython: Python 3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.version '3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0]' and PyPy: Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30) [PyPy 5.6.0 with GCC 6.2.0 20161109] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>> import sys >>>> sys.version '2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)\n[PyPy 5.6.0 with GCC ... With this commit on MicroPython we now have: MicroPython v1.18-371-g9d08eb024 on 2022-04-28; linux [GCC 11.2.0] v... Use Ctrl-D to exit, Ctrl-E for paste mode >>> import sys >>> sys.version '3.4.0; MicroPython v1.18-371-g9d08eb024 on 2022-04-28' Note that the start of the banner is the same as the end of sys.version. This helps to keep code size under control because the string can be reused by the compiler. Signed-off-by: Damien George <damien@micropython.org>	2022-04-28 15:23:17 +10:00
Damien George	ac2293161e	py/modsys: Add optional mutable attributes sys.ps1/ps2 and use them. ... This allows customising the REPL prompt strings. Signed-off-by: Damien George <damien@micropython.org>	2022-03-10 10:58:33 +11:00
Damien George	f2040bfc7e	py: Rework bytecode and .mpy file format to be mostly static data. ... Background: .mpy files are precompiled .py files, built using mpy-cross, that contain compiled bytecode functions (and can also contain machine code). The benefit of using an .mpy file over a .py file is that they are faster to import and take less memory when importing. They are also smaller on disk. But the real benefit of .mpy files comes when they are frozen into the firmware. This is done by loading the .mpy file during compilation of the firmware and turning it into a set of big C data structures (the job of mpy-tool.py), which are then compiled and downloaded into the ROM of a device. These C data structures can be executed in-place, ie directly from ROM. This makes importing even faster because there is very little to do, and also means such frozen modules take up much less RAM (because their bytecode stays in ROM). The downside of frozen code is that it requires recompiling and reflashing the entire firmware. This can be a big barrier to entry, slows down development time, and makes it harder to do OTA updates of frozen code (because the whole firmware must be updated). This commit attempts to solve this problem by providing a solution that sits between loading .mpy files into RAM and freezing them into the firmware. The .mpy file format has been reworked so that it consists of data and bytecode which is mostly static and ready to run in-place. If these new .mpy files are located in flash/ROM which is memory addressable, the .mpy file can be executed (mostly) in-place. With this approach there is still a small amount of unpacking and linking of the .mpy file that needs to be done when it's imported, but it's still much better than loading an .mpy from disk into RAM (although not as good as freezing .mpy files into the firmware). The main trick to make static .mpy files is to adjust the bytecode so any qstrs that it references now go through a lookup table to convert from local qstr number in the module to global qstr number in the firmware. That means the bytecode does not need linking/rewriting of qstrs when it's loaded. Instead only a small qstr table needs to be built (and put in RAM) at import time. This means the bytecode itself is static/constant and can be used directly if it's in addressable memory. Also the qstr string data in the .mpy file, and some constant object data, can be used directly. Note that the qstr table is global to the module (ie not per function). In more detail, in the VM what used to be (schematically): qst = DECODE_QSTR_VALUE; is now (schematically): idx = DECODE_QSTR_INDEX; qst = qstr_table[idx]; That allows the bytecode to be fixed at compile time and not need relinking/rewriting of the qstr values. Only qstr_table needs to be linked when the .mpy is loaded. Incidentally, this helps to reduce the size of bytecode because what used to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices. If the module uses the same qstr more than two times then the bytecode is smaller than before. The following changes are measured for this commit compared to the previous (the baseline): - average 7%-9% reduction in size of .mpy files - frozen code size is reduced by about 5%-7% - importing .py files uses about 5% less RAM in total - importing .mpy files uses about 4% less RAM in total - importing .py and .mpy files takes about the same time as before The qstr indirection in the bytecode has only a small impact on VM performance. For stm32 on PYBv1.0 the performance change of this commit is: diff of scores (higher is better) N=100 M=100 baseline -> this-commit diff diff% (error%) bm_chaos.py 371.07 -> 357.39 : -13.68 = -3.687% (+/-0.02%) bm_fannkuch.py 78.72 -> 77.49 : -1.23 = -1.563% (+/-0.01%) bm_fft.py 2591.73 -> 2539.28 : -52.45 = -2.024% (+/-0.00%) bm_float.py 6034.93 -> 5908.30 : -126.63 = -2.098% (+/-0.01%) bm_hexiom.py 48.96 -> 47.93 : -1.03 = -2.104% (+/-0.00%) bm_nqueens.py 4510.63 -> 4459.94 : -50.69 = -1.124% (+/-0.00%) bm_pidigits.py 650.28 -> 644.96 : -5.32 = -0.818% (+/-0.23%) core_import_mpy_multi.py 564.77 -> 581.49 : +16.72 = +2.960% (+/-0.01%) core_import_mpy_single.py 68.67 -> 67.16 : -1.51 = -2.199% (+/-0.01%) core_qstr.py 64.16 -> 64.12 : -0.04 = -0.062% (+/-0.00%) core_yield_from.py 362.58 -> 354.50 : -8.08 = -2.228% (+/-0.00%) misc_aes.py 429.69 -> 405.59 : -24.10 = -5.609% (+/-0.01%) misc_mandel.py 3485.13 -> 3416.51 : -68.62 = -1.969% (+/-0.00%) misc_pystone.py 2496.53 -> 2405.56 : -90.97 = -3.644% (+/-0.01%) misc_raytrace.py 381.47 -> 374.01 : -7.46 = -1.956% (+/-0.01%) viper_call0.py 576.73 -> 572.49 : -4.24 = -0.735% (+/-0.04%) viper_call1a.py 550.37 -> 546.21 : -4.16 = -0.756% (+/-0.09%) viper_call1b.py 438.23 -> 435.68 : -2.55 = -0.582% (+/-0.06%) viper_call1c.py 442.84 -> 440.04 : -2.80 = -0.632% (+/-0.08%) viper_call2a.py 536.31 -> 532.35 : -3.96 = -0.738% (+/-0.06%) viper_call2b.py 382.34 -> 377.07 : -5.27 = -1.378% (+/-0.03%) And for unix on x64: diff of scores (higher is better) N=2000 M=2000 baseline -> this-commit diff diff% (error%) bm_chaos.py 13594.20 -> 13073.84 : -520.36 = -3.828% (+/-5.44%) bm_fannkuch.py 60.63 -> 59.58 : -1.05 = -1.732% (+/-3.01%) bm_fft.py 112009.15 -> 111603.32 : -405.83 = -0.362% (+/-4.03%) bm_float.py 246202.55 -> 247923.81 : +1721.26 = +0.699% (+/-2.79%) bm_hexiom.py 615.65 -> 617.21 : +1.56 = +0.253% (+/-1.64%) bm_nqueens.py 215807.95 -> 215600.96 : -206.99 = -0.096% (+/-3.52%) bm_pidigits.py 8246.74 -> 8422.82 : +176.08 = +2.135% (+/-3.64%) misc_aes.py 16133.00 -> 16452.74 : +319.74 = +1.982% (+/-1.50%) misc_mandel.py 128146.69 -> 130796.43 : +2649.74 = +2.068% (+/-3.18%) misc_pystone.py 83811.49 -> 83124.85 : -686.64 = -0.819% (+/-1.03%) misc_raytrace.py 21688.02 -> 21385.10 : -302.92 = -1.397% (+/-3.20%) The code size change is (firmware with a lot of frozen code benefits the most): bare-arm: +396 +0.697% minimal x86: +1595 +0.979% [incl +32(data)] unix x64: +2408 +0.470% [incl +800(data)] unix nanbox: +1396 +0.309% [incl -96(data)] stm32: -1256 -0.318% PYBV10 cc3200: +288 +0.157% esp8266: -260 -0.037% GENERIC esp32: -216 -0.014% GENERIC[incl -1072(data)] nrf: +116 +0.067% pca10040 rp2: -664 -0.135% PICO samd: +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS As part of this change the .mpy file format version is bumped to version 6. And mpy-tool.py has been improved to provide a good visualisation of the contents of .mpy files. In summary: this commit changes the bytecode to use qstr indirection, and reworks the .mpy file format to be simpler and allow .mpy files to be executed in-place. Performance is not impacted too much. Eventually it will be possible to store such .mpy files in a linear, read-only, memory- mappable filesystem so they can be executed from flash/ROM. This will essentially be able to replace frozen code for most applications. Signed-off-by: Damien George <damien@micropython.org>	2022-02-24 18:08:43 +11:00
iabdalkader	644f4dcc94	shared/runtime/pyexec: Cleanup EXEC_FLAG flag constants. ... - Cleanup pyexec flags definitions so it's clear they are different. - Use mp_uint_t for exec_flags because it should be unsigned.	2022-01-06 13:06:39 +11:00
Jim Mussared	e0bf4611c3	py: Only search frozen modules when '.frozen' is found in sys.path. ... This changes makemanifest.py & mpy-tool.py to merge string and mpy names into the same list (now mp_frozen_names). The various paths for loading a frozen module (mp_find_frozen_module) and checking existence of a frozen module (mp_frozen_stat) use a common function that searches this list. In addition, the frozen lookup will now only take place if the path starts with ".frozen", which needs to be added to sys.path. This fixes issues #1804, #2322, #3509, #6419. Signed-off-by: Jim Mussared <jim.mussared@gmail.com>	2021-12-18 00:01:59 +11:00
Damien George	136369d72f	all: Update to point to files in new shared/ directory. ... Signed-off-by: Damien George <damien@micropython.org>	2021-07-12 17:08:10 +10:00
Damien George	4d546713ec	shared: Introduce new top-level dir and move 1st party lib code there. ... This commit moves all first-party code developed for this project from lib/ to shared/, so that lib/ now only contains third-party code. The following directories are moved as-is from lib to shared: lib/libc -> shared/libc lib/memzip -> shared/memzip lib/netutils -> shared/netutils lib/timeutils -> shared/timeutils lib/upytesthelper -> shared/upytesthelper All files in lib/embed/ have been moved to shared/libc/. lib/mp-readline has been moved to shared/readline. lib/utils has been moved to shared/runtime, with the exception of lib/utils/printf.c which has been moved to shared/libc/printf.c. Signed-off-by: Damien George <damien@micropython.org>	2021-07-12 17:07:36 +10:00