aboutsummaryrefslogtreecommitdiff
path: root/py/compile.c
AgeCommit message (Collapse)Author
2022-05-23py/emitnative: Access qstr values using indirection table qstr_table.Damien George
This changes the native emitter to access qstr values using the qstr indirection table qstr_table, but only when generating native code that will be saved to a .mpy file. This makes the resulting native code fully static, ie it does not require any fix-ups or rewriting when it is imported. The performance of native code is more or less unchanged. Benchmark results on PYBv1.0 (using --via-mpy and --emit native) are: N=100 M=100 baseline -> this-commit diff diff% (error%) bm_chaos.py 407.16 -> 411.85 : +4.69 = +1.152% (+/-0.01%) bm_fannkuch.py 100.89 -> 101.20 : +0.31 = +0.307% (+/-0.01%) bm_fft.py 3521.17 -> 3441.72 : -79.45 = -2.256% (+/-0.00%) bm_float.py 6707.29 -> 6644.83 : -62.46 = -0.931% (+/-0.00%) bm_hexiom.py 55.91 -> 55.41 : -0.50 = -0.894% (+/-0.00%) bm_nqueens.py 5343.54 -> 5326.17 : -17.37 = -0.325% (+/-0.00%) bm_pidigits.py 603.89 -> 632.79 : +28.90 = +4.786% (+/-0.33%) core_qstr.py 64.18 -> 64.09 : -0.09 = -0.140% (+/-0.01%) core_yield_from.py 313.61 -> 311.11 : -2.50 = -0.797% (+/-0.03%) misc_aes.py 654.29 -> 659.75 : +5.46 = +0.834% (+/-0.02%) misc_mandel.py 4205.10 -> 4272.08 : +66.98 = +1.593% (+/-0.01%) misc_pystone.py 3077.79 -> 3128.39 : +50.60 = +1.644% (+/-0.01%) misc_raytrace.py 388.45 -> 393.71 : +5.26 = +1.354% (+/-0.01%) viper_call0.py 576.83 -> 566.76 : -10.07 = -1.746% (+/-0.05%) viper_call1a.py 550.39 -> 540.12 : -10.27 = -1.866% (+/-0.11%) viper_call1b.py 438.32 -> 432.09 : -6.23 = -1.421% (+/-0.11%) viper_call1c.py 442.96 -> 436.11 : -6.85 = -1.546% (+/-0.08%) viper_call2a.py 536.31 -> 527.37 : -8.94 = -1.667% (+/-0.04%) viper_call2b.py 378.99 -> 377.50 : -1.49 = -0.393% (+/-0.08%) Signed-off-by: Damien George <damien@micropython.org>
2022-05-18py/compile: De-duplicate constant objects in module's constant table.Damien George
The recent rework of bytecode made all constants global with respect to the module (previously, each function had its own constant table). That means the constant table for a module is shared among all functions/methods/etc within the module. This commit add support to the compiler to de-duplicate constants in this module constant table. So if a constant is used more than once -- eg 1.0 or (None, None) -- then the same object is reused for all instances. For example, if there is code like `print(1.0, 1.0)` then the parser will create two independent constants 1.0 and 1.0. The compiler will then (with this commit) notice they are the same and only put one of them in the constant table. The bytecode will then reuse that constant twice in the print expression. That allows the second 1.0 to be reclaimed by the GC, also means the constant table has one less entry so saves a word. Signed-off-by: Damien George <damien@micropython.org>
2022-05-17py/compile: Allow new qstrs to be allocated at all compiler passes.Damien George
Prior to this commit, all qstrs were required to be allocated (by calling mp_emit_common_use_qstr) in the MP_PASS_SCOPE pass (the first one). But this is an unnecessary restriction, which is lifted by this commit. Lifting the restriction simplifies the compiler because it can allocate qstrs in later passes. This also generates better code, because in some cases (eg when a variable is closed over) the scope of an identifier is not known until a bit later and then the identifier no longer needs its qstr allocated in the global table. Code size is reduced for all ports with this commit. Signed-off-by: Damien George <damien@micropython.org>
2022-05-17py/emitnative: Put a pointer to the native prelude in child_table array.Damien George
Some architectures (like esp32 xtensa) cannot read byte-wise from executable memory. This means the prelude for native functions -- which is usually located after the machine code for the native function -- must be placed in separate memory that can be read byte-wise. Prior to this commit this was achieved by enabling N_PRELUDE_AS_BYTES_OBJ for the emitter and MICROPY_EMIT_NATIVE_PRELUDE_AS_BYTES_OBJ for the runtime. The prelude was then placed in a bytes object, pointed to by the module's constant table. This behaviour is changed by this commit so that a pointer to the prelude is stored either in mp_obj_fun_bc_t.child_table, or in mp_obj_fun_bc_t.child_table[num_children] if num_children > 0. The reasons for doing this are: 1. It decouples the native emitter from runtime requirements, the emitted code no longer needs to know if the system it runs on can/can't read byte-wise from executable memory. 2. It makes all ports have the same emitter behaviour, there is no longer the N_PRELUDE_AS_BYTES_OBJ option. 3. The module's constant table is now used only for actual constants in the Python code. This allows further optimisations to be done with the constants (eg constant deduplication). Code size change for those ports that enable the native emitter: unix x64: +80 +0.015% stm32: +24 +0.004% PYBV10 esp8266: +88 +0.013% GENERIC esp32: -20 -0.002% GENERIC[incl -112(data)] rp2: +32 +0.005% PICO Signed-off-by: Damien George <damien@micropython.org>
2022-04-14py/parse: Factor obj extract code to mp_parse_node_extract_const_object.Damien George
Signed-off-by: Damien George <damien@micropython.org>
2022-04-01py: Fix compiling and decoding of *args at large arg positions.Damien George
There were two issues with the existing code: 1. "1 << i" is computed as a 32-bit number so would overflow when executed on 64-bit machines (when mp_uint_t is 64-bit). This meant that *args beyond 32 positions would not be handled correctly. 2. star_args must fit as a positive small int so that it is encoded correctly in the emitted code. MP_SMALL_INT_BITS is too big because it overflows a small int by 1 bit. MP_SMALL_INT_BITS - 1 does not work because it produces a signed small int which is then sign extended when extracted (even by mp_obj_get_int_truncated), and this sign extension means that any position arg after *args is also treated as a star-arg. So the maximum bit position is MP_SMALL_INT_BITS - 2. This means that MP_OBJ_SMALL_INT_VALUE() can be used instead of mp_obj_get_int_truncated() to get the value of star_args. These issues are fixed by this commit, and a test added. Signed-off-by: Damien George <damien@micropython.org>
2022-03-31py/runtime: Allow multiple *args in a function call.David Lechner
This is a partial implementation of PEP 448 to allow unpacking multiple star args in a function or method call. This is implemented by changing the emitted bytecodes so that both positional args and star args are stored as positional args. A bitmap is added to indicate if an argument at a given position is a positional argument or a star arg. In the generated code, this new bitmap takes the place of the old star arg. It is stored as a small int, so this means only the first N arguments can be star args where N is the number of bits in a small int. The runtime is modified to interpret this new bytecode format while still trying to perform as few memory reallocations as possible. Signed-off-by: David Lechner <david@pybricks.com>
2022-03-31py/runtime: Allow multiple **args in a function call.David Lechner
This is a partial implementation of PEP 448 to allow multiple ** unpackings when calling a function or method. The compiler is modified to encode the argument as a None: obj key-value pair (similar to how regular keyword arguments are encoded as str: obj pairs). The extra object that was pushed on the stack to hold a single ** unpacking object is no longer used and is removed. The runtime is modified to decode this new format. Signed-off-by: David Lechner <david@pybricks.com>
2022-03-30py/compile: Only show raw code that is bytecode.Damien George
Signed-off-by: Damien George <damien@micropython.org>
2022-03-28py: Change jump opcodes to emit 1-byte jump offset when possible.Damien George
This commit introduces changes: - All jump opcodes are changed to have variable length arguments, of either 1 or 2 bytes (previously they were fixed at 2 bytes). In most cases only 1 byte is needed to encode the short jump offset, saving bytecode size. - The bytecode emitter now selects 1 byte jump arguments when the jump offset is guaranteed to fit in 1 byte. This is achieved by checking if the code size changed during the last pass and, if it did (if it shrank), then requesting that the compiler make another pass to get the correct offsets of the now-smaller code. This can continue multiple times until the code stabilises. The code can only ever shrink so this iteration is guaranteed to complete. In most cases no extra passes are needed, the original 4 passes are enough to get it right by the 4th pass (because the 2nd pass computes roughly the correct labels and the 3rd pass computes the correct size for the jump argument). This change to the jump opcode encoding reduces .mpy files and RAM usage (when bytecode is in RAM) by about 2% on average. The performance of the VM is not impacted, at least within measurment of the performance benchmark suite. Code size is reduced for builds that include a decent amount of frozen bytecode. ARM Cortex-M builds without any frozen code increase by about 350 bytes. Signed-off-by: Damien George <damien@micropython.org>
2022-03-16py/showbc: Remove global variables and make DECODE_PTR work correctly.Damien George
The bytecode state variables mp_showbc_code_start and mp_showbc_constants have been removed and made local variables passed into the various functions. As part of this, the DECODE_PTR macro is fixed so it extracts the relevant pointer from the child_table (a regression introduced in f2040bfc7ee033e48acef9f289790f3b4e6b74e5). Signed-off-by: Damien George <damien@micropython.org>
2022-03-16py/parse: Handle check for target small-int size in parser.Damien George
This means that all constants for EMIT_ARG(load_const_obj, obj) are created in the parser (rather than some in the compiler). Signed-off-by: Damien George <damien@micropython.org>
2022-03-16py/parse: Put const bytes objects in parse tree as const object.Damien George
Instead of as an intermediate qstr, which may unnecessarily intern the data of the bytes object. Signed-off-by: Damien George <damien@micropython.org>
2022-02-24py: Rework bytecode and .mpy file format to be mostly static data.Damien George
Background: .mpy files are precompiled .py files, built using mpy-cross, that contain compiled bytecode functions (and can also contain machine code). The benefit of using an .mpy file over a .py file is that they are faster to import and take less memory when importing. They are also smaller on disk. But the real benefit of .mpy files comes when they are frozen into the firmware. This is done by loading the .mpy file during compilation of the firmware and turning it into a set of big C data structures (the job of mpy-tool.py), which are then compiled and downloaded into the ROM of a device. These C data structures can be executed in-place, ie directly from ROM. This makes importing even faster because there is very little to do, and also means such frozen modules take up much less RAM (because their bytecode stays in ROM). The downside of frozen code is that it requires recompiling and reflashing the entire firmware. This can be a big barrier to entry, slows down development time, and makes it harder to do OTA updates of frozen code (because the whole firmware must be updated). This commit attempts to solve this problem by providing a solution that sits between loading .mpy files into RAM and freezing them into the firmware. The .mpy file format has been reworked so that it consists of data and bytecode which is mostly static and ready to run in-place. If these new .mpy files are located in flash/ROM which is memory addressable, the .mpy file can be executed (mostly) in-place. With this approach there is still a small amount of unpacking and linking of the .mpy file that needs to be done when it's imported, but it's still much better than loading an .mpy from disk into RAM (although not as good as freezing .mpy files into the firmware). The main trick to make static .mpy files is to adjust the bytecode so any qstrs that it references now go through a lookup table to convert from local qstr number in the module to global qstr number in the firmware. That means the bytecode does not need linking/rewriting of qstrs when it's loaded. Instead only a small qstr table needs to be built (and put in RAM) at import time. This means the bytecode itself is static/constant and can be used directly if it's in addressable memory. Also the qstr string data in the .mpy file, and some constant object data, can be used directly. Note that the qstr table is global to the module (ie not per function). In more detail, in the VM what used to be (schematically): qst = DECODE_QSTR_VALUE; is now (schematically): idx = DECODE_QSTR_INDEX; qst = qstr_table[idx]; That allows the bytecode to be fixed at compile time and not need relinking/rewriting of the qstr values. Only qstr_table needs to be linked when the .mpy is loaded. Incidentally, this helps to reduce the size of bytecode because what used to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices. If the module uses the same qstr more than two times then the bytecode is smaller than before. The following changes are measured for this commit compared to the previous (the baseline): - average 7%-9% reduction in size of .mpy files - frozen code size is reduced by about 5%-7% - importing .py files uses about 5% less RAM in total - importing .mpy files uses about 4% less RAM in total - importing .py and .mpy files takes about the same time as before The qstr indirection in the bytecode has only a small impact on VM performance. For stm32 on PYBv1.0 the performance change of this commit is: diff of scores (higher is better) N=100 M=100 baseline -> this-commit diff diff% (error%) bm_chaos.py 371.07 -> 357.39 : -13.68 = -3.687% (+/-0.02%) bm_fannkuch.py 78.72 -> 77.49 : -1.23 = -1.563% (+/-0.01%) bm_fft.py 2591.73 -> 2539.28 : -52.45 = -2.024% (+/-0.00%) bm_float.py 6034.93 -> 5908.30 : -126.63 = -2.098% (+/-0.01%) bm_hexiom.py 48.96 -> 47.93 : -1.03 = -2.104% (+/-0.00%) bm_nqueens.py 4510.63 -> 4459.94 : -50.69 = -1.124% (+/-0.00%) bm_pidigits.py 650.28 -> 644.96 : -5.32 = -0.818% (+/-0.23%) core_import_mpy_multi.py 564.77 -> 581.49 : +16.72 = +2.960% (+/-0.01%) core_import_mpy_single.py 68.67 -> 67.16 : -1.51 = -2.199% (+/-0.01%) core_qstr.py 64.16 -> 64.12 : -0.04 = -0.062% (+/-0.00%) core_yield_from.py 362.58 -> 354.50 : -8.08 = -2.228% (+/-0.00%) misc_aes.py 429.69 -> 405.59 : -24.10 = -5.609% (+/-0.01%) misc_mandel.py 3485.13 -> 3416.51 : -68.62 = -1.969% (+/-0.00%) misc_pystone.py 2496.53 -> 2405.56 : -90.97 = -3.644% (+/-0.01%) misc_raytrace.py 381.47 -> 374.01 : -7.46 = -1.956% (+/-0.01%) viper_call0.py 576.73 -> 572.49 : -4.24 = -0.735% (+/-0.04%) viper_call1a.py 550.37 -> 546.21 : -4.16 = -0.756% (+/-0.09%) viper_call1b.py 438.23 -> 435.68 : -2.55 = -0.582% (+/-0.06%) viper_call1c.py 442.84 -> 440.04 : -2.80 = -0.632% (+/-0.08%) viper_call2a.py 536.31 -> 532.35 : -3.96 = -0.738% (+/-0.06%) viper_call2b.py 382.34 -> 377.07 : -5.27 = -1.378% (+/-0.03%) And for unix on x64: diff of scores (higher is better) N=2000 M=2000 baseline -> this-commit diff diff% (error%) bm_chaos.py 13594.20 -> 13073.84 : -520.36 = -3.828% (+/-5.44%) bm_fannkuch.py 60.63 -> 59.58 : -1.05 = -1.732% (+/-3.01%) bm_fft.py 112009.15 -> 111603.32 : -405.83 = -0.362% (+/-4.03%) bm_float.py 246202.55 -> 247923.81 : +1721.26 = +0.699% (+/-2.79%) bm_hexiom.py 615.65 -> 617.21 : +1.56 = +0.253% (+/-1.64%) bm_nqueens.py 215807.95 -> 215600.96 : -206.99 = -0.096% (+/-3.52%) bm_pidigits.py 8246.74 -> 8422.82 : +176.08 = +2.135% (+/-3.64%) misc_aes.py 16133.00 -> 16452.74 : +319.74 = +1.982% (+/-1.50%) misc_mandel.py 128146.69 -> 130796.43 : +2649.74 = +2.068% (+/-3.18%) misc_pystone.py 83811.49 -> 83124.85 : -686.64 = -0.819% (+/-1.03%) misc_raytrace.py 21688.02 -> 21385.10 : -302.92 = -1.397% (+/-3.20%) The code size change is (firmware with a lot of frozen code benefits the most): bare-arm: +396 +0.697% minimal x86: +1595 +0.979% [incl +32(data)] unix x64: +2408 +0.470% [incl +800(data)] unix nanbox: +1396 +0.309% [incl -96(data)] stm32: -1256 -0.318% PYBV10 cc3200: +288 +0.157% esp8266: -260 -0.037% GENERIC esp32: -216 -0.014% GENERIC[incl -1072(data)] nrf: +116 +0.067% pca10040 rp2: -664 -0.135% PICO samd: +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS As part of this change the .mpy file format version is bumped to version 6. And mpy-tool.py has been improved to provide a good visualisation of the contents of .mpy files. In summary: this commit changes the bytecode to use qstr indirection, and reworks the .mpy file format to be simpler and allow .mpy files to be executed in-place. Performance is not impacted too much. Eventually it will be possible to store such .mpy files in a linear, read-only, memory- mappable filesystem so they can be executed from flash/ROM. This will essentially be able to replace frozen code for most applications. Signed-off-by: Damien George <damien@micropython.org>
2021-09-10py/parse: Simplify parse nodes representing a list.Damien George
This commit simplifies and optimises the parse tree in-memory representation of lists of expressions, for tuples and lists, and when tuples are used on the left-hand-side of assignments and within del statements. This reduces memory usage of the parse tree when such code is compiled, and also reduces the size of the compiler. For example, (1,) was previously the following parse tree: expr_stmt(5) (n=2) atom_paren(45) (n=1) testlist_comp(146) (n=2) int(1) testlist_comp_3b(149) (n=1) NULL NULL and with this commit is now: expr_stmt(5) (n=2) atom_paren(45) (n=1) testlist_comp(146) (n=1) int(1) NULL Similarly, (1, 2, 3) was previously: expr_stmt(5) (n=2) atom_paren(45) (n=1) testlist_comp(146) (n=2) int(1) testlist_comp_3c(150) (n=2) int(2) int(3) NULL and is now: expr_stmt(5) (n=2) atom_paren(45) (n=1) testlist_comp(146) (n=3) int(1) int(2) int(3) NULL Signed-off-by: Damien George <damien@micropython.org>
2021-05-30py/compile: Raise an error on async with/for outside an async function.Jeff Epler
A simple reproducer is: async for x in (): x Before this change, it would cause an assertion error in mpy-cross and micropython-coverage.
2021-04-27py: Add option to compile without any error messages at all.Damien George
This introduces a new option, MICROPY_ERROR_REPORTING_NONE, which completely disables all error messages. To be used in cases where MicroPython needs to fit in very limited systems. Signed-off-by: Damien George <damien@micropython.org>
2020-07-25py/compile: Don't await __aiter__ special method in async-for.Jonathan Hogg
MicroPython's original implementation of __aiter__ was correct for an earlier (provisional) version of PEP492 (CPython 3.5), where __aiter__ was an async-def function. But that changed in the final version of PEP492 (in CPython 3.5.2) where the function was changed to a normal one. See https://www.python.org/dev/peps/pep-0492/#why-aiter-does-not-return-an-awaitable See also the note at the end of this subsection in the docs: https://docs.python.org/3.5/reference/datamodel.html#asynchronous-iterators And for completeness the BPO: https://bugs.python.org/issue27243 To be consistent with the Python spec as it stands today (and now that PEP492 is final) this commit changes MicroPython's behaviour to match CPython: __aiter__ should return an async-iterable object, but is not itself awaitable. The relevant tests are updated to match. See #6267.
2020-06-16py/compile: Implement PEP 526, syntax for variable annotations.Damien George
This addition to the grammar was introduced in Python 3.6. It allows annotating the type of a varilable, like: x: int = 123 s: str The implementation in this commit is quite simple and just ignores the annotation (the int and str bits above). The reason to implement this is to allow Python 3.6+ code that uses this feature to compile under MicroPython without change, and for users to use type checkers. In the future viper could use this syntax as a way to give types to variables, which is currently done in a bit of an ad-hoc way, eg x = int(123). And this syntax could potentially be used in the inline assembler to define labels in an way that's easier to read.
2020-06-16py/grammar.h: Consolidate duplicate sub-rules for :test and =test.Damien George
2020-06-16py/compile: Implement PEP 572, assignment expressions with := operator.Damien George
The syntax matches CPython and the semantics are equivalent except that, unlike CPython, MicroPython allows using := to assign to comprehension iteration variables, because disallowing this would take a lot of code to check for it. The new compile-time option MICROPY_PY_ASSIGN_EXPR selects this feature and is enabled by default, following MICROPY_PY_ASYNC_AWAIT.
2020-06-16py/compile: Convert scope test to SCOPE_IS_COMP_LIKE macro.Damien George
This macro can be used elsewhere.
2020-05-09py/parse: Make mp_parse_node_extract_list return size_t instead of int.Damien George
Because this function can only return non-negative values, and having the correct return type gives more information to the caller.
2020-04-23all: Format code to add space after C++-style comment start.stijn
Note: the uncrustify configuration is explicitly set to 'add' instead of 'force' in order not to alter the comments which use extra spaces after // as a means of indenting text for clarity.
2020-04-05all: Use MP_ERROR_TEXT for all error messages.Jim Mussared
2020-04-05py/objexcept: Allow compression of exception message text.Jim Mussared
The decompression of error-strings is only done if the string is accessed via printing or via er.args. Tests are added for this feature to ensure the decompression works.
2020-04-05py: Use preprocessor to detect error reporting level (terse/detailed).Jim Mussared
Instead of compiler-level if-logic. This is necessary to know what error strings are included in the build at the preprocessor stage, so that string compression can be implemented.
2020-02-28all: Reformat C and Python source code with tools/codeformat.py.Damien George
This is run with uncrustify 0.70.1, and black 19.10b0.
2020-02-06py/compile: Allow 'return' outside function in minimal builds.Petr Viktorin
A 'return' statement on module/class level is not correct Python, but nothing terribly bad happens when it's allowed. So remove the check unless MICROPY_CPYTHON_COMPAT is on. This is similar to MicroPython's treatment of 'import *' in functions (except 'return' has unsurprising behavior if it's allowed).
2019-11-21py/compile: Coalesce error message for break/continue outside loop.Petr Viktorin
To reduce code size.
2019-10-05py: Add new Xtensa-Windowed arch for native emitter.Damien George
Enabled via the configuration MICROPY_EMIT_XTENSAWIN.
2019-10-04py/compile: Disallow 'import *' outside module level.Petr Viktorin
This check follows CPython's behaviour, because 'import *' always populates the globals with the imported names, not locals. Since it's safe to do this (doesn't lead to a crash or undefined behaviour) the check is only enabled for MICROPY_CPYTHON_COMPAT. Fixes issue #5121.
2019-09-26py: Rename MP_QSTR_NULL to MP_QSTRnull to avoid intern collisions.Josh Lloyd
Fixes #5140.
2019-09-26py/compile: Use calculation instead of switch to convert token to op.Damien George
2019-08-30py: Integrate sys.settrace feature into the VM and runtime.Milan Rossa
This commit adds support for sys.settrace, allowing to install Python handlers to trace execution of Python code. The interface follows CPython as closely as possible. The feature is disabled by default and can be enabled via MICROPY_PY_SYS_SETTRACE.
2019-08-30py/compile: Improve the line numbering precision for lambdas.Damien George
Prior to this patch the line number for a lambda would be "line 1" if the body of the lambda contained only a simple expression (with no line number stored in the parse node). Now the line number is always reported correctly.
2019-08-28py: Add global default_emit_opt variable to make emit kind persistent.Damien George
mp_compile no longer takes an emit_opt argument, rather this setting is now provided by the global default_emit_opt variable. Now, when -X emit=native is passed as a command-line option, the emitter will be set for all compiled modules (included imports), not just the top-level script. In the future there could be a way to also set this variable from a script. Fixes issue #4267.
2019-08-19py/compile: Improve the line numbering precision for comprehensions.Milan Rossa
The line number for comprehensions is now always reported as the correct global location in the script, instead of just "line 1".
2019-05-06py: remove "if (0)" and "if (false)" branches.Jun Wu
Prior to this commit, building the unix port with `DEBUG=1` and `-finstrument-functions` the compilation would fail with an error like "control reaches end of non-void function". This change fixes this by removing the problematic "if (0)" branches. Not all branches affect compilation, but they are all removed for consistency.
2019-05-01py/asmthumb: Support asm_thumb code running on normal ARM processors.Damien George
With this change, @micropython.asm_thumb functions will work on standard ARM processors (that are in ARM state by default), in scripts and precompiled .mpy files. Addresses issue #4675.
2019-03-14py/compile: Check that arch is set when compiling native, viper or asm.Damien George
2019-03-14py/compile: Support multiple inline asm emitters.Damien George
2019-03-14py/compile: Add support to select the native emitter at runtime.Damien George
2019-03-08py: Add support to save native, viper and asm code to .mpy files.Damien George
This commit adds support for saving and loading .mpy files that contain native code (native, viper and inline-asm). A lot of the ground work was already done for this in the form of removing pointers from generated native code. The changes here are mainly to link in qstr values to the native code, and change the format of .mpy files to contain native code blocks (possibly mixed with bytecode). A top-level summary: - @micropython.native, @micropython.viper and @micropython.asm_thumb/ asm_xtensa are now allowed in .py files when compiling to .mpy, and they work transparently to the user. - Entire .py files can be compiled to native via mpy-cross -X emit=native and for the most part the generated .mpy files should work the same as their bytecode version. - The .mpy file format is changed to 1) specify in the header if the file contains native code and if so the architecture (eg x86, ARMV7M, Xtensa); 2) for each function block the kind of code is specified (bytecode, native, viper, asm). - When native code is loaded from a .mpy file the native code must be modified (in place) to link qstr values in, just like bytecode (see py/persistentcode.c:arch_link_qstr() function). In addition, this now defines a public, native ABI for dynamically loadable native code generated by other languages, like C.
2019-03-08py/emitnative: Consolidate where HASCONSTS is set to load-const-obj fun.Damien George
Simplifies the code and fixes handling of the Ellipsis const in native code generation (which also needs the constant table so must set this flag).
2019-03-05py: Replace POP_BLOCK and POP_EXCEPT opcodes with POP_EXCEPT_JUMP.Damien George
POP_BLOCK and POP_EXCEPT are now the same, and are always followed by a JUMP. So this optimisation reduces code size, and RAM usage of bytecode by two bytes for each try-except handler.
2019-03-05py: Fix VM crash with unwinding jump out of a finally block.Damien George
This patch fixes a bug in the VM when breaking within a try-finally. The bug has to do with executing a break within the finally block of a try-finally statement. For example: def f(): for x in (1,): print('a', x) try: raise Exception finally: print(1) break print('b', x) f() Currently in uPy the above code will print: a 1 1 1 segmentation fault (core dumped) micropython Not only is there a seg fault, but the "1" in the finally block is printed twice. This is because when the VM executes a finally block it doesn't really know if that block was executed due to a fall-through of the try (no exception raised), or because an exception is active. In particular, for nested finallys the VM has no idea which of the nested ones have active exceptions and which are just fall-throughs. So when a break (or continue) is executed it tries to unwind all of the finallys, when in fact only some may be active. It's questionable whether break (or return or continue) should be allowed within a finally block, because they implicitly swallow any active exception, but nevertheless it's allowed by CPython (although almost never used in the standard library). And uPy should at least not crash in such a case. The solution here relies on the fact that exception and finally handlers always appear in the bytecode after the try body. Note: there was a similar bug with a return in a finally block, but that was previously fixed in b735208403a54774f9fd3d966f7c1a194c41870f
2019-03-01py/compile: Add optimisation to compile OrderedDict inplace.Damien George
This optimisation eliminates the need to create a temporary normal dict. The optimisation is enabled via MICROPY_COMP_CONST_LITERAL which is enabled by default (although only has an effect if OrderdDict is enabled). Thanks to @pfalcon for the initial idea and implementation.
2019-02-26py/compile: Fix handling of unwinding BaseException in async with.Damien George
All exceptions that unwind through the async-with must be caught and BaseException is the top-level class, which includes Exception and others. Fixes issue #4552.
2019-02-25py: Eliminate warnings about unused arguments when debugging disabled.Damien George