aboutsummaryrefslogtreecommitdiff
path: root/py/lexer.h
AgeCommit message (Collapse)Author
2022-05-25py/builtin: Clean up and simplify import_stat and builtin_open config.Damien George
The following changes are made: - If MICROPY_VFS is enabled then mp_vfs_import_stat and mp_vfs_open are automatically used for mp_import_stat and mp_builtin_open respectively. - If MICROPY_PY_IO is enabled then "open" is automatically included in the set of builtins, and points to mp_builtin_open_obj. This helps to clean up and simplify the most common port configuration. Signed-off-by: Damien George <damien@micropython.org>
2021-08-14py: Implement partial PEP-498 (f-string) support.Jim Mussared
This implements (most of) the PEP-498 spec for f-strings and is based on https://github.com/micropython/micropython/pull/4998 by @klardotsh. It is implemented in the lexer as a syntax translation to `str.format`: f"{a}" --> "{}".format(a) It also supports: f"{a=}" --> "a={}".format(a) This is done by extracting the arguments into a temporary vstr buffer, then after the string has been tokenized, the lexer input queue is saved and the contents of the temporary vstr buffer are injected into the lexer instead. There are four main limitations: - raw f-strings (`fr` or `rf` prefixes) are not supported and will raise `SyntaxError: raw f-strings are not supported`. - literal concatenation of f-strings with adjacent strings will fail "{}" f"{a}" --> "{}{}".format(a) (str.format will incorrectly use the braces from the non-f-string) f"{a}" f"{a}" --> "{}".format(a) "{}".format(a) (cannot concatenate) - PEP-498 requires the full parser to understand the interpolated argument, however because this entirely runs in the lexer it cannot resolve nested braces in expressions like f"{'}'}" - The !r, !s, and !a conversions are not supported. Includes tests and cpydiffs. Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2020-06-16py/compile: Implement PEP 572, assignment expressions with := operator.Damien George
The syntax matches CPython and the semantics are equivalent except that, unlike CPython, MicroPython allows using := to assign to comprehension iteration variables, because disallowing this would take a lot of code to check for it. The new compile-time option MICROPY_PY_ASSIGN_EXPR selects this feature and is enabled by default, following MICROPY_PY_ASYNC_AWAIT.
2019-11-26py: Remove 3 obsolete commented-out lines from header files.Damien George
2019-09-26py: Add support for matmul operator @ as per PEP 465.Damien George
To make progress towards MicroPython supporting Python 3.5, adding the matmul operator is important because it's a really "low level" part of the language, being a new token and modifications to the grammar. It doesn't make sense to make it configurable because 1) it would make the grammar and lexer complicated/messy; 2) no other operators are configurable; 3) it's not a feature that can be "dynamically plugged in" via an import. And matmul can be useful as a general purpose user-defined operator, it doesn't have to be just for numpy use. Based on work done by Jim Mussared.
2019-09-26py/lexer: Reorder operator tokens to match corresponding binary ops.Damien George
2017-07-31all: Use the name MicroPython consistently in commentsAlexander Steffen
There were several different spellings of MicroPython present in comments, when there should be only one.
2017-07-18all: Unify header guard usage.Alexander Steffen
The code conventions suggest using header guards, but do not define how those should look like and instead point to existing files. However, not all existing files follow the same scheme, sometimes omitting header guards altogether, sometimes using non-standard names, making it easy to accidentally pick a "wrong" example. This commit ensures that all header files of the MicroPython project (that were not simply copied from somewhere else) follow the same pattern, that was already present in the majority of files, especially in the py folder. The rules are as follows. Naming convention: * start with the words MICROPY_INCLUDED * contain the full path to the file * replace special characters with _ In addition, there are no empty lines before #ifndef, between #ifndef and one empty line before #endif. #endif is followed by a comment containing the name of the guard macro. py/grammar.h cannot use header guards by design, since it has to be included multiple times in a single C file. Several other files also do not need header guards as they are only used internally and guaranteed to be included only once: * MICROPY_MPHALPORT_H * mpconfigboard.h * mpconfigport.h * mpthreadport.h * pin_defs_*.h * qstrdefs*.h
2017-02-17py/lexer: Convert mp_uint_t to size_t where appropriate.Damien George
2017-02-17py/lexer: Simplify handling of line-continuation error.Damien George
Previous to this patch there was an explicit check for errors with line continuation (where backslash was not immediately followed by a newline). But this check is not necessary: if there is an error then the remaining logic of the tokeniser will reject the backslash and correctly produce a syntax error.
2017-02-17py/lexer: Use strcmp to make keyword searching more efficient.Damien George
Since the table of keywords is sorted, we can use strcmp to do the search and stop part way through the search if the comparison is less-than. Because all tokens that are names are subject to this search, this optimisation will improve the overall speed of the lexer when processing a script. The change also decreases code size by a little bit because we now use strcmp instead of the custom str_strn_equal function.
2016-12-22py/lexer: Permanently disable the mp_lexer_show_token function.Damien George
The lexer is very mature and this debug function is no longer used. If it's really needed one can uncomment it and recompile.
2016-11-16py/lexer: Make lexer use an mp_reader as its source.Damien George
2016-04-13py: add async/await/async for/async with syntaxpohmelie
They are sugar for marking function as generator, "yield from" and pep492 python "semantically equivalents" respectively. @dpgeorge was the original author of this patch, but @pohmelie made changes to implement `async for` and `async with`.
2015-06-04unix: Allow to cat a script into stdin from the command line.Damien George
See issue #1306.
2015-03-19py: Allow to compile with extra warnings (sign-compare, unused-param).Damien George
2015-02-08py: Parse big-int/float/imag constants directly in parser.Damien George
Previous to this patch, a big-int, float or imag constant was interned (made into a qstr) and then parsed at runtime to create an object each time it was needed. This is wasteful in RAM and not efficient. Now, these constants are parsed straight away in the parser and turned into objects. This allows constants with large numbers of digits (so addresses issue #1103) and takes us a step closer to #722.
2015-01-07py: Put all global state together in state structures.Damien George
This patch consolidates all global variables in py/ core into one place, in a global structure. Root pointers are all located together to make GC tracing easier and more efficient.
2015-01-01py: Move to guarded includes, everywhere in py/ core.Damien George
Addresses issue #1022.
2014-12-27py: Move to guarded includes for compile.h and related headers.Paul Sokolovsky
2014-12-05py: Optimise lexer by exposing lexer type.Damien George
mp_lexer_t type is exposed, mp_token_t type is removed, and simple lexer functions (like checking current token kind) are now inlined. This saves 784 bytes ROM on 32-bit unix, 348 bytes on stmhal, and 460 bytes on bare-arm. It also saves a tiny bit of RAM since mp_lexer_t is a bit smaller. Also will run a bit more efficiently.
2014-07-30py: Change lexer stream API to return bytes not chars.Damien George
Lexer is now 8-bit clean inside strings.
2014-07-03lexer: Convert type (u)int to mp_(u)int_t.Damien George
2014-06-03py: Instead of having "debug on" var, have "optimization level" var.Paul Sokolovsky
This allows to have multiple "optimization" levels (CPython has two (-OO removes docstrings), we can have more).
2014-05-03Add license header to (almost) all files.Damien George
Blanket wide to all .c and .h files. Some files originating from ST are difficult to deal with (license wise) so it was left out of those. Also merged modpyb.h, modos.h, modstm.h and modtime.h in stmhal/.
2014-02-05Search paths properly on import and execute __init__.py if it exists.Damien George
2014-01-25Implement mp_parse_node_free; print properly repr(string).Damien George
2014-01-18Add source file name and line number to error messages.Damien George
Byte code has a map from byte-code offset to source-code line number, used to give better error messages.
2014-01-15Convert parse errors to exceptions.Damien George
Parser no longer prints an error, but instead returns an exception ID and message.
2014-01-12py: Improve memory management for parser; add lexer error for bad line cont.Damien George
2014-01-08Move lexerstr to main py directory (everyone uses it).Damien George
2014-01-04Add ellipsis object.Damien George
2014-01-03Basic implementation of import.Damien George
import works for simple cases. Still work to do on finding the right script, and setting globals/locals correctly when running an imported function.
2013-12-21Change object representation from 1 big union to individual structs.Damien
A big change. Micro Python objects are allocated as individual structs with the first element being a pointer to the type information (which is itself an object). This scheme follows CPython. Much more flexible, not necessarily slower, uses same heap memory, and can allocate objects statically. Also change name prefix, from py_ to mp_ (mp for Micro Python).
2013-10-20Integrate new lexer stream with stm framework.Damien
2013-10-20Abstract out back-end stream functionality from lexer.Damien
2013-10-09Improve indent/dedent error checking and reporting.Damien
2013-10-04Initial commit.Damien