Python 3, indentation and all, with a typed AST whose node names mirror CPython’s own. Four fifths of its test suite is for generation — because the reason this exists is that RPG, SAS and COBOL migrations have to emit idiomatic Python at the end.
CPython ships a free parser written by the people who define the language. If you need to analyze Python, alone, in a Python process, use ast or LibCST. This page is about the other case.
Python’s hardest properties are not in its parser at all. They are in the tokenizer, in a set of soft keywords that must not be reserved, and in a string literal that was outside the grammar entirely until Python 3.12.
Indentation is a lexical decision. The tokenizer keeps an indentation stack and synthesizes INDENT and DEDENT tokens; brackets suspend indentation entirely, a trailing backslash joins lines, and blank or comment-only lines are not indentation events at all. Get any of that wrong and the block structure of the file is wrong.
match and case are soft keywords. match = re.match(...) and def case(x) are ordinary, legal Python, and a parser that reserves those words breaks real code the day someone upgrades. CPython’s own grammar handles this with backtracking, which is a luxury an LL grammar does not have.
Assignment targets are arbitrarily nested patterns — a, (b, *rest) = f() — and star-unpacking appears in targets, calls, literals and comprehensions. Type annotations are expressions evaluated at definition time unless quoted, so x: Foo and x: "Foo" mean the same thing and produce different trees.
And since Python 3.9 the reference implementation uses a PEG parser. The official grammar is a PEG, so any ANTLR translation of it is exactly that — a translation. We would rather tell you where ours stops than describe it as “accommodating Python’s dynamic constructs”.
Input — structural pattern matching
def classify(shape) -> str:
match shape:
case Point(x=0, y=0):
return "origin"
case Point(x=0, y=y) | Point(x=y, y=0) if y > 0:
return "on an axis"
case [Point() as first, *rest] if len(rest) > 2:
return "polyline"
case {"kind": "circle", **extra}:
return "circle"
case _:
return "unknown"
# both of these are still legal Python
match = classify(Point(0, 0))
def case(x): return xas patterns, sequence patterns with *rest, mapping patterns with **extra, plus guards. The last two lines rebind match and case as ordinary names.Output — AST, by node class
PyModule ├── PyFunctionDef classify, annotated return │ └── PyMatch from match_stmt, line 2 │ ├── case_block — class pattern │ ├── case_block — or-pattern + guard │ ├── case_block — sequence pattern, star target │ ├── case_block — mapping pattern, ** rest │ └── case_block — wildcard ├── PyAssign target: match — an ordinary name └── PyFunctionDef case — an ordinary function
ast module, so a developer who knows ast.Module and ast.FunctionDef already knows this tree. One honest caveat: match is in the grammar but has no dedicated test, and we would rather you heard that from us.Two things are missing from the model and both of them will change how you design around it. Showing them is more useful than a page of features, so here they are, in the largest frame on the page.
Input — ordinary Python
# rounding is deliberate: finance signed off on banker's rounding
def summarize(rows):
total = sum(r.amount for r in rows)
label = f"{total:,.2f} — {'high' if total > 1000 else 'low'}"
return labeltotal and you find the assignment on line 3 and nothing on line 4. And the comment on line 1 is not in the tree at all: the lexer skips comments rather than routing them to a hidden channel, so they cannot be recovered. Two facts, stated plainly, because you will hit both in the first hour.Why the f-string is opaque. Before Python 3.12 an f-string was not part of the grammar at all. It was a single string token with an f prefix, and CPython pulled the replacement fields out afterwards with a separate hand-written mini-parser. That is why, before 3.12, you could not reuse a quote character inside the expression, backslashes were banned inside replacement fields, and nesting was limited.
PEP 701 changed the tokenizer contract. In Python 3.12 f-strings became formal grammar: the tokenizer emits dedicated start, middle and end tokens, replacement fields are parsed as ordinary expressions, quote reuse is legal and nesting is unlimited. It is one of very few times a mainstream language changed how a string literal tokenizes.
Where this engine stands. On the pre-701 model. An f-string is one literal node; the expressions inside it are not parsed and do not appear in the tree. So a rename refactoring has to treat f-string bodies as text, an analysis that counts references to a name will miss the ones inside them, and the 3.12 quote-reuse form will not lex.
And comments. Docstrings survive — they are string expressions, not comments. Hash comments do not. If your project needs comment-preserving round trips over Python source, say so in the first conversation; today that is a limitation of this engine and not something to discover in week three.
| Feature | Since | Status |
|---|---|---|
| Significant indentation, implicit and explicit line joining | core | Supported |
Decorators, comprehensions, generators, yield from, with, lambdas, star-unpacking |
≤ 3.7 | Supported |
| Function and variable annotations | 3.0, 3.6 | Supported |
async and await |
3.5 | Supported |
| f-strings as literals | 3.6 | Lexed as one opaque token — the expressions inside are not parsed |
Structural pattern matching, match and case |
3.10 | In the grammar and the AST, but with no dedicated test yet |
Walrus operator := |
3.8 | Not in the grammar |
Positional-only parameters, / |
3.8 | Not in the grammar |
Exception groups, except* |
3.11 | Not in the grammar |
| PEP 701 f-string grammar | 3.12 | Not implemented — quote-reuse forms will not lex |
Type parameter syntax and the type statement, PEP 695 |
3.12 | Not in the grammar |
| Python 2 | — | Not supported, and not planned |
The last row deserves a sentence rather than a mark. The grammar descends from a published Python 3 grammar; there is no print statement in it, no exec, no old-style classes, and only one file extension is registered. Python 2 reached end of life in January 2020. If you have a 2-to-3 problem, it is a semantics problem — strings against bytes, integer division, iterator protocols — and it is a services conversation, not a parser feature.
The tokenizer maintains a stack and synthesizes block tokens. Inside brackets, indentation is suspended entirely; a trailing backslash joins lines; comment-only lines are not events. All of it has to be right before the parser sees anything.
values = [
1, 2, # indentation suspended in here
3,
]
total = 1 + \
2match and case introduce a statement in one position and are ordinary names in another. Reserving them breaks working code; not reserving them makes the grammar ambiguous at exactly the point it matters.
match command.split():
case ["go", direction]: move(direction)
match = None # still a variableThe left of an assignment is not a name — it is an arbitrarily nested destructuring pattern with star-unpacking, and the same shapes reappear in for headers, calls, literals and comprehensions.
a, (b, *rest) = f()
for (k, (v1, v2)) in pairs: ...
merged = {**a, "x": 1}Imports resolve at run time against the interpreter’s path. Conditional imports, importlib and __import__ are all ordinary Python. No parser can give you a complete import graph, and one that claims to is guessing.
if settings.FAST:
from ._fast import encode
else:
encode = importlib.import_module(name).encodeThe same honesty applies one level up: a Python AST tells you what the source says, not what the program will do. Decorators that rewrite functions, metaclasses, monkey-patching and exec are run-time behavior and out of reach of any static parser, ours included.
Of roughly 104 automated tests in the module, 79 are for the code generator. That ratio is not an accident and it is not a gap — it is what the product is.
CPython’s ast module is a reader. It parses source into a tree, and deliberately does not give you a comfortable way to construct a tree programmatically and print idiomatic source back out. That is precisely what a migration needs at the far end.
An RPG-to-Python or SAS-to-PySpark pipeline reads the legacy source with one Strumenta engine, transforms in one model, builds a Python AST, fills the template-supplied parts through the placeholder system, and prints. The Python side of that is this module, and its 79 generation tests are the evidence for it.
It is a drop-in pipeline stage. The module ships Starlasu pipeline components for both directions — parse and generate — so the Python end of a migration is configuration rather than integration work.
And a light expression-type layer exists to support that generation: integer, string, decimal, date, time and datetime shapes with promotion rules, which is what you need when the source language had fixed-point decimals and indicators. It is transpiler support, not a Python type checker, and we are not going to call it type inference.
parse .py source → PyModule generate PyModule → .py source serialize PyModule → JSON | XML | LionWeb the same model the RPG, COBOL, SAS and SQL engines produce — so the transform in the middle is one traversal, not five adapters
Available as a JVM library, as a command-line tool that serializes the AST, and as a pipeline component. Reachable from Java, Kotlin, Python, TypeScript and C#.
A JVM library and a command-line tool, the generated documents that describe the model, and a license with support in it.
Kotlin on the JVM, built with ANTLR on Kolasu — the JVM implementation of Starlasu. In process you get the tree and the positioned issue list directly; on the command line you get the AST serialized as JSON or XML. Java 17 or later. A license file is registered at start-up; licenses are refreshed from a license service, so tell us early if the engine has to run air-gapped.
A typed AST rooted at a module, with node names that mirror CPython’s ast — modules, function and class definitions, assignments and augmented assignments, comprehensions, awaits, matches, tries. Positions on every node. Difficult input returns a partial tree plus positioned issues rather than an exception.
Build a Python tree programmatically, leave holes where a template supplies the content, fill them, and print idiomatic source. 79 tests stand behind it. This is the part of the product that a migration pipeline actually consumes.
An EMF metamodel and AST documentation are generated from the model itself, so the reference matches the code rather than a changelog. The engine supports LionWeb, so the model interchanges with LionWeb-compliant tooling, and it is reachable through bindings from Java, Kotlin, Python, TypeScript and C#.
Standard, Distribution or Service — with support included, not sold separately.
Standard for use inside your own organization, Distribution if the engine ships inside a product you sell, Service if it runs behind a service you operate. The guarantee runs a year and is extendable, or can be taken pay-as-you-go. Write to products@strumenta.com.
This is the whole argument, and it is not a claim about grammar quality. ast.parse is written by the people who define Python and it is always exactly current.
What ast gives you is Python, and only Python, only inside a CPython process, in a node shape that nothing else in your pipeline speaks. It does not compose with your RPG parser or your SAS parser, and it does not serialize into a model another team’s tooling can read.
Every Strumenta engine is built on Starlasu, so every one produces an AST in the same shape with the same traversal model — the visitor you wrote for RPG walks the Python tree. Every engine supports LionWeb, so the models leave the process intact. And every engine is reachable through bindings from Java, Kotlin, Python, TypeScript and C#.
So license this when Python is the target of a migration, or one language in a polyglot inventory. Do not license it to parse Python on its own — for that, ast, LibCST and tree-sitter-python are free, good, and we will tell you so on the first call.
Where the Python usually comes from
RPG & DDS engine →
The most common source of generated Python we see.
SAS engine →
SAS exits usually land in Python or PySpark.
COBOL engine →
The batch on the other side of the migration.
The whole catalog →
15+ engines, one AST framework underneath.
Seven things we would rather tell you now than have you find in week three. Two of them are on this page twice, because they matter that much.
# comment.except* and PEP 695 type parameters are not in the grammar today. match/case is, but has no dedicated test.ast or LibCST.:=, positional-only parameters, except* or PEP 695?No. The grammar is Python 3 and always has been: there is no print statement, no exec statement and no old-style-class syntax in it, and only one file extension is registered. Python 2 reached end of life in January 2020. If you have a 2-to-3 problem it is mostly a semantics problem — strings against bytes, integer division, iterator protocols — and that is a services conversation, not a parser feature.
Hash comments are not. The lexer skips them rather than putting them on a hidden channel, so they are not in the token stream and cannot be recovered from the tree. Docstrings do survive, because they are string expressions rather than comments. If comment-preserving round trips over Python source are essential to your project, raise it in the first conversation.
Not today. The engine uses the pre-PEP-701 model, where an f-string is a single literal token: the expressions inside the braces are not parsed and do not appear in the AST. So searching the tree for uses of a name will miss the ones inside f-strings, and a rename has to treat those bodies as text. PEP 701, in Python 3.12, made f-strings formal grammar; that model is not implemented here.
Python 3, roughly through 3.10 — including structural pattern matching, async/await, annotations, decorators and comprehensions. Not covered: the walrus operator, positional-only parameters, except*, the PEP 701 f-string grammar and PEP 695 type parameters. That is the real list; ask us before you plan around any of it.
ast is in the standard library?If you only need to analyze Python, you would not, and we will say so. The reason to license this one is that it writes Python: a code generator with 79 tests, string templates and a placeholder system, producing idiomatic source from a tree you build programmatically. That is what an RPG-to-Python or SAS-to-PySpark pipeline needs, and it is what ast deliberately does not provide. The second reason is composition — the same AST shape as our RPG, COBOL, SAS and SQL engines, so one traversal crosses all of them.
If Python is the target of a migration, send us a sample of the source language and what you want the output to look like. If Python is one language in a bigger inventory, tell us the others. And if it is Python alone, we will point you at the free tools and save you a purchase order. To see the model itself first, the Strumenta Playground parses Python in your browser — pick Python from the language list.
