STRUMENTA LANGUAGE ENGINES · TERADATA SQL

Teradata SQL, where the DDL is the physical design.

A 6,336-line grammar across four files that reads what nothing else reads: MULTISET, FALLBACK, journaling, DATABLOCKSIZE, CHECKSUM, MERGEBLOCKRATIO, MAP, PRIMARY INDEX, COMPRESS value lists, LOCKING ROW FOR ACCESS and COLLECT STATISTICS USING SAMPLE.

Teradata is not a language people write applications in. It is the language a warehouse is configured in — and every one of those clauses is a migration decision somebody has to make on purpose rather than discover in production.

The engine at a glance
GRAMMAR6,336 lines across four files
CORPUSexample set per release, v0.1.1–v0.1.12
ASTgenerated from the grammar, not hand-written
OUTPUTAST · JSON · EMF/XMI · LionWeb
RUNTIMEJVM library · CLI · Python API
ADJACENTTeradata → Spark SQL transpiler

A CREATE TABLE that is really a storage specification

TeradataSqlStatementsParser.g4 · 1,154 lines + TeradataSqlLexer.g4 · 2,083 lines

In every other dialect on our list, CREATE TABLE declares columns. In Teradata it also declares duplicate-row semantics, the fallback copy policy, whether the table is journaled before and after, the data block size in bytes, the checksum level, the merge block ratio, the hashing map, the primary index and the compression values for individual columns. Then it declares the columns.

Those clauses arrive in a comma-separated list before the column list, in almost any order, with values that are themselves keyword phrases: DEFAULT MERGEBLOCKRATIO, CHECKSUM = DEFAULT, MAP = TD_MAP1. It is a large, ambiguous keyword surface, and a generic SQL grammar has no rules for any of it — not partial rules, none.

Which is the whole commercial point. When a Teradata estate moves to Snowflake, Databricks, BigQuery or Synapse, those clauses are exactly the part somebody has to decide about. A PRIMARY INDEX becomes a clustering key or nothing at all. A COMPRESS value list becomes nothing. FALLBACK becomes nothing.

Parsing them means you can produce a report of what is being dropped, table by table, before the migration starts. Skipping them means finding out afterwards, from a query that used to take four seconds.

The same is true of the operational statements around the DDL. COLLECT STATISTICS USING SAMPLE 10 PERCENT and LOCKING ROW FOR ACCESS are not decoration — they are why the nightly batch finishes, and they have grammar rules of their own here.

Input — Teradata table definition

create_table_statement.sql

CREATE MULTISET TABLE prodeiw.SB_TRANS_EVENT ,FALLBACK ,
     NO BEFORE JOURNAL, NO AFTER JOURNAL,
     DATABLOCKSIZE = 524288 BYTES, CHECKSUM = DEFAULT,
     DEFAULT MERGEBLOCKRATIO, MAP = TD_MAP1
     (
      SB_Event_Id INTEGER NOT NULL,
      SB_Event_Processing_Dt DATE FORMAT 'YYYY-MM-DD',
      SB_Activity_Cd INTEGER COMPRESS (10251 ,10000 ,-10 ),
      SB_Host_Val CHAR(10) CHARACTER SET LATIN
          NOT CASESPECIFIC
     )
PRIMARY INDEX ( SB_Event_Id );
Seven marked clauses, none of which is a column. This body is taken from the shipped example corpus rather than written for this page — the shape is one the engine is tested against.

Output — parse tree, by grammar rule

intermediate rules elided

teradata_sql_compilation_unit
└── teradata_sql_statement
    ├── teradata_sql_table_multiplicity MULTISET
    ├── teradata_sql_fallback_table_option FALLBACK
    ├── teradata_sql_before_journal_table_option
    ├── teradata_sql_data_block_size_table_option524288 BYTES
    ├── teradata_sql_checksum_table_option DEFAULT
    ├── teradata_sql_merge_block_ratio_table_option
    ├── teradata_sql_map_table_option TD_MAP1
    ├── ⋯ column definitions
    └── teradata_sql_primary_index SB_Event_Id
These are the real rule names in the shipped grammar. The AST classes are generated from that grammar with Starlasu annotations rather than written by hand, so the model and the rules cannot drift apart; every node carries a position and difficult input returns an issue list, not an exception.

BTEQ: what actually happens to the dot-commands

This is the first question every Teradata prospect asks, and it deserves an exact answer rather than a reassuring one.

The lexer rule, as shipped

TeradataSqlLexer.g4

BTEQ_COMMAND
    : {this.getCharPositionInLine() == 0}?
      (PERIOD|EQ) .*? SEMICOLON? '\n' -> channel(HIDDEN)
    ;
Read it literally. A line that begins at column 0 with . or = is consumed to the end of the line and routed to a hidden channel. So a real BTEQ script does not break the parser and the SQL inside it parses — but the dot-commands themselves are not parsed. There is no node for .IF ERRORCODE, no branch structure, no .EXPORT target. The column-0 predicate is what keeps db.table and the = operator from being swallowed with them.

FastLoad, MultiLoad, TPump and TPT are a different matter: they are separate utility languages with their own control syntax, and they are out of scope — no token, no rule, no fixture. If your corpus is mostly MultiLoad scripts, this is not the product, and we would rather tell you on the first call.

What it handles

Four constructs that decide whether a parser survives a real warehouse export. None of them is exotic Teradata; all four are in the first thousand lines of any DDL dump.

COMPRESS value lists

A column definition can carry a parenthesized list of dozens of literal values — negative numbers, space-padded CHAR constants — and it is syntactically indistinguishable from any other parenthesized list without knowing where you are. It is also the clause that has no target-platform equivalent at all.

  cd INTEGER COMPRESS (10251 ,10000 ,-10 ),
  val CHAR(10) COMPRESS
      ('0001      ','SSVS01    ')

QUALIFY, a fourth filtering clause

Alongside WHERE and HAVING, evaluated after the window functions, which means the clause-ordering grammar is not the ANSI one. Teradata also spells row limits TOP n with no parentheses — the opposite of Transact-SQL, and a genuine source of silent mis-parses.

SELECT TOP 10 acct, amt,
       RANK() OVER (ORDER BY amt DESC) r
  FROM ledger
QUALIFY r <= 3;

LOCKING as a statement prefix

A modifier that comes before the statement it modifies, at database, table, view or row scope, with seven lock types including READ OVERRIDE and LOAD COMMITTED. LOCKING ROW FOR ACCESS SELECT … is a Teradata idiom with no equivalent anywhere else, and it is all over production reporting code.

LOCKING ROW FOR ACCESS
SELECT acct, bal FROM prodeiw.balances;

REPLACE as a DDL verb, and period types

Where every other dialect writes CREATE OR REPLACE or ALTER, Teradata writes REPLACE PROCEDURE, REPLACE VIEW, REPLACE FUNCTION — a one-word tell that this is not ANSI SQL. NORMALIZE with MEETS OR OVERLAPS brings temporal period semantics that the other four dialects simply do not have.

REPLACE VIEW prodeiw.v_open AS
SELECT NORMALIZE ON MEETS OR OVERLAPS
       acct, valid_period FROM prodeiw.spans;

Coverage, by construct

Every rule name in the middle column is a rule in the grammar we ship. The three “not supported” rows are the ones that decide deals, so they are on the page rather than in an appendix.
Construct Grammar rules Status
Table physical design — multiplicity, fallback, journaling, block size, checksum, merge block ratio, map, log, free space, isolated loading teradata_sql_table_multiplicity, teradata_sql_fallback_table_option, teradata_sql_data_block_size_table_option, teradata_sql_map_table_option Supported
Indexing and partitioning — primary, AMP, unique, no primary index, column and period partitioning teradata_sql_primary_index, teradata_sql_primary_amp_index, teradata_sql_partition_by_index, teradata_sql_column_partition Supported
Column attributes — COMPRESS lists, FORMAT, TITLE, CASESPECIFIC, character sets column definition rules Supported
Locking, at database, table, view or row scope teradata_sql_locking_statement, teradata_sql_lock_type Supported
Optimizer statistics, including sampling and thresholds teradata_sql_collect_statistics_statement, teradata_sql_collect_statistics_using_option Supported
Query surface — QUALIFY, TOP n, SAMPLE, NORMALIZE, EXPAND ON, MERGE, UPDATE … FROM, USING teradata_sql_merge_statement, teradata_sql_update_from_statement, teradata_sql_using_statement Supported
Stored procedures — CREATE/REPLACE PROCEDURE, labeled BEGIN…END, declarations, condition handlers, cursor declarations, IF/ELSEIF/ELSE, CALL teradata_sql_compound_statement, teradata_sql_handler_declaration, teradata_sql_cursor_declaration, teradata_sql_if_statement Supported
Functions, with the full option set — DETERMINISTIC, null-input behavior, security, parameter style, GLOP sets CREATE/REPLACE/ALTER/DROP FUNCTION Supported
BTEQ dot-commands BTEQ_COMMAND Recognized at column 0 and sent to a hidden channel — scripts parse, the commands themselves do not
Cursor manipulation — OPEN, FETCH, CLOSE teradata_sql_cursor_declaration Declaration only
Loops in stored procedures — WHILE, LOOP, REPEAT, FOR, ITERATE, LEAVE Not supported — no loop construct of any kind
Macros — CREATE MACRO, REPLACE MACRO, EXECUTE Not supported — there is no MACRO token in any grammar file
ALTER TABLE, GRANT/REVOKE, EXPLAIN, HELP, CREATE DATABASE/USER, dynamic SQL Not supported
FastLoad, MultiLoad, TPump, TPT control languages Out of scope — separate utility languages

We do not publish a certified Teradata release range, and we are not going to invent one. What can be said is what the grammar demonstrably accepts: the MAP = TD_MAP1 clause in the shipped corpus is a 16.10-and-later feature, so the grammar is at least that modern. Every release of the engine ships its own example set, and that set is the honest record of what each version added.

v0.1.1v0.1.2v0.1.3v0.1.4v0.1.5v0.1.6v0.1.7v0.1.8v0.1.9v0.1.10v0.1.11v0.1.12

One file that only this engine takes

We license five SQL-family engines. This is what the first page of a Teradata DDL export looks like, and exactly one of the five reads it.

Input — warehouse export, table plus statistics

prodeiw_ddl.sql

LOCKING ROW FOR ACCESS
CREATE MULTISET TABLE prodeiw.SB_TRANS_EVENT ,FALLBACK ,
     NO BEFORE JOURNAL, NO AFTER JOURNAL,
     DATABLOCKSIZE = 524288 BYTES, CHECKSUM = DEFAULT,
     DEFAULT MERGEBLOCKRATIO, MAP = TD_MAP1
     (
      SB_Event_Id INTEGER NOT NULL,
      SB_File_Number SMALLINT NOT NULL,
      SB_Event_Processing_Dt DATE FORMAT 'YYYY-MM-DD' NOT NULL,
      SB_Activity_Type_Cd INTEGER NOT NULL COMPRESS (10251 ,10000 ,-10 ),
      SB_Host_Activity_Val CHAR(10) CHARACTER SET LATIN
          NOT CASESPECIFIC COMPRESS ('0001      ','SSVS01    ')
     )
PRIMARY INDEX ( SB_File_Number );

COLLECT STATISTICS USING SAMPLE 10 PERCENT
   COLUMN (SB_Event_Processing_Dt) ON prodeiw.SB_TRANS_EVENT;
The table body is taken verbatim from the shipped example corpus, so this is a tested shape rather than an illustration. Nothing else on the list accepts it: Oracle PL/SQL has no MULTISET table, no journaling clauses and no PRIMARY INDEX; Transact-SQL spells physical design WITH (DISTRIBUTION = …); Firebird and our generic SQL engine have no equivalent surface at all.

You do not have to take this on trust. Paste a statement of your own into the Strumenta Playground, pick Teradata SQL, and read the tree the engine actually produces — including the issue list when it does not like something.

WHAT THIS ENGINE IS FOR

The tree arrives in Python, and Spark SQL comes out the other end.

Data engineers do not work in Kotlin. Two things follow from taking that seriously, and both of them are shipped code rather than a roadmap.

First, a generated Python API. The same metamodel that produces the Kotlin AST classes also generates Pylasu node classes and the unserializers to go with them, so a Python wrapper reads the parser’s JSON straight into typed Python objects. The model cannot drift between the two languages, because one grammar generates both. No other SQL engine in our catalog has that.

Second, a Teradata SQL to Spark SQL transpiler built on this parser. It is a separate component, and this page is not going to pretend it is a finished migration product — but the claim that Teradata trees can be walked into Spark SQL is demonstrated by code, not by a slide.

Be clear about the runtime: the Python package delegates parsing to the JVM engine and unserializes the result. A JVM is still required underneath. What you get in Python is the model and the API, not a pure-Python parser, and it is better to know that before your platform team designs around it.

# one JSON tree per source file
./gradlew run --args="parse prodeiw_ddl.sql"

# the metamodel, as EMF/XMI
teradata-sql-parser metamodel --output TeradataSql.xmi

# and the generated Python API
starlasu-tools nodegen --using pylasu \
                       --from TeradataSql.xmi
starlasu-tools unserializergen --using pylasu

The command line, the metamodel export, and the generator that produces the Pylasu node classes. The AST documentation is generated from the same grammar, so the document and the model are never out of step.

6,336
LINES OF ANTLR GRAMMAR, FOUR FILES
12
RELEASES WITH THEIR OWN EXAMPLE CORPUS
7
LOCK TYPES, AT FOUR SCOPES
2
LANGUAGES THE AST IS GENERATED FOR

What you receive

A jar, a Python package on top of it, the generated documents that describe the tree, and a person to write to. It runs inside your own network, with no connection back to us and none to the warehouse the DDL came from.

01 The engine

A JVM library that is also its own command-line tool

Kotlin on the JVM, built with ANTLR on Kolasu — the JVM implementation of Starlasu. Use it as a library, or run parse over a directory of exports and collect one JSON tree per file. The Python wrapper sits on top of the same jar.

library · CLI · Python wrapper
02 The model

An AST generated from the grammar, in two languages

A typed AST with positions on every node and a positioned issue list on difficult input. Export the metamodel as EMF/XMI, serialize trees as JSON, and generate the Pylasu Python classes from the same metamodel. The engine supports LionWeb, so the model travels to tooling we did not write.

JSON · EMF/XMI · Pylasu · LionWeb
03 The documents

Generated AST documentation, and the release corpus

The AST documentation is generated from the grammar rather than written alongside it. With it comes the example set for each release, which is the clearest record we have of what the engine accepts — and the one artifact we would rather you read than a coverage percentage.

AST document · examples per release
04 The license

Standard, Distribution or Service — with support included

Standard for use inside your own organization, Distribution if the engine ships inside a product you sell, Service if it runs behind a service you operate. Support is part of the license, not an upsell. The guarantee runs a year and is extendable, or can be taken pay-as-you-go.

three tiers · support included
Onboarding

Week one is running the command line over your own DDL export and reading the issue list — not waiting for an environment.

A warehouse export is a single large file or a directory of them, which is the easiest possible input to test with. If the integration needs help, that is what the support in the license is for; if it needs more than help, we do that work too, as training or architectural design alongside your team. Write to products@strumenta.com.

THE REST OF THE ESTATE

The warehouse is the destination, not the source.

Nothing loads a Teradata warehouse by hand. The rows arrive from a mainframe COBOL batch or an IBM i application, and the columns you are trying to trace were named by a program written twenty years before the warehouse existed.

“Where does this column come from?” is not a Teradata question. It starts in a COBOL copybook or an RPG file definition, passes through an extract, and lands in a table whose PRIMARY INDEX somebody chose to match a key that only makes sense upstream. An inventory that stops at the warehouse edge answers the easy half.

Every Strumenta engine is built on Starlasu, so a Teradata tree, a COBOL tree and an RPG tree have the same shape, the same traversal model and the same API. One tool walks all of them — and where the SQL is embedded in the host program, it is a node in that program’s own tree rather than a string somebody has to re-parse. That is precisely what you cannot get by gluing unrelated open-source parsers together.

Every engine also supports LionWeb, so the models interchange with LionWeb-compliant tooling instead of being locked inside one process — and every engine is available through bindings from Java, Kotlin, Python, TypeScript and C#. The host application picks the language; the parser does not.

Engines in the same estate

one model, one traversal

COBOL engine →
The batch that feeds the extract.

RPG & DDS engine →
The IBM i side, file definitions included.

Generic SQL engine →
The lakehouse queries you land on after migrating.

The whole catalog →
15+ engines, one AST framework underneath.

Why license this rather than bend a generic SQL grammar

For Oracle or Transact-SQL there is a free public grammar to start from, and it is a reasonable starting point. For Teradata there is not: the platform is a closed commercial product and its documentation sits behind a login. The realistic free option is a generic ANSI SQL grammar, so it is worth being concrete about where that stops.

What a generic ANSI SQL grammar does with the first page of a Teradata export. This is not a straw man — it is the normal way teams discover the problem.
In your export A generic ANSI SQL grammar This engine
CREATE MULTISET TABLE … FALLBACK, MAP = TD_MAP1 Syntax error at MULTISET A rule per option, in the tree
COMPRESS (10251, 10000, -10) on a column No such column attribute Parsed, so it can be reported as dropped
DATE FORMAT 'YYYY-MM-DD' inside a type A string literal where a type belongs Supported
LOCKING ROW FOR ACCESS SELECT … A statement that starts with a modifier Supported
A BTEQ script with dot-commands at column 0 Fails on the first .LOGON Dot-commands hidden, SQL inside them parses
A model your tooling can share with COBOL and RPG Raw parse-tree contexts Starlasu AST, LionWeb, Python and JVM APIs
Someone to call when a customer file fails The issue tracker, and good luck Support included in the license

Writing the physical-design DDL alone is weeks of careful reading against documentation you have to have access to, and getting the BTEQ column-0 rule right is the kind of detail that only surfaces when a real customer script fails. The method behind this work has a name: read the Chisel Method, or Parsers and transpilers for the three ways into it.

What to discuss before you license

Six things we would rather tell you now than have you find in week three. Two of them lose deals, and they are still here.

  • Macros are not supported. There is no MACRO token anywhere in the grammar, so CREATE MACRO, REPLACE MACRO and EXECUTE macroname do not parse. Teradata macros are a real and widely used feature, and in some estates they carry a large share of the business logic. Count yours before anything else; if the answer is “most of it”, this engine does not fit yet and we should talk about what it would take.
  • Stored procedures have no loops. WHILE, LOOP, REPEAT, FOR, ITERATE and LEAVE are absent — the compound statement, declarations, handlers, cursor declarations and IF/ELSEIF/ELSE are all there, and then the loops are not. Neither are SIGNAL/RESIGNAL or OPEN/FETCH/CLOSE. A procedure with a cursor loop in it will not parse today.
  • BTEQ is skipped, not understood. Dot-commands at column 0 go to a hidden channel so your scripts do not break. There is no node for them and no control flow. .IF ERRORCODE <> 0 THEN .QUIT 8 is invisible to any analysis you build on the tree.
  • The load utilities are out of scope. FastLoad, MultiLoad, TPump and TPT have their own control languages, and none of them is in this grammar.
  • Parts of the operational DDL are missing. ALTER TABLE, GRANT/REVOKE, EXPLAIN, HELP, CREATE DATABASE and CREATE USER are not in the grammar. An export that is mostly CREATE TABLE is the good case; an export that is mostly ALTER TABLE history is not.
  • This is a young, focused component. It is at an early version number and we are not going to dress it up as a mature platform. It does one job — reading warehouse DDL and query code so a migration can be sized and executed — and the release-by-release example corpus is the evidence for what it does today. We do not publish a test count for it, so we are not printing one.

What we will ask you

  1. Is your corpus DDL exports, BTEQ scripts, macro bodies or stored procedures — and roughly in what proportions?
  2. Do your procedures use loops or cursor manipulation?
  3. Are FastLoad, MultiLoad or TPump scripts in scope?
  4. Which Teradata release, and do you use period data types, NORMALIZE or column partitioning?
  5. What is the target platform — Spark SQL, where a transpiler already exists, or something else where you supply the generator?
  6. Kotlin, Java or Python for the consuming code? Remember the Python API still needs a JVM underneath.
  7. Do you want the tree as JSON, as EMF/XMI, as LionWeb, or as in-process objects?
  8. How many statements and how many distinct tables? Sizing changes the shape of the first engagement.

Straight answers

Can we feed it our BTEQ scripts as they are?

Yes, and the SQL inside them will parse. A line beginning at column 0 with a dot or an equals sign is consumed to the end of the line and sent to a hidden channel, so the dot-commands do not break the parse. What you will not get is a model of them: no node for .IF ERRORCODE, no branch structure, no .EXPORT target. If your analysis needs the script logic and not just the statements, that is extra work and we will scope it as such.

Does it handle Teradata macros?

No. There is no MACRO token in any of the grammar files, so CREATE MACRO, REPLACE MACRO and EXECUTE are not supported today. We know that matters — macros carry real logic in many estates — which is why it is on the page rather than in a footnote. Tell us what proportion of your corpus they are and we will tell you honestly whether to buy this now.

Can we work with the tree from Python?

Yes, and it is generated rather than hand-maintained: the same metamodel that produces the Kotlin classes generates the Pylasu node classes and their unserializers, and a Python wrapper reads the parser’s JSON into typed Python objects. One caveat that matters for your architecture — parsing still happens on the JVM. The Python package delegates to it. There is no pure-Python Teradata parser here.

Does it convert Teradata SQL to something else?

There is a Teradata SQL to Spark SQL transpiler built on this parser, as a separate component. It is real code and it is the reason we can talk about targets rather than only about analysis. It is not a finished, warranted migration product, and this engine does not write Teradata SQL back out from a modified tree at all. For any other target, you get the model and write the generator — which is a normal and estimable piece of work.

What is the first useful thing we can do with it?

Parse the DDL export and count. Tables by multiplicity, by index strategy, by partitioning; which tables carry COMPRESS lists that the target platform has no equivalent for; which have no statistics collected. That inventory is normally the first paid engagement in a warehouse migration, it is the artifact the bid is built on, and it needs nothing but the export and the command line.

Send us a DDL export and a BTEQ script, if you have one

Bring us the export, and we will count what parses.

A representative slice — the CREATE TABLE statements, a few stored procedures, one real BTEQ job — and we will come back with what the engine reads, how many macros and loops are in the way, and what an inventory of your warehouse would actually cost. If the answer is that this is not the right product yet, we will say that.

Scroll to Top