Skip to content

Macro mode ​

Add the plugin once, and your parser imports get evaluated and compiled at build time. The combinator import you mark with { type: 'macro' } disappears entirely, leaving an optimized TableProgram artifact in its place. That artifact imports the shared parseman/table runtime — it doesn't ship the combinator tree or the compiler.

You still call run() / parse() to execute the grammar, so Parséman remains an ordinary runtime import in your code. What the macro removes is the grammar-construction machinery — every combinator, the whole compiler — not the small driver that runs the result.

If you're shipping a package that contains a compiled parser, import the driver from parseman/run instead of the main entry. It's three modules instead of the whole library.

1. Register the plugin ​

Parséman ships an unplugin-based plugin, so one export adapts to every major bundler.

ts
import parseman from 'parseman/plugin'

export default {
  plugins: [parseman()],
}
js
import parseman from 'parseman/plugin'

export default {
  plugins: [parseman.rollup()],
}
js
const parseman = require('parseman/plugin').default

module.exports = {
  plugins: [parseman.webpack()],
}

2. Import with with { type: 'macro' } ​

ts
import { literal, sequence, choice, regex, transform } from 'parseman' with { type: 'macro' }

Same combinators, no other changes needed. The plugin walks each initializer, evaluates it at build time, and replaces it with an inline function.

Import attributes ​

The with { … } suffix is import attributes — standard JavaScript syntax from the TC39 import-attributes proposal. It attaches metadata to an import (or export … from) statement. The canonical standardized use is JSON modules:

ts
import data from './config.json' with { type: 'json' }

type: 'macro' isn't a TC39-defined module type. It's a bundler convention — the attribute tells the build tool to evaluate the import at compile time and inline the result, instead of bundling it for runtime. The same with { type: 'macro' } pattern shows up in Bun, Parcel, and unplugin-macros. Parséman's plugin keys off it the same way: see the attribute, compile the combinator tree, strip the attribute so the import stays valid for the interpreter fallback.

Older runtimes may still accept the earlier assert { type: 'macro' } spelling — import assertions, the syntax this replaced. TypeScript has parsed with { … } on imports since 5.3.

TypeScript config ​

TypeScript only accepts import-attribute syntax when module is set to esnext, nodenext, or preserve. With anything else you'll get:

text
TS2823: Import attributes are only supported when the --module option is set to
esnext, nodenext, or preserve.

For a bundler-based project the usual pairing is:

json
{
  "compilerOptions": {
    "module": "preserve",
    "moduleResolution": "bundler"
  }
}

What gets emitted ​

The bundle receives a version-stamped TableProgram: compact numeric rows, constant and callback pools, and bindings to the shared table runtime. The runtime links those rows to closure bodies once. Proven hot shapes — disjoint literal choices, scalar terminals, selected lexical programs — get direct bodies without introducing a second semantic parser alongside the table one.

  • Disjoint first characters → an indexed character dispatch instead of trying each arm.
  • Composed grammars dispatch too. When you compose([...]) separately compiled grammars, a choice arm that references a rule in another artifact keeps its first-char dispatch. Each artifact is compiled on its own, so the referenced rule's first-set isn't known yet at that point — the guard gets emitted as a placeholder and resolved at fuse time against the final rule's first-set. That stays correct even when a later artifact overrides a rule with a different first-set (open recursion). The dispatch survives the artifact boundary instead of degrading to an arm-by-arm scan.
  • Regex parsers → use a scalar recognizer where provably equivalent, otherwise a sticky /pattern/y stored in the artifact pool.
  • Failure paths allocate no objects.

For a deep dive on which regex shapes become scan loops, into what, and why some stay on the engine, see Under the hood: regex lowering.

What gets compiled ​

The plugin compiles combinator trees end to end:

  • All the core combinators — literal, regex, sequence, choice, many, oneOrMore, optional, sepBy, transform, skip, token, not, scanTo, balanced.
  • rules() factories, including mutually recursive ones — encoded as named table rules whose links are resolved after the complete rule map is known.
  • parser({ trivia }) / noTrivia() wrappers.
  • node() rules — CST capture, named field() capture, trivia logging, unwrap / collapse, and all — with build / transform callbacks retained in the artifact pool.

A full grammar built as a rules() factory of node() rules compiles end to end. Each rule stays independently callable, and static site labels let the linker skip capture, trivia, line, or recovery machinery wherever the final graph proves it's not needed.

Both binding forms compile:

ts
const { value } = rules(…)   // each rule becomes a top-level function
const grammar   = rules(…)   // an object literal of compiled rules; grammar.value(…) works

Grammar-wide rules(...) settings are read at the call site, which means one authored factory can produce several standalone macro artifacts:

ts
import { rules } from 'parseman' with { type: 'macro' }
import { grammarFactory } from './grammar.js'

export const grammar = rules({ trivia: rw }, grammarFactory)
export const grammarWithLines = rules({ trivia: rw, trackLines: true }, grammarFactory)
export const cstGrammar = rules({ trivia: rw, trackLines: true, hostMode: 'cst' }, grammarFactory)

trackLines fills in line/column fields on spans as CST nodes are created; hostMode: 'cst' builds through the CST host even when the grammar has direct AST builders. The macro emits each call site independently, so importing grammar doesn't drag in the line-aware or CST artifact along with it. See One factory, several macro artifacts and Line/column spans.

Sharing terminals across grammars ​

A grammar can import a terminal, a skip set, an options object or a boundary string from another module and use it anywhere the macro evaluates the grammar: in a rule body, in a combinator's arguments, or in a rules({ trivia, scanSkip }) option.

ts
// shared/scan-skip.ts — built with the plugin
import { literal, regex, sequence } from 'parseman' with { type: 'macro' }
export const blockComment = regex(/\/\*(?:[^*]|\*(?!\/))*\*\//)
export const doubleQuoted = sequence(literal('"'), regex(/(?:[^"\\]|\\[\s\S])*/), literal('"'))
export const scanSkip = [blockComment, doubleQuoted]
ts
// less/grammar.ts
import { rules, literal, scanTo } from 'parseman' with { type: 'macro' }
import { scanSkip } from '@my/shared/scan-skip'

export const grammar = rules({ scanSkip }, g => ({ Body: scanTo(literal(';')) }))

The exporting module is read, never run:

  • A relative import resolves to its source, and the declaration is evaluated the same way a local one would be.
  • A package import resolves to its published entry, which is a compiled artifact. A compiled terminal is a parser function, not a combinator, so the macro carries each terminal another module can reach — exported, or named by an exported value — with its combinator IR attached under Symbol.for('parseman.combinatorIR'). Plain values (strings, option objects, arrays of terminals) are kept as written. Both sides must be built by the same parseman version. A bundled entry works too: a bundler may emit each top-level const as var, and a top-level let/var the module never reassigns or redeclares is read like a const.

A terminal's node() builder can call a helper its own module imports. The compiled grammar imports that helper too, with the path re-spelled from the importing module. A helper that is a relative file inside another package has no stable path from outside it, so that fails the build; have the package import the helper by package name instead.

Each grammar gets its own copy of an imported terminal. Under compose() and composeLeaf(), scanSkip still applies only to the rules of the grammar that declared it, so each dialect in a family can name the one shared set by import.

A default or namespace import isn't resolved. Import the binding by name.

Parsers that close over external variables the evaluator can't resolve are left as-is. The plugin compiles what it can and leaves the rest to the interpreter, with a build warning pointing at exactly what it skipped — see When the plugin can't compile something below.

Code size — what to expect ​

Compiling trades bundle size for speed: a combinator grammar becomes a serialized table plus any assembly bodies selected for proven shapes, and that artifact ships in your bundle alongside the shared table runtime.

Measure in bytes, not lines. Generated lines get longer as well as more numerous, so a line-count multiplier understates the growth — GraphQL once moved +15.6% in bytes while its generated line count moved only +3.4%. The number that actually matters is raw generated bytes per byte of grammar source, because raw bytes are what V8 parses at import time.

The budget ​

Every shipped variant of a real grammar stays within 5× raw generated bytes per byte of its grammar source (the runtime and size contract, rule 4). The size-gate CI job enforces it on jess's four grammars, from a vendored snapshot bundled the way jess ships them (pnpm size:guard:jess), and fails any variant over the ceiling. The same job runs pnpm size:guard, which holds parseman's own probe fixtures to their committed sizes in bench/size-baseline.json.

Measured ​

pnpm size:probe measures the artifact users actually ship. Byte counts are deterministic, so it reproduces them on any machine, and pnpm size:guard fails CI when a probe grows past its ceiling in bench/size-baseline.json. A few representative rows, at 0.57.0:

ShapeSourceGeneratedRatioGzip
5 node() sites735 B2,011 B2.7×862 B
17 node() sites2,127 B5,924 B2.8×1,645 B
33 node() sites4,015 B11,092 B2.8×2,619 B
3-level composition1,984 B2,176 B1.1×978 B
trivia-enabled559 B1,698 B3.0×813 B

How it scales ​

Generated size is linear in node() call sites, not superlinear. The probe's 4-to-32-site series adds about 324 bytes per additional node() site, with roughly 391 bytes of fixed overhead. That replaces the obsolete direct-source-codegen estimate of about 4.2 kB per site.

That's why the source-relative multiplier looks like it's exploding while the per-node cost stays flat: the multiplier is a denominator artifact. A dense real grammar packs far more node() sites into each source byte than a toy fixture does. It also means the source-relative ratio isn't comparable between grammars that compose — a composing grammar's source doesn't contain the node sites it emits. Compare a grammar to its own baseline, not to another grammar.

The biggest cost: emitting the same grammar more than once ​

A grammar factory is usually compiled more than once. A CSS parser that has to serve both a build tool and an editor typically calls composeLeaf four times over the same recognition pieces, differing only in trackLines and hostMode:

ts
export const cssGrammar              = composeLeaf([...pieces, rules({ trivia }, f)])
export const cssLineGrammar          = composeLeaf([...pieces, rules({ trivia, trackLines: true }, f)])
export const cssCstGrammar           = composeLeaf([...pieces, rules({ trivia, hostMode: 'cst' }, f)])
export const cssDiagnosticCstGrammar = composeLeaf([...pieces, rules({ trivia, hostMode: 'cst', trackLines: true }, f)])

A module's terminal grammar — its composeLeaf(), or a compose() in a module that declares no composeLeaf() — gets a speed variant on its default AST/no-lines table: generated code for its hottest sites, within the plugin's assemblyBudget: a multiple of the module's own code, comments and whitespace aside, 0.5 by default, 0 for none. The closure engine runs every other site, and the two hand over at each boundary. The right budget is per grammar: a larger one parses faster in steady state and costs import and first-parse time, so measure both, and keep the artifact within 5× its source. The line-tracking and CST variants keep the compact closure representation, so four public variants don't mean four assemblies. It's still one TableProgram semantic implementation; the generated functions are a build-time materialization of part of the same table.

Sites are ranked by a static estimate of how often they run. The plugin's optional assemblyProfile: { entry, files } ranks them by measured calls instead: the build parses files from rule entry on the closure engine and keeps the sites with the most calls per byte, counting the calls a site saves from crossing between engines when a neighbour is already kept ("sites next to each other"). It changes which sites are generated, never what a parse returns (test/unit/table-selective-assembly.test.ts, like every claim below; the quoted names are its tests). A corpus file that doesn't parse in full or parses only by recovering, a corpus with nothing in it, or a reducer or gate that throws, fails the build ("a corpus that cannot run fails the build"). A file with only whitespace counts as empty: it's skipped, and a corpus of nothing else fails ("skips a whitespace-only file in a corpus that has content"). A build also fails where grammars were profiled and none has the entry rule ("a build where no grammar has the entry rule"). A direct transformMacro caller gets assemblyProfile: 'no-entry' on the result instead, and has to read it ("tells a direct caller that no grammar has the entry rule"). esbuild can't list a build's modules, so under it the check reads every module the plugin instance has seen, and a later build on that instance that dropped the module with the rule still passes ("reads every module the plugin has seen under esbuild, which can't list a build's modules"). A Vite dev server doesn't run the check ("a Vite dev server doesn't fail on a mistyped entry"). A callback the corpus doesn't reach never runs, so nothing checks it ("never runs a callback the corpus doesn't reach").

Callback source runs at build time with no module around it, so a grammar whose reducers or gates read imported helpers passes them as scope: assemblyProfile: { entry, files, scope: { valueStopOf } }. The names are visible as globals during the corpus run only ("scope gives the corpus run the helpers a gate imports, for that run only"). Without scope the build fails ("fails on a gate that reads an imported helper").

  • A name that's already a global fails the build ("refuses a name that is already a global, leaving it alone"). The opposite slip is silent: a helper named like a real global and left out of scope uses the global, so a grammar's own escape left out of scope would run the global escape ("runs the global when a helper named like one is left out of scope").
  • Direct node() builders don't run in the corpus run. A plain node stands in for each, so a helper only a builder reads doesn't fail the build, and the counts don't depend on what the builder does ("a plain node stands in for a builder carried from another module").

Each extra variant costs real bytes, but no longer a full copy. Measured by pnpm size:probe, holding the grammar constant and varying only the number of variants:

VariantsGeneratedvs 1 variant
11,309 B1.00×
22,136 B1.63×
43,782 B2.89×

Variant growth stays sublinear, but it isn't free. Measure the variants you actually ship — the selective assembly applies only to a terminal grammar's default AST/no-lines table, while the tracked and CST siblings stay compact.

Other measured axes, same probe and same baseline:

  • trivia enabled vs. disabled: 1.18×
  • hostMode: 'cst' vs. 'ast': effectively equal in the probe (1,906 vs. 1,901 bytes)
  • composition amortizes fixed structure: the three-level case comes to 2,176 generated bytes

Two things still keep this in perspective:

  • The combinator library and compiler don't ship in the artifact. Macro output imports the shared table runtime, so you're shipping the serialized parser plus that runtime — not the combinator tree and compiler.

    Shipping a compiled parser ​

    For a library whose published package contains a compiled grammar, import the driver from the dedicated entry:

    ts
    import { run, buildLineIndex, offsetToLineCol } from 'parseman/run'

    buildLineIndex and offsetToLineCol turn the offsets in a result's spans and errors into line and column numbers, so reporting a position doesn't need the main entry either.

    Its whole module closure is three files — the driver, the recovery helpers it hands to tolerant parses, and the capture buffer those use. The main entry pulls in the combinator set, compile(), the first-set analysis, and the CST builders, none of which a compiled parser touches. test/unit/run-entry-closure.test.ts pins that closure by module list, so it can't quietly grow.

  • Compression isn't the only budget. Gzip approximates transfer cost, but raw bytes still affect module parse time and memory. The size gate records both.

If bundle size matters more than raw throughput for a given grammar, reach for the interpreter (zero generated code, zero setup) or call compile() at runtime instead of using the macro. See the three modes.

Source maps and per-mode debugging are covered in Debugging compiled grammars.

When the plugin can't compile something ​

If the plugin meets a macro-imported declaration it can't compile statically — it closes over a runtime value, or isn't a recognized combinator shape — it:

  1. leaves that declaration for the interpreter,
  2. strips the with { type: 'macro' } attribute so the import stays valid, and
  3. emits a build warning ([parseman] file:line — …) pointing right at it.

So a silent fallback never goes unnoticed. You'll see exactly which rule dropped to the interpreter, and why.

Some things fail the build instead of falling back, because there's no correct fallback:

  • a rules({ trivia }) or rules({ scanSkip }) option the plugin can't evaluate, an options spread that could hide one, or a trailing rules(factory, options) argument that isn't an object literal. Building without the option would change what the grammar accepts.
  • an imported binding the plugin can't resolve, wherever the grammar uses it. At runtime a package's terminal is a compiled parser, which the interpreter can't use either.

The error names the option or declaration, the grammar, and each binding it couldn't resolve, with the reason.

Released under the MIT License. Commercial support available on request.