Skip to content

Extending grammars ​

Two grammars often overlap almost entirely: a base language and a dialect that adds or tweaks a few rules. Think JSON versus a lenient JSON with comments and trailing commas, or CSS versus a Less/Sass superset. Rather than copy the base and edit it, compose it — take the base grammar and fuse your changes on top.

compose() ​

compose([...]) fuses grammars into one runnable parser. Later entries override earlier ones by rule name:

ts
import { rules, regex, choice, compose, run } from 'parseman'

const base = rules(g => ({
  Value: choice(g.Num, g.Word),
  Num:   regex(/[0-9]+/),
  Word:  regex(/[a-z]+/),
}))

// A dialect that only redefines Num (must end with '!').
const dialect = rules(() => ({ Num: regex(/[0-9]+!/) }))

const parser = compose([base, dialect])
run(parser.Value, '12!')   // ✅ matches — via the overridden Num
run(parser.Value, '12')    // ✗ no match — dialect's Num needs '!'
run(parser.Value, 'abc')   // ✅ Word still works

A grammar (the result of rules(...)) is composable as-is. There's no wrapper to opt into and no special export — every unlisted rule is inherited, and the listed ones override.

Override is open-recursive ​

This is the key property, and it's what a plain object merge can't give you: overriding a rule reroutes every reference to it, including references inside the base's own rules. Above, base.Value calls g.Num — and after compose, that call resolves to the dialect's Num. Composition re-binds all rule references in one shared scope, so the base's internals see your overrides too.

The references it re-binds are the g.X references in each composed grammar's own rules() factory. A factory that returns a rule object taken from another grammar (Entry: base.Entry) brings that grammar's references with it, so compose() refuses one whose graph references a rule the composition also defines: the interpreter would keep base's rule while compile() binds the composition's, and they'd accept different input. compile() of a plain rule map refuses it the same way. Compose base itself instead, so its references resolve to the composition's rules. A borrowed rule that references nothing the composition defines is fine (test/parity/interpreted-fuse-parity.test.ts).

Re-reduce or extend an inherited rule: prior ​

An override replaces a rule outright. To change only part of it, take the factory's second parameter, prior. prior.X is the definition of X that the grammars composed before this one gave it, with a top-level node() unwrapped to its body. Use it to build a different node from the same grammar, or to add an arm to an inherited choice, without copying the rule:

ts
// [verify]
import { choice, compose, node, regex, rules, run, type PriorRules } from 'parseman'

const base = rules(g => ({
  Value: node('Value', choice(g.Num, g.Word), children => ({ base: children[0].value })),
  Num:   regex(/[0-9]+/),
  Word:  regex(/[a-z]+/),
}))

// The same grammar for Value, a different node.
const reduced = rules((g, prior: PriorRules<typeof base>) => ({
  Value: node('Value', prior.Value, children => ({ dialect: children[0].value })),
}))

// Value's choice plus one arm.
const extended = rules((g, prior: PriorRules<typeof base>) => ({
  Value: node('Value', choice(prior.Value, regex(/@[a-z]+/)), children => ({ dialect: children[0].value })),
}))

run(compose([base, reduced]).Value, '12').value
// → { dialect: '12' }

run(compose([base, extended]).Value, '@x').value
// → { dialect: '@x' }

prior.X's own g. references stay bound to the composition, so it reads every override, including ones this grammar makes. prior.X comes from the pieces composed before this one, never from this piece or a later one. A composition where no earlier piece defines X refuses prior.X, at runtime and in the build, and a grammar run before anything is composed under it throws the same message on every engine.

Annotate the parameter with PriorRules<typeof base>, as above. prior.X is then typed, and a misspelled rule fails to compile. Without it the parameter answers any string, and under noUncheckedIndexedAccess each read is Combinator<unknown> | undefined.

Keep the node() when you extend an inherited node. prior.X is the node's body, so X: choice(prior.X, arm) is a rule with no node: the inherited node and its reducer are gone from every path through X, and the children reach the enclosing node instead. Write X: node('X', choice(prior.X, arm), reduce).

prior.X is the earlier definition itself, placed in the new rule. So it parses and encodes exactly as the rule written out by hand would, at no extra cost. A grammar built on its own, before anything is composed under it, ships prior.X in its carried pieces, and the composition that imports it fills it in (test/unit/rules-prior.test.ts).

Assembling one grammar from parts of several ​

To borrow a piece of another grammar — say a mixin rule from one dialect and a loop from another — factor the reusable bit into its own small rules({ trivia }) grammar and compose() it in. A piece only needs to define its own rules. Anything it references by name (values, identifiers, whitespace) resolves to the composing grammar's versions, so a borrowed rule automatically picks up the host grammar's syntax and trivia:

ts
// A package exports the mixin machinery as a standalone composable grammar…
export const mixins = rules({ trivia }, (g) => ({
  MixinCall: sequence(g.Selector, literal('('), g.args, literal(')')),
  // …references g.Selector / g.args by name — the composing grammar supplies them.
}))

// …and a consumer composes just that piece in:
const parser = compose([css, mixins, myDelta])

Because references resolve by name across compose(), you don't have to extract a dependency closure — you just name the shared rules, and the host grammar supplies them, along with its trivia, since the composing grammar always wins.

Shared shapes: one shape, many bindings ​

The same mechanism lets you factor out a composite shape whose leaves differ per dialect. A ratio is <value> '/' <value> in every CSS dialect — what counts as a <value> is what differs (a number; a number or an interpolation). Write the shape once, leave the leaf as a hole, and let each consumer bind it:

ts
// @scope/shapes — the shape, with a hole
import { rules, literal, sequence } from 'parseman' with { type: 'macro' }
export const ratio = rules(g => ({ Ratio: sequence(g.Value, literal('/'), g.Value) }))

// a dialect binds its own Value, and builds its own tree on top
import { composeLeaf, node, regex, rules } from 'parseman' with { type: 'macro' }
import { ratio } from '@scope/shapes'
export const parser = composeLeaf([ratio, rules(g => ({
  Value:    regex(/[0-9]+/),                       // …or /@\{[a-z]+\}|[0-9]+/ next door
  Document: node('Document', g.Ratio, (children, _f, span) => ({ type: 'Ratio', children, span })),
}))])

A grammar with a hole isn't a runnable parser on its own — ratio.Ratio can't resolve Value until something supplies it — so its exported value stays an ordinary rules(…) map. It still ships fully compiled: the macro stamps its compiled pieces onto the value, and the consumer's compose() / composeLeaf() fuses them statically, with no base source and no runtime composition involved. A hole that nothing binds is a build error, not a silent drop.

Under composeLeaf, the usual rule still applies: every grammar before the final local one must be recognition-only. A shape can leave holes, but it can't carry reductions of its own — the semantics belong to the leaf that owns the tree.

The gating diagnostic follows the same logic: whether a shape's choice dispatches on the first character depends on what gets bound, so the shape itself isn't warned about it. The answer is computed — and reported — at each compose() / composeLeaf() that binds the hole.

There is one engine you ship ​

Under the macro, compose() / composeLeaf() lower to one static table: that's the compiled artifact you ship. Without the macro, they link the pieces into one interpreter grammar (see the execution modes below), and compile() turns that into a table at runtime when you want one. Either way the composition is the same grammar, with the same overrides and the same parse.

Building trees: swap the output shape ​

If your grammar's node() rules build an AST, compose() still lets a caller choose a different tree at parse time without changing the grammar — just pass a build host as ctx.build. cstBuildHost yields a uniform positioned CST from any grammar:

ts
import { compose, cstBuildHost, run } from 'parseman'

const parser = compose([base])
run(parser.Value, '12')                          // → the grammar's own AST
run(parser.Value, '12', { build: cstBuildHost }) // → a positioned CST node

This is how the same composed grammar can serve an evaluator (its own AST) and a language service (a CST with spans). Use node(..., { collapse: true }) for a grammar-local transparent wrapper, or cstBuildHost({ collapse }) when the caller should choose the public CST policy — it hides one-child wrapper rules without a separate post-processing walk. See incremental re-parsing for driving this in an editor.

No base source required ​

Here's the part that matters most for reuse: composing a grammar never needs the base grammar's source. When you build with the macro, an exported grammar automatically carries its compiled, composable form on the value — so import { base } is all a consumer needs. A downstream package just imports the compiled grammar and composes it:

ts
// @scope/base  →  ships a compiled grammar
import { rules, regex, choice } from 'parseman' with { type: 'macro' }
export const base = rules(g => ({ Value: choice(g.Num, g.Word), Num: regex(/[0-9]+/), Word: regex(/[a-z]+/) }))

// @scope/dialect  →  extends it, importing the COMPILED base
import { rules, regex, compose } from 'parseman' with { type: 'macro' }
import { base } from '@scope/base'
export const parser = compose([base, rules(() => ({ Num: regex(/[0-9]+!/) }))])

The dialect's build reads the base's compiled grammar — never its TypeScript source, and it never recompiles it. There's no "ship your source for speed" tradeoff here; a published, compiled-only package composes just fine.

Under rollup, rolldown or Vite, the composable form is build-only data, so it isn't in the artifact a parse loads. The plugin writes it beside each output chunk as <chunk>.parseman.json (lib/index.js → lib/index.js.parseman.json), and a downstream build reads that file. Ship it with the package. An ancestor that comes from another package is recorded as a reference to that package's export and resolved from the artifact's location, so it isn't copied into every descendant. Nothing at runtime reads carried pieces, and a composed artifact no longer imports its ancestors just to forward them. An entry's export is traced through its re-exports, export * included, to the module that declares it, and only that module's pieces are written for it: an export traced to a module that recorded none (a precompiled dependency, say) gets no sidecar entry and keeps its inline pieces. An ancestor the build bundles, rather than keeping external, is written out in full. Each sidecar records a digest of its artifact, and a downstream build that finds a different artifact beside it fails, as it does when a compiled grammar's pieces cannot be found at all: neither falls back to a runtime compose(). Under other bundlers the pieces stay inline on the value (test/unit/carried-pieces-sidecar.test.ts).

How this behaves in each execution mode ​

compose() works whether a grammar runs interpreted, via compile(), or via the macro:

  • Macro (build): compose([...]) is fused at build time into one static table. It needs no base grammar source (the pieces travel with the imported grammar as carried IR, beside the artifact or on its value, which the plugin re-lowers in the bundler and links in one namespace, as the runtime link below does) and constructs no code at runtime, so it ships in strict-CSP contexts like browser extensions or some CDNs with no extra configuration. An imported build-compiled grammar has to resolve at build time, including through a bundled upstream (esbuild's inlined ancestors, renamed imports and /* @__PURE__ */ keys are followed). If it can't, the build fails and names the upstream, because a runtime compose() can't link a compiled grammar (test/unit/compose-unresolved-upstream.test.ts).
  • Runtime (interpreter): compose([...]) only links. Each piece's rules() factory runs again against one shared namespace, a later piece's rule winning by name, so the result is the interpreter grammar one rules() over all the pieces would build. It builds no table, evaluates no source and never touches the pieces, so it runs under any CSP, and it links lazily: compose() itself only reads the pieces' rule names. Runtime compose() takes interpreter grammars — rules() maps and other runtime compositions. It refuses a build-compiled grammar, because linking one would mean evaluating its carried IR; compose that at build time instead.
  • compile() (runtime, opt-in): pass the composition to compile() for a table. compile(composed) encodes it once and specialises the table with new Function; under a strict CSP that throws EvalError and the same table runs on the closure assembler.

All three produce the same parse, with overrides resolved across the whole set, and report the same expected set when it fails (test/unit/compose-expected-parity.test.ts). test/unit/csp-runtime-paths.test.ts runs every runtime path under --disallow-code-generation-from-strings and asserts that nothing but compile() ever reaches Function or eval; test/parity/interpreted-fuse-parity.test.ts pins the runtime link to the compiled table.

Released under the MIT License. Commercial support available on request.