CST / AST nodes

For grammars that produce a typed syntax tree, support incremental re-parsing, or care about trivia, wrap each named rule in node(combinator, build?, opts?). There are two ways to get a tree out:

• A plain CST — omit build and let the library build a uniform positioned node for every rule. Fastest to write; see Just want a plain CST? below.
• Your own AST — pass a build callback that constructs whatever node shape you want from the captured children. Covered first.

Either way, Parséman captures the rule's terminals into children / rawChildren. When that node owns trivia capture, it also records trivia as flat [start, end, insertIdx, …] entries in triviaLog. Capture is the library's job — you don't wrap terminals to recover their spans, and you don't reconstruct trivia. It is collected as the parser runs, in both the interpreter and the compiled build.

import { rules, parser, node, regex, literal, sequence, many, trivia } from 'parseman'
import type { Combinator } from 'parseman'

// Any node shape works as long as it satisfies NodeLike (see below).
type N = { _tag: 'node'; type: string; span: { start: number; end: number }; state: unknown; children: unknown[] }
const ws = trivia(regex(/\s+/))

export const { Expr, Num } = rules<{ Expr: Combinator<N>; Num: Combinator<N> }>(g => {
  const num = node(regex(/[0-9]+/),
    (children, raw, span) => ({ _tag: 'node', type: 'Num', span, state: null, children: [...children] }))

  const expr = node(
    parser({ trivia: ws }, sequence(g.Num, many(sequence(literal('+'), g.Num)))),
    (children, raw, span) => ({ _tag: 'node', type: 'Expr', span, state: null, children: [...children] }))

  return { Expr: expr, Num: num }
})

Expr.parse('1 + 2 + 3', 0, { trackLines: false })
// value is a Node whose children are the captured Num sub-nodes and '+' leaves

What build receives

build(children, fields, span, rawChildren, triviaLog, state):

• children — structural items in source order: spanned CSTLeaf terminals ({ _tag: 'leaf', value, span }) and sub-nodes (whatever a nested node()'s build returned). A build that returns a bare string is recorded by the parent as a spanned leaf, so single-item "collapsing" rules keep their source span.
• fields — named captures from field(name, parser) inside this node, or undefined when the node has no captured fields or the build does not declare this parameter. A repeated field name becomes an array of captures.
• span — the full source span matched by this node.
• rawChildren — structural children only (same items as children, without trivia tokens).
• triviaLog — flat [start, end, insertIdx, …] entries for whitespace/comments consumed between terms. insertIdx is the rawChildren index before which the trivia was consumed. Pass the tree to buildTriviaIndex(tree, input) for a before/after map — useful for whitespace-sensitive syntax (e.g. CSS div p vs div.p).
• state — a snapshot of ctx.state at parse time (see Context-sensitive parsing).

Wrap a rule's inner combinator in parser({ trivia }, combinator) so trivia-skipping is baked in regardless of which rule you start from; the macro compiles the wrapper (and all capture) away to flat JS.

Scoped trivia ownership

There are three deliberately separate concerns:

• parser({ trivia, captureTrivia: true }, child) activates recording while child runs. It is useful for a local grammar scope; the surrounding parser state resumes when that scope ends.
• node(child, build, { captureTrivia: true }) owns the per-node log. The log belongs to that node's build/CST boundary, not to every combinator below it.
• node(child, build, { trailingTrivia: true }) is the narrow document-boundary form: after child succeeds, it commits one final run of the active trivia into that same node's log. It also enables that node's capture frame, even when its build/host would otherwise elide trivia.
• A direct build callback that declares its fifth triviaLog parameter retains the established capture behavior. That arity analysis is a performance optimization and compatibility rule, not a second parser-wide capture API.

Plain combinators such as sequence() and many() do not own nodes, so they do not expose or retain a triviaLog. Put the ownership boundary at node() instead of building a side channel or reconstructing source gaps later.

const ws = trivia(regex(/[ \t]+/))

const Pair = node(
  'Pair',
  parser({ trivia: ws }, sequence(regex(/[a-z]+/), literal(':'), regex(/[0-9]+/))),
  (_children, _fields, _span, _rawChildren, triviaLog) => ({ type: 'Pair', triviaLog }),
  { captureTrivia: true },
)

// Only Pair owns these offsets; the inner sequence and terminals remain plain combinators.
Pair.parse('name : 1', 0, { trackLines: false })

Use parser-level activation for a nested region only when that region is inside an owning node(). For example, node(... parser({ captureTrivia: true }, child) ...) records the inner region for that node without making unrelated sibling nodes retain trivia.

Terminal document trivia

many() correctly rolls back trivia before a failed next item: that run is terminal, not between two siblings. A document root that needs to retain a final comment can opt in without a source-gap scan:

const Document = node('Document', many(rule), undefined, { trailingTrivia: true })

Use this only at a meaningful terminal boundary, normally the document root. A block body followed by } already has a real following term, so its final trivia belongs to that block's normal node log; do not opt every nested node in. This keeps ordinary sibling-gap ownership unchanged and avoids a parser-wide capture tax.

Capture follows your build's arity

Building fields, triviaLog, and cloning state per node isn't free — on a value-dense grammar the per-token trivia-log push alone is a large slice of parse time. Parséman skips the capture your build never asks for: a build that declares only (children) gets no fields, trivia log, or state clone; (children, fields, span) gets named fields and spans but still skips raw children, trivia, and state; declaring triviaLog keeps the log; declaring state keeps the clone. This is inferred from the function's parameter list at compile time — you don't opt in.

The same inference runs at parse time for a structural node(parser) whose AST is built by an injected ctx.build host: Parséman reads the host's arity (build(type, children, fields, span, rawChildren, triviaLog, state)) and elides the trivia/state/field capture the host doesn't take.

Keep build hosts plain-positional

The arity check is conservative-by-necessity: Function.length under-counts a rest ((...args) =>) or default ((a, b = 1) =>) parameter, and can't see through a bound function. Parséman detects rest/default params and arguments and falls back to full capture (correct, just not the fast path). So a host written (type, ...args) => that reads args[4] still gets its trivia — but to keep the elision, declare plain positional parameters and drop the ones you don't use.

transform(p, fn) is still the tool for plain value-mapping (no children/trivia). node() is for CST/AST rules — it adds the capture transform doesn't. Both compile under the macro.

Just want a plain CST?

If you don't need a custom AST, omit build. A node(combinator) with no build callback is structural: it constructs its node through a host you supply via ctx.build, so the same grammar produces a plain CST for tooling (host set) and its own AST for evaluation (host unset, or a build callback). Inside rules(), the object key is the node type. Pass the built-in cstBuildHost and every rule becomes a uniform positioned node:

// [verify]
import { rules, node, regex, literal, sequence, many, parser, trivia, run, cstBuildHost } from 'parseman'

const ws = trivia(regex(/\s+/))
const g = rules(gg => ({
  Expr: node(parser({ trivia: ws }, sequence(gg.Num, many(sequence(literal('+'), gg.Num))))),
  Num:  node(regex(/[0-9]+/)),
}))

const r = run(g.Expr, '1 + 2', { build: cstBuildHost })

r.ok
// → true
r.value.type
// → 'Expr'
r.value.span
// → { start: 0, end: 5 }
r.value.children.length
// → 3

// No host at all: the SAME default CST node, spans included.
run(g.Expr, '1 + 2').value.span
// → { start: 0, end: 5 }

r.value is the CST — every node the same NodeLike shape, terminals as CSTLeaf:

Use node('Type', parser) when a rule needs an explicit public type or when the node is a local/manual helper outside rules().

{
  _tag: 'node', type: 'Expr', span: { start: 0, end: 5 }, state: null,
  children: [
    { _tag: 'node', type: 'Num', span: { start: 0, end: 1 }, state: null,
      children: [{ _tag: 'leaf', value: '1', span: { start: 0, end: 1 } }] },
    { _tag: 'leaf', value: '+', span: { start: 2, end: 3 } },
    { _tag: 'node', type: 'Num', span: { start: 4, end: 5 }, state: null,
      children: [{ _tag: 'leaf', value: '2', span: { start: 4, end: 5 } }] },
  ],
}

Visit it with createVisitor, and turn its trivia into a before/after lookup with buildTriviaIndex.

That is the whole of it for a structural grammar. No hostMode, no compile options — write productions, omit build, pass a host. If this is what you came for, you are done.

What you get with no host at all

Omitting build and the host is not an error and does not fall back to something lesser. A structural node with no ctx.build constructs the same default CST node — { _tag: 'node', type, span, state, children }, spans included. For the grammar above the two trees are byte-identical.

cstBuildHost earns its place for two other reasons: it is what makes a grammar that does have build callbacks produce a uniform CST anyway, and it takes a collapse option for public syntax trees.

Line/column spans

CST nodes carry offset spans by default:

{ span: { start: 12, end: 18 } }

Line/column fields are opt-in. Enable them at the parse call, at a parser(...) wrapper, or for a whole rules() grammar:

parse(root, input, { trackLines: true })
parser({ trackLines: true }, root)
rules({ trackLines: true }, factory)

With trackLines enabled, CST spans are filled as nodes are created:

{
  span: {
    start: 12,
    end: 18,
    startLine: 2,
    startColumn: 5,
    endLine: 2,
    endColumn: 11,
  }
}

Use rules({ trackLines: true }, factory) when the grammar itself is the artifact you ship, especially under the macro. A line-aware macro artifact is standalone: consumers run the compiled grammar and receive CST nodes with line/column spans without re-parsing or walking the tree afterward.

Keep trackLines off for the normal fast path. Offset-only spans are enough for most AST work, and the default emits no line-indexing code in macro output. If you only need line/column for a few diagnostics after parsing, keep offset spans and use the line-index helpers in the API reference.

When the grammar has its OWN builders

A grammar with no direct builders serves either consumer. The moment you add one, you need a second compilation for the CST consumer. That is the whole reason hostMode exists, and it is why nothing above this section mentioned it.

The "host set or host unset" switch above is a genuine per-parse choice for structural nodes — that is the node(parser) contract and it is unchanged.

A node with its own build callback is different. The callback owns the result, so the artifact has to be told at COMPILE time which consumer it is for:

const factory = (g) => ({ /* … the whole grammar, written once … */ })

export const grammar    = rules({ trivia: rw }, factory)                    // eval AST
export const cstGrammar = rules({ trivia: rw, hostMode: 'cst' }, factory)   // positioned CST

One source, two compilations — and each bundle tree-shakes away the artifact it does not import, so the eval path carries no CST capture and the tooling path carries no eval builders. The same option exists on compile and compose.

hostMode defaults to 'ast', so a grammar with build callbacks runs them, which is almost always what you want and is why nothing before this point had to mention the option. You add hostMode: 'cst' for the second artifact, not the first. This two-artifact pattern is the advanced case: most first grammars are structural and never need it.

Why not decide per parse? hostMode does not just pick which builder runs; it decides what every node captures (children, raw children, trivia log, fields, state). A runtime switch would keep all of that live on both paths, so an eval parse would pay for CST capture it never reads.

Getting it wrong is an error, not a degraded tree: driving an 'ast' artifact with a positioned-CST host throws, naming the fix. Without that check the builder's own object travels into the tree as a non-CST child, a CST child filter drops it, and the node simply vanishes from an otherwise successful parse.

Unwrapping and collapsing wrapper rules

Layered grammars accumulate "wrapper" rules that exist only for structure — an expression precedence ladder (Sum → Product → Primary), or a selector-list rule that wraps a single selector. When such a rule matches just one child, the wrapper node is noise: you want to be that child, not box it.

The unwrap and collapse options do that for grammar-local wrapper rules. Both skip build for a one-child match; zero or two-plus children go through build as normal. The difference is the shape of a single captured leaf:

• { unwrap: true } returns the leaf's string value.
• { collapse: true } returns the original CSTLeaf object, span included.

Set at most one of the two options on a given node().

import { node, choice, sequence, literal, regex } from 'parseman'

// A precedence level that's transparent when there's no operator.
const sum = node('Sum',
  sequence(product, many(sequence(literal('+'), product))),
  (children, raw, span) => ({ _tag: 'node', type: 'Sum', span, state: null, children: [...children] }),
  { unwrap: true },
)

Use collapse for the same grammar-local wrapper behavior when the single child must stay in CST form:

const componentValue = node('ComponentValue',
  choice(g.Function, g.Block, regex(/[^\s{}()[\];]+/)),
  (children, raw, span) => ({ _tag: 'node', type: 'ComponentValue', span, state: null, children: [...children] }),
  { collapse: true },
)

If the regex arm matches alone, unwrap would return the bare token string; collapse returns { _tag: 'leaf', value, span }.

Children captured	Result
0	build runs normally
1	build skipped — unwrap returns a leaf's string value; collapse returns the child exactly; sub-nodes are returned as-is
2+	build runs normally

So 2 parses to a bare Product node (no redundant Sum wrapper), while 2 + 3 produces a real Sum node with its children. You get readable layered rules without paying a build call per transparent layer — and without hand-writing if (children.length === 1) return children[0] in every wrapper builder.

unwrap and collapse produce identical results under the interpreter, compile(), and the macro build: the compiled output emits a children.length === 1 ? <single-child> : build(…) ternary, and the plugin reads static { unwrap: true } / { collapse: true } literals as the 4th argument.

Projecting one semantic child

Some AST rules are not one-child wrappers. They recognize punctuation, comments, or other syntactic scaffolding that a CST host must still see, while the AST value is one semantic child. A parenthesized expression is the small version:

const paren = node('Paren',
  sequence(literal('('), expr, literal(')')),
  { project: 1 },
)

AST mode returns captured child 1; the ( and ) leaves are still captured inside the node frame. Compile the same grammar with hostMode: 'cst', and the positioned-CST host receives the full child list, raw children, spans, fields, and trivia exactly like an ordinary direct node.

Projection is semantic value shaping, so a selected leaf becomes its string value, matching unwrap. A selected sub-node is returned as-is.

This deliberately lives on node() rather than in a separate value wrapper: node() is the grammar boundary that owns CST capture, host-mode selection, serialization, specs, coverage, and tracing. A projection option is plain serializable data, so composed grammars can carry it through IR without callback source.

Reach for project only when the selected child index is part of the grammar shape. The API is intentionally just a number. If the rule needs predicates such as "first value child", "all statement children", exact CST-leaf projection, or string reconstruction from several tokens, use a normal build callback.

Grammar collapse vs host collapse

node(..., { collapse: true }) is a grammar-local decision for one wrapper rule. For a public CST parser, cstBuildHost({ collapse }) lets the caller apply a host-wide collapse policy without changing the grammar's AST/value behavior.

Collapsing public CST wrappers

When a structural grammar is also a public CST parser, you may want the same transparent wrapper policy without changing AST/value behavior. Pass a configured CST host:

import { cstBuildHost, run } from 'parseman'

const r = run(g.Stylesheet, source, {
  build: cstBuildHost({ collapse: ['SelectorList', 'ComponentValue'] }),
})

This is CST-shaped collapse, not value unwrap:

Option	Use it for	One-child leaf result
node(..., { unwrap: true })	AST/value wrapper rules	the leaf's string value
node(..., { collapse: true })	grammar-local structural wrapper rules	the original CSTLeaf object, span included
cstBuildHost({ collapse })	caller-selected public CST wrapper policy	the original CSTLeaf object, span included

collapse only considers successful one-child nodes whose raw child list is also one item, so trivia-only matches and multi-token nodes keep their wrapper. The policy can be:

• true — collapse every one-child structural CST wrapper.
• ['RuleName', ...] — collapse only named node types.
• (type, child, children, rawChildren) => boolean — decide from the grammar type and captured CST children.

Because the policy lives on the build host, a composed grammar can expose a compact public CST while the evaluator keeps using the grammar's own AST builders. The interpreter, compile(), and macro output all check the policy while the node is being built, so there is no separate tree-normalization pass.

The NodeLike contract

Any AST your build callbacks produce participates in incremental re-parsing as long as it satisfies NodeLike — that's the whole contract:

type NodeLike = {
  readonly _tag: 'node'
  readonly type: string          // the rule name — used as the registry key on re-parse
  readonly span: Span
  readonly state: unknown        // ctx.state snapshot at parse time; replayed on edit
  readonly children: ReadonlyArray<{ readonly _tag: string }>
}

children only needs items carrying a _tag so traversal can tell sub-nodes (_tag: 'node') from anything else. The type string must match the rule name in the registry so edit() can re-parse the right rule.

The built-in CST leaf/node/error shapes are also exported as types — CSTNode, CSTLeaf, CSTError, CSTTrivia — if you'd rather use them directly. See the types reference.

Walking the tree

The tree is plain objects, so you can recurse it yourself. For typed traversal, createVisitor(grammar, spec) dispatches by concrete node type and by tags declared on node(..., { tags }). The same call works with an interpreted rules() grammar or a compiled/macro/compose() grammar.

Tagging related node types

Use tags when several concrete node types belong to the same semantic family. The node's type stays specific, while each tag gives visitors another stable dispatch key:

import { choice, literal, node, rules, sequence } from 'parseman'

const grammar = rules(g => ({
  Statement: choice(g.AtRuleWithBlock, g.AtRuleStatement, g.Declaration),

  AtRuleWithBlock: node(
    'AtRuleWithBlock',
    sequence(literal('@media'), literal('{'), literal('}')),
    { tags: ['AtRule', 'Statement'] },
  ),

  AtRuleStatement: node(
    'AtRuleStatement',
    sequence(literal('@import'), literal(';')),
    { tags: ['AtRule', 'Statement'] },
  ),

  Declaration: node(
    'Declaration',
    sequence(literal('color'), literal(':'), literal('red'), literal(';')),
    { tags: ['Statement'] },
  ),
}))

Tags are grammar metadata by default. They are available to createVisitor(...) without adding a tags property to every CST node.

Creating a visitor

Pass the grammar and a handler object to createVisitor. The returned function visits a tree depth-first:

import { createVisitor, cstBuildHost, parseDoc } from 'parseman'

const doc = parseDoc(grammar, 'Statement', '@media{}', { build: cstBuildHost })
const tree = doc.tree

const visit = createVisitor(grammar, {
  type: {
    AtRuleWithBlock(node) {
      // Only block-form at-rules.
    },
    Declaration(node) {
      // Only declarations.
    },
  },
  tag: {
    AtRule(node) {
      // Both AtRuleWithBlock and AtRuleStatement.
    },
    Statement(node) {
      // At-rules and declarations.
    },
  },
  enter(node) {},
  leave(node) {},
})

if (tree) visit(tree)

type keys are checked against the grammar's CST node types. tag keys are checked against tags declared by the grammar, and a node with multiple tags runs each matching tag handler in declared tag order. enter runs before type/tag handlers and leave runs after children. Return false from enter to skip a node's children.

The visitor does not need node.tags at runtime; it reads the grammar's reflection table and maps a node's type to its declared tags. That keeps the default CST shape lean and lets the same visitor work for interpreted grammars, compiled grammars, macro artifacts, and composed grammars.

Materializing tags on CST nodes

When a downstream tool wants to inspect tags directly from the tree, opt in at the CST host:

import { cstBuildHost, run } from 'parseman'

const result = run(grammar.AtRuleWithBlock, '@media{}', {
  build: cstBuildHost({ tags: true }),
})

if (result.ok) {
  result.value.tags
  // => ['AtRule', 'Statement']
}

The materialized array is the rule's static tag array. Untagged nodes still omit the property, and the default cstBuildHost omits all tags properties.

Next

• Wire your rule registry into a live-editing document in Incremental re-parsing.
• Thread parse-time context through nodes in Context-sensitive parsing.