Performance gates
parseman has four performance gates. They answer different questions, and the difference is the whole point.
| gate | asks | cost |
|---|---|---|
pnpm perf:guard | did parseman's own microbenchmarks move, especially the retained interpreter baseline? | seconds |
pnpm perf:guard:grammars | did a known cost AXIS move? | ~1.5-2 min |
pnpm perf:workloads | did realistic parsing get slower, on any axis at all? | ~2.5-3 min |
pnpm bench:macro-optimize | did the shipping Jess macro parsers stay correct and within their measured speed band? | several minutes |
Which check gives the verdict (#175)
Rule: a perf gate measures parseman under the V8 and Node defaults a user gets. No V8 flag, no NODE_OPTIONS, no v8.setFlagsFromString in a measured process. Pretenuring, GC timing and code layout are part of how a user's parser runs, so their effects are real regressions or real wins, and a gate that switched them off would not see them.
| check | measures | role in CI |
|---|---|---|
pnpm perf:xproc:gate | every density case and every workload, HEAD vs the previous release, each the plain parse (no trivia recorded); three more rows time the parse that records every trivia gap and are reported, never gated; one parser per process, each side's own dist/ build under plain node; 100–900 rounds per case (bench/xproc-gate.json) in 20 shards, pinned to Node 26 | the verdict, via xproc-perf |
pnpm perf:xproc:gate --peak | the workloads vs the fastest release on record, same design, the committed allowance as the floor | the verdict for the peak clause |
pnpm perf:guard:grammars, pnpm perf:workloads, pnpm perf:workloads:peak | the same cases, in one process, interleaved, with --no-compilation-cache | not in CI: local diagnostics only, and their output says so |
pnpm perf:guard, pnpm bench:macro-optimize | as below | as below |
The in-process gates are fast and good at saying WHERE a cost is. They put four instances of a case in one heap, though, and V8's per-instance choices land on one side of a pair. On identical code that read as a +3.4% head-side bias on expected/none and an ~8% build-position effect on json/document, and it failed 8 runs in 20. They also turn V8's compilation cache off, which the rule forbids in a measurement, so they do not run in CI: a number posted on a PR gets read as a verdict. The cross-process gate has none of those problems. Its design, its layout randomisation and its round count are in perf-harness-interleaving.md.
Of the four tools below, perf:guard runs in the pre-commit hook, and the macro gate runs on every code PR as a required result of test. Its identity check and a failed measurement fail the job; its timing rows stay advisory. Since #175 the grammar and workload A/B tools are local diagnostics, and the cross-process gate covers their cases in CI. The first three are self-contained in this repository. The macro gate also uses the pinned Jess commit named in bench/jess/macro-optimize-config.json; CI checks it out automatically.
The last two are not redundant, and which one you reach for depends on what you are doing. perf:workloads is the one that FINDS a regression: it parses real stylesheets and reports the time, so a cost on any axis — including one nobody has thought of — shows up. perf:guard:grammars is the one that EXPLAINS it: it holds everything constant except a named axis, so the spread across a sweep tells you what kind of cost you are looking at.
Neither substitutes for the other, and the history says so in both directions. The sweep read flat on fix(expect) when it had only one axis. The workload gate reads a real +49.6% Less regression as +2%…+9%, because a realistic parse spends most of its time on work the regression does not touch — which is exactly why the amplifying sweep is worth keeping.
These four are GATES: they answer "did it move?" The grammar and workload gates compare two builds in the same run. The fast guard compares against a committed baseline: interpreted rows block outside the measured 15% cross-machine tolerance, preserving the setup-free gains, while smaller compiled movement is reported for investigation and judged by the workload and comparison-chart gates.
The macro gate covers a different shipping shape. Jess loads Parseman's macro output, not the runtime compile() path exercised by the grammar and workload gates. It first requires full three-way result identity, then times the candidate against the 0.57.0 selective assembly with a separate matching A/A control. It was anchored on 0.50.2 until 0.57.0 replaced the whole-grammar assembly 0.50.2 embedded, which the runtime and size contract's rule 4 removes; bench/jess/macro-optimize-config.json records the reference. A run is invalid if any configured fixture is missing, either leg has the wrong engine or source path, the A/A control moves more than its measured 20% CI ceiling, or any paired candidate row is more than 3% slower. The A/A result validates the instrument but does not numerically rescale a candidate measured in another process: doing that turned a valid 1.013x paired CSS row into a synthetic 1.045x failure by dividing it by an independent 0.970x control. Across the four clean GitHub runs used to calibrate this gate, every raw paired candidate row was at or below 1.026x, while a valid all-winning run saw 15.7% separate-process A/A movement. Paired-vs-solo drift uses the same distinction, with a 20% runner floor while the candidate bar remains 3%. Invalid and regressed runs exit nonzero; the JSON report explains the rejection.
When a target is stated in absolute milliseconds against a named fixture — as the table lowering's is — the instrument is canonical-fixture-benchmark.md (pnpm bench:less), which is a MEASUREMENT rather than a gate and carries a load ceiling and a printed protocol for that reason.
The sweep gate — pnpm perf:guard:grammars
Why it exists
perf:guard measures fixtures/css/decls.css (47 bytes) and fixtures/css/selector.css (34 bytes), in microseconds, against a committed baseline. It is the interpreter's quick regression ratchet and a cheap tripwire for a catastrophic compiled-parser change.
It is not a gate on parseman's actual goal. During the 0.34.0 cycle it passed on every PR. It passed at 0.34.0. And 0.34.0 made a downstream Less grammar parse 25% slower — discovered by the consumer instead of at the source.
The mechanism was not()'s probe-leak fix (correct, and kept) emitting six unconditional capture-buffer length stores per probe. Assigning array.length runs V8's length setter — a backing-store trim decision — and costs the same whether or not the value changes. A rollback overwhelmingly restores a length that never moved, so the store was pure overhead on the common path.
The microbenchmarks execute not() about 20 times per KB and did not move. The Less grammar executes it about 600 times per KB and moved 25%.
That gap — trigger versus goal — is what perf:guard:grammars closes.
What it measures
Not a stylesheet. Two sweeps over one grammar shape (bench/grammar-density/grammar.ts) and one ~38 KB input, each holding everything constant except one axis.
rollback/* — speculative probes per byte. 0, 1, 4 and 16 negative lookaheads in front of every value term. The conversion is MEASURED by instrumenting the emitted artifact, not estimated: 3,556 / 14,224 / 56,896 not() executions over the 37.7 KB input, i.e. 94 per KB per guard — 0 / 94 / 377 / 1508. The real grammars in the 0.34.0 event measured css 20, jess 121, less 599, so all three land inside the sweep and dense sits 2.5x above the worst of them.
An earlier revision of this document said
x 42, which putdenseat ~672/KB and made less (599) look like a grammar sitting at the very edge of the bracket. It was not, and arollback/extremecase added to "extend the range" on that reading has been removed. If the input generator changes, re-measure this constant rather than scaling the old one.
expected/* — derived expected-set width at a losing choice. 0, 1 and 4 optional terms in front of every choice arm, all drawing on the same operand alphabet, so deriving through the nullable prefix re-reaches the same tokens once per term per arm. The emitted total-failure path is _ctx._fx = [...arm0, ...arm1, …], and those arrays feed the enclosing choice's concat, so the cost compounds up the nesting rather than adding.
none (0 terms) is a disjoint-arm baseline: with no nullable prefix its arms gate on distinct first characters, so it differs from the other two in dispatch as well as in width. narrow (1) and wide (4) both carry a prefix and so share a dispatch shape, differing only in width — read the width axis across those two.
This second axis exists because the first version of this gate had only the first, and 0.35.0 shipped a 32% Less regression straight through it — see "Watching it go red" below. A gate parameterised on one axis only ever catches that axis.
The point is not to look like CSS. It is to hold everything constant except the axis under test, so the result is legible: a per-execution cost shows up as an ordering across a sweep, and a per-site or per-input cost does not.
Every case builds nodes with trivia capture, which is load-bearing rather than decorative: the emitted rollback is gated on ctx.capturing, and each truncation is gated on its sink being non-null at run time. A grammar with no node() compiles the rollback away and measures nothing.
The grammar also wraps its declaration rule in attempt, so the six-sink transactional restore is exercised alongside not's. The 0.34.0 fix touched 33 emission sites; a gate that watches only one of them will pass the next regression in the other 32.
How it measures
A/B against a pinned reference commit of this repo, both sides loaded and interleaved in one process, rotating order per round.
Self-calibrating by construction: nothing machine-specific is stored, so it reads the same on a laptop and on a shared CI runner. That matters more than it sounds — comparing separate processes on this hardware produced 9.4 ms and 26 ms for the same case in consecutive launches, and it is the reason an older nightly-workflow proposal (never adopted) had to keep its timing tolerance at 40%. An interleaved A/B has no cross-machine term at all.
The reference side is a git worktree at the pinned sha with this repo's node_modules symlinked in — nothing needs a per-sha dependency tree, because src/index.ts is loaded through tsx and tsx only transpiles. The grammar file is copied from the working tree into that worktree, so both sides compile byte-identical grammar input and the only difference is parseman itself. The gate then asserts both sides produce the same parse result before timing either: the cheapest way for one side to look fast is to stop doing work.
Repetitions per timed sample are calibrated at startup so every case's sample lands near 4 ms. Without that the sparse cases run in ~0.5 ms, where timer granularity and a single GC pause dominate — self-vs-self noise measured 3.5% there against 0.4% on the dense case. Calibration runs on the reference side only and the same count is applied to both, so it cannot favour one.
Per-case, never aggregated
Replaying 0.34.0 against 0.33.0:
| case | probes/value | median delta |
|---|---|---|
rollback/none | 0 | +1.2% |
rollback/sparse | 1 | +54.9% |
rollback/medium | 4 | +89.0% |
rollback/dense | 16 | +112.8% |
Any aggregate shows something mild. The spread is the finding: it says the cost is per-execution, which is exactly how the real regression was diagnosed (css 20 / jess 121 / less 599 not() per KB, ordering −1.6% / +6.6% / +25.5%).
Replaying the 0.35.0 regression — --ref=9c6fee2 --head-ref=a464372, the merge of fix(expect) — reads the other way round, which is the whole argument for having both axes. Five replay runs, because one is not a measurement:
| case | median delta, 5 runs | pairs won | |
|---|---|---|---|
rollback/none … rollback/dense | 0–16 probes | −17.3% … +6.2%, no consistent sign | 0–12 / 12 |
expected/none | 0 opt/arm | −0.7% … +1.2% | 5–9 / 12 |
expected/narrow | 1 opt/arm | +1.8% … +21.1% | 0–3 / 12 |
expected/wide | 4 opt/arm | +6.9% … +39.3%, fired 5 of 5 | 0/12, every run |
expected/wide is the only case that breaches on every run. expected/narrow breaches on two of five, which is the right ordering — it carries a nullable prefix too, just one term of it. The rollback sweep — the entire gate as first landed — never fires consistently on a change that cost jess's Less grammar 32% of its parse time.
The wide spread inside each row is the machine, not the change. See the threshold caveat below: these runs were taken at load average ~5, where the harness's own self-vs-self floor measures far worse than the quiet-machine figure the thresholds were calibrated against.
Median AND min AND win rate
A single median is not a measurement. The first attempt at measuring the real regression produced a wrong number that way. The gate reports all three and requires two independent signals to fire: a case fails only when it breaches the median or min threshold and loses at least 3/4 of its interleaved pairs.
Win rate alone is lopsided even at noise-level deltas; a real regression loses nearly every pair — the 0.34.0 replay won 0/12 on all three guarded cases.
…and a MAJORITY OF PASSES — the fix that made the verdict honest
That was still not enough, and the failure was not subtle.
This gate carried its own copy of the measurement loop, written before bench/ab-harness.ts existed. That copy sampled the two sides as contiguous blocks rotated per round over the concatenated case list, so ref|expected/narrow and head|expected/narrow sat seven positions apart and never shared GC state, cache state or position in the run. Run with both sides pinned to the same commit — a byte-identical src/, at load average 8.1:
rollback/sparse median −9.2% min −6.5% won 12/12
expected/narrow median +23.3% min +10.6% won 0/12 FAILA 32-point spread and a hard FAIL between a build and itself. So won 0/12 was proving nothing: it did not discriminate a regression from noise, because the two sides were never measured under the same conditions. That is the signature CI hit on 40ce56b and 1c6f6a8 — two commits touching zero files under src/.
The fix was not a new statistic. ab-harness.ts had already diagnosed exactly this for the workload gate, and this gate now uses it:
- Adjacent pairing with alternating order — the two sides of a case are measured back to back, so they share GC and cache state, and which one goes first alternates.
- Three independent passes, strict majority — a burst lands in one pass, a regression lands in all of them.
- The sign test (
signTest.medianPct: 3) — a consistent win rate well under 50% is unambiguous even when the percentage is small.
--self was added so the noise floor can be re-measured wherever you are.
Validated in both directions
A gate is only known to work when it has been watched pass on identical source and fail on a known regression. Both, deliberately on a badly contended box:
| run | load average | result |
|---|---|---|
--self ×3 | 23→31, 31→26, 26→19 | PASS, no case false-failed (worst single pass +12.4% median, absorbed by the majority rule) |
identical src/ via --ref | 27.8→20.4 | PASS, 0/3 breaches on all 7 cases |
| injected +86% | 9→21 | FAIL, 7/7 cases, 3/3 passes |
| injected +4.5% | 19.7→19.5 | FAIL, 2 cases |
| injected +2.2% | 20→27.8 | FAIL, 2 cases |
| injected +1.1% | 22.5→30.6 | FAIL, expected/none 3/3 passes on a consistent 2/12 win rate |
The injection is a pure per-byte spin cost, calibrated at 0.936 ns/iteration (min of 40 × 2M) against a 38,625-byte input and each case's min-of-120 baseline.
Smallest reliably detectable regression: ~1–2% per case. At the bottom of that band it is the sign test carrying detection, not the percentage — the medians are noise-inflated at these load averages and should not be read as effect sizes. Note also that the injection is a flat per-byte cost and is therefore not amplified by these synthetic cases the way a genuine rollback-density or expected-set-width regression is; a regression riding either axis is caught proportionally smaller.
…and the passes were not independent — the COMPILATION LOTTERY (0.46.0)
Everything above is true and none of it was sufficient, because the majority rule and the sign test both rest on an assumption nobody had tested: that the 12 pairs of a pass are 12 independent trials, and that the passes are independent of each other. Neither was true. Both gates compiled each side once, before the pass loop, and reused those two instances for every pass.
Two independently compiled instances of identical code do not run at identical speed. V8 tiers, inlines and assigns feedback to them separately, and whichever instance wins stays the winner for the life of the process. So a pass does not resolve twelve coin flips; it resolves one draw of a compilation lottery, twelve times. The sign test's effective sample size was 1, and its quoted p ≈ 0.003 was a fiction.
Measured on this repo's density cases, every side byte-identical, three independently compiled pairs measured in the same loop at the same rotated run positions so that nothing but the instances differed:
| case | pair 0 | pair 1 | pair 2 |
|---|---|---|---|
rollback/none | 12/12, −7.8% | 5/12, +1.0% | 6/12, +1.0% |
expected/wide | 11/12, −6.0% | 7/12, +0.6% | 6/12, −0.0% |
expected/none (another run) | 0/12, +7.8% | 8/12, −0.1% | 8/12, −1.3% |
A perfect 12/12 and a perfect 0/12, with ±8% medians, on code that cannot differ. That single number is both of this gate's failure modes at once:
- a draw against the head side is a false FAIL at any percentage threshold, and reusing the instances made all three passes agree on it, so the majority rule endorsed it rather than absorbing it;
- a draw for the head side makes the case BLIND — a win rate whose null sits near 0.9 can never come down to a flat 0.25 ceiling, so a real regression there reads green, and any speedup it reports is the draw talking.
Two hypotheses were tested and refuted on the way to that. It is not the two module graphs: under --self both sides resolve to the same directory, so import() returns the same module object and the same compile function (asserted, both true) — and the skew is still there. It is not a stable property of a case either: the skew lands on a different case in almost every session, which is why an offline per-case calibration table would have been wrong.
Found later: the lottery had two concrete causes, and resampling as first built could reach neither. V8's compilation cache hands identical
new Functionsource back as the same compiled code and feedback, so a "recompiled" instance of byte-identical emitted source was the instance before it; and the parser's runtime helpers are optimized once per module graph, which passes never rebuilt. The--selfskew above fits the first cause — one module graph, one source text — though it was not reproduced to confirm.measurePasses()now turns the cache off and builds every instance from a fresh module graph.And a third, found after those two were fixed (#175). V8's allocation-site pretenuring flips a coin per instance: an instance whose allocation sites get pretenured runs 13–25% slower for the rest of the pass. On
expected/nonethe coin fell more often on the head side, so identical code read +3.4% and failed 7 runs in 20. That is V8 behaviour users get too, so it is not switched off. The verdict moved to the cross-process gate instead. See Three mechanisms behind it.
The fix is resampling, in measurePasses() (bench/ab-harness.ts).
- Both sides are recompiled every pass.
passesindependent passes are finally independent draws of the thing that dominates, which is what the majority rule always claimed to be doing.passeswas raised 3 → 5 at the same time — the response the--selfblock prescribes for a gate that reads the machine, and the response that tightens rather than widens. - A CONTROL pair measures the null win rate, in-process, every run. Two independently compiled reference instances — identical code, so every pair it wins is instrument and not compiler — measured in the same passes, at the same rotated positions, with the gate pair and the control pair alternating which gets compiled first (the first-compiled pair draws the skew disproportionately). Its pooled win rate is printed per case.
- The win-rate ceiling is null-relative, not absolute. A case is judged at
null − (0.5 − winRateCeiling). A case whose null is 50% is judged at exactly the configured 25%, so the calibration can never loosen an unbiased case; a case the instrument favours or disfavours is judged at the same distance from its own null instead of at an absolute rate it can never reach. - Calibration no longer warms one side.
calibrate()parses ~14 times before the pass loop and used to do it on the very instances the reference side then raced with. The repetition count it produces was always applied to both sides; the warming was not. It now runs on a throwaway instance set.
The null is measured rather than assumed because it is not stable, and it is worth reading. On a self-check after the fix, rollback/none drew 12/12 at −9.5% median in its first pass and 6/12, 8/12, 4/12, 6/12 in the other four: under the old design that first draw was the whole run, and the case would have reported a −9.5% "speedup" against a byte-identical tree. Any reading of that shape — 0.46.0's post-revert rollback/none at −22.5%…−11.4% is the live example — is the lottery, and the null column now says so on the same screen.
Validated on the null, five runs of each gate
Both gates, --self (the reference against itself, so every number below is instrument), on a shared machine at load average 2.9–8.6 — which is what this box looks like with other lanes on it.
| run | perf:guard:grammars --self | perf:workloads --self |
|---|---|---|
| 1 | 0/35 passes breached, worst +1.7% median / +1.7% min | 0/25, +12.6% / +2.0% |
| 2 | 0/35, +1.3% / +2.4% | 0/25, +1.8% / +2.4% |
| 3 | 0/35, +2.1% / +1.5% | 0/25, +2.2% / +3.5% |
| 4 | 1/35, +10.9% / +10.5% | 0/25, +1.7% / +2.7% |
| 5 | 0/35, +1.9% / +1.6% | 0/25, +2.0% / +4.2% |
No case false-failed in any of the ten runs. Measured null win rates stayed inside 36.7%–70.0% throughout, so the calibrated ceilings ranged 11.7%–45.0%.
Run 4 is the one worth reading, because it is the old failure caught in the act. rollback/none drew 0/12 at +10.9% median and +10.5% min in its first pass — a textbook false-fail signature, on byte-identical source — and its other four passes read 7/12, 8/12, 6/12, 6/12 at −1.4%…+0.0%. Under the old design that first draw was the run: all three passes shared those instances, all three would have breached, and the gate would have reported a hard FAIL against a build and itself. Resampling turned it into 1 breach of 5, which the majority rule absorbs.
The mirror image showed up in the first density self-check: rollback/none drew 12/12 at −9.5% in one pass and 6/12, 8/12, 4/12, 6/12 in the rest. That is where a quotable "speedup" against an identical tree comes from, and it is the same lottery with the sign flipped.
Thresholds, and where they came from
medianPct: 6, minPct: 6, winRateCeiling: 0.25, signTest: 3, passes: 5.
The scorer has a versioned reduction contract. paired-ratio-v2 takes the median of aligned HEAD/REF sample ratios and, for the floor signal, the median of aligned within-sample-minimum ratios. Ratio-of-aggregate medians/minima is the retired aggregate-v1 method: it discards the adjacency the harness measured and can report a 60% regression for a pair series whose aligned median is exactly flat. Changing the scorer invalidates percentage noise calibration. Both perf configs therefore name their scorer and fail closed for normal gates until a RED-proven --self A/A run revalidates the thresholds.
winRateCeiling is the ceiling for a null of 0.5; the ceiling actually applied is that number shifted onto the case's measured null, per the section above.
Measured, not guessed. The same build compared against itself through this harness (--ref=<head-sha> --head-ref=<head-sha>, 4 rounds × 3 runs, calibrated samples, interleaved with per-round rotation) moved the per-case median by at most 1.9% and the min by at most 1.0%. 6% is over 3× the worse of those.
That 1.9% was measured on a quiet machine, and is the number the block-sampled harness reported before the pairing fix. On a contended one the single-pass floor is far larger — up to +12.4% median in the runs above. The percentage thresholds were not widened to cover that; the majority-of-passes rule absorbs it instead, which is what keeps the gate sensitive at 1–2% while not firing on a busy runner. Widening the threshold to cover a burst would blind the gate to exactly the band it exists to watch.
What it does NOT catch — read this before trusting a green
Below about 3% per case is under the harness's resolution. More rounds would buy resolution at the cost of a slower gate, i.e. a gate that gets skipped. That trade was made deliberately. More rounds do not help anyway: the 12 pairs of a pass share one pair of instances. This gate is advisory since #175; the cross-process gate, which resolves a 2% cost, gives the verdict.
It is amplified, not more sensitive in real terms. The same 0.34.0 change that cost the real Less grammar +25.5% costs
rollback/dense+112.8% — about 4.4×, because the synthetic case is almost nothing but probes. Read a number here as roughly a quarter of itself on a real grammar: a 6% breach is something like a 1.5% real-grammar regression. That amplification is the gate's value (it sees small things early) and its limit (it cannot tell you what a given delta costs a particular consumer). Theexpected/*amplification is milder —expected/wideread +25.6% on a change worth +49.6% to jess's Less grammar, so under 1×; a breach there is at least as bad in real terms.It watches two axes, and only two.
rollback/*moves speculative probes per byte;expected/*moves derived expected-set width at a losing choice. A regression in trivia scanning, first-set dispatch, node construction or the interpreter can move every case in BOTH sweeps together — the gate will report that, but neither spread will explain it, because nothing is being held constant against it. No gate covers first-set dispatch on its own: thecomposeLeafdispatch assertionperf:guardonce carried asserted a property of textual fusion and went with the source lowering (bench/perf-guard.ts).The history here is the argument for reading this bullet literally rather than as boilerplate: the gate shipped with one axis and a regression rode the other through it in the very next PR. A third axis is an entry in
DENSITY_CASES; adding one is cheaper than the release that finds it for you.The 1.9% / 1.0% noise floor is a QUIET-MACHINE figure. Re-measured self-vs-self (
--ref=X --head-ref=X) at load average ~5, the worst case moved the median 8.3% and the min 7.6% — past the 6% thresholds. Interleaving and per-round rotation cancel a steady load; they do not cancel a bursty one at ~3 ms samples. In practice this shows up as a case breaching once and not breaching on a re-run. A single red run is not a regression. Re-run it, and re-run self-vs-self alongside if the machine is busy; a real regression loses 0-of-12 pairs every time, which is what distinguishes it from load.Losing 0-of-12 pairs is not by itself proof either. The two sides share a heap and a JIT profile, so a case near an optimization cliff can be pushed to a different optimization outcome by the mere PRESENCE of the other side's code — including by a branch it never executes. Compared against ITSELF at
d4f107f,expected/narrowread +25.0% median, +21.2% min, 0 of 12 pairs in one run of four and sat inside ±2.6% in the other three;rollback/nonewon 12 of 12 twice in the same four runs, which would mask a real regression rather than invent one. Cross-process the same self-versus-self pair reads neutral. Before calling a red a regression, confirm it withpnpm perf:xproc— see perf-harness-interleaving.md for the tells and the recipe.The synthetic grammar is not a correctness corpus. It asserts only that both sides parse identically; it says nothing about whether that parse is right.
The broad workload gate — pnpm perf:workloads
Why a third gate exists
The sweep gate above is parameterised. That is its strength for attribution and its structural limit for detection: a gate parameterised on one axis only ever catches that axis.
This is not a theoretical worry. perf:guard:grammars landed with a rollback-density sweep, and the very next merge regressed a different axis and went straight through it. fix(expect) derived expectations through nullable prefixes — a correctness fix, to error-message quality — and cost jess's Less grammar 49.6% of its parse time, because expected sets are built on every failed arm rather than only when an error is reported, so they sit in the hot path of a backtracking parser. Nobody would have predicted that from the diff. That is precisely the argument for a gate that is not targeted at anything.
So this gate is deliberately unparameterised. It parses realistic input with realistic grammars and reports the time. A regression on any axis — one already known, or the next one — shows up as time.
What it measures
Five workloads, in bench/workloads/, each ~50 KB:
| workload | grammar | what it is there for |
|---|---|---|
less/stylesheet | bench/workloads/less.ts | high speculative rollback, wide derived expected sets, full CST with trivia capture |
less/mixins | same grammar | the same grammar over input weighted towards mixin calls, guards and arithmetic — separates "the grammar got slower" from "this shape of source got slower" |
css/stylesheet | examples/css/parser.ts | the same problem domain at LOW rollback density |
graphql/document | examples/graphql/parser.ts | non-CSS, and non-CST: builds plain values through transform, so capture is off entirely |
json/document | examples/json/parser.ts | the same, at a much simpler grammar shape |
The corpora are hand-authored (bench/workloads/fixtures/app.less, site.css) and repeated with a per-copy identifier prefix to reach size. They are not copied from any third-party project.
less/stylesheet versus less/mixins is the one pair that varies exactly one thing — same grammar, different input mix. The cross-dialect rows are not controls: css/stylesheet differs from less/stylesheet in both grammar and input, so a difference between them narrows a cause but does not isolate one. Isolation is what perf:guard:grammars is for; say what a row is before reading it as a control.
Everything it needs is in this repository
No sibling checkout, no clone, no network. An earlier attempt at this gate cloned a downstream repository and was rejected for exactly that reason: a gate a contributor cannot run is a gate that does not run.
Vendoring is the cost of that. bench/workloads/less.ts is a Less-dialect grammar written for this purpose, in the same spirit as examples/css/parser.ts, which is already an adaptation of the same downstream project's CSS grammar. It is not a conformant Less parser and nothing asserts that it is. Its contract is narrower and stated in the file: it must exercise the compiler the way a real stylesheet grammar does.
Per-workload, never aggregated
Replaying 0.34.0's fix(not), less/stylesheet moves +41.8% and css/stylesheet moves −0.5% in the same process. Any mean of those five rows is mild and passes. Every row is thresholded on its own and the failure message names rows.
That ordering is also the check that the workloads are worth having: the real event measured less +25.5% and css −1.6%, and the gate reproduces the sign and the ordering, not just a number.
How it measures
The same A/B machinery as the sweep gate — a pinned reference commit of THIS repo, both sides in one process, no stored timings — factored into bench/ab-harness.ts so the two gates cannot drift apart on the parts that make a measurement a measurement. Two things are specific to this gate:
Sides are measured in adjacent, order-alternated pairs. Not as two blocks with a rotation. This is not a refinement: measured the block way, the reference side of a 50 KB CST workload read 38% slower than an identical build of itself. The workloads allocate heavily, whichever side runs first in a round eats the previous side's garbage, and a rotation by one over ten entries never moves a case far enough to cancel it. Directional bias of that size does not add noise — it MASKS a regression on the head side. Pairing dropped the self-vs-self floor from 38% to 2%.
Three independent passes, majority verdict. A workload fails only when a strict majority of passes breach. This is the gate's answer to "regression, or busy machine?", and it is enforced rather than documented — see the threshold section.
Thresholds, and where they came from
medianPct 5 minPct 5 winRateCeiling 0.25
signTest: winRateCeiling 0.25, medianPct 1.5, minPct 1.5
passes 5, majority requiredwinRateCeiling is the ceiling for a null of 0.5. Both gates share measurePasses(), so the ceiling actually applied here is that number shifted onto each workload's measured null — see the compilation lottery, which is where passes went from 3 to 5 and where the passes started being independent of each other at all.
Measured with pnpm perf:workloads --self, which runs the reference against itself, on a machine at load average 5–9 — because that is what a shared runner looks like, and the quiet-machine figure is the one that gets a gate laughed at the first time it false-fails:
- worst single-pass median +9.9%, worst single-pass min +3.4%
- worst absolute swing in either direction 12.3%
- passes that breached: 0 of 15. No workload false-failed.
What this calibration does NOT cover.
--selfruns a commit against itself, so both sides compile to the same-sized code image. It therefore measures machine noise, timer granularity and GC drift — and is blind to the interleaving artifact that arises when two differently-sized code images share one heap and one JIT profile, which is every real A/B. A clean--selfmeans "the harness is not noisy today", never "this A/B number is real". See Single-process interleaving artifacts.
A 9.9% swing past a 5% threshold that does not breach is the design working. Every breach rule requires the WIN RATE as well as a percentage, and a noise pass swings its percentage while keeping its win rate near 50%. The percentage thresholds are not what make this safe at 5%; the conjunction is.
The sign test, and why it had to exist
The percentage rules alone caught fix(not) (+35–42%) and could not reliably catch fix(expect), which costs the realistic Less workloads +2%…+9% — genuinely under a 5% threshold. Widening the threshold to reach it would have been exactly backwards: it would blind the gate to the band it exists to watch, where 0.34.0's css row moved −1.6% and 0.35.0's repayment was −12.3%.
What separated signal from noise in that data was not the magnitude, it was the direction. Sides are measured in adjacent order-alternated pairs, so under the null hypothesis "these two builds are the same" each pair is a coin flip. Across the five fix(expect) replay runs the less/* rows lost 1–4 of 12 pairs, pass after pass, while the unaffected rows sat at 5–9. Losing 3 of 12 has p ≈ 0.07 per pass; required in a majority of passes.
The
p ≈ 0.07and the1 in 500above are WRONG as stated, and are kept because the reasoning they motivated is right and the correction is the point. Both numbers assume the 12 pairs are independent trials and that the passes are independent of each other. Until 0.46.0 neither held: one compiled pair of instances served every pass, and a single draw of the compilation lottery produces 0/12 or 12/12 on byte-identical code.measurePasses()recompiles per pass and measures the null, which is what makes a win rate mean what this section says it means. Thefix(expect)evidence stands — it was reproduced across five separate runs, i.e. five separate draws.
So a workload also breaches when it loses ≤ 25% of its pairs AND is ≥ 1.5% slower on both median and min. Both a percentage floor and the win rate are needed, so a build that is merely a consistent hair slower does not fail — it has to be consistently slower and measurably slower on the min, which is the sample least disturbed by a burst.
Watching it go red
A gate nobody has watched fail is not known to work. All three known regressions, five runs each, on a machine at load average 5–9.
1. fix(not) — 0.34.0's unconditional rollback stores (+33.9% real)
pnpm perf:workloads --ref=3175734 --head-ref=fbeb43e --allow-parse-diff| workload | median delta | pairs won | verdict |
|---|---|---|---|
less/stylesheet | +37.2% … +43.9% | 0–2 / 12 | FAIL, 5 of 5 runs, 3/3 passes every run |
less/mixins | +34.4% … +42.4% | 0–2 / 12 | FAIL, 5 of 5 runs, 3/3 passes every run |
css/stylesheet | −3.3% … +2.2% | 2–9 / 12 | ok, 5 of 5 |
graphql/document | −1.3% … +4.8% | 1–9 / 12 | ok, 5 of 5 |
json/document | −0.9% … +0.7% | 5–11 / 12 | ok, 5 of 5 |
Re-validated after the 0.46.0 lottery fix (5 passes, recompiled per pass, null-calibrated ceilings), same command, load average 6.0 → 4.5:
| workload | median delta, 5 passes | pairs won | measured null / ceiling | verdict |
|---|---|---|---|---|
less/stylesheet | +36.7% … +40.5% | 0/12 every pass | 66.7% / 41.7% | FAIL, 5/5 passes |
less/mixins | +36.7% … +39.8% | 0–1/12 | 48.3% / 23.3% | FAIL, 5/5 passes |
css/stylesheet | −1.0% … +1.0% | 5–10/12 | 45.0% / 20.0% | ok |
graphql/document | −0.3% … +1.7% | 4–9/12 | 40.0% / 15.0% | ok |
json/document | −0.9% … +1.0% | 3–7/12 | 56.7% / 31.7% | ok |
Note less/stylesheet: its null landed at 66.7%, so its ceiling was raised to 41.7% — and the regression still won 0 of 12 pairs in every pass. Calibrating the ceiling upward for a case the instrument favours does not cost detection; it is the case that would have been BLIND at a flat 25% if the draw had gone further.
--allow-parse-diff is required here and only here: fix(not) fixed a trivia-log rollback leak, so the pre-fix side genuinely records a different trivia count. Refusing to measure that would mean the one replay that proves the gate works is the one replay it cannot run. The flag is rejected outright unless a pinned --head-ref is being replayed, so it can never soften a gating run.
2. fix(expect) — 0.35.0's nullable-prefix derivation (+49.6% real)
This is the one perf:guard:grammars read flat on before its expected/* sweep was added.
pnpm perf:workloads --ref=9c6fee2 --head-ref=a464372| workload | median delta | pairs won | verdict |
|---|---|---|---|
less/stylesheet | −2.3% … +8.5% | 0–10 / 12 | FAIL in 2 of 5 runs |
less/mixins | −1.8% … +13.9% | 1–5 / 12 | FAIL in 3 of 5 runs |
css/stylesheet | −13.5% … +51.1% | 3–10 / 12 | ok, 5 of 5 |
graphql/document | −3.1% … +10.0% | 1–9 / 12 | ok, 5 of 5 |
json/document | −8.2% … +4.8% | 1–11 / 12 | ok, 5 of 5 |
At least one less/* row failed in 4 of the 5 runs, with no false failure on any other row. State that honestly rather than as "caught": this is the weak detection of the three, it sits near the gate's resolution floor, and one run in five was green. If this were the only gate watching that axis it would not be enough — which is why expected/wide in the sweep gate, which fires 5 of 5, is not redundant with it.
3. The fixed state — stays green
pnpm perf:workloads # HEAD (with the fix(expect) dedup) vs the 0.35.0 referenceThree runs, --ref=3562f78 --head-ref=e1b92e9 (0.35.0 as shipped, versus the same tree with the dedup):
| workload | median delta | verdict |
|---|---|---|
less/stylesheet | −5.0% … +4.1% | ok, 3 of 3 |
less/mixins | −4.2% … +7.3% | ok, 3 of 3 |
css/stylesheet | −1.4% … +1.6% | ok, 3 of 3 |
graphql/document | −3.0% … +0.8% | ok, 3 of 3 |
json/document | −0.7% … +0.7% | ok, 3 of 3 |
Green on every row on every run. Two of the three runs read the less/* rows as an improvement (−5.0% … −0.3% median, winning 8–11 of 12 pairs), which is the expected sign: 0.35.0 shipped the regression and the dedup removes it. The third run read them flat-to-slightly-positive without breaching — the honest reading of that is that a 4-6% repayment is close enough to this gate's floor that it shows up as "not slower" rather than as a clean win.
What it does NOT catch — read this before trusting a green
Below roughly 1.5–2% per workload it reads green. That is the sign test's percentage floor. There is no amplification here to buy resolution with: unlike the sweep, these are end-to-end parses of realistic input, so a reading IS the downstream cost rather than a multiple of it. The flip side is that a number here needs no translation.
A regression that shows up in only half the passes reads green. That is the direct, deliberate cost of the majority rule, bought to stop false failures on a shared runner. A gate that false-fails gets ignored, and then the real failures get ignored with it.
It cannot attribute. It says "less/* got 20% slower". It does not say why. That is
perf:guard:grammars' job, and the two are meant to be read together.The corpora are repeated, not 50 KB of unique source. The construct mix is what the measurement depends on and that comes from the hand-authored corpus, but the repetition means branch predictors and inline caches see more regular input than a real 50 KB file would. A regression that only appears on genuinely irregular input would be understated.
Five workloads is five shapes. Nothing here parses a template language, a programming language with real operator precedence, or anything with heavy
expect/recovery use. A regression confined to those is invisible. Adding a workload is an entry inbuildWorkloads(); adding one is cheaper than the release that finds it for you.The vendored Less grammar is a benchmark, not a conformance corpus. It asserts that both sides parse identically. It says nothing about whether that parse is correct.
This gate is single-process and multi-workload too, and it has not been audited for the interference documented in perf-harness-interleaving.md. One observation: a workload measured clean under
--only=cssand breached when run alongside the other four. That is a question, not a finding — there is no control behind it, and the pairing rule already exists because these workloads disturb each other through GC. Stated so nobody has to rediscover it.A workload can be realistic and still structurally blind. The first draft of
bench/workloads/less.tsrouted every value alternative through a named rule, which is a perfectly reasonable way to write a grammar — and it read FLAT on thefix(expect)replay, because a rule reference is a function boundary and the enclosing choice never sees the widened set. Same dialect, same vocabulary, same input, opposite answer, from nothing but where the rule boundaries were drawn.bench/workloads/fxprobe.tsmeasures that exposure directly; re-run it when the grammar changes, and do not assume a flat reading means a flat cost.
Running them
pnpm perf:guard:grammars # the sweep gate
pnpm perf:workloads # the broad gate
pnpm perf:workloads --quick # 3 passes x 2 rounds x 2 runs — TRIAGE ONLY, does not gate
pnpm perf:workloads --only=less # substring filter on workload id
pnpm perf:workloads:describe # what each workload parses, and whether it reaches EOF
pnpm perf:guard:grammars --quick # 3 passes x 2 rounds x 2 runs — TRIAGE ONLY, does not gate
pnpm perf:xproc:gate --peak # the peak clause, as CI runs it (one shard's worth
# without --shard-out; CI runs 10 and aggregates)
# Validate the gate itself. A gate nobody has watched fail is not known to work.
pnpm perf:guard:grammars --ref=7f1ddcd --head-ref=c7780e4 # 0.34.0 without the fix — expect RED
pnpm perf:guard:grammars --ref=7f1ddcd --head-ref=fdf4e90 # 0.34.0 with the fix — expect GREEN
pnpm perf:guard:grammars --ref=<sha> --head-ref=<same sha> # the noise floor
# Confirm a suspected regression ACROSS PROCESSES before believing it. Not a
# gate — see docs/design/perf-harness-interleaving.md.
pnpm perf:xproc --ref=<sha> --head-ref=<sha> --case=expected/narrow
# Same for the broad gate. `replay.sh` runs each five times, because one run is
# not evidence at this noise floor.
bench/workloads/replay.sh 3175734 fbeb43e 5 --allow-parse-diff # fix(not) — expect RED
bench/workloads/replay.sh 9c6fee2 a464372 5 # fix(expect) — expect RED
pnpm perf:workloads --self # the noise floorQuick mode still recompiles both sides three times. One compiled parser pair is one draw of V8's instance-level optimisation lottery, not an independent sample: a one-pass Less self-check false-failed byte-identical code at +12.6%. The quick shape therefore spends its small budget on three independent pairs and reports the median delta across those passes first. The full pass range remains visible as noise context; do not reject a borderline experiment from one endpoint.
The reference is the previous release, resolved when the gate runs: the newest stable v* tag whose source HEAD contains, other than HEAD itself (previousRelease in bench/ab-harness.ts). A release tag sits on main when the merged commit was released as it stood, or on the release commit scripts/release.mjs stamp makes beside main, whose first parent is the merged commit; either one names the source. --ref=<sha> overrides it for a one-off comparison. The macro gate is the exception: it is pinned in bench/jess/macro-optimize-config.json together with the Jess commit it builds against, and moving it is a deliberate bench change.
When HEAD's src/ is the reference's, the gate reports and does not fail. Every merge that changes src/ publishes, so a pull request that leaves src/ alone is measured against identical code (the version literal in src/version.ts, which a release commit stamps, aside; a local run without --head-ref compares the working tree). Both sides are built from src/ and share one install, so that run is an A/A control. A breach there is the instrument, not a regression. CI measured it: a workflow-only PR, against v0.58.0 with the same src/, read json/document at +1.9% median, breached 4/5 passes, and failed the required check. The run still measures and prints its table, so a change to the harness is exercised, and any case that breached is named as the noise floor on that runner (sameSource and identicalSourceNote in bench/ab-harness.ts).
The per-PR report: this change against its own base
The gate answers "is this slower than the previous release?". A PR also wants "what did THIS change do?". On a pull request, xproc-perf writes its per-case table (median and mean change, both p-values, the verdict) to the job summary ($GITHUB_STEP_SUMMARY). That table is against the previous release. Every merge that changes src/ publishes, so the previous release is usually the PR's base. macro-perf also prints its candidate-against-base table, which is report only.
The in-process tools keep their --merge-base=<sha> report for local use. It still skips the measurement when the PR changes nothing under src/, or when the base's src/ equals the reference's.
Why the reference is not a pinned sha
It was, until publishing moved to every merge. Each config carried a referenceSha and a note saying "bump this to the released sha at every release, in the release PR". It was missed ten releases running: the density gate was still anchored to v0.33.0 and the workload gate to v0.35.0 when 0.45.0 was prepped.
Nothing went red, which is the point. A stale anchor does not fail — it compares against a baseline that has already absorbed every regression since, so the accumulated headroom becomes the error bar. rollback/dense read -62.0% against v0.33.0: that path could have got 2.6x slower in this release and the gate would still have printed ok. The absolute-baseline rule was satisfied in letter while its RESOLUTION was destroyed, silently.
scripts/check-changelog.mjs then enforced the bump on release PRs. With a release on every merge there is no release PR to enforce it on, and a hand edit per merge would be the same chore made more frequent. Resolving the anchor from the tags makes staleness impossible instead of detectable: every PR is measured against the release immediately before it, which is the gate's question, "is this release faster than the last one?"
A newer anchor is a stricter gate, and the previous release may surface a regression an older one was hiding. That is the gate working. Land the number visibly.
A missing reference is a hard failure, never a skip: CI checks out with fetch-depth: 0 so the tags and their commits are present, and if no release tag is reachable, the gate says so and exits non-zero. "The gate did not run" must never render as green — and the test aggregate treats a SKIPPED perf job as a failure for the same reason, unless changes said the PR was documentation-only.
The bars an engine change has to meet
A refactor of the engines (src/table/assemble.ts, src/table/emit-assembly.ts, and what they call) that calls itself reuse has to meet all of these. A "reuse" refactor of the left-corner routes moved decisions known at compile time into parse-time checks and added calls on the hot path. On the closure engine the routed path ran up to 25.9% more instructions. An earlier local run had read "neutral", because two of its legs ran one source tree through a symlink.
| bar | gate | command |
|---|---|---|
| 1. One definition in source; no parallel copy of a piece | INV-13, plus the review item below | pnpm check:invariants |
| 2. No duplicated function in the emitted or printed output | duplicate-function gate, in size-gate | pnpm dupfn:guard |
| 3. No more function calls per parse than the previous release | call-count gate, calls | pnpm callcount:guard |
| 4. Nothing known at compile time becomes a parse-time check | call-count gate, branches, plus the review item below | pnpm callcount:guard |
| Speed is confirmed on real, independent builds | assertIndependentLegs in every A/B harness, then xproc-perf | pnpm perf:xproc:gate |
Review checklist
- Bar 4. For every test the change adds or moves inside a piece or an emitted function, ask: can this value change during a parse? If it cannot (an option, the host mode, a grammar shape, anything fixed when the table is encoded or assembled), decide it at assembly by selecting a different piece. Name each such test in the PR.
- Bar 1. Look for a piece that was copied and then edited. INV-13 only sees exact copies.
Call-count gate
An instrumentation build (bench/engine-guards/instrument.ts) adds a counter to every function entry and every branch test in a copy of each side's bundle, and to every new Function body as it is compiled. src/ is not edited, so published code pays nothing. One parse is counted until two in a row agree. The counts are identical across processes and on Node 22, 24 and 26, so a single extra call per parse fails.
It counts 37 rows: each workload's plain parse under emitted/ and under closure/ (Function refusing, as under a CSP), the parse that records all trivia (rootTrivia: { select: 'all' }) as <workload>/allTrivia for the workloads that have one (Less and CSS), and jess's vendored grammars under macro/. The macro/*/ast rows are the path jess ships: the variants its parse and /positions entries use, called with the options they pass, with reducers building real nodes through a vendored bundle of @jesscss/core (fixtures/jess-grammars/core/core.js). That bundle's calls and tests are counted too, and it only changes when the snapshot is refreshed. A row fails when HEAD counts more than the previous release, the reference perf:xproc:gate uses. bench/call-count-baseline.json must equal what HEAD counts, so a change in counts is a reviewed line in the diff. Bank a drop with pnpm callcount:baseline.
pnpm callcount:guard --cold --ref=<ref> counts the emitted/ and closure/ rows another way: compiling the workload's grammar and its first parse, together, on both sides. A row above the reference fails. It has no baseline and takes no reasons, and it has no macro/ rows, since a jess grammar is built once for all of its fixtures. CI does not run it; a change to what compiling costs states its table.
An increase that is the price of a change needs bench/call-count-reasons.json: { "against": "<previous release tag>", "rows": { "<row> <metric>": { "why": "<why>", "limit": <most it may add per parse> } } }. An entry must say why and state a limit, and an increase above the limit fails. An entry that excuses no increase fails, so it cannot wait for a later one. An entry only applies against the release it names, so it stops excusing anything once that change ships; a file left over from an earlier release prints a notice rather than failing, so a release does not turn every open PR red.
A count only means something over a whole parse. Every parse counted must succeed and consume its input (assertWorkloadFullyConsumed), and a jess parse must also build a Stylesheet with no recovered error or diagnostic; anything else stops the gate rather than reading as fewer calls. An empty input fails, and so does a run that counted no rows for a family. Over these rows, bench/call-count-baseline.json can still rise with no reason entry while HEAD is at or below the release, so giving back an unreleased win is seen only in review. A row in the baseline at the merge base with origin/main may not leave it: both sides count the rows the gate lists, so cutting one from the gate and the baseline would otherwise pass. Removing or renaming a row needs an owner's bypass; adding one is allowed.
Run on that refactor's two reuse commits against their parents, with macro/css/ast then the only AST row and its reducers stubbed, it fails both: ce6b3fbf (+1,527 calls on macro/css/ast/site.css) and fb769bac (up to +22,029 calls and +128,922 branch tests per parse on the closure rows; +1.1–2.1% on the macro/ rows).
What branches catches: a test added to a piece, including one that is always false and adds no call, which is the common form of bar 4. What it cannot see:
whether a test could have been decided at compile time. It sees more tests, not why. A test added for a correct reason fails too and needs a reason entry;
a new test that replaces an old one, or an increase cancelled by a drop elsewhere in the same row;
work that is not a call or a test: allocation, property reads, a call site turned megamorphic, a builtin such as
charCodeAt.?.is not counted;anything the fixtures do not reach.
site.cssandapp.lesshold no SCSS-only or.jess-only syntax, so those dialects' own productions are not counted.compile time and the first parses. Only the settled per-parse cost is counted, so 200,000 extra calls while compiling or on the first parse can show as +1 branch test, or nothing.
What calls cannot see: what V8 does with a call. An inlined call costs nothing and still counts, and a call that blocks inlining costs far more than one. A count says the work moved; xproc-perf says what it cost.
A trap in the reference build. The reference side runs HEAD's copy of bench/jess/real-grammars.ts and bench/workloads over the release's src/. A PR that moves a module they name (such as src/compiler/linker.ts or src/plugin/index.ts) leaves the reference side unable to build. The gate fails, no reason entry can argue it, and it stays red until a release contains the new path. Keep a re-export at the old path until that release.
A race at release. A PR with a reason entry against vN that merges after vN+1 publishes leaves that entry stale on main: it excuses nothing, the increase fails against vN+1, and publishing pauses until a follow-up updates against.
Duplicate-function gate
bench/dup-function-guard.ts normalises every function in 20 artifacts: the emitted code of the workload grammars and jess's macro-built bundles (scripts/function-shapes.mjs: comments, whitespace and the function's own binding names dropped; free names and property names kept). A shape that occurs n times is n−1 duplicates. bench/dup-function-baseline.json holds each artifact's count: more fails, and fewer fails until it is banked with pnpm dupfn:baseline. It runs in size-gate and prints the function count beside the duplicate count.
In CI the committed baseline is also compared with the merge base with origin/main, and any artifact's count rising fails, so a new copy cannot be written into the baseline with it. There is no reason file. An artifact at the merge base may not leave the baseline either, so renaming a row cannot hide a rise. A rise that is right, such as a grammar snapshot refresh that brings in new reducers, and a real removal or rename both need an owner's bypass. Adding an artifact is allowed. Every count in the baseline must be an integer. The gates check a row's key, not what it measures, so pointing a row at another grammar or shrinking a fixture is left to review.
The count is per artifact, so it can net out: when assembly selection shifts, sites that stop being printed can hide new copies elsewhere in the same artifact.
It starts from a backlog: 21 duplicated functions in the emitted CSS and 56 in the emitted Less grammar, for example. It cannot see a copy that differs in one constant or a free name (which may be correct), a copy in another artifact, or a body under 60 characters. It can over-report a pair that differs only in var against let, which share a shape. Those bundles also carry jess's own reducers, so a duplicate there can be the grammar author's.
INV-13
See the invariant gate. It is the structural half of bar 1, and it starts from three findings, entered as DEBT: rollbackNeeded and needsTriviaRollback, bindRunners twice, and runAdjTerms and runDirectTerms. Issue #252 tracks removing them.
Independent legs
assertIndependentLegs (bench/ab-harness.ts) prints each leg's real path and refuses to run when two legs are one directory, by device and inode, so a symlink, a case variant and a bind mount are all caught. perf:xproc:gate, perf:xproc, perf:guard:grammars, perf:workloads, callcount:guard and bench/jess/ab.ts (which bench:macro-optimize drives) call it, and test/bench/engine-guards.test.ts fails if a harness that materialises a second side does not. A declared --self run is the one exception. bench/jess/fixture.ts compares engines with every leg at HEAD by design, so it is not covered. Two separate checkouts of one commit pass this check; sameSource is what reports those as A/A.
The pretenuring check — pnpm perf:pretenure
Every process must parse in the fast mode, whatever the bundle layout. V8 decides per object-literal allocation site whether to allocate straight into old space, from where scavenges fall during warm-up. When the objects a parse keeps come from literals, identical processes split into a fast mode and a slow one that runs dozens of mark-compacts, and appending dead code to the bundle re-rolls which is which (#224). Parseman builds its tree objects with constructors so that cannot happen (src/cst/plain.d.ts).
bench/pretenure-check.ts builds dist/ and appends 0, 970, 4,000 and 15,000 bytes of never-called code to dist/index.js. It then runs seven cases in fresh plain-node processes:
css/stylesheet,expected/narrowandrollback/medium, whose fixture builders copy what they keep;css/cst, a CST host (cstBuildHost()nodes over parseman's leaves and spans);nolines/docandlines/doc, builder-less default nodes without and with line tracking;spans/doc, a builder that keeps the span parseman passes it.
The check fails if any process runs more than maxMarkCompacts (3) mark-compacts (bench/pretenure-check.json). The fast mode runs 1–2 on the machine that set the threshold. Fast-mode counts depend on heap sizing, and a smaller machine may read 3–4. Every run prints a calibration line with the pad-0 counts per case: read it before trusting a failure. On main before #230 the same check read 49–69 on Node 26. It is not in CI: run it on Node 24 and Node 26 for any change to what a parse allocates, about 6 minutes each.
Each worker checks every parse before it reports a count. The parse must succeed and reach the end of its input (trailing whitespace aside), and the first parse must build the expected root (Stylesheet, or Doc with 3,000 items). A parse that fails or stops early would measure a smaller workload and pass for the wrong reason, so the worker refuses to count it and the case fails. --plant=truncated proves it: every input gains a fragment its grammar cannot consume, and the density and doc cases fail. The CSS grammar is permissive and accepts the fragment.
--plant=<site> turns one converted site class back into an object literal in the staged dist/, and the run must then FAIL. On Node 26 it does for span, leaf, node, line-span and children. field and buffer cannot make it fail, because no tree keeps a field capture or a capture buffer: they die with their node. Node 24 pretenures fewer sites, so line-span fails on Node 26 only; run the plants on Node 26.
The peak clause — pnpm perf:xproc:gate --peak
The blind spot every per-step gate has
The release policy is "each release must be faster than the last unless the slowdown is deliberate and documented". Everything above enforces that per step, and a per-step test cannot see a slow bleed: five consecutive 1% losses are each inside the noise floor, no step gets flagged, and the sum is a real 5% regression.
That is measured, not hypothetical. The standalone version sweep at ~/parseman-perf-probe/ (40 rounds, 2,800 samples, 0 digest mismatches) found −3.9% over 0.28.1→0.32.0 and −5.1% over 0.28.0→0.34.0 on its probe grammar, with almost every individual step inside its noise floor. A per-step gate would have caught none of it.
The clause
A release may not sit below the committed broad-workload release baseline by more than the noise floor. The baseline lives in bench/workloads/config.json:
"peak": { "sha": "bf03092", "version": "0.48.0", "allowancePct": 5 }- Absolute, not differential.
shanames a commit, never a stored millisecond count, so the comparison is re-measured on whatever machine runs it and reads the same on a laptop and a CI runner. allowancePctis tolerance, not budget. It is the measured noise floor of this harness — the same 5% the per-release thresholds use, from the same self-check.- Structurally stricter than the per-release rule: median and min must both breach, not either. A per-release gate is watching for a change that just happened and should be twitchy; the peak clause answers "are we below the best we have ever been", which is worth answering only when both statistics agree. The win-rate conjunction and the majority-of-passes rule are kept.
Demonstrated at load average 98 → 106, where json/document swung from −20.1% to +67.6% median across passes while its min held at −2.9%…−0.7% and its win rate at 5–8 of 12 — verdict correctly ok. That is the whole design in one row: the median is the statistic a contended runner destroys; min and win rate are not.
The original peak was seeded, not swept — and cannot be imported
Stated rather than left to be discovered. 0.45.0 was the starting record. It was not the winner of a measured sweep across all releases. That sweep was attempted at 0.46.0 and abandoned: the machine sat at load average 70–90, where triage runs of this very gate reported the same workload anywhere from −86% to +131%. A number a control cannot reproduce is worse than no number.
What was measured, and matters more:
HEAD runs ~45–50% faster than 0.28.0, 0.36.0 and 0.38.0 on
css/stylesheet.
So the sweep's 0.28.0 peak is a property of its own instrument — a 10-node monolithic fused probe grammar — and must not be copied into this config. A realistic composed grammar and a synthetic monolithic one do not peak in the same place. On this workload set there is no evidence of a historical peak above HEAD. Whatever peak a different instrument reports is evidence about that instrument's shape. Re-measure; never import.
0.48 is the first deliberate operational reset of that record. Its release CI measured the canonical TableProgram 98–249% slower than the seeded 0.45 baseline across these five broad workloads. The owner accepted that architectural reset after correctness, artifact-size, package-size, and external-competitor gates passed, so bf03092 is the new baseline for detecting additional regressions. The exact 0.45 drawdowns remain in the 0.48 changelog, and pinned-0.46 Jess parity remains a separate internal objective; resetting this gate does not rewrite either historical comparison.
Re-baselining is a deliberate, committed diff — and it is enforced
Moving peak forward is how a genuine improvement or an explicitly accepted new normal becomes the new bar. Moving it backward, or widening allowancePct, makes a slower build the reference — which may be a legitimate trade, and is also exactly how a regression gets laundered into a baseline.
scripts/check-changelog.mjs §D therefore:
- validates every
peakblock structurally — sha resolvable as a commit, version parseable, allowance a positive number (a malformed record is a gate that silently does not gate); - requires any edit to one to be named in a CHANGELOG entry the PR adds, calling out backward moves and widened allowances by name. An entry already on the base does not count:
main's Unreleased can hold entries a release has already shipped.
It runs on every PR, because a peak can move at any time and the moment it matters is the moment it moves. It has no hatch — release-exempt does not waive it. This is deliberate repetition of a lesson already paid for: both referenceSha fields carried "bump this at every release" in a comment and were missed for ten consecutive releases.
When it fires
Do not widen the threshold to make a build pass. Either fix the regression, or re-baseline in the open: move the peak, or change bench/xproc-gate.json, with the number and the reason in the CHANGELOG, which scripts/check-changelog.mjs §D and §E require. An honest "3% is the cost of correctness, 22% was avoidable" is a result. "Should be faster now" is not.
