docs/proposals/helper-detection-unification.md
GitHubProposal: Unified helper detection — expression-site registry + pattern-as-JS matching
Status: CLOSED / mostly rejected. Phase 1 (inline-vs-declaration detection de-dup) landed and stuck — a real ~260-line reduction with no behavior change. Everything past it (corpus matching, then a skeleton-pattern + checker engine) was prototyped in shadow mode, measured against real bundles, and reverted: the migratable subset is ~10–14 of ~209 detection functions, far below the scale a shared matcher needs, and the engine net-increased LOC. The detection code is large because the problem is irreducibly semantic (~93% marker-based or stateful). Read ../learnings/helper-detection-pattern-engine.md for the conclusion and don't re-run this. The engine, corpus matcher, and the shadow-mode comparison harness built during the investigation were all removed afterward (they validated a rejected approach); reconstruct from git history if ever needed. The sections below are the original proposal, preserved for context.
Problem
Helper detection works (see helper-detection.md) but its cost has outgrown its design. Measured against the current tree (numbers approximate, attribution by function ranges):
rules/transpiler_helper_utils.rsis ~4,900 lines, of which ~77% (~3,760 lines) is hand-written body-shape matchers — 13 predicates covering 17TranspilerHelperKindvariants. Two dominate: the slicedToArray family (~1,280 lines including inline-form extraction) and asyncToGenerator (~770 lines). The infrastructure around them (LocalHelperContext, lifecycle utilities, import-path classification) is comparatively small and healthy.- Six rules carry ~1,000 additional lines of private detection:
un_es6_class(~400),un_sliced_to_array(~250),un_object_spread(~150, esbuild-specific),un_object_rest(~100),un_interop_require_default(~80),un_class_call_check(~50). - The duplication has one dominant shape: the central scanner only detects helpers as module-level declarations, but rules also need to recognize the same helpers at expression sites (inline IIFE/arrow forms). The interop, classCallCheck, objectWithoutProperties, and slicedToArray shapes are each matched twice — once as a declared function in the scanner, again as an inline expression inside the consuming rule.
So there are two distinct problems:
- One query surface where rules need two.
collect_transpiler_helpers()answers "which module-level bindings are helpers?" but not "is this expression an inlined helper?". Rules answer the second question privately. - Matcher expression cost. Hand-written Rust AST destructuring costs roughly 10x the lines of the JS it recognizes; 40–66% of matcher LOC is let-else/match-pyramid navigation rather than semantic checks. Every new helper, variant, or Babel version pays this again.
Phase 1 — expression-site channel on LocalHelperContext
Mechanical consolidation, no new matching technology:
Add an expression-site query to the helper context, reusing the existing body matchers on
FnExpr/ArrowExprnodes:/// Classify an expression as an inlined transpiler helper, if it is one. /// Covers IIFE forms like `((e) => e && e.__esModule ? e : {default: e})(x)`. pub fn classify_inline_helper_call( call: &CallExpr, ) -> Option<(TranspilerHelperKind, &Expr /* the wrapped argument */)>Migrate the six rules' private detection onto it. The rewrite halves of those rules are untouched; only their "is this the helper?" code moves or is deleted.
esbuild-specific spread helpers currently detected inside
un_object_spreadmove behind the same registry with their own kind variants, so "what counts as a helper" has a single home even when the shape is bundler-flavored.
Expected effect: ~1,000 lines consolidated, the four concrete duplications removed, and — more importantly — a single choke point through which all detection flows, which phase 2 requires.
Every migrated rule keeps its existing unit tests; inline-form regression tests move to the shared matcher's test file rather than being re-asserted per rule.
Phase 2 — pattern-as-JS matching engine (shadow mode)
Replace hand-written predicates with patterns written as JavaScript source, matched by one generic engine:
- Corpus: vendored verbatim from what transpilers actually publish —
@babel/helpers(across the versions we care about, including loose mode),tslib(1.x/2.x),@swc/helpers. One snippet per known variant of each helper, tagged with itsTranspilerHelperKind. Adding a helper or a new Babel version is pasting its source, not writing Rust. - Engine: structural equality modulo binding renaming (alpha-equivalence —
the generalization of what
MatchContextalready does per-matcher), plus a small metavariable/ellipsis convention for the genuinely variable spots (e.g.$ARGfor "any expression", statement ellipsis for tolerated extra guards). Estimated one-time cost: ~600–900 lines, with property that all future helpers are data. - Shared normalization: both corpus and candidates are run through the same
Stage-1 normalization slice (
UnBracketNotation,UnIndirectCall,UnminifyBooleans,RemoveVoid, …) before comparison. This is the insight that changes the calculus relative to the original design notes: wakaru now owns a canonicalizer, so exact-tree-modulo-renaming covers most of what relaxed predicates were hand-tolerating. Residual variance (ternary vs if/else single-return) becomes either one matching-time canonicalization or one more corpus entry. - Performance: candidates are pre-filtered by cheap signals (param count,
statement count, presence of marker atoms like
__esModule) before unification, so the engine runs on a handful of candidates per module — same scan cadence as today's lazily-builtLocalHelperContext.
Shadow mode: the engine runs alongside the existing predicates across the
full fixture suites and the matrices in scripts/repro/, logging every
disagreement (helper found by one side only). No pipeline output changes. The
disagreement log is the decision artifact.
What stays bespoke
Equality-plus-corpus will not reach:
- asyncToGenerator / regenerator-adjacent shapes — deep structural drift across versions; the existing relaxed marker-based matcher stays.
- slicedToArray compositions — minifiers inline the Babel sub-helpers
(
arrayWithHoles,iterableToArrayLimit,nonIterableRest) into each other. The corpus carries the sub-helpers individually (Babel ships them individually), which may decompose some of this naturally, but the composed inline-extraction path keeps its bespoke code until shadow data says otherwise.
The target for replacement is the simple-to-medium tier: the interops, extends, classCallCheck, possibleConstructorReturn, assertThisInitialized, toConsumableArray, objectSpread, objectWithoutProperties, defineProperty, taggedTemplateLiteral. That tier alone is a large majority of matcher count and roughly half the matcher LOC.
Phase 3 — replace at parity, per helper
A predicate is deleted only when shadow mode shows, for that helper kind:
- zero missed detections across all fixtures and repro matrices, and
- zero new detections that are not verified true positives (a new true positive found by the corpus is a win, recorded as a fixture).
Helpers that never reach parity keep their hand-written matcher, documented as the deliberate exception. Partial adoption is an acceptable end state — the engine plus corpus must justify itself per helper, not all-or-nothing.
Relation to prior design decisions
helper-detection.md rejected a custom IR, CFG hashing, and version auto-detection — this proposal is none of those. It is adjacent to the rejected "general AST pattern DSL", so the difference is worth stating: that rejection was made when matchers were few and small, and the hard part of fingerprinting ("stable canonicalization") had no answer. Today the matcher corpus is ~3,800 lines and growing with every supported transpiler version, and Stage 1 is the canonicalizer. The fixed cost of one unifier now amortizes; the marginal cost of the status quo no longer does. Prior art in this exact domain: webcrack detects Babel helpers via template matchers with capture variables; ast-grep demonstrates pattern-as-code unification in Rust (not directly reusable — it operates on tree-sitter trees, not mid-pipeline SWC ASTs).
Risks
- Strictness misses unseen variants. Equality is less forgiving than a relaxed predicate. Mitigations: corpus breadth is cheap (paste the variant); shadow mode finds gaps before anything is deleted; a diagnostic for "function that looks helper-shaped but matched nothing" can surface unknown variants from real bundles over time.
- Ellipsis/metavariable semantics creep. The convention must stay small — if a pattern needs sophisticated control-flow tolerance, that helper belongs in the bespoke tier, not in a more powerful engine.
- Two systems during transition. Shadow mode means running both. The per-helper parity gate keeps the transition window finite and the disagreement log makes the cutover auditable.
UPDATE (validated) — from corpus matching to a skeleton-pattern + checker engine
The phases above were prototyped end-to-end in shadow mode against real bundles (an external helper-bearing corpus) and the repro analysis. The data redirected the effort. This section supersedes the corpus-replacement plan (Phase 3 above) with the approach the experiments actually justify.
What the experiments found
Corpus / exact-canonical matching has a low ceiling. Canonicalize-then-compare
(alpha-rename bound idents → normalize → string-compare against vendored
snippets) reached exact parity with zero false positives only on small,
structurally-stable helpers: interopRequireDefault, classCallCheck, and —
after fixing a free-vs-bound canonicalization artifact and adding an isolated
UnConditionals normalization pass — objectSpread and the non-inlined
OR-dispatcher forms of slicedToArray/toConsumableArray. It is fundamentally
hopeless on interopRequireWildcard, objectWithoutProperties, and any
fully-inlined-sub-helper body: those vary combinatorially per minifier and
are not canonical. Replaceable LOC via this route ≈ 22% of the matcher core, and
only the cheap matchers — it does not touch the giants (slicedToArray ~1.3k,
asyncToGenerator ~0.8k) that dominate the LOC. Rejected as the primary lever.
The real footprint is bigger than the matcher core. A full inventory found
~209 detection functions / ~17.5k LOC of helper/shape detection spread across
transpiler_helper_utils.rs and the rules (un_es6_class, un_regenerator,
un_object_rest, un_async_await, un_object_spread, …). ~55% of that, by line
count, is navigation/extraction boilerplate (let-else pyramids, match-arm
tree-walking, Box/Option unwrapping to reach the nodes worth checking).
A skeleton-pattern + semantic-checker engine removes the boilerplate, not the
semantics — and that is where the lines are. A prototype SWC-native engine
(holes bind to (sym, SyntaxContext) or subtrees; repeated holes enforce
same-binding identity) reimplemented two matchers as pattern + checker, proven
equivalent to the originals with zero disagreements:
classCallCheck: 54 → 21 LOC (61% reduction)interopRequireDefault: 92 → 54 LOC (41%; dragged by an imperative if/return↔ternary desugar — see alternation, below)- marginal cost per additional fixed-shape matcher: ~21–25 LOC
The binding-identity precision ($obj repeated must be the same binding) is
preserved through SyntaxContext — the precision a tree-sitter/ast-grep textual
matcher would lose, and the reason this is SWC-native rather than off-the-shelf.
Scope criterion (what the engine is and isn't for)
- IN (fixed-shape): a helper recognizable as a fixed skeleton modulo
binding-renaming plus a small residual semantic check. These compress ~45–60%.
Candidates:
classCallCheck,interopRequireDefault, the simpleinteropRequireWildcardforms,possibleConstructorReturn,assertThisInitialized,defineProperty,taggedTemplateLiteral,extends,objectSpread, the self-containedtoConsumableArrayform. - OUT (marker-based / signal-accumulation): helpers matched by "has marker X
somewhere in an unbounded body" (
Array.isArray+Symbol.iteratorforslicedToArray; promise/generator signals forasyncToGenerator/__awaiter; theun_regeneratorstate machines; theun_es6_classcomposition context). A skeleton does not fit these; they stay bespoke (checker-heavy). This is where most of the 17.5k lines live, so the realistic whole-codebase reduction is ~25–40%, not 55% — a strong cut on the fixed-shape subset, a wash on the giants.
Engine design
- Patterns are authored as skeletons with
$NAMEholes over the SWC node kinds the in-scope matchers need (if/throw/return statements, unary!, paren, binaryinstanceof/&&, conditional?:, member incl. computed string-literal props, call,new, one-prop object literal, ident). - Holes bind to a concrete
(sym, SyntaxContext)(ident holes) or a subtree (expr holes). A repeated hole requires the same binding at every occurrence (alpha/binding identity, reusingMatchContextsemantics). - Alternation primitive (required): a pattern is a set of skeletons; match
succeeds if any alternative matches. This absorbs structural variants
(if/return vs ternary) declaratively, eliminating the imperative desugar that
dragged
interopRequireDefaultdown to 41%. Without it the engine under-delivers; with it the fixed-shape subset compresses toward the 60% case. - Checker: a small per-helper closure over the bound holes verifies residual
semantics the skeleton can't (e.g.
$I/$Care the two params;$Eis the globalTypeError). The engine preserves the captured ctxt, so checkers may optionally be stricter than today's name-only global checks. - Builds on existing primitives:
match_context.rs(slots),helper_matcher.rs(binding identity),expr_utils.rs(structural compare) — ~80% of the foundation already exists.
Migration discipline (per matcher)
- Author the pattern(s) + checker as a
_v2function alongside the original. - Equivalence test: over rule-test inputs + synthetic variants + negatives (and
the external bundle corpus where convenient), assert
_v2agrees with the production matcher on every input — zero disagreements required. - Only then route production detection through
_v2and delete the original. - Full pipeline tests + snapshots must stay green (behavior-preserving), and the per-matcher LOC delta is recorded.
Partial adoption is the intended end state: migrate the fixed-shape subset, leave the marker-based giants bespoke and documented as the deliberate exception.