lexer

Lex (published to npm as lex) is a tiny, dependency-free JavaScript lexer modeled after the classic Unix flex scanner generator. Instead of generating code from a grammar file, it lets you build a lexer at runtime by registering regular-expression rules directly against a Lexer instance, each paired with an action function that runs when the pattern matches.

The library implements flex-style behavior in plain JavaScript: rules compete for the longest match, numbered start conditions let a single lexer behave differently depending on mode (useful for things like string interpolation or indentation-sensitive syntax), and an action can call this.reject = true to decline its own match and fall through to the next-best rule. It ships as a single ~150-line file with no runtime dependencies, and is commonly wired up as a custom scanner for parser generators like Jison.

What You Get

• A Lexer constructor with addRule(pattern, action, start) and setInput(input) methods that support method chaining
• Automatic normalization of regex flags (sticky/global, multiline, ignoreCase, unicode) so rules work consistently across JS engines
• Flex-style numbered start conditions (inclusive and exclusive) for building stateful, mode-switching lexers
• A reject mechanism that lets an action decline its match and fall back to the next longest-matching rule
• Support for returning multiple tokens from a single action (e.g. for Python-style INDENT/DEDENT handling)
• A configurable defunct handler for unmatched characters, with a sensible default that throws a descriptive error

Common Use Cases

• Writing a custom scanner/lexer stage ahead of a hand-rolled or generated parser (e.g. as the lexer for a Jison grammar)
• Tokenizing small domain-specific languages, config formats, or template syntaxes embedded in an application
• Implementing indentation-sensitive tokenization (Python-style INDENT/DEDENT) via multi-token actions
• Building mode-switching tokenizers (e.g. treating text differently inside vs. outside a string or code block) using start conditions

Under The Hood

Architecture The entire engine lives in a single file exposing one constructor, Lexer, whose instance state (tokens, rules, a remove counter) is held in closures rather than instance properties. addRule normalizes each pattern’s regex flags (forcing sticky/global support, preserving multiline/ignoreCase/unicode) and stores rule objects keyed by optional start-condition arrays. The public lex() method drives a scan loop: it calls a private scan() function that tests every active rule’s regex against the current index, collects all matches, and sorts them so the longest match wins (except for rules flagged global, which are deliberately left unsorted to support first-match-wins patterns). Rejection support (this.reject = true inside an action) lets a match be declined mid-loop, causing lex() to fall back to the next candidate and rewind its index bookkeeping. There is a single abstraction layer — closures over private arrays and counters — so the whole state machine would need to move together if the core matching strategy changed; there’s no dependency injection or layering beyond that.

Tech Stack Lex is plain ES5 JavaScript with zero runtime dependencies ("dependencies": {} in package.json) and no build step — it ships lexer.js directly as its main entry. Distribution predates modern bundler tooling: it supports npm, RingoJS (via ringo-admin), and browser use via Bower and Component, but has no ES module output, no TypeScript types, and no bundler or lockfile configuration.

Code Quality No test files, test framework, or CI configuration exist anywhere in the repository. There’s no linter or formatter config either. Error handling is limited to a single default defunct handler that throws a descriptive Error for unmatched input; there is no other explicit error handling in the ~150-line source. Naming is terse and reflects a pre-ES6 style (remove, index, matches), consistent with the library’s 2012–2018 development window, and cross-realm array detection is done manually via Object.prototype.toString.call() rather than Array.isArray.

API Design The public surface is small and consistent: addRule(pattern, action, start), setInput(input), and lex(), with addRule/setInput both chainable via return this. Callers don’t need engine-specific regex knowledge — Lex normalizes sticky/global flags automatically. The flex-style start-condition system and the per-action reject fallback are unusual, thoughtful touches for a library this small, letting users implement stateful, mode-aware tokenizers (shown in the README via a Python-indentation example) without hand-rolling a separate state machine. Getting started requires only a few lines of code, though the API predates newer iterator/generator-based tokenizer conventions.