wink-nlp-utils

wink-nlp-utils is a JavaScript library from the winkjs family that provides over 30 focused functions for preparing raw text before it enters an NLP or machine-learning pipeline. It works on both strings (names, sentences, paragraphs) and token arrays, covering tasks like case conversion, punctuation and HTML-tag stripping, elision handling, sentence-boundary detection, n-gram generation, Porter2 stemming, phonetic encoding (Soundex), and stop-word removal.

Rather than bundling these into one opinionated pipeline, wink-nlp-utils exposes each transformation as an independent, composable function under string and tokens namespaces, so callers chain only the preprocessing steps they need. It’s commonly used as a preprocessing layer ahead of packages like wink-bm25-text-search or wink-naive-bayes-text-classifier, and its maintainers now recommend wink-nlp for full NLP pipelines, positioning this package specifically for lower-level text-preparation utilities.

What You Get

• String utilities for case conversion, trimming, and removing extra spaces, punctuation, special characters, and HTML tags
• Elision handling — split or remove contractions like “isn’t”, or amplify negation words for sentiment-aware preprocessing
• Sentence-boundary detection and a rule-based tokenizer that tags tokens as word/number/url/email/emoji/etc.
• N-gram, edge n-gram, bag-of-n-grams, and set-of-n-grams generation from strings
• Porter2 stemming and Soundex/phonetic encoding for both single strings and token arrays
• Token-array utilities: stop-word removal, bag-of-words/set-of-words, bigrams, and negation propagation

Common Use Cases

• Cleaning raw scraped or user-submitted text (removing HTML tags, extra whitespace, special characters) before indexing or search
• Building a preprocessing pipeline ahead of a text classifier or search engine such as wink-naive-bayes-text-classifier or wink-bm25-text-search
• Generating training corpora for chatbots via composeCorpus’s bracket-expansion syntax
• Normalizing tokens (stemming, stop-word removal, case-folding) for bag-of-words feature extraction
• Extracting a person’s name from a formatted string (e.g., stripping titles and degrees)

Under The Hood

Architecture wink-nlp-utils follows a flat, facade-based architecture: the entry point (src/wink-nlp-utils.js) builds a single prepare object with string and tokens namespaces, then requires and attaches roughly 30 independent single-purpose modules (string-lower-case.js, string-tokenize.js, tokens-bow.js, etc.), each exporting exactly one function. There is no shared base class or internal dependency graph between these modules beyond a handful of shared regex/helper files (util_regexes.js, helper-return-words-filter.js, helper-return-indexer.js) required directly by the functions that need them. This means each transformation is independently testable and swappable — replacing string-stem.js, for instance, would only affect prepare.string.stem and prepare.tokens.stem, not the rest of the namespace. The tradeoff is that cross-cutting behavior must be duplicated or imported per-file rather than composed through a shared pipeline abstraction.

Tech Stack The library is plain CommonJS JavaScript with no TypeScript and no build/bundle step — it ships src/*.js directly as its main entry. Its own runtime dependencies are entirely from the winkjs family: wink-nlp and wink-eng-lite-web-model power the detailed tokenizer, wink-porter2-stemmer implements stemming, wink-tokenizer and wink-helpers provide lower-level tokenization/regex primitives, and wink-distance is available for string-similarity use elsewhere in the family. Tooling is dated but complete: ESLint (eslint:recommended plus an extensive custom ruleset) for linting, JSDoc for API documentation generation (the docs/ directory is committed, generated via npm run docs), Mocha/Chai for tests, and nyc/Istanbul for coverage reporting to Coveralls, with CI on Travis CI against Node 16.17.1.

Code Quality Testing is extensive: test/wink-nlp-utils-specs.js alone runs over a thousand lines, exercising every exported function against tables of expected inputs/outputs including shared null/undefined/numeric edge cases, and .nycrc.json enforces a 99.5% coverage floor across branches, lines, functions, and statements via check-coverage. Error handling is minimal and implicit — invalid inputs (null, undefined, wrong type) are allowed to throw JavaScript’s own runtime errors rather than being caught and re-thrown as descriptive, typed errors. Naming is consistent and camelCase throughout, and every public function carries a JSDoc block with @alias, @param, @return, and @example tags, which also generates the published API docs. There is no TypeScript, so there are no compile-time type guarantees for consumers.

API Design The public API is deliberately flat and composable: a single top-level require('wink-nlp-utils') exposes two namespaces, string and tokens, and callers pick only the functions they need rather than running a fixed pipeline — e.g., tokenize, then removeWords, then stem, called explicitly in sequence. Several functions offer both a plain-array and a “detailed” mode (e.g., tokenize(sentence, true) returns tagged token objects instead of bare strings), keeping the default call site simple while still exposing richer output on demand. Naming aliases (e.g., bow/bagOfWords, soc/setOfChars) let callers use either a terse or descriptive form. Getting started requires no configuration beyond require() — there’s no client instantiation, connection, or schema step, keeping boilerplate to a single line per function call.