stopword

stopword is a JavaScript module for Node.js and the browser that strips stopwords — common, low-information words like “the”, “a”, and “is” — from an array of text tokens. It covers 62 languages with curated stopword lists sourced from the stopwords-iso and more-stoplists projects, spanning everything from English and French to less commonly supported languages like Somali, Sotho, and Lugbara.

The library exposes one function, removeStopwords(tokens, stopwords), which defaults to English but accepts any of the bundled per-language arrays (imported by ISO 639-3 code) or a fully custom list. It’s distributed as CommonJS, ESM, and UMD/minified bundles — including a jsDelivr CDN build — so it drops into Node scripts, bundled front-end apps, or plain <script> tags without a build step, and ships with an official @types/stopword package for TypeScript projects.

What You Get

• A single removeStopwords(tokens, stopwords?) function that filters an array of word tokens against a stopword list, defaulting to English.
• Individually importable stopword arrays for 62 languages, keyed by ISO 639-3 code (e.g. eng, fra, swe, zho), plus regional variants like Brazilian Portuguese and Gurmukhi Punjabi.
• Prebuilt CJS, ESM, and UMD/minified bundles ready for Node.js, bundlers, or direct <script> inclusion via jsDelivr.
• A numeric-token list (_123) for stripping digits (including Farsi, Korean, and Myanmar numerals) when paired with a tokenizer.

Common Use Cases

• Preprocessing text before feeding it into a search index or full-text search engine.
• Cleaning tokenized text before keyword extraction, topic modeling, or frequency analysis in NLP workflows.
• Normalizing multilingual user-generated content across dozens of supported languages.
• Building lightweight, dependency-free stopword removal directly in the browser via the UMD/CDN bundle.

Under The Hood

Architecture The library is intentionally minimal: src/stopword.js re-exports a single removeStopwords function alongside 60+ per-language stopword arrays defined in individual sibling modules (stopwords_eng.js, stopwords_fra.js, etc.), each just a flat exported array of strings. removeStopwords itself is a one-line Array.filter that lower-cases each token and checks membership in the chosen stopword array, defaulting to the English list when none is supplied. There’s no internal state, class hierarchy, or plugin system — the “architecture” is really a data-and-function pairing, with Rollup (rollup.config.js) handling the packaging into CJS/ESM/UMD bundles under dist/ so the same source ships in whatever module format a consumer needs.

Tech Stack Pure JavaScript with no runtime dependencies — words-n-numbers and batr appear only as devDependencies used in tests and the demo. Builds are produced with Rollup, code style is enforced with the standard linter (auto-fixed via npm run lint), and the package publishes CJS (stopword.cjs.js), ESM (stopword.esm.mjs), and UMD/minified bundles, with the UMD build also served from jsDelivr’s CDN. TypeScript consumers get types from a separate, community-maintained @types/stopword package rather than inline .d.ts files.

Code Quality Tests run via ava across three suites — test.cjs.js, test.esm.mjs, and a headless-browser ui-test.js (run under xvfb) — exercising both module formats and browser usage. CI (.github/workflows/tests.yml) runs this matrix across three Node versions (lts/-1, lts/*, node) on every push and PR, and npm test chains linting, building, and all three test suites together so a broken build or style violation fails fast. There’s no static type-checking on the source itself, but the flat, single-function design leaves little surface for logic bugs.

API Design The public surface is deliberately tiny and predictable: one function plus a set of consistently named, individually importable stopword arrays keyed by ISO 639-3 code. Getting started requires no configuration — removeStopwords(tokens) works out of the box against English, and swapping languages or merging custom terms is a matter of passing a different array as the second argument. The tradeoff is limited flexibility (only exact, case-insensitive token matching is supported, with no stemming or partial matching), but that keeps the learning curve close to zero.