newspaper3k

newspaper3k is a Python 3 library for news article scraping and full-text extraction. Given a URL, it downloads the page, parses out the main article text, authors, publish date, top image, all images, embedded videos, and other metadata, without you writing site-specific selectors.

Beyond single articles, it can build an entire news source by discovering category and article URLs from a homepage, downloading articles multi-threaded, and running lightweight NLP to extract keywords and summaries. It supports over 30 languages with automatic language detection, making it a common building block for news-monitoring and content-curation pipelines.

What You Get

• Full-text article extraction from raw HTML without per-site selectors
• Structured metadata: authors, publish date, top image, all images, and embedded videos
• Source building that discovers category and article URLs from a homepage
• A multi-threaded download framework for scraping many articles at once
• Lightweight NLP for keyword and summary extraction, plus 30-plus language support with auto-detection

Common Use Cases

• Building news-monitoring and media-tracking pipelines
• Extracting clean article text for NLP, search indexing, or datasets
• Curating and summarizing articles across many publications
• Collecting article metadata (authors, dates, images) at scale

Under The Hood

Architecture The package centers on two entry-point abstractions in newspaper/: Article (article.py) for a single URL and Source (source.py) for a whole publication. A pipeline of specialized modules handles each stage: network.py downloads HTML, parsers.py wraps lxml parsing, extractors.py and cleaners.py isolate the main content and metadata, images.py/videos.py pull media, nlp.py computes keywords and summaries, and outputformatters.py renders text. mthreading.py coordinates concurrent downloads and configuration.py exposes a Config object for proxies, user agents, and timeouts.

Tech Stack Pure Python 3, powered by lxml for fast HTML parsing and requests for downloading, with Pillow for image handling and NLTK-style corpora for the NLP steps. Packaging is classic setuptools via setup.py, with a download_corpora.py helper for the NLP data files.

Code Quality The codebase is cleanly decomposed by responsibility (network, parse, extract, clean, nlp, output) and ships a test suite under tests/ (unit_tests.py plus benchmarks and fixture data). Being a long-lived project it carries some legacy setup friction around native lxml/Pillow dependencies, documented thoroughly in the README, but the module boundaries make each extraction stage easy to follow.

API Design The public API is deliberately requests-inspired and beginner-friendly: Article(url), then download(), parse(), and optional nlp(), with results exposed as plain attributes. A single Config object threads proxy, user-agent, and threading settings through both Article and newspaper.build(), and helpers like fulltext(html) cover one-off extraction. Onboarding is fast for anyone who has used requests.