SWIRL

SWIRL is an open-source federated metasearch and RAG platform that distributes queries across databases, SaaS services, search engines, and enterprise apps simultaneously—without extracting, copying, or indexing any data. Instead of building a central index, SWIRL reaches out to each source at query time, collects results, and uses spaCy and NLTK to re-rank them by semantic relevance before presenting a unified view.

Designed for organizations where data sovereignty and security are non-negotiable, SWIRL supports 40+ pre-built connectors covering Microsoft 365 (SharePoint, OneDrive, Outlook, Teams), developer tools (GitHub, Jira, Confluence), databases (BigQuery, Snowflake, PostgreSQL, MongoDB), vector stores (Pinecone, Qdrant), and public sources (arXiv, Google News). Each connector is a configurable SearchProvider that maps query syntax and result fields to SWIRL’s unified schema.

SWIRL adds RAG on top of federated search without requiring a separate ETL pipeline: after retrieving and re-ranking results, it passes the top documents to an OpenAI or Azure OpenAI model to generate cited, context-aware answers. This combination—real-time federation plus in-context RAG—means teams can ask questions in natural language and get answers sourced from their live internal systems.

Deployable in minutes via a single Docker Compose command, SWIRL also exposes its full capability through a REST API and a Model Context Protocol (MCP) server, making it directly accessible to AI agents like Claude Desktop, Claude Code, and Cursor—enabling them to answer questions against private enterprise sources through a standard protocol with no additional data plumbing.

What You Get

• Federated Search Across 40+ Connector Types - Fan out a single query to SharePoint, OneDrive, Outlook, Teams, GitHub, Jira, Confluence, BigQuery, Snowflake, MongoDB, Elasticsearch, OpenSearch, Pinecone, Qdrant, arXiv, Google News, and more—simultaneously, in parallel, without touching the source data.
• LLM Re-Ranking Without a Vector Database - Uses spaCy’s en_core_web_lg model and NLTK stemming to score and re-rank federated results by cosine similarity to the query, replacing the need for a separate vector index or embedding pipeline.
• Inline RAG with Cited Sources - After ranking, passes the top results to an OpenAI or Azure OpenAI model for context-aware answer generation with source attribution—no preprocessing or ingestion pipeline required.
• Model Context Protocol (MCP) Server - The companion swirl-mcp-server exposes six MCP tools (search, rag_answer, list_providers, create_search, get_results, list_searches) so Claude Desktop, Claude Code, and Cursor can query private enterprise sources directly through a standard agent protocol.
• Configurable Processor Pipeline - Chain pre-query processors (spell correction, GenAI query rewriting, synonym expansion), result processors (date detection, deduplication, relevancy scoring, PII removal), and custom mapping stages per SearchProvider.
• Duplicate Detection via Cosine Similarity - Identifies and removes semantically near-identical results across providers using configurable similarity thresholds on title and body fields, reducing noise in federated result sets.
• Search Subscription and Continuous Monitoring - Subscribe to any search to receive ongoing updates as new matching results appear across connected sources, useful for ticket monitoring, news tracking, or alerting workflows.
• Result Mixers for Custom Ordering - Select from relevancy, date, or round-robin (stack) mixing strategies to control how federated results from multiple sources are merged and ordered for end users.
• Microsoft 365 OAuth2 Integration - Full MSAL-based OAuth2 authentication for M365 Outlook, OneDrive, SharePoint, and Teams, with tag-prefix shortcuts (sharepoint:, teams:, outlook:) for scoped queries within M365.
• Galaxy UI with AI Summary - Built-in web interface displaying federated results with AI-generated summaries, source attribution, author fields, date sorting, result selection, and chart visualizations for structured data responses.

Common Use Cases

• Enterprise Knowledge Search Without Migration - An IT team deploys SWIRL to unify search across SharePoint, Confluence, and Google Drive, giving employees one search bar that returns ranked results from all three—without migrating content or building a new index.
• AI Agent with Access to Private Data - A developer configures SWIRL’s MCP server so a Claude-based AI agent can answer questions by searching internal Jira tickets, GitHub issues, and Confluence pages in real time, with no data leaving the company’s network.
• Compliance and Legal Discovery - A legal team uses SWIRL to search across email archives, OneDrive, and a PostgreSQL case database simultaneously, applying PII removal and relevancy ranking to surface the most pertinent documents quickly.
• Developer Productivity and Onboarding - Engineers query SWIRL to search GitHub repositories, Jira, and internal wikis together, getting ranked results that surface relevant code, tickets, and documentation in a single response, reducing context-switching during debugging or onboarding.
• Research and Competitive Intelligence - An analyst uses SWIRL to federate queries across arXiv, PubMed (via European PMC), Google News, and internal reports, receiving a ranked, deduplicated result set with an AI-generated summary of key findings.
• Customer Support Knowledge Base - A support team connects SWIRL to their help documentation, ticket system, and CRM, enabling agents to ask natural-language questions and receive RAG-generated answers with source links from live internal systems.

Under The Hood

Architecture SWIRL is built on a layered Django monolith where domain concerns—SearchProviders, Connectors, Processors, and Mixers—each occupy distinct module boundaries with well-defined interfaces. Celery dispatches search requests asynchronously using task groups, enabling true parallel federation across all configured providers simultaneously. A declarative processor pipeline handles pre-query transformation, per-result field mapping, semantic relevancy scoring, and deduplication as composable stages that can be configured independently per SearchProvider. The Mixer abstraction provides pluggable result ordering strategies—relevancy, date, and round-robin—applied as a final pass over the merged result set. The recent addition of a Model Context Protocol adapter decouples AI agent access from the REST API, exposing federated search as native agent tools without requiring changes to the core platform.

Tech Stack The backend runs Python 3.13 on Django 5.x with Django REST Framework, served via Daphne (ASGI) with WebSocket support through Django Channels for real-time result streaming. Celery with Redis as the message broker handles asynchronous search federation, orchestrated through docker-compose for single-command deployment. Semantic relevancy uses spaCy’s en_core_web_lg model for cosine similarity scoring combined with NLTK for stemming, stopword handling, and sentence tokenization. Connector targets span Elasticsearch, OpenSearch, Google BigQuery, Snowflake, MongoDB, PostgreSQL, SQLite3, Pinecone, and Qdrant for vector search. RAG generation uses the OpenAI Python SDK with configurable model selection including GPT-4.1 and Azure OpenAI equivalents. GitHub Actions enforces quality through a QA suite, unit tests, Docker build validation, spell checking, and URL verification workflows.

Code Quality The project maintains a comprehensive test suite spanning 15 test modules covering authentication flows, query processing and transformation, RAG integration, middleware, API request handling, and Microsoft 365-specific scenarios. Tests use pytest fixtures with realistic JSON payload data files, Django TestCase for ORM-dependent paths, and extensive mocking to isolate external dependencies including OpenAI and Microsoft Graph calls. Error handling is explicit throughout the connector and authenticator layers, with structured logging via Celery’s task logger and custom exception classes. Naming follows Django and Python conventions consistently across the codebase. The absence of type annotations in several core modules limits static analysis coverage, but the combination of realistic fixture data, external dependency isolation, and multi-workflow CI demonstrates solid engineering discipline.

What Makes It Unique SWIRL’s defining technical innovation is its zero-extraction federated model: it re-ranks results from 40+ heterogeneous source types at query time using spaCy cosine similarity and NLTK stemming, without ever indexing or copying a document. Where conventional RAG systems require preprocessing pipelines to ingest data into a vector database, SWIRL preserves data sovereignty by querying sources in-place and applying LLM-powered relevancy as a real-time post-processing stage. The addition of an MCP server is a forward-looking integration that exposes this federation capability directly to AI agents through a standard protocol—enabling Claude, Cursor, and other MCP clients to query private enterprise systems without any additional plumbing or data movement, a pattern that no traditional enterprise search platform currently provides.

Self-Hosting

SWIRL is licensed under Apache License 2.0, a permissive open-source license that allows free use, modification, and redistribution for both personal and commercial purposes. There are no copyleft obligations—you can embed SWIRL in a commercial product or internal platform without being required to open-source your modifications. The only material requirement is that you preserve copyright notices and the original license text when distributing. This makes it genuinely enterprise-safe from a legal standpoint without the compliance complexity of LGPL or AGPL alternatives.

Running SWIRL yourself requires a meaningful operational footprint: Python 3.13, Redis (for the Celery message broker), and a spaCy language model download at startup. The recommended path is Docker Compose, which bundles everything into a reproducible single-command deployment, but the container is stateless by default—credentials, SearchProvider configurations, and search results are not persisted across restarts unless you configure external PostgreSQL and Redis volumes. For production use, the team’s documentation covers persistent installation, Kubernetes deployment, and database migration, but these require deliberate setup. You become responsible for uptime, security patching, Redis availability, and keeping the spaCy model and connector credentials current as source APIs evolve.

SWIRL’s team offers an enterprise managed service for organizations that want to offload operational responsibility. The community edition you self-host does not include SLAs, managed upgrades, enterprise SSO beyond the documented Microsoft OIDC path, high-availability clustering, or dedicated support—contact with issues is via Slack and GitHub. The enterprise tier adds these managed-service guarantees along with enhanced connector coverage and priority support. For teams that can accept the operational burden, the community edition is fully functional and production-capable; for organizations that cannot staff the operational overhead, the managed offering removes that friction at a cost that is not publicly listed.