Open Scouts

Open Scouts is an open-source, AI-powered web monitoring platform that lets users build “scouts” — automated agents that run on a defined schedule to search the web for specific information and send email notifications when results are found. Whether tracking local restaurant openings, monitoring AI industry news, watching competitor product announcements, or surfacing academic research, scouts work continuously so users don’t have to.

The platform is built on Next.js 15 and React 19, using Supabase as the core backend for authentication, real-time database, and edge function execution. Scouts are dispatched from a pg_cron + pg_net pipeline in PostgreSQL, triggering isolated Deno-based edge functions that orchestrate OpenAI GPT-4 agent loops with Firecrawl for web search and scraping. Each successful run generates a one-sentence AI summary stored with pgvector embeddings for semantic search across historical executions.

Users interact with scouts through a conversational chat interface powered by the Vercel AI SDK, refining goals and reviewing results in real time via Supabase Realtime subscriptions. Individual Firecrawl API keys can be stored per user, with usage tracked to their own account — making Open Scouts production-ready for self-hosting without requiring a shared provider key.

Open Scouts is a reference implementation from the Firecrawl team showing how to build scalable, AI-driven monitoring pipelines on open infrastructure. Its dispatcher architecture is designed to support thousands of concurrent scouts while maintaining execution isolation, rate-limit safety, and automatic stuck-execution cleanup.

What You Get

• Conversational Scout Creation - Users describe what they want monitored in plain language (e.g., “Scout for new Indian restaurants near me”) and an OpenAI GPT-4 agent automatically generates optimized search queries and scout configuration via function calling.
• Scalable Dispatcher Architecture - A pg_cron + pg_net pipeline triggers one isolated Deno edge function per scout every minute, so thousands of scouts can run concurrently without shared state or resource contention.
• Firecrawl-Powered Web Search & Scraping - Scouts execute multi-step search and scrape loops using Firecrawl’s v2 API, with geo-targeted queries, blacklisted-domain filtering, 60-second per-request timeouts, and automatic content trimming to 2,000 characters.
• Per-User Firecrawl API Keys - Each user enters their own Firecrawl API key in Settings; usage is tracked to their personal account and keys are stored securely in Supabase Vault, with server-side fallback for shared self-hosting deployments.
• AI-Generated Execution Summaries with Semantic Search - Every successful scout run produces a one-sentence OpenAI summary stored as a pgvector(1536) embedding, enabling semantic retrieval of past results by meaning rather than exact keyword match.
• Email Alerts via Resend - HTML email notifications are sent automatically when scouts find results, with test-email support and per-domain verification for Resend’s free tier (3,000 emails/month).
• Real-Time Result Dashboard - Supabase Realtime subscriptions push scout execution status and results to the UI as they arrive, with a step-by-step visual display of search and scrape tool calls.
• Consecutive Failure Auto-Disable - Scouts that fail three times in a row are automatically deactivated, with stuck executions (running >10 minutes) cleaned up by a separate pg_cron cleanup job every five minutes.

Common Use Cases

• Tracking local business openings - A food blogger sets up a scout to monitor for new Indian restaurants in their city; the scout searches weekly via Firecrawl with geo-targeted queries and emails links to new locations the moment they appear.
• Following AI industry news - A startup founder monitors for mentions of “LLM regulation” or “open-source AI models” across news sites and blogs, receiving rich HTML email digests on a daily or every-3-days cadence.
• Competitive product intelligence - A SaaS product manager watches competitor pricing and feature announcement pages, getting notified whenever Firecrawl detects new content that matches the scout’s defined goal.
• Academic research alerts - A PhD student configures a scout to surface new papers on “transformers in healthcare” from arXiv, university blogs, and preprint servers, with AI-generated one-sentence summaries delivered to their inbox.
• Real estate and listing monitoring - A buyer sets up scouts to track new property listings in specific neighborhoods, using the platform’s geolocation support to narrow searches to a city or district.
• Open-source project monitoring - A developer tracks GitHub discussions, blog posts, and changelog pages for specific libraries, turning open-ended web content into structured email updates without building a custom scraper.

Under The Hood

Architecture Open Scouts separates concerns cleanly across three execution tiers. The Next.js App Router frontend handles user interaction and real-time display via Supabase Realtime subscriptions, with React Server Components for auth-gated pages and client components for the chat and result panels. The Next.js API layer handles scout creation, chat streaming with the Vercel AI SDK, and manual execution triggers, each route performing parallel auth and ownership checks. The actual monitoring work runs entirely in Supabase: a pg_cron job calls a PostgreSQL dispatcher function every minute, which queries for due scouts and fires isolated pg_net HTTP POST requests to individual Deno edge functions — one per scout. This dispatcher pattern decouples scheduling from execution and allows horizontal scaling without shared worker state. A second pg_cron job handles cleanup of stuck executions.

Tech Stack The frontend is Next.js 15 with React 19 and TypeScript, using the App Router, Vercel AI SDK for streaming chat, Framer Motion and Motion for animations, and a comprehensive Radix UI + shadcn/ui component layer styled with Tailwind CSS v3. The backend uses Supabase for PostgreSQL (with pgvector for 1536-dimensional embeddings, pg_cron for scheduling, pg_net for HTTP dispatch, and Supabase Vault for encrypted credential storage), Deno-based Supabase Edge Functions for agent execution, Firecrawl’s v2 search and scrape API for web content retrieval, OpenAI GPT-4 with function calling for agent orchestration and embeddings, Resend for transactional email, and PostHog for server-side analytics with client-side flushing.

Code Quality The codebase is written in strict TypeScript with Zod schemas for API route validation and well-typed Supabase query results. No test files exist anywhere in the repository, leaving authentication flows, dispatcher logic, and agent tool execution entirely unverified by automated tests. Error handling is explicit in API routes (parallel auth and DB calls with structured 401/403/400 responses) but more generic in edge functions where error propagation relies on console logs rather than typed error hierarchies. Component organization is clean and modular, with custom hooks, context providers, and shared utilities in well-named directories. Linting is configured via ESLint with the Next.js preset and Prettier for formatting, though CI configuration is absent.

What Makes It Unique The most architecturally distinctive element is the SQL-native dispatcher: scout scheduling and dispatch happen entirely inside PostgreSQL using pg_cron + pg_net + vault-stored secrets, with no Node.js scheduler process or external queue. This means the scheduling layer has zero cold-start latency and inherits Supabase’s managed uptime. The per-user Firecrawl API key model — with vault storage, automatic partner key fallback, usage logging, and invalid-key detection — gives multi-tenant deployments clean account-level billing isolation without a dedicated key management service. The AI SDK chat interface is tightly integrated with scout execution history, letting users refine monitoring goals conversationally while viewing live step-by-step tool call traces as the agent searches and scrapes.

Self-Hosting

Open Scouts is licensed under the MIT License, as declared in package.json (no separate LICENSE file is present in the repository). MIT is a permissive open-source license: you can use, modify, distribute, and deploy the software commercially with no copyleft obligations. The only requirement is that the copyright notice be preserved in distributions. There are no open-core restrictions, enterprise gating, or feature flags separating a free and paid tier in the codebase.

Self-hosting Open Scouts requires meaningful infrastructure setup compared to a simple web app. You need a Supabase project with four extensions enabled: pg_cron, pg_net, vector, and supabase_vault. The setup script automates table creation, cron job configuration, vault secret storage, and edge function secret sync, but it runs against a live Supabase project rather than a local database, so you must link the Supabase CLI before proceeding. Two edge functions (scout-cron and send-test-email) must be deployed to Supabase Cloud — local edge function execution is not currently documented for production use. You also need active API accounts with OpenAI, Firecrawl, and Resend, and must configure Google Cloud Console credentials if you want Google OAuth. Ongoing operational responsibility includes monitoring cron job health (no alerting is built in), handling Firecrawl rate limits (the dispatcher sleeps 60 seconds between scout dispatches to stay within 6 req/min), and managing Resend domain verification for sending to arbitrary email addresses on the free tier.

There is no commercially managed version of Open Scouts — it is a reference implementation from the Firecrawl team. The hosted demo at openscouts.firecrawl.dev exists for exploration, but there is no official SaaS offering with SLAs, managed upgrades, priority support, or high-availability guarantees. Self-hosters accept full responsibility for uptime, database backups, secret rotation, and keeping up with upstream changes. The partner Firecrawl API key integration (which auto-provisions per-user keys) is noted in the README as a closed-beta enterprise feature not available publicly, so self-hosters should plan on users providing their own Firecrawl credentials.