Open Scouts is an open-source, AI-driven web monitoring platform that lets users create automated agents—called ‘scouts’—to continuously search the web for specific information and notify them via email when results are found. Built with Next.js, Supabase, and OpenAI, it combines semantic search, scheduled web scraping via Firecrawl, and real-time alerts to solve the problem of manual, time-consuming online monitoring. It’s designed for developers, researchers, and professionals who need to track dynamic web content—like new restaurant listings, AI news, or competitor updates—without manually refreshing pages. The platform supports user isolation, per-user API key management, and scalable execution via serverless edge functions.
What You Get
- AI-Powered Scouts - Create automated agents that interpret natural language queries (e.g., ‘Find new Indian restaurants near me’) and use OpenAI to generate search strategies, then execute them via Firecrawl for web scraping.
- Scheduled Web Monitoring - Scouts run on customizable intervals (hourly, daily, weekly) using pg_cron in Supabase to trigger executions without manual intervention.
- Email Alerts with Rich Content - Receive HTML-formatted email notifications via Resend when scouts find relevant results, including links and AI-generated summaries.
- Semantic Search on Execution Summaries - Each scout execution generates a one-sentence summary embedded with pgvector, enabling future search and filtering of past results by meaning, not just keywords.
- Per-User API Key Management - Users can securely add their own Firecrawl API key via the Settings UI, ensuring usage is tracked to their account and avoiding shared key limitations.
- Scalable Dispatcher Architecture - Uses pg_cron + pg_net to dispatch hundreds of scouts concurrently via isolated Deno edge functions, each with 256MB memory and 400s timeout for reliable execution.
- Real-Time Dashboard - View all scout results, history, and summaries in real-time using Supabase Realtime subscriptions with a React 19 + Next.js 15 frontend.
- Google OAuth and Email Auth - Supports secure user authentication via Supabase Auth with both email/password and Google OAuth options for seamless onboarding.
- Serverless Edge Functions - Scout execution logic runs in isolated Deno functions deployed to Supabase Cloud, ensuring security and scalability without managing servers.
Common Use Cases
- Building a personal AI news monitor - Track breaking developments in AI, cryptocurrency, or tech by creating scouts with queries like ‘latest LLM papers on arXiv’ or ‘new AI startup funding rounds’, and receive daily email summaries.
- Monitoring local business trends - Set up scouts to find newly opened restaurants, cafes, or retail stores in your city by querying ‘new Indian restaurants in Austin’ and get alerts with addresses and reviews.
- Competitor tracking → automated intelligence - Monitor competitor websites for product launches, pricing changes, or blog updates by defining scouts with specific keywords; receive summaries instead of raw HTML.
- DevOps teams managing external service alerts - Use Open Scouts to track status pages, GitHub releases, or blog updates from third-party tools (e.g., ‘OpenAI API status changes’) and trigger internal notifications via email.
- Research assistants automating literature reviews - Create scouts to find recent academic papers, conference announcements, or industry reports on niche topics and aggregate results into a digestible email format.
- Content curators building automated newsletters - Schedule scouts to find trending content on specific topics and use email alerts to populate weekly newsletters without manual searches.
Under The Hood
This project is a modern web application built with TypeScript and Next.js, designed to deliver a responsive, type-safe user interface with integrated AI-powered features. It emphasizes component reusability and structured API interactions, leveraging a range of UI libraries and tools to support rich functionality.
Architecture
This codebase adopts a monolithic architecture with clear modular boundaries, emphasizing separation of concerns within the Next.js ecosystem. The system utilizes API routes for backend interactions and maintains a layered structure that supports scalable UI development.
- Strong emphasis on component-based organization with reusable UI elements
- API-driven architecture using Supabase for authentication and data handling
- Clear separation between frontend components, shared utilities, and backend logic
Tech Stack
Built with TypeScript and Next.js 15, the application integrates a variety of modern frontend tools and libraries for enhanced performance and user experience.
- TypeScript ensures type safety and improves long-term maintainability
- UI is powered by Radix UI, Tailwind CSS, and Lucide React for rich component design
- Next.js provides routing and server-side rendering capabilities for fast page loads
- ESLint and Prettier are configured to enforce code quality standards
Code Quality
While the project demonstrates some consistency in coding patterns and error handling, it suffers from limited automated testing and signs of technical debt.
- Error handling is implemented through try/catch blocks across multiple modules
- Code style shows variation, with minimal enforcement of consistent formatting practices
- Technical debt is evident in duplicated logic and direct database access patterns
- Linting and formatting tools are present but not fully enforced across the codebase
What Makes It Unique
This project introduces a distinctive blend of AI-driven web scraping and automated monitoring features, setting it apart from typical web applications.
- Integrates AI-powered data collection with automated scout execution workflows
- Offers comprehensive admin controls for managing and optimizing scraping operations
- Combines continuous monitoring with intelligent data extraction capabilities