Second Me

Second Me is an open-source AI identity platform that lets you build a personalized digital twin by training a local language model on your own memories, documents, voice notes, and reflections. Unlike cloud-based AI assistants that store your data on remote servers, Second Me keeps everything on your machine—training, inference, and storage all happen locally using Qwen2.5 base models fine-tuned with LoRA adapters and served via llama.cpp or MLX on Apple Silicon.

The platform’s core innovation is its three-layer Hierarchical Memory Modeling (HMM) pipeline: raw documents and notes are ingested and analyzed at the L0 layer, synthesized into structured identity facets called “shades” at the L1 layer using GraphRAG and ChromaDB, and then used to fine-tune a personalized language model at the L2 layer. This means your Second Me doesn’t just retrieve your memories—it embeds your perspective, reasoning style, and personality directly into model weights.

Once trained, your AI self can operate in multiple modes: a Playground for personal conversations and idea exploration, a Roleplay mode where it adopts different personas to represent you in simulated scenarios, and a Space mode for connecting with other Second Me instances on a permissioned decentralized network. An MCP server integration also exposes your AI self as a tool callable by external agents and applications.

Built for developers, researchers, and privacy-conscious users who want genuine AI personalization without data lock-in, Second Me runs on Windows, macOS (with MLX acceleration for M-series chips), and Linux, with Docker Compose setups for CPU, CUDA, and Apple Silicon deployment targets.

What You Get

• Hierarchical Memory Modeling (HMM) Pipeline - A three-layer L0/L1/L2 architecture that ingests raw documents and voice notes, clusters them into identity facets called ‘shades’ using GraphRAG and ChromaDB, then fine-tunes a Qwen2.5 model with LoRA adapters to embed your personality at the weights level.
• Local-First Training and Inference - All training runs on your hardware using Hugging Face Transformers, PEFT, and TRL, with GGUF conversion for efficient inference via llama.cpp (CPU/CUDA) or MLX on Apple Silicon M-series chips—no data ever leaves your machine.
• Roleplay Mode - Your AI self can switch personas to represent you in different simulated scenarios such as AMAs, speed dating conversations, or brainstorming sessions, with consistent identity coherence maintained by the Me-Alignment Algorithm.
• AI Space and Decentralized Network - Publish your Second Me to a permissioned network where it can collaborate with other AI selves, answer questions on your behalf, or participate in multi-agent workflows without exposing your raw data.
• MCP Server Integration - Two included MCP server implementations (local and public) expose your AI self as a tool callable by external agents, Claude, and other MCP-compatible applications.
• Multi-Format Memory Ingestion - Accepts text notes, Markdown files, PDFs (via pdfplumber and PyMuPDF), web links, and audio voice memos, all processed through a unified L0 insight extraction pipeline.
• Cross-Platform Docker Deployment - Dedicated Dockerfiles and Compose configurations for CPU (Linux/Windows), CUDA (GPU-accelerated), and Apple Silicon, with a Makefile orchestrating platform-specific builds.
• Real-Time Training Progress via WebSocket - A Flask-Sock WebSocket layer streams live training metrics and status updates to the Next.js frontend, giving users visibility into fine-tuning progress without polling.

Common Use Cases

• Personal Knowledge Twin for Researchers - An academic loads years of research notes, paper annotations, and voice memos into Second Me, producing an AI assistant that reasons in their specific domain vocabulary and can answer questions in their own analytical style.
• Privacy-First AI Delegate for Professionals - A consultant trains Second Me on client interaction notes and methodology documents, then deploys their AI self on the network to handle initial discovery conversations while they are unavailable—without sharing raw data with any cloud provider.
• Digital Presence for Communities and Creators - A blogger or domain expert publishes their Second Me to the decentralized network, letting readers interact directly with an AI that reflects their published opinions and reasoning, enabling asynchronous engagement at scale.
• Offline AI Assistant for Sensitive Environments - A legal or medical professional runs Second Me fully air-gapped on a local machine, using personal case notes and reference materials to build a private assistant that respects data residency and confidentiality requirements.
• Experimental AI Identity Development - A developer or researcher uses Second Me’s L0/L1/L2 pipeline as a framework for experimenting with personalized fine-tuning approaches, testing how different memory structures and training configurations affect identity coherence in small language models.

Under The Hood

Architecture Second Me implements a deliberate three-tier processing hierarchy that separates memory ingestion, identity synthesis, and model training into distinct layers with explicit data contracts between them. The L0 layer handles heterogeneous document intake—text, PDFs, audio, links—extracting insights and embeddings; L1 synthesizes these into structured identity facets using GraphRAG-powered clustering; L2 consumes the synthesized data to fine-tune a base language model via LoRA adapters. A Flask application with twelve domain-specific route blueprints serves as the HTTP gateway, with service and repository classes keeping HTTP handling decoupled from business logic. Training is executed as a spawned subprocess with WebSocket channels streaming progress back to the client, and ChromaDB handles vector storage for the retrieval-augmented components. The architecture cleanly separates the compute-heavy training workload from the inference-time serving path, though the dependency on filesystem-resident model artifacts creates an implicit coupling between training and deployment.

Tech Stack The backend runs on Python 3.12 with Flask 3.x and Flask-Sock for WebSocket support, using Flask-Pydantic for request validation and SQLAlchemy with asyncio extensions for database persistence. Model training uses Hugging Face Transformers, PEFT for LoRA adapter management, and TRL’s SFTTrainer for supervised fine-tuning on Qwen2.5 base models, with a bundled gguf-py library handling GGUF conversion for deployment via llama.cpp on CPU or CUDA, and MLX for Apple Silicon inference. Memory intelligence relies on OpenAI-compatible API calls (configurable to any endpoint), LangChain and ChromaDB for retrieval-augmented generation, Microsoft’s GraphRAG for knowledge graph construction, and sentence-transformers for semantic embedding. The frontend is built on Next.js with TypeScript, Ant Design for UI components, and Zustand for state management. Deployment is orchestrated through Docker Compose with separate Dockerfiles for CPU, CUDA, and Apple Silicon environments, with a Makefile providing a unified build interface.

Code Quality Test coverage is limited and concentrated in the bundled gguf-py library’s metadata and quantization modules, with no test suite covering the application’s own training pipeline, L0/L1/L2 generators, or API routes. Type annotations are used consistently throughout the Python backend with Pydantic models enforcing API request/response contracts, and the TypeScript frontend maintains strict typing. Error handling is explicit in the more critical paths—particularly around model loading and training—with custom exception patterns and defensive null checks, though some areas fall back to broad exception catches with logging. The ruff linter is present as a dev dependency and a modular service-repository pattern shows architectural discipline, but there is no CI configuration or pre-commit enforcement in the repository.

What Makes It Unique The fundamental novelty in Second Me is treating personal AI fine-tuning as a continuous identity operation rather than a one-time customization. The “shade” abstraction—personality facets extracted by clustering memory embeddings—goes beyond typical retrieval-augmented systems by driving a supervised fine-tuning step that encodes these facets at the model-weights level, not just at query time. This means the resulting AI self reflects the user’s perspective structurally, not just through document retrieval. The Me-Alignment Algorithm maintaining identity coherence across roleplay scenarios, the MCP server exposing the AI self to external tool ecosystems, and the decentralized network layer enabling AI-to-AI interaction with permissioned identity sharing combine to form an infrastructure layer for personal AI that has no direct open-source equivalent.

Self-Hosting

Second Me is released under the Apache License 2.0, one of the most permissive open-source licenses available. You are free to use it commercially, modify it, distribute it, and incorporate it into proprietary products without any copyleft obligation—the license does not require you to open-source changes you make. The only requirements are preserving copyright notices and the license text in distributions. This makes it unambiguously safe for enterprise self-hosting, internal tooling, or building commercial products on top of the platform.

Running Second Me yourself demands meaningful infrastructure investment. At minimum you need a machine with sufficient RAM to hold a quantized small language model in memory—the project’s own reference table suggests at least 8 GB for usable model sizes, with 16–32 GB recommended for models that deliver good conversational quality. The training step requires more headroom still: LoRA fine-tuning on even a 1–3B parameter model is CPU-intensive (hours on consumer hardware) or requires a CUDA-capable GPU. You are responsible for managing the Docker containers, keeping base models updated as the project evolves, handling backups of your model artifacts and memory database, and monitoring disk usage as training data accumulates. There is no managed update path—you pull new commits and rebuild.

There is no paid cloud tier or hosted offering from the Second Me team at this time, though the project’s roadmap references cloud training and deployment solutions as a future direction. What you trade for complete data sovereignty is operational convenience: no SLA, no managed backups, no auto-scaling, no support contract. Community support is available through a Discord server and GitHub Issues, both of which appear active, but resolution time is community-paced. Users who need guaranteed uptime or cannot maintain infrastructure should monitor the project’s roadmap for any managed offering before committing to self-hosted production use.