LMSYS Chatbot Arena
Nonprofit running Chatbot Arena, a crowdsourced benchmark platform for comparing large language models.
Updated April 2026
Overview
- Website
- chat.lmsys.org
- Founded
- 2023
- Headquarters
- Berkeley, CA
- Segment
- Model Distribution & Serving
Product overview
LMSYS builds and maintains Chatbot Arena, a crowdsourced platform where users compare responses from anonymous LLMs via pairwise battles to generate Elo-based leaderboards reflecting real-world preferences. It evaluates over 100 models from providers like OpenAI, Anthropic, Google, and open-source ones like Llama, attracting millions of users and hundreds of thousands of votes. Distinct from static benchmarks by using live, community-driven human evaluations on diverse real prompts, with all infrastructure open-sourced on GitHub.
Revenue model
501(c)(3) nonprofit sustained through tax-deductible donations and sponsorships including cash, GPUs, and cloud credits from NVIDIA, Google Cloud, a16z, AMD, and Hugging Face. No paid services, subscriptions, or API pricing.
Moat
The key competitive moat of LMSYS Chatbot Arena is its massive crowdsourced dataset of over 1 million human preference votes on real-world prompts, which powers the de facto industry-standard Elo leaderboard that's more cited than traditional benchmarks like MMLU and provides near-live feedback that model developers prioritize for iteration. This creates powerful network effects—more users and models improve data quality and leaderboard relevance, generating proprietary high-quality preference data (e.g., Arena-Hard subsets) that's hard to replicate without equivalent scale, alongside high switching costs for the AI community now reliant on it for dynamic, non-overfittable evaluations.