A carnival-style horse race in which 3-4 language models answer the same trivia questions. Each correct answer moves a horse one step, and the first model to reach the winning distance wins the derby.
Why I built it: I wanted to measure two things together: how smart a model is, and how fast its inference endpoint answers. Most benchmarks report only one of them. In LLM Derby, every model gets the same questions at the same time, so you see accuracy and speed side by side. A smart but slow model can lose to a fast model that is mostly right. You can also race the same model on two providers to compare their endpoints.
You do not need to install anything. It runs entirely in the browser. Bring an API key from any supported provider, or pick Simulated (no API calls) to try it immediately without a key.
Watch a race
Screenshots





Posts
I write about how I build and race LLM Derby, and what I learn about models, providers, and evals.
Features
- Race 3 or 4 models head-to-head - single race, best of 3, or best of 5
- Any provider, per horse - Nebius Token Factory, OpenAI, Anthropic, Google Gemini, Together AI, Groq, Fireworks AI, OpenRouter, DeepInfra, Mistral AI, Cerebras, Hugging Face, GMI Cloud, Baseten, or any custom OpenAI-compatible endpoint
- Tunable races - winning distance, per-question time limit, and difficulty levels (Easy / Medium / Hard / Expert), optionally rising from easy to hard
- Live race track with an announcer, streaks, and sound effects
- Question board showing every model’s answer, time, and token count side by side
- Results and analytics - podium, accuracy, time to first token, latency, tokens/sec, token usage and estimated cost, plus a “how the race unfolded” chart and accuracy by difficulty
- Export full results (JSON), answers (CSV), or a results image (PNG)
Questions and grading
The question bank has 266 questions across Easy, Medium, and Hard. Each question has one main answer and a list of accepted aliases.
Sample questions
| Level | Question | Answer |
|---|---|---|
| Easy | Which element has the chemical symbol Fe? | Iron |
| Easy | The Atacama Desert is mostly located in which country? | Chile |
| Medium | How many times does the letter “s” appear in the word “Mississippi”? | 4 |
| Medium | Which planet rotates on its side, with an axial tilt of about 98 degrees? | Uranus |
| Hard | How many squares of any size are there on a standard 8×8 chessboard? | 204 |
| Hard | In how many different ways can you make 50 cents using pennies, nickels, dimes and quarters? | 49 |
| Expert (generated each race) | What is 48173 × 9362? | 450995626 |
| Expert (generated each race) | In the Gregorian calendar, what day of the week is 14 March 2087? | Friday |
LLM Derby generates new Expert questions for every race. They include big multiplications, calendar math, counting, string puzzles, number bases, primes, Collatz, and more. Code computes the answers. The questions are new every time, so models cannot memorize them.
LLM Derby asks each model to end its reply with ANSWER: <short answer>. It then normalizes the answer (case, punctuation, accents, leading articles, number words) and compares it with the accepted answers.
Privacy
API keys stay in memory for the current browser tab only. The game never saves them, and sends them only to the provider you choose. There is no backend. For live demos, use a separate key with a spending limit.