LLM Derby logo

A carnival-style horse race in which 3-4 language models answer the same trivia questions. Each correct answer moves a horse one step, and the first model to reach the winning distance wins the derby.

Why I built it: I wanted to measure two things together: how smart a model is, and how fast its inference endpoint answers. Most benchmarks report only one of them. In LLM Derby, every model gets the same questions at the same time, so you see accuracy and speed side by side. A smart but slow model can lose to a fast model that is mostly right. You can also race the same model on two providers to compare their endpoints.

You do not need to install anything. It runs entirely in the browser. Bring an API key from any supported provider, or pick Simulated (no API calls) to try it immediately without a key.

Watch a race

Screenshots

Posts

I write about how I build and race LLM Derby, and what I learn about models, providers, and evals.

Read the LLM Derby posts →

Features

  • Race 3 or 4 models head-to-head - single race, best of 3, or best of 5
  • Any provider, per horse - Nebius Token Factory, OpenAI, Anthropic, Google Gemini, Together AI, Groq, Fireworks AI, OpenRouter, DeepInfra, Mistral AI, Cerebras, Hugging Face, GMI Cloud, Baseten, or any custom OpenAI-compatible endpoint
  • Tunable races - winning distance, per-question time limit, and difficulty levels (Easy / Medium / Hard / Expert), optionally rising from easy to hard
  • Live race track with an announcer, streaks, and sound effects
  • Question board showing every model’s answer, time, and token count side by side
  • Results and analytics - podium, accuracy, time to first token, latency, tokens/sec, token usage and estimated cost, plus a “how the race unfolded” chart and accuracy by difficulty
  • Export full results (JSON), answers (CSV), or a results image (PNG)

Questions and grading

The question bank has 266 questions across Easy, Medium, and Hard. Each question has one main answer and a list of accepted aliases.

Sample questions

LevelQuestionAnswer
EasyWhich element has the chemical symbol Fe?Iron
EasyThe Atacama Desert is mostly located in which country?Chile
MediumHow many times does the letter “s” appear in the word “Mississippi”?4
MediumWhich planet rotates on its side, with an axial tilt of about 98 degrees?Uranus
HardHow many squares of any size are there on a standard 8×8 chessboard?204
HardIn how many different ways can you make 50 cents using pennies, nickels, dimes and quarters?49
Expert (generated each race)What is 48173 × 9362?450995626
Expert (generated each race)In the Gregorian calendar, what day of the week is 14 March 2087?Friday

LLM Derby generates new Expert questions for every race. They include big multiplications, calendar math, counting, string puzzles, number bases, primes, Collatz, and more. Code computes the answers. The questions are new every time, so models cannot memorize them.

LLM Derby asks each model to end its reply with ANSWER: <short answer>. It then normalizes the answer (case, punctuation, accents, leading articles, number words) and compares it with the accepted answers.

Privacy

API keys stay in memory for the current browser tab only. The game never saves them, and sends them only to the provider you choose. There is no backend. For live demos, use a separate key with a spending limit.