How the machines work

Member lesson

Benchmarks, and why they mislead

Lesson 12 of 13 in How the machines work.

LLM & AI Agent Benchmarks vs Reality: Why AI Applications Break · IBM Technology
Lesson

Every model launch comes with a chart, and the chart is almost never about you. Preference arenas measure which image a stranger picks in two seconds, which rewards contrast, saturation and drama. Your job may be a flat-lit product on a plain ground where drama is exactly the failure.

On the left, a spotlit winner's podium holds a dramatic, high-contrast print while small figures clap. On the right, a person at a plain table lays out ten small prints of the same simply lit jar in a row and marks each with a pencil.
The podium rewards drama. Ten quiet test briefs from your own work tell you whether a model suits you.
Member

The rest of this is for members.

You have read the opening of this lesson. The rest of it comes with a membership.

One membership covers all four courses and Brands.

  • All lessons in the four Learn courses
  • All brand breakdowns in the Brands section
  • All quizzes, flashcards and crosswords
  • An AI teacher on every lesson, for up to 60 questions a day
  • All course material published during the membership

Price

Membership is from $12 a month on the Yearly plan, which is charged at $144 once a year. Lifetime access is $59, paid once.Membership is from ₹200 a month on the Yearly plan, which is charged at ₹2,400 + GST once a year. Lifetime access is ₹3,999 + GST, paid once.

Check what stuck

Members only

You can't take this check yet.

Members answer these 3 questions and keep the ticket for this lesson. Every lesson has one.

  1. Question 1, locked

    Which of these words means this? “A standard test set. Useful for comparing models, poor at predicting whether one suits your job.”

    • Vision model
    • VAE
    • Benchmark
    • Compositionality
  2. Question 2, locked

    Which of these words means this? “A ranking from head-to-head human preference votes. Measures what people prefer at a glance.”

    • Hallucination
    • Sample
    • Determinism
    • Elo / arena score
  3. Question 3, locked

    Which of these words means this? “When the test questions were in the training data. Inflates a score without improving anything.”

    • Fine-tuning
    • Captioning
    • Embedding
    • Contamination

Elsewhere in the library

Hear about new courses.

Leave an email and we will tell you when something new lands. We do not write often.