How the machines work

Member lesson

Transformers and attention

Lesson 5 of 13 in How the machines work.

Attention in transformers, step-by-step | Deep Learning Chapter 6 · 3Blue1Brown
Lesson

Before 2017 a model reading a sentence read it in order and struggled to connect a pronoun to a noun eleven words back. Attention changed the shape of the problem. Every position gets to look at every other position and decide how much each one matters, all at once, which is both a better model of language and a far better fit for the hardware.

abottlelitfromthe leftevery word weighs every other wordwhich is why word order and emphasis do anything at all
Five words, each connected to every other, with two connections drawn heavier.
A round table seen from above with seven objects on it, including a pear, a key, a shoe and a cup. Threads connect every object to every other. The threads leading to the pear are thick and coloured; the rest are thin.
Everything looks at everything. Give it one clear subject, like the pear, and that is where the pull goes.
Member

The rest of this is for members.

You have read the opening of this lesson. The rest of it comes with a membership.

One membership covers all four courses and Brands.

  • All lessons in the four Learn courses
  • All brand breakdowns in the Brands section
  • All quizzes, flashcards and crosswords
  • An AI teacher on every lesson, for up to 60 questions a day
  • All course material published during the membership

Price

Membership is from $12 a month on the Yearly plan, which is charged at $144 once a year. Lifetime access is $59, paid once.Membership is from ₹200 a month on the Yearly plan, which is charged at ₹2,400 + GST once a year. Lifetime access is ₹3,999 + GST, paid once.

Check what stuck

Members only

You can't take this check yet.

Members answer these 3 questions and keep the ticket for this lesson. Every lesson has one.

  1. Question 1, locked

    Which of these words means this? “A mechanism that lets every part of the input weigh every other part, rather than reading strictly left to right.”

    • Seed
    • Contamination
    • Attention
    • Tokenizer
  2. Question 2, locked

    Which of these words means this? “The architecture built around attention. Under almost everything in this course, image models included.”

    • Memorisation
    • Batch
    • Token cost
    • Transformer
  3. Question 3, locked

    Which of these words means this? “One attention calculation. A layer runs many in parallel, each learning to care about something different.”

    • Memorisation
    • Batch
    • Head
    • Token cost

Elsewhere in the library

Hear about new courses.

Leave an email and we will tell you when something new lands. We do not write often.