How the machines work

Member lesson

Tokens, embeddings and how words become directions

Lesson 4 of 13 in How the machines work.

Transformers, the tech behind LLMs | Deep Learning Chapter 5 · 3Blue1Brown
Lesson

A model cannot read. Your prompt is cut into tokens, each token is looked up as a list of numbers, and those numbers are what the rest of the system sees. The cutting is not by word: common words are one token, unusual ones are several, and a brand name you invented last week may be four fragments that mean nothing together.

pixels inencodelatentsmall, and organiseddenoiseheredecodepixels outfine detail is lost on the way IN, not fumbled on the way outwhich is why small type never survives
Pixels compressed into a small latent space, worked on there, then decoded back to pixels.
Scissors cut a strip of small drawings into pieces. On a corkboard the pieces are pinned in groups of related things: citrus fruits together, a boat with an oar and sail, a teapot with a cup. One squiggle has been cut into scraps that fall to the floor unpinned.
Related things land near each other. The squiggle falls apart, as a made-up brand name does.
Member

The rest of this is for members.

You have read the opening of this lesson. The rest of it comes with a membership.

One membership covers all four courses and Brands.

  • All lessons in the four Learn courses
  • All brand breakdowns in the Brands section
  • All quizzes, flashcards and crosswords
  • An AI teacher on every lesson, for up to 60 questions a day
  • All course material published during the membership

Price

Membership is from $12 a month on the Yearly plan, which is charged at $144 once a year. Lifetime access is $59, paid once.Membership is from ₹200 a month on the Yearly plan, which is charged at ₹2,400 + GST once a year. Lifetime access is ₹3,999 + GST, paid once.

Check what stuck

Members only

You can't take this check yet.

Members answer these 3 questions and keep the ticket for this lesson. Every lesson has one.

  1. Question 1, locked

    Which of these words means this? “A chunk of text, usually part of a word. Models read and are billed in these.”

    • Denoising step
    • Sampler
    • Model
    • Token
  2. Question 2, locked

    Which of these words means this? “The thing that cuts text into tokens. Different models cut differently, which is why prompts do not transfer cleanly.”

    • Denoising step
    • Sampler
    • Model
    • Tokenizer
  3. Question 3, locked

    Which of these words means this? “The part of an image model that reads your prompt. Often the real reason two models understand you differently.”

    • Unit economics
    • Text encoder
    • Cross-attention
    • Vision model

Elsewhere in the library

Hear about new courses.

Leave an email and we will tell you when something new lands. We do not write often.