AI creative workflows

Member lesson

Failure, retries and what to do when a run dies

Lesson 9 of 10 in AI creative workflows.

AWS re:Invent 2024 - Try again: The tools and techniques behind resilient systems (ARC403) · AWS Events
Lesson

Model APIs fail. They time out, they rate-limit, they return something malformed, and a pipeline that assumes success will stop at three in the morning and be discovered at nine.

platform timeoutrequest timeoutretry budgettries × waitroomeach inner limit must end before the outer one
Three nested time limits: the platform's timeout holds the request timeout, which holds the retry budget; each inner limit must be smaller than the one around it.
Two sets of nested hourglasses. In the first, a small hourglass fits easily inside a larger glass one. In the second, the inner hourglass is too big and has cracked the outer glass, spilling sand across the table.
The inner timer must be smaller than the outer. Otherwise it breaks where you can't see it.
Member

The rest of this is for members.

You have read the opening of this lesson. The rest of it comes with a membership.

One membership covers all four courses and Brands.

  • All lessons in the four Learn courses
  • All brand breakdowns in the Brands section
  • All quizzes, flashcards and crosswords
  • An AI teacher on every lesson, for up to 60 questions a day
  • All course material published during the membership

Price

Membership is from $12 a month on the Yearly plan, which is charged at $144 once a year. Lifetime access is $59, paid once.Membership is from ₹200 a month on the Yearly plan, which is charged at ₹2,400 + GST once a year. Lifetime access is ₹3,999 + GST, paid once.

Check what stuck

Members only

You can't take this check yet.

Members answer these 3 questions and keep the ticket for this lesson. Every lesson has one.

  1. Question 1, locked

    Which of these words means this? “A temporary fault. Worth retrying, briefly.”

    • Ladder
    • Runbook
    • Transient failure
    • Seed
  2. Question 2, locked

    Which of these words means this? “A fault retrying cannot fix. Retrying it costs money and time.”

    • Decision record
    • Transient failure
    • Permanent failure
    • Dry run
  3. Question 3, locked

    Which of these words means this? “Waiting longer between each retry. Stops a retry storm making an outage worse.”

    • Human gate
    • Review budget
    • Backoff
    • Sweep

Elsewhere in the library

Hear about new courses.

Leave an email and we will tell you when something new lands. We do not write often.