transformers

Rob Miles explains on Computerphile how language models generate plausible text and how transformers improved on recurrent networks, with a look at what comes next.

3Blue1Brown's visual explanation of how transformers and GPT-style language models work, tracing data through tokens, embeddings and attention.

3Blue1Brown's visual walkthrough of the attention mechanism inside transformers, covering self-attention, multiple heads and cross-attention. It is chapter 6 of his deep learning series.

Yannic Kilcher walks through the Attention Is All You Need paper, which introduced the Transformer, a sequence model based solely on attention without recurrence or convolutions. He reviews the architecture and the paper's translation results.

Alex O'Connor shows that ChatGPT cannot draw a wine glass filled to the brim, and ties the failure to David Hume's missing shade of blue. It asks what the failure says about how image models learn.

A Welch Labs explainer on neural scaling laws and the compute-efficient frontier, asking why model loss seems to hit a line it cannot cross. It draws an analogy to an ideal gas law for AI.

Musician Benn Jordan explains and demonstrates adversarial noise that poison-pills music files against AI training and scraping.

A guest video by Welch Labs on 3Blue1Brown explaining how text-to-image and video models work. It covers CLIP, shared embedding spaces, diffusion models (DDPM, DDIM), conditioning, guidance and negative prompts.

Rob Miles discusses GPT-3 on Computerphile, covering how scaling parameters kept improving results over GPT-2 and how the model nearly learned arithmetic without anyone knowing exactly how.