transformers

Rob Miles explains on Computerphile how language models generate plausible text and how transformers improved on recurrent networks, with a look at what comes next.

3Blue1Brown's visual explanation of how transformers and GPT-style language models work, tracing data through tokens, embeddings and attention.

3Blue1Brown's visual walkthrough of the attention mechanism inside transformers, covering self-attention, multiple heads and cross-attention. It is chapter 6 of his deep learning series.

Yannic Kilcher walks through the Attention Is All You Need paper, which introduced the Transformer, a sequence model based solely on attention without recurrence or convolutions. He reviews the architecture and the paper's translation results.

A short cartoon explaining how language models like ChatGPT work, from autocomplete and conditional probabilities to neural networks, gradient descent and backpropagation. It shows how networks act as universal function approximators.

Kurzgesagt examines how AI-generated content is flooding the internet, from video and music to news and books. It looks at how this makes verifying what is real harder.

Alex O'Connor shows that ChatGPT cannot draw a wine glass filled to the brim, and ties the failure to David Hume's missing shade of blue. It asks what the failure says about how image models learn.

A Welch Labs explainer on neural scaling laws and the compute-efficient frontier, asking why model loss seems to hit a line it cannot cross. It draws an analogy to an ideal gas law for AI.

Rob Miles explains "glitch" tokens, odd strings in large language model vocabularies that make the model produce gibberish. He covers how they likely arise from tokenizer training data, such as Reddit usernames, that never appeared in the model's own training.

Musician Benn Jordan explains and demonstrates adversarial noise that poison-pills music files against AI training and scraping.

Chubbyemu presents a case study of a man who followed AI health advice, replaced salt with sodium bromide and developed bromism with severe neurological effects. It is told as a dramatized medical case by a doctor.

Musician Benn Jordan uses AI to detect AI-generated music and digs into music industry data, including streaming payouts and sample library copyright problems. He covers ways to tell whether a track is human-made.

Computerphile video in which Mike Pound explains a University of Maryland paper on watermarking LLM output. The method biases token choices so generated text can later be detected statistically.

A guest video by Welch Labs on 3Blue1Brown explaining how text-to-image and video models work. It covers CLIP, shared embedding spaces, diffusion models (DDPM, DDIM), conditioning, guidance and negative prompts.

Rob Miles discusses GPT-3 on Computerphile, covering how scaling parameters kept improving results over GPT-2 and how the model nearly learned arithmetic without anyone knowing exactly how.