transformers

Rob Miles explains on Computerphile how language models generate plausible text and how transformers improved on recurrent networks, with a look at what comes next.

Leo Dirac explains how transformer models have replaced LSTMs in NLP. He gives background from bag of words through RNNs and LSTMs, then a technical deep dive into multi-headed self-attention and positional encoding, with sample code.

3Blue1Brown's visual explanation of how transformers and GPT-style language models work, tracing data through tokens, embeddings and attention.

3Blue1Brown's visual walkthrough of the attention mechanism inside transformers, covering self-attention, multiple heads and cross-attention. It is chapter 6 of his deep learning series.

A university lecture from CS480/680 by Pascal Poupart on attention mechanisms and transformer networks, worked through on a chalkboard.

Andrej Karpathy builds a GPT from scratch in code, following Attention Is All You Need and GPT-2/3, in about two hours using PyTorch. He connects it to ChatGPT and assumes familiarity with his earlier makemore videos.

Yannic Kilcher walks through the Attention Is All You Need paper, which introduced the Transformer, a sequence model based solely on attention without recurrence or convolutions. He reviews the architecture and the paper's translation results.

A short cartoon explaining how language models like ChatGPT work, from autocomplete and conditional probabilities to neural networks, gradient descent and backpropagation. It shows how networks act as universal function approximators.

Dwarkesh Patel interviews Google's Jeff Dean and Noam Shazeer about 25 years at the company, from PageRank and MapReduce to the Transformer, mixture of experts and Gemini. They also discuss where AI is heading.

Lex Fridman's long conversation with Andrej Karpathy on Tesla's self-driving approach, Optimus, neural networks, transformers, AGI and more. It also touches on his OpenAI work and working with Elon Musk.

Ilya Sutskever gives a Simons Institute talk on why neural networks generalize, framing learning as finding short descriptions, like compression. It was part of the Large Language Models and Transformers program.

Tesla's 2021 AI Day presentation covering vision, planning and control, auto-labeling, simulation, the Dojo training hardware and the Tesla Bot announcement, followed by a Q&A.

Andrej Karpathy's three-hour-plus general-audience deep dive into how ChatGPT-style LLMs are built, covering the full training stack from pretraining to post-training. It also gives mental models for their quirks, like hallucination, and how to use them well.

Jeremy Howard's 90-minute practical tour of language models. It covers how they work, GPT-4's capabilities and limits, using the OpenAI API, code writing, data analysis and fine-tuning.

Andrej Karpathy gives a one-hour general-audience introduction to large language models, covering what they are, how they are trained, comparisons to operating systems, and security challenges such as jailbreaks.

Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to tokenization, and covers Unicode byte encodings along the way.

Sebastien Bubeck presents the Sparks of AGI paper at MIT, showing early experiments with GPT-4 across many domains and asking whether artificial intelligence has arrived. It includes the TikZ unicorn example.

Tristan Harris and Aza Raskin of the Center for Humane Technology present why current AI capabilities pose catastrophic societal risks. They argue companies are racing to deploy without adequate safety, and that institutions must adapt. It was given before GPT-4's launch.

Sentdex demonstrates GAN Theft Auto, a GameGAN-based neural network that recreates GTA V so the network itself is the playable environment. He shows the results and releases the code.

Corridor Digital makes a roughly seven-minute anime short using Stable Diffusion and related AI tools, based on a technique they teach in a longer tutorial. It was an early showcase of AI-assisted animation.

Geoffrey Hinton's 2025 Royal Institution Discourse on what AI has learned from biological intelligence and how it might surpass it. He explains neural networks and argues that human language use resembles what backpropagation-trained networks do.

Mario Zechner explains why he built pi, a minimal coding agent, out of frustration with existing agent harnesses. He also discusses how AI agents are straining open source projects and argues for slowing down.

Andrej Karpathy walks through how GPT assistants are trained, from tokenization and pretraining to supervised finetuning and RLHF, and gives practical prompting strategies and a view of the tooling ecosystem.

Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.

Yannic Kilcher explains how he trained a language model on three years of 4chan's /pol/ board and let it post on the site. He evaluates how good the model was and discusses the ethics and fallout.

Tom Scott describes trying ChatGPT to fix his Gmail labels and being unsettled by how capable it was. He argues that we cannot know where we are on the technology curve, comparing it to Napster.

Lex Fridman's long podcast conversation with Chris Lattner, creator of Swift, LLVM and MLIR and co-founder of Modular. Topics include the Mojo language, the future of programming and AI, and hardware.

Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.

Nicholas Carlini of Anthropic talks at [un]prompted 2026 about how adversaries could misuse current LLMs for attacks. He shows recent models finding zero-day vulnerabilities in heavily tested software, including a SQL injection in Ghost.

Lex Fridman interviews MIT psycholinguist Edward Gibson on human language, syntax, grammar and how large language models relate to them. The conversation runs nearly three hours.

A short demo of an open-source Krita plugin that uses Stable Diffusion to generate and edit images in place. It adds ControlNet scribble and line-art support. You select an area, push a button, and new matching content appears.

OpenAI's 2021 live demo of Codex translating natural language into code, from Hello World to building a game and a Microsoft Word plugin.

Kurzgesagt examines how AI-generated content is flooding the internet, from video and music to news and books. It looks at how this makes verifying what is real harder.

Bloomberg Technology interviews Google engineer Blake Lemoine about the experiments that led him to believe LaMDA was sentient and why he was placed on administrative leave. He also raises concerns about AI ethics at Google.

Brandon Sanderson's keynote on AI-generated art, arguing that the artistic process and the journey matter more than the finished product. He explains why he rebels against generative AI in creative work.

Freya Holmer delivers a long critique of generative AI, arguing it degrades the internet by flooding it with machine-generated junk. The video grew out of her frustration finding reliable information about a technical file format.

Alex O'Connor shows that ChatGPT cannot draw a wine glass filled to the brim, and ties the failure to David Hume's missing shade of blue. It asks what the failure says about how image models learn.

A Welch Labs explainer on neural scaling laws and the compute-efficient frontier, asking why model loss seems to hit a line it cannot cross. It draws an analogy to an ideal gas law for AI.

Full episode of Sassy Justice, a satirical local-news show from the South Park creators starring a deepfaked consumer advocate, Fred Sassy. It uses face-swapped public figures to comment on the dangers of manipulated video.

Jon Gjengset spends about three hours livecoding complex changes in a Rust codebase with agentic AI tools, testing his hypothesis that AI is weak at non-pattern work. He documents where Claude Code helps and where it struggles.