language models

Rob Miles explains on Computerphile how language models generate plausible text and how transformers improved on recurrent networks, with a look at what comes next.

A short cartoon explaining how language models like ChatGPT work, from autocomplete and conditional probabilities to neural networks, gradient descent and backpropagation. It shows how networks act as universal function approximators.

Jeremy Howard's 90-minute practical tour of language models. It covers how they work, GPT-4's capabilities and limits, using the OpenAI API, code writing, data analysis and fine-tuning.

Andrej Karpathy gives a one-hour general-audience introduction to large language models, covering what they are, how they are trained, comparisons to operating systems, and security challenges such as jailbreaks.

Rob Miles discusses GPT-3 on Computerphile, covering how scaling parameters kept improving results over GPT-2 and how the model nearly learned arithmetic without anyone knowing exactly how.

Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.

Lex Fridman interviews MIT psycholinguist Edward Gibson on human language, syntax, grammar and how large language models relate to them. The conversation runs nearly three hours.

Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.

Guy Steele's 1998 OOPSLA keynote on designing a programming language that its users can grow. He builds the talk using only one-syllable words and terms he has defined, to demonstrate the idea.

Jonathan Blow's first talk on designing a new programming language for games, which became Jai. He explains the problems with C++ and why Go, Rust and D do not fit game development.

Evan Czaplicki, creator of Elm, examines how programming languages are funded. He looks at where compiler engineers' salaries come from, why Go has about 5 engineers while Dart has about 30, and how this limits language design.

Guy Steele's classic 1998 OOPSLA keynote on how a programming language must grow with its user community. He delivers it in a playful style, starting with only monosyllabic words and defining new terms as he goes.

Chris Lonsdale presents a TEDx talk on how adults can learn a language in about six months, centering on immersion, comprehensible input and a practical learning methodology.

A SXSW talk by W Watson that uses Lacanian psychoanalysis, with nods to Zizek, to critique popular programming languages. It argues that people choose and defend languages for emotional and ideological reasons rather than rational ones.

Shriram Krishnamurthi's PWLConf 2019 talk on the formal notion of expressive power of programming languages, based on Felleisen's paper. It explains how to tell when a feature adds power by whether emulating it requires global rewriting.

Stephen Wolfram gives a 13-minute tour of the Wolfram Language, a symbolic, functional programming language with a built-in knowledgebase. He demos querying large datasets, interactivity and deployment.

Robert Sapolsky's Stanford lecture on language, comparing human and animal communication, ape language projects like Koko and Kanzi, and how language is acquired. It is part of his human behavioral biology course.

Brian Kernighan gives a talk at the University of Nottingham on how to succeed in language design without really trying. He draws on his experience with awk and other languages.

Michael Arntzenius's Strange Loop talk on Datafun, a functional query language that brings database-style querying into a general-purpose language. It discusses bridging database theory and programming languages.

A George Carlin stand-up bit from Doin' It Again about soft language and euphemism, showing how softened words hide uncomfortable facts. It touches on topics like shell shock and aging.

A 10-minute clip from a Jonathan Blow Twitch stream in which he argues that game engines do not really need separate scripting languages. He questions the value of embedded interpreters when the main language can do the job.

A 15-minute clip of linguist Stephen Krashen discussing language acquisition, drawn from the 1980s. It presents his input hypothesis and the distinction between acquisition and learning.

Jonathan Blow demos his prototype programming language, covering the base language and compile-time execution, including running a game while compiling. The video is about 70 minutes of demo followed by Q&A.

A lecture from the 1980s by linguist Stephen Krashen on how people acquire language. He explains his comprehensible input hypothesis for second-language acquisition.

Paul Biggar's Google Tech Talk on compiling and optimizing scripting languages, drawn from his work on the phc ahead-of-time PHP compiler. He covers runtime typing, aliasing and code generation challenges, and argues AOT can compete with JIT.

Rob Pike's October 2009 Google Tech Talk introducing Go, a new experimental systems language designed for fast builds, concurrency and garbage collection. It covers the goals, syntax and type system.

A Polish linguist tests whether speakers of Bulgarian, Polish and Croatian can understand Interslavic, a constructed language built from Slavic languages. It shows how well they follow it without prior study.

An animated ranking of the most popular programming languages from 1965 to 2019, built from popularity indexes, GitHub access data and historical surveys.

A sketch from the Norwegian NRK show Uti Var Hage poking fun at the Danish language and how hard it is to understand.

A short excerpt of Rob Pike's Lang.NEXT 2014 talk where he describes Go's target audience as young Google programmers who are not researchers. The language has to be easy for them to understand and adopt.

3Blue1Brown's visual explanation of how transformers and GPT-style language models work, tracing data through tokens, embeddings and attention.

Geoffrey Hinton's 2025 Royal Institution Discourse on what AI has learned from biological intelligence and how it might surpass it. He explains neural networks and argues that human language use resembles what backpropagation-trained networks do.

Andrew Kelley, creator of Zig, introduces the language at the Recurse Center's Localhost series in 2018. He covers its goals of robustness, optimality and clarity, C interop, cross-compilation and explicit allocators.

A 2008 Google Tech Talk in which Slava Pestov introduces Factor, a concatenative language influenced by Forth, Lisp and Smalltalk. It covers the interactive environment, metaprogramming, and the language's modern high-level features and libraries.

Ian Piumarta's 2007 Stanford EE380 lecture on building a dynamic programming environment from the ground up, from the Viewpoints Research Institute's work. It covers the design of a language and system with a recursive design, parsing through code generation, and the COLA architecture.

Scott Wlaschin's NDC Oslo talk on four 1970s languages, SQL, Prolog, ML and Smalltalk. It explains their philosophies and what practical principles they still offer modern developers.

A 2001 panel with Paul Graham, John Maeda, Jonathan Rees and Guy Steele on programming language design, part of the Dynamic Languages Wizards Series.

Chandler Carruth's CppNow 2023 talk describes Carbon as a successor language to C++ rather than a superset, focusing on C++ interop and a path to memory safety in large existing codebases. It explains why Carbon takes a different approach from Cpp2 and Circle.

Carol Nichols' Philly ETE 2019 talk on what makes Rust unique. It covers memory safety, how editions let the language evolve without breaking code, and who relies on Rust in production.

Douglas Crockford's code::dive 2022 talk on what the next programming language should look like. He presents ideas behind his Misty language, including message-passing actors.