karpathy

Lex Fridman's long conversation with Andrej Karpathy on Tesla's self-driving approach, Optimus, neural networks, transformers, AGI and more. It also touches on his OpenAI work and working with Elon Musk.

Dwarkesh Patel interviews Andrej Karpathy on why reinforcement learning is flawed, why AGI may blend into steady GDP growth, why self-driving took so long, and the future of education. The conversation runs about two and a half hours.

Andrej Karpathy, then Tesla's head of AI, presents at ScaledML 2020 on how Tesla builds full self-driving using vision-based neural nets and fleet data collection. He covers training custom detectors and neural nets gradually replacing traditional code.

Andrej Karpathy's Databricks talk on Software 2.0, the shift from human-written code to code produced by neural network training. It covers how developer work moves toward curating datasets, the advantages and challenges, and opportunities for new tooling.

Andrej Karpathy's keynote at AI Startup School laying out software 1.0, 2.0 and 3.0, with LLMs seen as utilities, fabs and operating systems. It covers LLM quirks, partial-autonomy apps and keeping a human in the generate-and-verify loop.

A No Priors podcast conversation with Andrej Karpathy covering self-driving (Tesla versus Waymo), the Optimus robot, bottlenecks in AI development, and his education project Eureka Labs.

Andrej Karpathy, then Tesla's head of AI, gives a CVPR 2021 workshop keynote on how Autopilot's neural networks work. A large part covers the move to a vision-only stack, including why radar was dropped and how vision produces depth estimates.

Andrej Karpathy's CVPR 2021 presentation on Tesla's Autopilot, covering its vision-based approach, sensor choices and the scale of Tesla's deep learning compute cluster. This upload is a lower-quality copy of the talk.

Andrej Karpathy's first Neural Networks: Zero to Hero lecture, which builds micrograd, a tiny autograd engine, from scratch in Python. It explains backpropagation and neural network training step by step, assuming only basic Python and some high-school calculus.

Andrej Karpathy's three-hour-plus general-audience deep dive into how ChatGPT-style LLMs are built, covering the full training stack from pretraining to post-training. It also gives mental models for their quirks, like hallucination, and how to use them well.

Andrej Karpathy gives a one-hour general-audience introduction to large language models, covering what they are, how they are trained, comparisons to operating systems, and security challenges such as jailbreaks.

Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.

Andrej Karpathy builds a GPT from scratch in code, following Attention Is All You Need and GPT-2/3, in about two hours using PyTorch. He connects it to ChatGPT and assumes familiarity with his earlier makemore videos.

Andrej Karpathy walks through how GPT assistants are trained, from tokenization and pretraining to supervised finetuning and RLHF, and gives practical prompting strategies and a view of the tooling ecosystem.

Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to tokenization, and covers Unicode byte encodings along the way.

Tesla's 2021 AI Day presentation covering vision, planning and control, auto-labeling, simulation, the Dojo training hardware and the Tesla Bot announcement, followed by a Q&A.

The first lecture of Stanford's CS231n Winter 2016 course on convolutional neural networks for visual recognition, introducing the course and the historical context of computer vision.

Tesla's roughly four-hour Autonomy Day investor event covers its in-house self-driving chip, vision-only approach, fleet data engine and robotaxi plans, with talks by Andrej Karpathy and others.

3Blue1Brown's first chapter on deep learning explains what neurons and layers are and the math underlying a basic neural network, using handwritten digit recognition with visual animations.

Feature-length documentary following DeepMind's AlphaGo program as it prepares for and plays the 2016 match against Go champion Lee Sedol. It covers the team, the games, and the emotional reaction of the Go world to an AI beating a top human.

3Blue1Brown's visual explanation of how transformers and GPT-style language models work, tracing data through tokens, embeddings and attention.

James Mickens's humorous USENIX Security keynote argues that improving security and trusting opaque technology like machine learning is far harder than the industry admits.

The first lecture of Andrew Ng's Stanford CS229 Machine Learning course, giving an overview of the course. It introduces supervised and unsupervised learning, learning theory and reinforcement learning.

SethBling shows MarI/O, a program using neural networks and the NEAT genetic algorithm to learn to play Super Mario World. The description links the source code, the paper and background reading.

Tom7 explains and demos software that learns to play NES games by finding score patterns in memory and then searching for inputs that raise it. It is the video companion to his SIGBOVIK paper.

The first lecture of David Silver's reinforcement learning course, introducing the field and its basic concepts. The course follows the Sutton and Barto textbook.

Jeremy Howard's 90-minute practical tour of language models. It covers how they work, GPT-4's capabilities and limits, using the OpenAI API, code writing, data analysis and fine-tuning.

Andrew Ng presents unsupervised feature learning and deep learning, showing how algorithms can learn representations from unlabeled data instead of hand-designed features. He gives examples in vision and audio, including visualizations of features learned by successive layers.

Lex Fridman has a three-hour conversation with Joscha Bach about artificial consciousness, the workings of the mind, intelligence and the possibly simulated nature of reality.

Martin Gorner's Devoxx talk introduces deep learning with TensorFlow, walking through neural networks for MNIST, how to choose an architecture and how to diagnose networks that fail to train.

Valve's John McDonald presents at GDC 2018 on how Valve uses deep learning (VACnet) to detect cheaters in Counter-Strike: Global Offensive. He covers training on human-reviewed replays and how model output is confirmed by manual review.

Geoffrey Hinton's Google Tech Talk linking machine learning to biology, explaining dropout as a form of model averaging and why sexual reproduction and stochastic spiking neurons may prevent overfitting.

A long Machine Learning Street Talk interview with Kenneth Stanley, creator of NEAT and novelty search, about his book on why greatness cannot be planned. They discuss open-endedness, the tyranny of objectives, serendipity and the evolution of ideas.

The first lecture of Caltech's CS 156 Machine Learning course by Yaser Abu-Mostafa, introducing the learning problem and supervised, unsupervised and reinforcement learning. It is one of 18 lectures in the Learning from Data course.

Veritasium explains why analog computing may suit AI workloads, covering old analog computers, The Analog Thing and Mythic's analog chips that use flash memory for neural network inference. It argues digital hardware is hitting limits.

Tom Scott-style absurdist research by tom7 for SIGBOVIK 2021, using machine learning to discover letterforms that are 'uppercaser' than uppercase A and lowercaser than lowercase. It is a comedic, technically inventive deep dive with fonts and a paper.

The first episode of Google's Machine Learning Recipes series. It introduces machine learning and builds a first supervised-learning classifier in about six lines of Python using scikit-learn.

3Blue1Brown's visual walkthrough of the attention mechanism inside transformers, covering self-attention, multiple heads and cross-attention. It is chapter 6 of his deep learning series.

A GDC 2018 talk by Valve's John McDonald on VACNet, a deep learning system that analyzes CS:GO gameplay to detect cheaters. It covers how the model is trained on human-reviewed Overwatch cases and how its output feeds human moderation.

Leo Dirac explains how transformer models have replaced LSTMs in NLP. He gives background from bag of words through RNNs and LSTMs, then a technical deep dive into multi-headed self-attention and positional encoding, with sample code.