learning

Machine Learning Street Talk interviews Paul Lessard about Categorical Deep Learning, which uses category theory and monads to make neural network architectures more interpretable and composable. They also discuss neural networks' limits in reasoning and unbounded computation.

Valve's John McDonald presents at GDC 2018 on how Valve uses deep learning (VACnet) to detect cheaters in Counter-Strike: Global Offensive. He covers training on human-reviewed replays and how model output is confirmed by manual review.

Dwarkesh Patel interviews Andy Matuschak on how he learns, why memorization matters for understanding, and why most learning tools fail. The conversation also covers handling confusion and the role of tutoring.

Veritasium's Derek Muller gives a Perimeter Institute livestream on how AI might change teaching and learning. He draws on past technology shifts and cognitive science to argue that learning requires effortful repetition.

Lesson 1 of fast.ai's Practical Deep Learning for Coders, taught by Jeremy Howard. It takes coders from running code in a Jupyter notebook straight into training a real image model.

Tom Scott-style absurdist research by tom7 for SIGBOVIK 2021, using machine learning to discover letterforms that are 'uppercaser' than uppercase A and lowercaser than lowercase. It is a comedic, technically inventive deep dive with fonts and a paper.

3Blue1Brown's visual walkthrough of the attention mechanism inside transformers, covering self-attention, multiple heads and cross-attention. It is chapter 6 of his deep learning series.

Bill Dally of NVIDIA gives a Hot Chips 2023 keynote on the challenges of building hardware for deep learning. He traces how GPUs have been adapted for machine learning, especially training.

Bill Dally of NVIDIA gives a University of Washington lecture on trends in deep learning hardware. It covers numeric formats, specialized instructions, energy per operation and co-design of hardware and software.

3Blue1Brown's visual explanation of how transformers and GPT-style language models work, tracing data through tokens, embeddings and attention.

UC Berkeley EECS colloquium by OpenAI's John Schulman on reinforcement learning from human feedback. He covers why RL helps with truthfulness, hallucination, and the limits of behavior cloning.

Lex Fridman interviews George Hotz, founder of Comma.ai, in a three-hour conversation. Topics include self-driving, Tesla and Waymo, crypto, GPT-3, immortality and book recommendations.

The first lesson of MangoJelly's FreeCAD 0.20 beginner course, covering setup, preferences and navigation. The series teaches fundamentals through complete workflows such as sketching, Part Design and STL export.

Veritasium debunks the idea of learning styles, such as being a visual learner, with an interview with Daniel Willingham and a memory experiment. It argues that a multimodal approach helps everyone.

Yosh trains a reinforcement learning agent to drive Trackmania until he can no longer beat it. The video explains how the training works and the design choices involved.

Tom Scott, who never learned to ride a bike, gets lessons from Mike Boyd, a YouTuber known for learning skills quickly. The video follows an adult's attempt to learn cycling in a short time.

Toyota Research Institute presents an approach for teaching robots new dexterous behaviors from human demonstration. It explains why this capability matters and what advances made it possible.

Veritasium examines which experts have real expertise, drawing on chess research and work by Klein and Kahneman. It explains that expertise comes from pattern recognition built through deliberate practice in valid, feedback-rich environments.

A 2023 University of Toronto conversation between Geoffrey Hinton and Fei-Fei Li on responsible AI development. They look back at the history of deep learning and ImageNet and discuss the ethics and risks of AI.

Jose Valim introduces Nx, the project bringing numerical computing to Elixir, covering multi-dimensional tensors, automatic differentiation and JIT compilation to CPU and GPU.

Jonathan Blow gives a roughly 90-minute talk, followed by Q&A, on how video games could actually improve education, in contrast to what is currently being tried.

Dwarkesh Patel interviews Richard Sutton, reinforcement learning pioneer, Turing Award winner and author of The Bitter Lesson. Sutton argues LLMs are a dead end because they cannot learn on the job, and that continual learning from experience will require a new architecture.

A short Unitree promo for its G1 humanoid robot, priced from $16K, showing agile motion, dexterous hands and imitation and reinforcement learning.

Dwarkesh Patel interviews Andrej Karpathy on why reinforcement learning is flawed, why AGI may blend into steady GDP growth, why self-driving took so long, and the future of education. The conversation runs about two and a half hours.

The video argues that machine-learning-based aimbots, which read the screen and drive the controls externally, could make cheating in FPS games undetectable. It explains how conventional aimbots work, how the new approach differs, and speculates on possible detection.

A long Machine Learning Street Talk interview with Kenneth Stanley, creator of NEAT and novelty search, about his book on why greatness cannot be planned. They discuss open-endedness, the tyranny of objectives, serendipity and the evolution of ideas.

Feature-length documentary following DeepMind's AlphaGo program as it prepares for and plays the 2016 match against Go champion Lee Sedol. It covers the team, the games, and the emotional reaction of the Go world to an AI beating a top human.

Andrej Karpathy, then Tesla's head of AI, presents at ScaledML 2020 on how Tesla builds full self-driving using vision-based neural nets and fleet data collection. He covers training custom detectors and neural nets gradually replacing traditional code.

A Veritasium video on AlphaFold, DeepMind's protein structure prediction system, and the protein design work it enabled. It features interviews with John Jumper and with David Baker's protein design group and explains why predicting protein folding matters.

A Welch Labs explainer on neural scaling laws and the compute-efficient frontier, asking why model loss seems to hit a line it cannot cross. It draws an analogy to an ideal gas law for AI.

A 42-minute CBS interview with Geoffrey Hinton at the Vector Institute in 2023, covering the history of neural networks and the present and future of AI. He discusses AGI timelines and whether language models truly understand.

Physicist and YouTuber Angela Collier argues in this hour-long video that AI as commonly described does not really exist. She covers the limits of the tools, biased datasets, and why they should not be used to make decisions.

Robert Miles covers a paper where researchers ran real versions of his earlier mesa-optimizer thought experiments. They show inner misalignment, where an AI learns the wrong goal in training and misbehaves when deployed.

Ilya Sutskever gives a Simons Institute talk on why neural networks generalize, framing learning as finding short descriptions, like compression. It was part of the Large Language Models and Transformers program.

Andrej Karpathy builds a GPT from scratch in code, following Attention Is All You Need and GPT-2/3, in about two hours using PyTorch. He connects it to ChatGPT and assumes familiarity with his earlier makemore videos.

Jim Keller's talk at the 2020 Scaled Machine Learning conference on Moore's Law in the age of AI chips. He compares scalar, vector and GPU-style parallel approaches to computation.

A five-minute C-SPAN clip of Dr. Jared Cooney Horvath testifying to a US Senate committee that screen time, including in schools, harms children's learning and cognitive development.

Andrej Karpathy's first Neural Networks: Zero to Hero lecture, which builds micrograd, a tiny autograd engine, from scratch in Python. It explains backpropagation and neural network training step by step, assuming only basic Python and some high-school calculus.

Andrej Karpathy's three-hour-plus general-audience deep dive into how ChatGPT-style LLMs are built, covering the full training stack from pretraining to post-training. It also gives mental models for their quirks, like hallucination, and how to use them well.

Jeremy Howard's 90-minute practical tour of language models. It covers how they work, GPT-4's capabilities and limits, using the OpenAI API, code writing, data analysis and fine-tuning.