backpropagation

Conal Elliott presents his paper on automatic differentiation, deriving a simple, generalized AD algorithm from a natural specification. Varying the representation of derivatives yields forward and reverse mode algorithms that avoid the complexity and statefulness of backpropagation.

James Mickens's humorous USENIX Security keynote argues that improving security and trusting opaque technology like machine learning is far harder than the industry admits.

Andrew Ng presents unsupervised feature learning and deep learning, showing how algorithms can learn representations from unlabeled data instead of hand-designed features. He gives examples in vision and audio, including visualizations of features learned by successive layers.

Andrej Karpathy, then Tesla's head of AI, presents at ScaledML 2020 on how Tesla builds full self-driving using vision-based neural nets and fleet data collection. He covers training custom detectors and neural nets gradually replacing traditional code.

Andrej Karpathy gives a one-hour general-audience introduction to large language models, covering what they are, how they are trained, comparisons to operating systems, and security challenges such as jailbreaks.

Valve's John McDonald presents at GDC 2018 on how Valve uses deep learning (VACnet) to detect cheaters in Counter-Strike: Global Offensive. He covers training on human-reviewed replays and how model output is confirmed by manual review.

Geoffrey Hinton's Google Tech Talk linking machine learning to biology, explaining dropout as a form of model averaging and why sexual reproduction and stochastic spiking neurons may prevent overfitting.

A GDC 2018 talk by Valve's John McDonald on VACNet, a deep learning system that analyzes CS:GO gameplay to detect cheaters. It covers how the model is trained on human-reviewed Overwatch cases and how its output feeds human moderation.

Ali Rahimi's NIPS 2017 Test-of-Time award talk, in which he argues that modern machine learning has become alchemy, working without being rigorously understood. He calls for more rigor and understanding in the field.

Max Tegmark's MIT CBMM talk on why deep learning works so well. He argues that the laws of physics and the symmetries in physical data make it compressible into small neural networks, tying physics to network structure.

George Hotz of Comma AI gives a 2017 conference talk on lessons from building a self-driving system. He argues about how driving should be defined and why the common approach to autonomous vehicles is flawed.

Andrej Karpathy walks through how GPT assistants are trained, from tokenization and pretraining to supervised finetuning and RLHF, and gives practical prompting strategies and a view of the tooling ecosystem.

Andrej Karpathy, then Tesla's head of AI, gives a CVPR 2021 workshop keynote on how Autopilot's neural networks work. A large part covers the move to a vision-only stack, including why radar was dropped and how vision produces depth estimates.

Peter Norvig's 2007 Google talk on theorizing from data, covering what huge datasets can do for language understanding, translation, information extraction and inference. It argues that more data often beats cleverer models.

Simon Knowles, co-founder and CTO of Graphcore, talks about "intelligence processors", chips designed specifically for machine learning. The talk was given at the RAAIS 2017 summit and covers Graphcore's architecture.

Tesla's roughly four-hour Autonomy Day investor event covers its in-house self-driving chip, vision-only approach, fleet data engine and robotaxi plans, with talks by Andrej Karpathy and others.

Joscha Bach's 31c3 talk argues that the mind should be understood as an information-processing system and that even the physical universe can be seen as computational. It tours the philosophy underlying artificial intelligence and cognitive science.

Tesla's full AI Day 2022 presentation, covering the Tesla Bot hardware and software, Full Self-Driving planning and perception, the occupancy network, data labeling, simulation and Dojo training infrastructure.

Bill Dally of NVIDIA gives a Hot Chips 2023 keynote on the challenges of building hardware for deep learning. He traces how GPUs have been adapted for machine learning, especially training.

Andrew Ng speaks at the Stanford MSx Future Forum about how AI is transforming industry after industry, comparing its impact to electricity. The talk also touches on evil-AI worries, jobs and basic income.

Peter Norvig, then Director of Research at Google, talks at Startup School 2008 about leveraging large amounts of data and about technical challenges in Google projects such as image search. A central idea is that data can beat hand-written programming.

Geoff Hinton's 2010 Google Tech Talk on deep belief networks, learning layers of representation one at a time from unlabeled data, and extending them with multiplicative interactions.

Chris Stucchio presents the ethical principles behind algorithmic decision making and shows with examples from lending, criminology and education that they cannot all be satisfied at once. The talk frames AI fairness as a set of quantitative tradeoffs.

Jose Valim introduces Nx, the project bringing numerical computing to Elixir, covering multi-dimensional tensors, automatic differentiation and JIT compilation to CPU and GPU.

UC Berkeley EECS colloquium by OpenAI's John Schulman on reinforcement learning from human feedback. He covers why RL helps with truthfulness, hallucination, and the limits of behavior cloning.

Jeff Hawkins presents his view that the neocortex works on principles different from traditional computers. He describes sparse distributed representations and how they are learned, then shows how Numenta applies them to predict from high-velocity data streams.

Jim Keller's talk at the 2020 Scaled Machine Learning conference on Moore's Law in the age of AI chips. He compares scalar, vector and GPU-style parallel approaches to computation.

A Google Tech Talk by Laurens van der Maaten, co-creator of t-SNE, on visualizing high-dimensional data by mapping it to two dimensions. It explains how the technique works and how it compares with other visualization approaches.

Dragan Djuric presents Clojure libraries for GPU and numerical computing, including ClojureCL, ClojureCUDA and Neanderthal. He shows how a dynamic REPL workflow can be combined with low-level hardware acceleration for machine learning and data science.

Demis Hassabis of DeepMind speaks at Google Zeitgeist, saying human-level AI is decades away but the debate should start now. He covers DeepMind's focus on game-playing agents.

Chris Lattner and Brennan Saeta present Swift for TensorFlow at the TensorFlow Dev Summit 2019, covering language-integrated automatic differentiation and tooling for machine learning in Swift.

A 92-minute RMIT University seminar in which Ken Stanley argues that setting an objective can block achieving it. He presents objective-less, novelty-driven search as an alternative approach to innovation and discovery, drawing on evolutionary computation.

A Google talk from the Chisel community in which engineers recount designing the Edge TPU with the Chisel hardware construction language. It covers the productivity gains and the problems they hit.

Andrej Karpathy's Databricks talk on Software 2.0, the shift from human-written code to code produced by neural network training. It covers how developer work moves toward curating datasets, the advantages and challenges, and opportunities for new tooling.

Ilya Sutskever gives a Simons Institute talk on why neural networks generalize, framing learning as finding short descriptions, like compression. It was part of the Large Language Models and Transformers program.

Jim Keller, CTO of Tenstorrent, talks at Samsung Foundry's SAFE 2021 about building AI silicon through modular, provable design. The talk also touches on Tenstorrent's RISC-V Ascalon core.

A 2007 Google Tech Talk introducing Polyworld, an artificial life simulator used to evolve artificial intelligence by designing worlds rather than minds. It presents results from ongoing attempts to evolve intelligent behavior.

Tesla's 2021 AI Day presentation covering vision, planning and control, auto-labeling, simulation, the Dojo training hardware and the Tesla Bot announcement, followed by a Q&A.

Ali Rahimi accepts the NIPS 2017 Test of Time Award and argues that machine learning has become alchemy, with practice running ahead of understanding.