optimization

Kaze Emanuar searches for the best sine approximation for the Nintendo 64, comparing the lookup tables in Mario 64 and Ocarina of Time with polynomial alternatives. It walks through numerics and 3rd, 4th and 5th order approximations.

Matt Parker follows the community effort to speed up his slow Wordle-solving code, which ended up running hundreds of millions of times faster. He walks through the techniques contributors used.

A walkthrough of the Quake III fast inverse square root, explaining where the constant 0x5f3759df comes from. It covers IEEE 754 floats, the bit hack that approximates a logarithm, and a Newton's method refinement.

3Blue1Brown's visual explanation of how transformers and GPT-style language models work, tracing data through tokens, embeddings and attention.

Veritasium explains why analog computing may suit AI workloads, covering old analog computers, The Analog Thing and Mythic's analog chips that use flash memory for neural network inference. It argues digital hardware is hitting limits.

3Blue1Brown's visual walkthrough of the attention mechanism inside transformers, covering self-attention, multiple heads and cross-attention. It is chapter 6 of his deep learning series.

Yannic Kilcher walks through the Attention Is All You Need paper, which introduced the Transformer, a sequence model based solely on attention without recurrence or convolutions. He reviews the architecture and the paper's translation results.

3Blue1Brown's second neural network chapter explains cost functions and gradient descent, showing how a network learns by minimizing error during training.

Rob Miles explains on Computerphile how language models generate plausible text and how transformers improved on recurrent networks, with a look at what comes next.

A Veritasium video on AlphaFold, DeepMind's protein structure prediction system, and the protein design work it enabled. It features interviews with John Jumper and with David Baker's protein design group and explains why predicting protein folding matters.

A Welch Labs explainer on neural scaling laws and the compute-efficient frontier, asking why model loss seems to hit a line it cannot cross. It draws an analogy to an ideal gas law for AI.

Yosh trains a reinforcement learning agent to drive Trackmania until he can no longer beat it. The video explains how the training works and the design choices involved.

Tom 7 (suckerpinch) delivers a long, comedic digression on gradient descent and floating-point imprecision. He shows how rounding behavior can supply the nonlinearity needed to build a neural network.

A guest video by Welch Labs on 3Blue1Brown explaining how text-to-image and video models work. It covers CLIP, shared embedding spaces, diffusion models (DDPM, DDIM), conditioning, guidance and negative prompts.

Rob Miles discusses GPT-3 on Computerphile, covering how scaling parameters kept improving results over GPT-2 and how the model nearly learned arithmetic without anyone knowing exactly how.

3Blue1Brown visualizes high-dimensional spheres to explain a surprising puzzle about how a sphere fits in a box as dimensions grow. It builds geometric intuition for counterintuitive properties of high-dimensional space.