optimization

Kaze Emanuar explains how he rewrote and optimized the Super Mario 64 source code so it runs far faster on real N64 hardware. He covers performance bottlenecks and fixes in detail for his ROM hacks.

Matthias Wandel tests old and new code on a range of PCs, including a new mini PC, to measure how much faster modern hardware really is. He also shows optimization techniques such as branchless code, with benchmark programs on GitHub.

Kaze Emanuar searches for the best sine approximation for the Nintendo 64, comparing the lookup tables in Mario 64 and Ocarina of Time with polynomial alternatives. It walks through numerics and 3rd, 4th and 5th order approximations.

Matt Parker follows the community effort to speed up his slow Wordle-solving code, which ended up running hundreds of millions of times faster. He walks through the techniques contributors used.

Wes Bos breaks down the techniques behind the McMaster-Carr website's speed: server-rendered HTML, prefetching, CDN and service worker caching, preloading, critical CSS and fixed-size images. He also notes its old-school jQuery and YUI JavaScript.

A walkthrough of the Quake III fast inverse square root, explaining where the constant 0x5f3759df comes from. It covers IEEE 754 floats, the bit hack that approximates a logarithm, and a Newton's method refinement.

3Blue1Brown's visual explanation of how transformers and GPT-style language models work, tracing data through tokens, embeddings and attention.

Veritasium explains why analog computing may suit AI workloads, covering old analog computers, The Analog Thing and Mythic's analog chips that use flash memory for neural network inference. It argues digital hardware is hitting limits.

3Blue1Brown's visual walkthrough of the attention mechanism inside transformers, covering self-attention, multiple heads and cross-attention. It is chapter 6 of his deep learning series.

A short cartoon explaining how language models like ChatGPT work, from autocomplete and conditional probabilities to neural networks, gradient descent and backpropagation. It shows how networks act as universal function approximators.

A Veritasium video on AlphaFold, DeepMind's protein structure prediction system, and the protein design work it enabled. It features interviews with John Jumper and with David Baker's protein design group and explains why predicting protein folding matters.

A Welch Labs explainer on neural scaling laws and the compute-efficient frontier, asking why model loss seems to hit a line it cannot cross. It draws an analogy to an ideal gas law for AI.

Yosh trains a reinforcement learning agent to drive Trackmania until he can no longer beat it. The video explains how the training works and the design choices involved.

Tom 7 (suckerpinch) delivers a long, comedic digression on gradient descent and floating-point imprecision. He shows how rounding behavior can supply the nonlinearity needed to build a neural network.

A guest video by Welch Labs on 3Blue1Brown explaining how text-to-image and video models work. It covers CLIP, shared embedding spaces, diffusion models (DDPM, DDIM), conditioning, guidance and negative prompts.

Robert Miles covers a paper where researchers ran real versions of his earlier mesa-optimizer thought experiments. They show inner misalignment, where an AI learns the wrong goal in training and misbehaves when deployed.

Rob Miles discusses GPT-3 on Computerphile, covering how scaling parameters kept improving results over GPT-2 and how the model nearly learned arithmetic without anyone knowing exactly how.