distributed training

The first lecture in Martin Kleppmann's 8-lecture distributed systems series from the University of Cambridge undergraduate course. It introduces the subject, and accompanying lecture notes are available.

An AWS re:Invent 2014 talk on the core distributed systems primitives behind AWS and Amazon.com, such as consensus, key-value stores and transactional logging. It describes how AWS builds large services from a few common building blocks.

Heidi Howard's Reactive Summit 2018 talk on generalizing Paxos by weakening its original requirements. She shows Paxos is one point on a broad spectrum of consensus approaches.

A Ray Summit 2022 keynote fireside chat between OpenAI's Greg Brockman and Anyscale's Robert Nishihara. They discuss how OpenAI's training infrastructure evolved and its use of Ray.

Doug McGuff presents the biochemistry behind high-intensity strength training and the paleo diet, including how the body stores and burns fat. It is a roughly hour-and-a-half full-length lecture.

Will Wilson of FoundationDB explains how the team tested their distributed database using deterministic simulation. By running the whole system in a simulator with injected faults and reproducible seeds, they could find rare concurrency and network bugs.

Paul Chiusano's Strange Loop 2019 talk introduces Unison, an open-source functional language for distributed, elastic systems. It covers content-addressed code identified by hashes of the syntax tree, a codebase manager, and algebraic effects called abilities.

Peter Alvaro's Strange Loop 2015 talk argues that query languages let you state what you mean, and applies that to distributed programming. It covers Datalog, Dedalus and the Bloom lineage for reasoning about distributed systems.

Martin Kleppmann's QCon London 2018 talk on keeping data consistent in distributed systems without a central leader. It surveys CRDT algorithms for syncing devices and collaborative editing over peer-to-peer networks.

An episode of Gwynne Dyer's 1983 documentary series War, about how military basic training turns recruits into soldiers. It covers the psychological side of conditioning.

Footage of FAA altitude chamber hypoxia training for pilots in which a participant passes out from oxygen deprivation. It shows how quickly and subtly cognition fails at altitude.

A short clip from an ACM and Computer History Museum interview in which Leslie Lamport discusses writing his paper on time, clocks and the ordering of events. He explains its link to special relativity.

The opening lecture of MIT 6.824 Distributed Systems (Spring 2020), introducing the course and core challenges of distributed systems. It begins a playlist that goes on to cover Raft and other consensus papers in Go.

Bryan Hunter's GigCityElixir 2023 talk on HCA Healthcare's Waterpark system, built on Elixir/Erlang. It describes how the architecture is distributed and fault-tolerant, with a multi-datacenter cluster and runtime code swapping.

Rick Hartley gives a roughly two-hour live training on proper grounding in PCB design. He explains how energy actually flows in the space between conductors, and how that governs noise, EMI, return paths and signal reflections.

Linus Torvalds gives a Google tech talk on git, the source control system he created two years earlier. He argues for distributed version control over centralized systems such as CVS and Subversion.

John Ousterhout lectures on the Raft consensus algorithm, made for the Raft user study comparing how students learn Raft and Paxos. It covers leader election and log repair.

A 1973 AT&T training film made for new employees at Bell Labs' Holmdel Computing Center, when Unix was in its third version and running at 16 computer centers in the Bell System. It shows how users submitted jobs and used the center's machines.

Yosh trains a reinforcement learning agent to drive Trackmania until he can no longer beat it. The video explains how the training works and the design choices involved.

A QCon London 2020 talk by Tesla engineers on the architecture of Tesla's Virtual Power Plant. It covers aggregating distributed batteries and solar with vertically integrated hardware and software across cloud and edge.

A 1953 US Navy training film on the basic mechanisms in fire control computers: shafts, gears, cams and differentials. It shows how these mechanical parts compute ship and target firing solutions.

Andrej Karpathy's three-hour-plus general-audience deep dive into how ChatGPT-style LLMs are built, covering the full training stack from pretraining to post-training. It also gives mental models for their quirks, like hallucination, and how to use them well.

Rich Hickey's 2013 talk applies language-design thinking to whole systems, covering names, queues, data, failure, and the interfaces between programs. It ends with a call to action for building systems holistically.

Andrej Karpathy gives a one-hour general-audience introduction to large language models, covering what they are, how they are trained, comparisons to operating systems, and security challenges such as jailbreaks.

A 1930s Chevrolet training film explaining the principles and development of the differential gear. It builds the idea step by step from first principles to the finished mechanism.

Leslie Lamport argues that programmers, like architects, should write rigorous specifications before coding, especially for concurrent and distributed systems. The talk introduces the case for writing specs (the TLA+ mindset) as a way of thinking above the code.

Bungie's David Aldridge explains the networking programming behind Halo: Reach in a 2011 GDC session. It covers how gameplay stays in sync online, including forecasting and state reconciliation.

Rick Beato argues that adults cannot develop true perfect pitch, which he ties to critical-period brain plasticity and language acquisition. He says it can only be developed in young children, and discusses the research behind this.

Robert Miles explains reward modelling, in which an AI learns what humans want from their preference comparisons instead of a hand-written reward function. He walks through the 2017 paper on deep reinforcement learning from human preferences.

Jason Scott of Archive Team describes how a loose band of volunteers rushes to copy sites like Geocities, Friendster and Yahoo Video before their owners delete them. The talk covers the tactics they use and the argument that this data is history worth saving.

A roughly 45-minute vintage US military instructional film that gives a comprehensive overview of celestial navigation, covering sextant use and the theory behind lines of position.

Andrej Karpathy walks through how GPT assistants are trained, from tokenization and pretraining to supervised finetuning and RLHF, and gives practical prompting strategies and a view of the tooling ecosystem.

FortNine's tongue-in-cheek guide to why drivers fail to see motorcyclists and how riders can make themselves more visible. It touches on perception psychology, including the role of saccades.

Alex O'Connor shows that ChatGPT cannot draw a wine glass filled to the brim, and ties the failure to David Hume's missing shade of blue. It asks what the failure says about how image models learn.

Rick Beato discusses a theory of why some children develop perfect pitch, based on early exposure to music. He draws on his experience with his own children.

Part one of an American Airlines training video presented by Captain Warren VanderBurgh on automation dependency. It grew out of a program on wake turbulence avoidance, and this restored version has the audio cleaned up.

Jimmy Bogard presents a post-mortem of a real microservices migration that turned into a slow distributed monolith. He covers the modeling, development and production problems and how the team recovered.

An AWS re:Invent 2018 session on how DynamoDB was built as a hyper-scale NoSQL database. It discloses underpinnings of the service and how it is run for over 100,000 customers.

Tech Ingredients shows how to build cheap, high-performance flat-panel speakers using audio exciters on sheet materials such as foam-core board. It explains how distributed-mode speakers work and walks through experiments to find good designs.

Jeff Hawkins presents his view that the neocortex works on principles different from traditional computers. He describes sparse distributed representations and how they are learned, then shows how Numenta applies them to predict from high-velocity data streams.