distributed training

An AWS re:Invent 2014 talk on the core distributed systems primitives behind AWS and Amazon.com, such as consensus, key-value stores and transactional logging. It describes how AWS builds large services from a few common building blocks.

Heidi Howard's Reactive Summit 2018 talk on generalizing Paxos by weakening its original requirements. She shows Paxos is one point on a broad spectrum of consensus approaches.

Will Wilson of FoundationDB explains how the team tested their distributed database using deterministic simulation. By running the whole system in a simulator with injected faults and reproducible seeds, they could find rare concurrency and network bugs.

Paul Chiusano's Strange Loop 2019 talk introduces Unison, an open-source functional language for distributed, elastic systems. It covers content-addressed code identified by hashes of the syntax tree, a codebase manager, and algebraic effects called abilities.

Peter Alvaro's Strange Loop 2015 talk argues that query languages let you state what you mean, and applies that to distributed programming. It covers Datalog, Dedalus and the Bloom lineage for reasoning about distributed systems.

Martin Kleppmann's QCon London 2018 talk on keeping data consistent in distributed systems without a central leader. It surveys CRDT algorithms for syncing devices and collaborative editing over peer-to-peer networks.

Bryan Hunter's GigCityElixir 2023 talk on HCA Healthcare's Waterpark system, built on Elixir/Erlang. It describes how the architecture is distributed and fault-tolerant, with a multi-datacenter cluster and runtime code swapping.

Linus Torvalds gives a Google tech talk on git, the source control system he created two years earlier. He argues for distributed version control over centralized systems such as CVS and Subversion.

A QCon London 2020 talk by Tesla engineers on the architecture of Tesla's Virtual Power Plant. It covers aggregating distributed batteries and solar with vertically integrated hardware and software across cloud and edge.

Rich Hickey's 2013 talk applies language-design thinking to whole systems, covering names, queues, data, failure, and the interfaces between programs. It ends with a call to action for building systems holistically.

Andrej Karpathy gives a one-hour general-audience introduction to large language models, covering what they are, how they are trained, comparisons to operating systems, and security challenges such as jailbreaks.

Leslie Lamport argues that programmers, like architects, should write rigorous specifications before coding, especially for concurrent and distributed systems. The talk introduces the case for writing specs (the TLA+ mindset) as a way of thinking above the code.

Bungie's David Aldridge explains the networking programming behind Halo: Reach in a 2011 GDC session. It covers how gameplay stays in sync online, including forecasting and state reconciliation.

Jason Scott of Archive Team describes how a loose band of volunteers rushes to copy sites like Geocities, Friendster and Yahoo Video before their owners delete them. The talk covers the tactics they use and the argument that this data is history worth saving.

Andrej Karpathy walks through how GPT assistants are trained, from tokenization and pretraining to supervised finetuning and RLHF, and gives practical prompting strategies and a view of the tooling ecosystem.

Jimmy Bogard presents a post-mortem of a real microservices migration that turned into a slow distributed monolith. He covers the modeling, development and production problems and how the team recovered.

An AWS re:Invent 2018 session on how DynamoDB was built as a hyper-scale NoSQL database. It discloses underpinnings of the service and how it is run for over 100,000 customers.

Jeff Hawkins presents his view that the neocortex works on principles different from traditional computers. He describes sparse distributed representations and how they are learned, then shows how Numenta applies them to predict from high-velocity data streams.

Ted Young of LightStep proposes trace testing, using distributed tracing data to unify testing and observability. The KubeCon-era talk argues for testing against trace data rather than treating tests and monitoring separately.

Martin Kleppmann's GOTO Berlin 2016 talk on how collaborative editors, offline apps, and multi-datacenter databases handle concurrent updates. It covers conflict resolution approaches including operational transformation and CRDTs, with a worked look at text CRDTs.

Zach Tellman's Clojure West talk covers the basics of queueing theory and its practical use in systems, including backpressure, how queues break down under load, and pitfalls of naive core.async use.

Will Wilson explains autonomous testing: using deterministic simulation and fuzzing, building on FoundationDB's approach, to test software continuously. It sets out the ideas behind his later company Antithesis.

Martin Kleppmann surveys current research on CRDTs, the algorithms for merging concurrent edits across offline replicas. He covers the hard parts such as rich text, moves and merge results that are technically correct but poor.

Tesla's full AI Day 2022 presentation, covering the Tesla Bot hardware and software, Full Self-Driving planning and perception, the occupancy network, data labeling, simulation and Dojo training infrastructure.

Martin Kleppmann's Strange Loop talk on database transactions, covering why they were dropped by NoSQL, their reputation for being slow, and newer algorithms that make serializable transactions perform and scale. It touches on consistency, linearizability and the CAP theorem.

Joe Armstrong's Strange Loop 2013 talk on building large scalable systems from small isolated parallel components. It covers detecting and correcting errors at run time, changing code while it runs, and how Erlang embodies these principles.

An AWS re:Invent 2018 talk on how AWS limits the blast radius of failures. It covers design techniques such as cell-based architecture and shuffle sharding.

David Mazieres presents the Stellar Consensus Protocol at Google, introducing federated Byzantine agreement. The model lets participants choose whom to trust rather than assuming fixed membership.

Martin Kleppmann's Local-First Conf talk reviewing where the local-first idea came from, where it stands five years after the essay, and what the community should work toward. He argues for generic sync infrastructure.

A QCon talk from Netflix on running a large microservices architecture and building resilience into it. It touches on tooling such as Hystrix and circuit breakers.

Andrej Karpathy, then Tesla's head of AI, presents at ScaledML 2020 on how Tesla builds full self-driving using vision-based neural nets and fleet data collection. He covers training custom detectors and neural nets gradually replacing traditional code.

Valve's John McDonald presents at GDC 2018 on how Valve uses deep learning (VACnet) to detect cheaters in Counter-Strike: Global Offensive. He covers training on human-reviewed replays and how model output is confirmed by manual review.

Joran Dirk Greef's Systems Distributed '23 talk on TigerStyle, the engineering methodology behind TigerBeetle. It covers designing safer systems in less time through up-front design, assertions, and deterministic simulation.

Cardiologist James O'Keefe argues in a TEDx talk that moderate running confers the best longevity benefits, while extreme endurance training can damage the heart. He draws on data and his own experience as a lifelong runner.

Simon Peyton Jones explains nested data parallelism and Data Parallel Haskell, an approach to exploiting multicore and distributed hardware while keeping a simple single-control-flow programming model.

David Epstein argues in this TED talk that improving athletic records owe much to technology, training and the selection of specialized body types, not just humans getting inherently better. He gives counter-intuitive examples across sports.

Mario Zechner explains why he built pi, a minimal coding agent, out of frustration with existing agent harnesses. He also discusses how AI agents are straining open source projects and argues for slowing down.

Admiral William McRaven's 2014 University of Texas commencement address, built around lessons from Navy SEAL training. It famously opens with the case for making your bed every morning.

Bill Dally of NVIDIA gives a Hot Chips 2023 keynote on the challenges of building hardware for deep learning. He traces how GPUs have been adapted for machine learning, especially training.

Moxie Marlinspike's 36C3 talk examines why distributed and decentralized technology is hard to build into consumer products, from the perspective of developing Signal. It argues that ecosystems move slowly and federation causes problems.