scale

James Hamilton, AWS VP and distinguished engineer, gives an insider view of the innovations behind the AWS cloud at re:Invent 2014. He covers custom hardware, networking, regions and availability zones, and infrastructure at scale.

John Graham-Cumming of Cloudflare talks at dotScale 2015 about handling roughly 10 trillion log lines a month. The talk touches on Cloudflare's hardware, its logging stack and its use of Lua in Nginx.

A PyData London 2018 talk on how JPMorgan scaled its Athena Python platform over ten years to about 35 million lines of code and thousands of developers. It covers infrastructure, tooling and culture for analytics and visualization at that scale.

An AWS re:Invent 2018 session on how DynamoDB was built as a hyper-scale NoSQL database. It discloses underpinnings of the service and how it is run for over 100,000 customers.

Hyrum Wright shares lessons from five years of mass code migrations in Google's huge C++ codebase. He covers why to migrate, how hard it really is, and how Google updates core libraries and their users continually.

Dropbox engineer Kevin Modzelewski gives a Stanford colloquium on Dropbox's history and the technical problems it faced scaling, including detailed database schema decisions.

Joe Armstrong's Strange Loop 2013 talk on building large scalable systems from small isolated parallel components. It covers detecting and correcting errors at run time, changing code while it runs, and how Erlang embodies these principles.

Google engineers describe Google's internal build system, a unified, language-agnostic system used across the company. It was presented as a 2012 NYC tech talk.

SpaceX engineers present at NVIDIA's GPU Technology Conference on a multi-physics code that simulates the full-scale Raptor methane engine. It covers GPU-based turbulent combustion modeling with adaptive grids and wavelet compression.

Bryan Cantrill's Scale By The Bay 2019 talk on software after Moore's Law. He covers the end of easy scaling and the move toward specialized, emerging architectures.

A Foresight Institute workshop video with two talks: Shawn Douglas on nanoscale instruments for visualizing small proteins, and Bret Victor on Dynamicland. Victor's part shows biomolecular design built in his Realtalk system with physical objects.

Psychologist Robert Epstein presents a Stanford EE380 colloquium on how internet algorithms such as search rankings can manipulate opinions, purchases and votes at massive scale. He also describes a new monitoring system for tracking such influence.

A dotScale 2014 talk by Robert Kennedy, a former healthcare.gov SRE who later worked at Google, about the engineering effort to rescue the troubled site. It explains the technical and organizational problems behind the failed launch.

An F8 2015 talk touring the tool-chain Facebook uses to let thousands of engineers build mobile apps, including its move to Mercurial, an Atom-based IDE and continuous integration.

Anthony Marcar describes lessons from three years of using Clojure at WalmartLabs, including scaling a team and a large multi-project codebase in an enterprise setting.

Bryan Cantrill argues that programming languages and platforms reflect core values, and compares languages through that lens, with Rust as a main example. He draws on his experience with Solaris, DTrace and ZFS.

Luis von Ahn, creator of CAPTCHA and reCAPTCHA, explains how small contributions from many people online can digitize books. He then introduces Duolingo as a way to translate the web while people learn languages.

Matt Ranney of Uber talks at GOTO Chicago 2016 about what Uber learned from growing past 1,000 production microservices. He covers regrets, sprawl and organizational effects.

Rick Reed of WhatsApp speaks at MeetBSD about running a messaging service on FreeBSD and Erlang at huge scale. He covers the stack, tuning and the challenges of millions of connections per host.

A PyCon 2012 talk by early YouTube engineer Mike Solomon on how YouTube scaled its architecture, which is mostly Python.

A Google Tech Talk by Raymond Blum on how Google backs up massive amounts of data such as Gmail and Picasa. It covers scaling challenges, failure types, and approaches that did not work, including tape.

Rick Reed of WhatsApp explains at Erlang Factory 2014 how the service scaled to billions of messages with a very small engineering team. He covers the FreeBSD and Erlang stack, Mnesia usage, server counts and bottleneck work.

A 2007 Google Tech Talk in which Cliff Click of Azul Systems presents a lock-free concurrent hash table that scales to 768 CPUs. He explains why it is correct by reasoning in a different way than usual happens-before and memory-order arguments.

James Hamilton's Tuesday Night Live keynote at AWS re:Invent 2016, touring AWS infrastructure at scale including networking, power and custom silicon. It also features NASA JPL's Tom Soderstrom.

Martin Kleppmann's Strange Loop talk on database transactions, covering why they were dropped by NoSQL, their reputation for being slow, and newer algorithms that make serializable transactions perform and scale. It touches on consistency, linearizability and the CAP theorem.

Cuong Do of YouTube describes how a handful of engineers scaled the site, covering hardware, software and database scalability challenges during its rapid growth. The talk was given at the 2007 Seattle Conference on Scalability.

Google's Amin Vahdat gives the ONS 2015 keynote on Google's datacenter networking, including its custom hardware, software-defined networking and very high bisection bandwidth.

A QCon talk from Netflix on running a large microservices architecture and building resilience into it. It touches on tooling such as Hystrix and circuit breakers.

Bryan Cantrill of Oxide Computer Company gives a Stanford EE380 talk on rethinking the server. He argues that hyperscaler-style rack-scale machines with open firmware, RISC-V and Rust should be available to everyone, not just the giants.

Sasa Juric explains what makes Erlang and Elixir suited to server-side systems, focusing on the BEAM concurrency model rather than syntax or ecosystem. He demonstrates processes, supervision, and fault tolerance.

Will Wilson of FoundationDB explains how the team tested their distributed database using deterministic simulation. By running the whole system in a simulator with injected faults and reproducible seeds, they could find rare concurrency and network bugs.

Martin Kleppmann argues that databases are global shared mutable state and proposes treating them as an ever-growing log of immutable facts. Derived data such as caches, indexes and views is then built by stream processors like Apache Samza. The talk draws on Kafka and LinkedIn's experience with commit logs.

Rich Hickey's 2013 talk applies language-design thinking to whole systems, covering names, queues, data, failure, and the interfaces between programs. It ends with a call to action for building systems holistically.

Rob Pike's Google I/O 2012 talk shows how goroutines and channels solve tricky concurrency problems with simple Go code. It features a playful goroutine demo with gophers passing messages and a search engine example.

Martin Kleppmann's GOTO Berlin 2016 talk on how collaborative editors, offline apps, and multi-datacenter databases handle concurrent updates. It covers conflict resolution approaches including operational transformation and CRDTs, with a worked look at text CRDTs.

Paul Chiusano's Strange Loop 2019 talk introduces Unison, an open-source functional language for distributed, elastic systems. It covers content-addressed code identified by hashes of the syntax tree, a codebase manager, and algebraic effects called abilities.

Peter Alvaro's Strange Loop 2015 talk argues that query languages let you state what you mean, and applies that to distributed programming. It covers Datalog, Dedalus and the Bloom lineage for reasoning about distributed systems.

Daniel Azuma's ElixirConf 2018 talk on whether Docker and OTP conflict. He shows they can coexist and uses Horde and DeltaCrdt, with a live tank-game demo, to keep state alive across container kills.

Bryan Hunter's GigCityElixir 2023 talk on HCA Healthcare's Waterpark system, built on Elixir/Erlang. It describes how the architecture is distributed and fault-tolerant, with a multi-datacenter cluster and runtime code swapping.

Leslie Lamport argues that programmers, like architects, should write rigorous specifications before coding, especially for concurrent and distributed systems. The talk introduces the case for writing specs (the TLA+ mindset) as a way of thinking above the code.