scale

James Hamilton, AWS VP and distinguished engineer, gives an insider view of the innovations behind the AWS cloud at re:Invent 2014. He covers custom hardware, networking, regions and availability zones, and infrastructure at scale.

John Graham-Cumming of Cloudflare talks at dotScale 2015 about handling roughly 10 trillion log lines a month. The talk touches on Cloudflare's hardware, its logging stack and its use of Lua in Nginx.

A PyData London 2018 talk on how JPMorgan scaled its Athena Python platform over ten years to about 35 million lines of code and thousands of developers. It covers infrastructure, tooling and culture for analytics and visualization at that scale.

An AWS re:Invent 2018 session on how DynamoDB was built as a hyper-scale NoSQL database. It discloses underpinnings of the service and how it is run for over 100,000 customers.

SpaceX engineers present at NVIDIA's GPU Technology Conference on a multi-physics code that simulates the full-scale Raptor methane engine. It covers GPU-based turbulent combustion modeling with adaptive grids and wavelet compression.

Bryan Cantrill's Scale By The Bay 2019 talk on software after Moore's Law. He covers the end of easy scaling and the move toward specialized, emerging architectures.

A Foresight Institute workshop video with two talks: Shawn Douglas on nanoscale instruments for visualizing small proteins, and Bret Victor on Dynamicland. Victor's part shows biomolecular design built in his Realtalk system with physical objects.

A dotScale 2014 talk by Robert Kennedy, a former healthcare.gov SRE who later worked at Google, about the engineering effort to rescue the troubled site. It explains the technical and organizational problems behind the failed launch.

An F8 2015 talk touring the tool-chain Facebook uses to let thousands of engineers build mobile apps, including its move to Mercurial, an Atom-based IDE and continuous integration.

Anthony Marcar describes lessons from three years of using Clojure at WalmartLabs, including scaling a team and a large multi-project codebase in an enterprise setting.

Bryan Cantrill argues that programming languages and platforms reflect core values, and compares languages through that lens, with Rust as a main example. He draws on his experience with Solaris, DTrace and ZFS.

Matt Ranney of Uber talks at GOTO Chicago 2016 about what Uber learned from growing past 1,000 production microservices. He covers regrets, sprawl and organizational effects.

Rick Reed of WhatsApp speaks at MeetBSD about running a messaging service on FreeBSD and Erlang at huge scale. He covers the stack, tuning and the challenges of millions of connections per host.

A PyCon 2012 talk by early YouTube engineer Mike Solomon on how YouTube scaled its architecture, which is mostly Python.

Rick Reed of WhatsApp explains at Erlang Factory 2014 how the service scaled to billions of messages with a very small engineering team. He covers the FreeBSD and Erlang stack, Mnesia usage, server counts and bottleneck work.

A 2007 Google Tech Talk in which Cliff Click of Azul Systems presents a lock-free concurrent hash table that scales to 768 CPUs. He explains why it is correct by reasoning in a different way than usual happens-before and memory-order arguments.

Martin Kleppmann's Strange Loop talk on database transactions, covering why they were dropped by NoSQL, their reputation for being slow, and newer algorithms that make serializable transactions perform and scale. It touches on consistency, linearizability and the CAP theorem.

Cuong Do of YouTube describes how a handful of engineers scaled the site, covering hardware, software and database scalability challenges during its rapid growth. The talk was given at the 2007 Seattle Conference on Scalability.

Google's Amin Vahdat gives the ONS 2015 keynote on Google's datacenter networking, including its custom hardware, software-defined networking and very high bisection bandwidth.

A QCon talk from Netflix on running a large microservices architecture and building resilience into it. It touches on tooling such as Hystrix and circuit breakers.

Sasa Juric explains what makes Erlang and Elixir suited to server-side systems, focusing on the BEAM concurrency model rather than syntax or ecosystem. He demonstrates processes, supervision, and fault tolerance.

Will Wilson of FoundationDB explains how the team tested their distributed database using deterministic simulation. By running the whole system in a simulator with injected faults and reproducible seeds, they could find rare concurrency and network bugs.

Martin Kleppmann argues that databases are global shared mutable state and proposes treating them as an ever-growing log of immutable facts. Derived data such as caches, indexes and views is then built by stream processors like Apache Samza. The talk draws on Kafka and LinkedIn's experience with commit logs.

Rob Pike's Google I/O 2012 talk shows how goroutines and channels solve tricky concurrency problems with simple Go code. It features a playful goroutine demo with gophers passing messages and a search engine example.

Martin Kleppmann's GOTO Berlin 2016 talk on how collaborative editors, offline apps, and multi-datacenter databases handle concurrent updates. It covers conflict resolution approaches including operational transformation and CRDTs, with a worked look at text CRDTs.

Paul Chiusano's Strange Loop 2019 talk introduces Unison, an open-source functional language for distributed, elastic systems. It covers content-addressed code identified by hashes of the syntax tree, a codebase manager, and algebraic effects called abilities.

A standing whiteboard conversation at Lang.NEXT 2012 among Carl Hewitt, Erik Meijer and Clemens Szyperski about the actor model. Hewitt, who originated the model, explains its foundations.

Peter Alvaro's Strange Loop 2015 talk argues that query languages let you state what you mean, and applies that to distributed programming. It covers Datalog, Dedalus and the Bloom lineage for reasoning about distributed systems.

Daniel Azuma's ElixirConf 2018 talk on whether Docker and OTP conflict. He shows they can coexist and uses Horde and DeltaCrdt, with a live tank-game demo, to keep state alive across container kills.

Leslie Lamport argues that programmers, like architects, should write rigorous specifications before coding, especially for concurrent and distributed systems. The talk introduces the case for writing specs (the TLA+ mindset) as a way of thinking above the code.

Zach Tellman's Clojure West talk covers the basics of queueing theory and its practical use in systems, including backpressure, how queues break down under load, and pitfalls of naive core.async use.

Dan Robinson, then CTO of Heap, explains how Heap runs its product analytics backend on PostgreSQL and CitusDB. It covers conversion funnels, retention and cohort queries across tens of billions of events and 50 TB of data, plus Postgres and Kafka together and the indexing strategy.

A 2018 GDC panel where Raph Koster, Starr Long, Richard Garriott and Rich Vogel look back on what went right and wrong building and running Ultima Online. They share candid stories from the early days, including player behavior and server architecture.

Will Wilson explains autonomous testing: using deterministic simulation and fuzzing, building on FoundationDB's approach, to test software continuously. It sets out the ideas behind his later company Antithesis.

Dr. Richard Cook, a physician and patient safety researcher, presents his ideas on how complex systems fail, at Velocity 2012. He argues for designing for resiliency rather than trying to exhaustively handle every failure mode.

James Long's dotJS 2019 talk on how CRDTs enable local-first apps and how simple they can be to implement. He shows how to use them to build robust offline-capable frontends.

Heidi Howard's Reactive Summit 2018 talk on generalizing Paxos by weakening its original requirements. She shows Paxos is one point on a broad spectrum of consensus approaches.

John Ousterhout lectures on the Raft consensus algorithm, made for the Raft user study comparing how students learn Raft and Paxos. It covers leader election and log repair.

Andrew Tanenbaum's FOSDEM 2010 talk on MINIX 3, a microkernel OS built for high reliability and security. It shows how the system detects and recovers from device driver crashes, and addresses microkernel performance concerns.

Martin Kleppmann's QCon London 2018 talk on keeping data consistent in distributed systems without a central leader. It surveys CRDT algorithms for syncing devices and collaborative editing over peer-to-peer networks.