scale

A short film in which friends build the first true-to-scale model of the solar system, with full planetary orbits, on a Nevada dry lakebed. It conveys the real distances and sizes involved.

James Hamilton, AWS VP and distinguished engineer, gives an insider view of the innovations behind the AWS cloud at re:Invent 2014. He covers custom hardware, networking, regions and availability zones, and infrastructure at scale.

John Graham-Cumming of Cloudflare talks at dotScale 2015 about handling roughly 10 trillion log lines a month. The talk touches on Cloudflare's hardware, its logging stack and its use of Lua in Nginx.

A PyData London 2018 talk on how JPMorgan scaled its Athena Python platform over ten years to about 35 million lines of code and thousands of developers. It covers infrastructure, tooling and culture for analytics and visualization at that scale.

An AWS re:Invent 2018 session on how DynamoDB was built as a hyper-scale NoSQL database. It discloses underpinnings of the service and how it is run for over 100,000 customers.

Hyrum Wright shares lessons from five years of mass code migrations in Google's huge C++ codebase. He covers why to migrate, how hard it really is, and how Google updates core libraries and their users continually.

A satirical animated dialogue in which a naive developer insists MongoDB is web scale while a database expert mocks him. It parodies the NoSQL versus relational debate.

Dropbox engineer Kevin Modzelewski gives a Stanford colloquium on Dropbox's history and the technical problems it faced scaling, including detailed database schema decisions.

Google engineers describe Google's internal build system, a unified, language-agnostic system used across the company. It was presented as a 2012 NYC tech talk.

SpaceX engineers present at NVIDIA's GPU Technology Conference on a multi-physics code that simulates the full-scale Raptor methane engine. It covers GPU-based turbulent combustion modeling with adaptive grids and wavelet compression.

Bryan Cantrill's Scale By The Bay 2019 talk on software after Moore's Law. He covers the end of easy scaling and the move toward specialized, emerging architectures.

Psychologist Robert Epstein presents a Stanford EE380 colloquium on how internet algorithms such as search rankings can manipulate opinions, purchases and votes at massive scale. He also describes a new monitoring system for tracking such influence.

A dotScale 2014 talk by Robert Kennedy, a former healthcare.gov SRE who later worked at Google, about the engineering effort to rescue the troubled site. It explains the technical and organizational problems behind the failed launch.

An F8 2015 talk touring the tool-chain Facebook uses to let thousands of engineers build mobile apps, including its move to Mercurial, an Atom-based IDE and continuous integration.

Anthony Marcar describes lessons from three years of using Clojure at WalmartLabs, including scaling a team and a large multi-project codebase in an enterprise setting.

Bryan Cantrill argues that programming languages and platforms reflect core values, and compares languages through that lens, with Rust as a main example. He draws on his experience with Solaris, DTrace and ZFS.

Luis von Ahn, creator of CAPTCHA and reCAPTCHA, explains how small contributions from many people online can digitize books. He then introduces Duolingo as a way to translate the web while people learn languages.

AWS's short promo for Snowmobile, a 45-foot shipping container pulled by a semi truck that moves up to 100PB of data to AWS.

A visual scale comparison of stars and cosmic objects, from planets up to some of the largest known stars, set to music. It is meant to convey the vastness of the universe.

Jeff Dean of Google describes his experiences building Google's systems and how its infrastructure evolved over time. The Stanford talk covers search, scaling tipping points and lessons from building large-scale software.

The Eames Office's 1977 film Powers of Ten starts at a picnic in Chicago and zooms out by a factor of ten every ten seconds to the edge of the universe. It then zooms inward to a proton in a carbon atom.

Numberphile explains the long and short scales for naming large numbers, covering millions, billions, trillions, milliards and the Greek myriad, with James Grime and Tony Padilla.

Matt Ranney of Uber talks at GOTO Chicago 2016 about what Uber learned from growing past 1,000 production microservices. He covers regrets, sprawl and organizational effects.

Rick Reed of WhatsApp speaks at MeetBSD about running a messaging service on FreeBSD and Erlang at huge scale. He covers the stack, tuning and the challenges of millions of connections per host.

A PyCon 2012 talk by early YouTube engineer Mike Solomon on how YouTube scaled its architecture, which is mostly Python.

A Google Tech Talk by Raymond Blum on how Google backs up massive amounts of data such as Gmail and Picasa. It covers scaling challenges, failure types, and approaches that did not work, including tape.

A recording of Richard Feynman's talk on the possibilities of manipulating matter at very small scales, widely seen as an early vision of nanotechnology. This is a later version of his 1959 "There's Plenty of Room at the Bottom" lecture.

Rick Reed of WhatsApp explains at Erlang Factory 2014 how the service scaled to billions of messages with a very small engineering team. He covers the FreeBSD and Erlang stack, Mnesia usage, server counts and bottleneck work.

A 2007 Google Tech Talk in which Cliff Click of Azul Systems presents a lock-free concurrent hash table that scales to 768 CPUs. He explains why it is correct by reasoning in a different way than usual happens-before and memory-order arguments.

James Hamilton's Tuesday Night Live keynote at AWS re:Invent 2016, touring AWS infrastructure at scale including networking, power and custom silicon. It also features NASA JPL's Tom Soderstrom.

Martin Kleppmann's Strange Loop talk on database transactions, covering why they were dropped by NoSQL, their reputation for being slow, and newer algorithms that make serializable transactions perform and scale. It touches on consistency, linearizability and the CAP theorem.

An 8-minute explainer on the Lightning Network and how it relates to Segwit, Bitcoin and Litecoin. It covers how off-chain payment channels route payments to scale blockchain transactions.

Google's Amin Vahdat gives the ONS 2015 keynote on Google's datacenter networking, including its custom hardware, software-defined networking and very high bisection bandwidth.

A QCon talk from Netflix on running a large microservices architecture and building resilience into it. It touches on tooling such as Hystrix and circuit breakers.

Cody's Lab builds a 200-billionth-scale model to show how far away the nearest stars are, using parallax and light speed. A short astronomy explainer on interstellar distances.

Technology Connections shows a cheap method for capturing analog video: a composite-to-HDMI upscaler feeding an HDMI capture dongle, aimed at digitizing home movies and VHS tapes.

Sasa Juric explains what makes Erlang and Elixir suited to server-side systems, focusing on the BEAM concurrency model rather than syntax or ecosystem. He demonstrates processes, supervision, and fault tolerance.

Will Wilson of FoundationDB explains how the team tested their distributed database using deterministic simulation. By running the whole system in a simulator with injected faults and reproducible seeds, they could find rare concurrency and network bugs.

Martin Kleppmann argues that databases are global shared mutable state and proposes treating them as an ever-growing log of immutable facts. Derived data such as caches, indexes and views is then built by stream processors like Apache Samza. The talk draws on Kafka and LinkedIn's experience with commit logs.

Rich Hickey's 2013 talk applies language-design thinking to whole systems, covering names, queues, data, failure, and the interfaces between programs. It ends with a call to action for building systems holistically.