How Things Fall Over
Curated by Claude · 12 videos · 7 hr 16 min
HN's reading list for the day after an outage. It opens with short, funny disaster stories, moves into detailed incident accounts and the craft of debugging under pressure, then to designs that limit the damage. It ends with Richard Cook's two short Velocity talks, which several commenters say they rewatch every few months.
Play all on YouTube- 12:26
How Toy Story 2 Almost Got Deleted: Stories From Pixar Animation: ENTV
551K views • 14 years ago@HOLLYWOODPixar staff tell the story of how Toy Story 2 was almost deleted from the studio's computers and was saved by a home computer backup.Two minutes to start: Pixar nearly lost a film to one command and was saved by a home-computer backup, a cautionary tale HN tells to anyone in storage or SRE.
- 210:20
Dev Deletes Entire Production Database, Chaos Ensues
3.6M views • 3 years agoKevin FangKevin Fang comically retells GitLab's 2017 incident in which an engineer deleted the production database, based on the company's postmortems. It walks through the confusion and failed backups.Kevin Fang's comic but accurate retelling of GitLab's 2017 incident, where every backup layer failed in turn, which HN prescribes for a bad ops day.
- 327:21
dotScale 2014 - Robert Kennedy - Life in the Trenches of healthcare.gov
13K views • 12 years agodotconferencesA dotScale 2014 talk by Robert Kennedy, a former healthcare.gov SRE who later worked at Google, about the engineering effort to rescue the troubled site. It explains the technical and organizational problems behind the failed launch.A rescue engineer's first-hand account of a failure that was organizational, not technical: nobody owned the whole site.
- 437:07
Who Destroyed Three Mile Island? - Nickolas Means | The Lead Developer Austin 2018
762K views • 8 years agoLeadDevNickolas Means recounts the 1979 Three Mile Island accident minute by minute, from the control rod insertion to the operator's decision to switch off emergency cooling. He argues that finding systemic causes is more useful than blaming individuals.A minute-by-minute account of a nuclear accident, which HN uses to push back whenever an outage gets blamed on human error.
- 5
52:35Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017
101K views • 9 years agoGOTO ConferencesBryan Cantrill's GOTO Chicago 2017 talk on debugging production systems under pressure, centered on the incident where an operator's mistyped command rebooted an entire Joyent datacenter. It covers human factors, automation and architecture.The postmortem HN rates above almost every written one: a mistyped command reboots a whole Joyent datacenter, and Cantrill walks through what came after.
- 6
49:09Debugging with the Scientific Method - Stuart Halloway
25K views • 10 years agoClojureTVStuart Halloway of Cognitect argues for debugging with the scientific method: form hypotheses, try to falsify them, and write everything down. The talk is given to the Clojure community but applies to any language.A calmer counterpart on method (form hypotheses, try to falsify them, write everything down), which several commenters say changed how they debug.
- 7
46:37Zebras All the Way Down - Bryan Cantrill, Uptime 2017
49K views • 9 years agoCode & SupplyBryan Cantrill's Uptime 2017 talk on the engineering challenges of the stateful data path, including storage, firmware and hardware bugs. It uses the 'zebras vs. horses' diagnostic heuristic to explain why rare failures keep appearing.Cantrill again, on the rare failures (firmware, disks, opaque hardware) that keep turning out to be the real cause, a talk HN calls entertaining and sobering.
- 8
55:44AWS re:Invent 2018: How AWS Minimizes the Blast Radius of Failures (ARC338)
37K views • 7 years agoAmazon Web ServicesAn AWS re:Invent 2018 talk on how AWS limits the blast radius of failures. It covers design techniques such as cell-based architecture and shuffle sharding.Moving from response to design: cell-based architecture and shuffle sharding, which HN links whenever someone asks how to contain an outage.
- 9
45:31The Do's and Don'ts of Error Handling • Joe Armstrong • GOTO 2018
74K views • 8 years agoGOTO ConferencesJoe Armstrong, inventor of Erlang, gives a GOTO Chicago 2018 talk on building fault-tolerant systems. He covers what to do and avoid when handling errors, arguing for letting processes fail and recover.Erlang's creator makes the case for letting things crash and recover, a conceptual talk rather than a code-heavy one.
- 10
1:01:59TigerStyle! (Or How To Design Safer Systems in Less Time) by Joran Dirk Greef
87K views • 3 years agoTigerBeetleJoran Dirk Greef's Systems Distributed '23 talk on TigerStyle, the engineering methodology behind TigerBeetle. It covers designing safer systems in less time through up-front design, assertions, and deterministic simulation.The most demanding answer here: TigerBeetle's method of assertions, up-front design and deterministic simulation for software meant to be defect-free.
- 11
28:21Velocity 2012: Richard Cook, "How Complex Systems Fail"
60K views • 14 years agoO'ReillyDr. Richard Cook, a physician and patient safety researcher, presents his ideas on how complex systems fail, at Velocity 2012. He argues for designing for resiliency rather than trying to exhaustively handle every failure mode.A patient-safety physician explains to ops engineers why complex systems always run partly broken, and one HN commenter rewatches it every couple of months.
- 12
19:10Velocity NY 2013: Richard Cook, "Resilience In Complex Adaptive Systems"
47K views • 13 years agoO'ReillyRichard Cook's short Velocity 2013 talk on resilience in complex adaptive systems, arguing systems operate near the edge of failure and that the surprise is how rarely they fail. He presents a model of how pressures push systems toward failure boundaries.The capstone, in 19 minutes: Cook's model of pressures pushing a system toward the edge of failure, which one commenter called among the most important talks they had seen.