HackerTube
HomeLongsLikedHistoryWatch laterCategoriesPlaylistsSettings

Velocity 2012: Richard Cook, "How Complex Systems Fail"

O'Reilly3 HN recommenders · recommended in 4 months, 2013-2021
YouTube
60K views · 14 years ago · talk · evergreen

Dr. Richard Cook, a physician and patient safety researcher, presents his ideas on how complex systems fail, at Velocity 2012. He argues for designing for resiliency rather than trying to exhaustively handle every failure mode.

HN people recommend it as a must-watch for anyone running complex systems, and some rewatch it regularly to remember its principles.

Cloud, DevOps & reliabilityDistributed systems#resilience#operations#safetyengineering#incidentanalysis#richardcook

7 comments from Hacker News

@rramadass2021 · in “Root cause of failure, root cause of success”

Huh? See link and watch ▶ this video The "chain analogy" is just that i.e. a Metaphor. You need to map it to your studies in Systems Science and Complexity Science . Start with the works of Henry Petroski. Then look at the works of Nassim Taleb , in particular Antifragile . Finally, do a search for "Failure in Complex Systems" and you get tons of journal articles one of which i have listed above.

CitesView on HN
@wenc2020 · in “How Complex Systems Fail (1998)”

The accompanying talk by Richard I Cook at the O'Reilly Velocity conference is here: ▶ this video When I worked in operations, I would watch this talk every couple of months to remind myself of the principles. The principles are summarized in the list, but the talk adds a bit more context. I find that this context matters for seeing how the principles might be applied.

Strongly recommendsView on HN
@wenc2019 · in “A Lifetime of Systems Thinking”

To build on this discussion, what are some highlights in that book that you found useful? I ask because I've read some systems thinking books (e.g. Systemantics) that were difficult to apply in real life. I come from the perspective of someone with a systems/theory builder personality. The only systems thinking book that I found remotely practical was The Fifth Discipline by Peter Senge. The most useful piece of short writing on systems thinking that I've come across is "How Complex Systems Fail" [1, 2], which talks about designing systems for resiliency, and not for rigid notions of reliability. [1] "How Complex Systems Fail" link [2] Its accompanying O'Reilly conference talk ▶ this video

RecommendsView on HN
@wenc2019 · in “DIB Guide: Detecting Agile BS (2018) [pdf]”

> We should probably be doing modelling and formal verification and stuff like that. This is interesting. Is this a hypothetical notion or can you share any personal experience of having worked on any projects where this is done successfully? I bring this up because there is typically a narrow band of applicability of purely theoretical constructs in most complex systems (AWS & Azure teams both use formal methods [1] to find bugs and to increase confidence in critical algorithms but I don't believe they do this system-wide. Also the verification is only done on the abstraction, not the actual implementation.). Richard Cook gave a talk on "How Complex Systems Fail" [2] which addresses this point, and advocates for designing for resiliency as opposed to designing to exhaustively handle all failure modes (which are impossible to predict). Design people often design from some idealized notion of inputs and/or worst case scenarios. Ops people know what can go wrong will go wrong, which is why ops is there to handle the deviations from the norm. Real world design exposes lift points, reveals controls and supports continuous maintenance. Highly reliable systems are evolved, not just designed. [1] link [2] ▶ this video

RecommendsView on HN
@hedwall2017 · in “Ask HN: What are some 'must watch' talks?”

Richards Cooks How complex systems fail from velocity a couple of years ago ▶ this video and the paper that's the basis for it is also nice to have handy link

RecommendsView on HN
@aristus2015 · in “Rush to Put Death Records Online Lets Anyone be 'Killed'”

The field is called "Operations Research". I had the pleasure of meeting Dr Cook in the speaker lounge right before he gave this keynote at Velocity 2012: ▶ this video

CitesView on HN
@rjzzleep2013 · in “Stop Fetishizing Failure”

except every single good thing we have is built upon failure. got a gsm phone? broadband? how about a modem? a pc? basic chemistry? cryptography? the very notion of cryptography is embracing the failure of having private data leaked. did you know 802.11 was initially made for infrared? do you remember how much it sucked? maybe if would have progressed faster if it was out in the wild faster. that's how it's progressing now. yeah, you may not like it. and every now and then I hear the idea that "when you're dealing with critical things, like healthcare where someone's life may depend on it, you cannot afford failure". except that's not true. those systems actually embrace failure more than anything else. google reliability engineering.or look at this: link there is a nice talk on failure from richard cook at the velocity conf: ▶ this video

RecommendsView on HN
Velocity NY 2013:   Richard Cook, "Resilience In Complex Adaptive Systems"19:10

Velocity NY 2013: Richard Cook, "Resilience In Complex Adaptive Systems"

O'Reilly
47K views • 13 years ago
Velocity 2010:  James Hamilton, "Datacenter Infrastructure Innovation"28:31

Velocity 2010: James Hamilton, "Datacenter Infrastructure Innovation"

O'Reilly
9K views • 16 years ago
Mikey Dickerson Velocity NY 2014 Keynote: "One Year After healthcare.gov..."28:14

Mikey Dickerson Velocity NY 2014 Keynote: "One Year After healthcare.gov..."

O'Reilly
19K views • 12 years ago
Mastering Chaos - A Netflix Guide to Microservices53:14

Mastering Chaos - A Netflix Guide to Microservices

InfoQ
2.3M views • 9 years ago
"Systems that run forever self-heal and scale" by Joe Armstrong (2013)1:10:23

"Systems that run forever self-heal and scale" by Joe Armstrong (2013)

Strange Loop Conference
85K views • 5 years ago
AWS re:Invent 2014 | (SPOT302) Under the Covers of AWS: Its Core Distributed Systems46:15

AWS re:Invent 2014 | (SPOT302) Under the Covers of AWS: Its Core Distributed Systems

Amazon Web Services
10K views • 11 years ago
Building Software Systems At Google and Lessons Learned1:22:45

Building Software Systems At Google and Lessons Learned

Stanford
257K views • 15 years ago
AWS re:Invent 2018: How AWS Minimizes the Blast Radius of Failures (ARC338)55:44

AWS re:Invent 2018: How AWS Minimizes the Blast Radius of Failures (ARC338)

Amazon Web Services
37K views • 7 years ago
AWS re:Invent 2014: AWS Innovation at Scale with James Hamilton (SPOT301)47:51

AWS re:Invent 2014: AWS Innovation at Scale with James Hamilton (SPOT301)

Amazon Web Services
70K views • 11 years ago
CRDTs and the Quest for Distributed Consistency43:39

CRDTs and the Quest for Distributed Consistency

InfoQ
68K views • 8 years ago
Lecture 1: Introduction1:19:35

Lecture 1: Introduction

MIT 6.824: Distributed Systems
921K views • 6 years ago
Inside a Google data center5:28

Inside a Google data center

Google Workspace
22.6M views • 11 years ago
Avoiding Microservice Megadisasters - Jimmy Bogard45:27

Avoiding Microservice Megadisasters - Jimmy Bogard

NDC Conferences
439K views • 9 years ago
Velocity 2011:  Jon Jenkins, "Velocity Culture"15:14

Velocity 2011: Jon Jenkins, "Velocity Culture"

O'Reilly
52K views • 15 years ago
Zach Tellman - Everything Will Flow40:13

Zach Tellman - Everything Will Flow

ClojureTV
27K views • 11 years ago
Seattle Conference on Scalability: YouTube Scalability52:41

Seattle Conference on Scalability: YouTube Scalability

Google TechTalks
30K views • 19 years ago
RustConf 2017 - Closing Keynote: Safe Systems Software and the Future of Computing by Joe Duffy42:45

RustConf 2017 - Closing Keynote: Safe Systems Software and the Future of Computing by Joe Duffy

Confreaks
2K views • 9 years ago
AWS re:Invent 2016: Tuesday Night Live with James Hamilton1:33:27

AWS re:Invent 2016: Tuesday Night Live with James Hamilton

Amazon Web Services
133K views • 9 years ago
Stanford Seminar - The Soul of a New Machine: Rethinking the Computer1:26:02

Stanford Seminar - The Soul of a New Machine: Rethinking the Computer

Stanford Online
76K views • 6 years ago
"I See What You Mean" by Peter Alvaro52:29

"I See What You Mean" by Peter Alvaro

Strange Loop Conference
59K views • 11 years ago