Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017

I was! (Along with co-author Dave Pacheco.) And I still have the dream that we'll finish it one day: we had written probably a third of it, but then life intervened in various dimensions. And indeed, as part of our preparation to write our book (which we titled The Joy of Debugging ), we read Wheeler's Debugging . On the one hand, I think it's great to have anything written about debugging, as it's a subject that has not been treated with the weight that it deserves. But on the other, the "methodology" here is really more of a collection of aphorisms; if folks find it helpful, great -- but I came away from Debugging thinking that the canonical book on debugging has yet to be written. Fortunately, my efforts with Dave weren't for naught: as part of testing our own ideas on the subject, I gave a series of presentations from ~2015 to ~2017 that described our thinking. A talk that pulls many of these together is my GOTO Chicago talk in 2017, on debugging production systems.[0] That talk doesn't incorporate all of our thinking, but I think it gets to a lot of it -- and I do think it stands at a bit of contrast to Wheeler's work. [0] ▶ this video

Bryan Cantrill draws some excellent parallels between this incident and software systems in his "Debugging Under Fire" talk (beware of semi-automated systems): ▶ this video

I'm reminded of Bryan Cantrill's talk "Debugging Under Fire"[1], which includes a retrospective of sorts about an entire datacenter rebooting.[2] That is a pretty large-scale disaster, but even that is a rung below a continent-wide outage. Poor "Bill" must have saw the proverbial light when he heard some folks wanted to trash the DIASTER script. [1]: ▶ this video [2]: link

The best postmortem I've seen was actually a conference talk: "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" by Bryan Cantrill (in the GOTO conference, 2017) Here's a YouTube video of it: ▶ this video Here's the slides: link It goes into detail about a pretty bad outage (when an entire data center was brought down), the human aspects, automation, how they handled it, the various risks, architectures, how things fail and about software development in general.

Time to share one of my favorite talks (and speakers) ever - "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" (Bryan Cantrill, GOTO 2017) ▶ this video

If it were so obvious, I am wondering why you need to worry about doing it wrong. You talk about trade-offs as if they are fixed in time. That is not the case at all in anything but toy systems. Maybe you need to find someone who can design a quality micro-service? You have no idea what I do and what my competencies are, yet, you are assuming you know a lot more about system design than I do and I am missing something simple. Here is another take from Bryan Cantrill [ ▶ this video ]. Very likely, he does not know anything about services either, I guess.

I’m surprised I haven’t seen this linked yet, but Bryan Cantrill of dtrace, Sun, lawnmower, Joyent, etc, fame gave an amazing talk for Monktoberfest 2016, titled “Oral Tradition in Software Engineering”, which features The Story of Mel [1]. Highly recommend checking it out — there are loads of little gems and stories like this throughout. All of his other presentations are great too and definitely worth a listen if you like this sort of thing [2]. A couple of my favorites are “Fork Yeah! The Rise and Development of Illumos” [3] and “Debugging Under Fire: Keep your Head when Systems have Lost their Mind” [4]. [1]: link [2]: link [3]: link [4]: ▶ this video

> All I say is that you must relay on externals if A-E are all on the same network as it may go down. Thankfully, it's not too hard to take advantage of multiple networks in a hybrid/multi-cloud setup nowadays! Though, depending on the necessary access controls and auditing, such a setup might require slightly more work. You do bring up an excellent point, though, about how it's a serious single point of failure in many systems out there, because personally I've also seen many setups like that (the majority of them, actually): I do suspect that in many cases that is indeed done for ease of use/convenience, even if it may lead to downtime. Of course, in some cases downtime is acceptable, so I cannot argue that it can also make sense to choose such a simpler setup - for example, for having your own company's applications/monitoring for development environments all on the same network. Though if this topology is retained at scale, things can get a bit interesting. On a similar note, I recall Bryan Cantrill doing an interesting presentation "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" that talked about restarting their whole data center and the implications of that: ▶ this video

I suppose it all depends on how much infra needs to be stood up for the absolute necessities of the business to operate. Does the company need that internal ticketing system in place to process external client transactions? Probably not, but it'll need it eventually (so maybe that moves to 2nd tier restore process?). My company's RTO is 24hrs to processing new client transactions. Restoring old ones will definitely take longer, but at least new ones can proceed. If your own company's RTO is 2w, that sounds like a lot needs to be in place. Part of the business continuity/ disaster recovery is getting management to sign off on those types of numbers, big or small. Make sure they're realistic. You're right that this type of recovery is not fun. Bryan Cantrill gives a great presentation about managing an outage ( ▶ this video ). One of my biggest takeaways, if you're looking at a sweeping outage and a long haul of a recovery, do sleep management asap with your team. Dead tired people are more likely to make brain dead decisions.

Perhaps not famous, but Bryan Cantrill, who gives my favorite talks, has an interesting and funny talk on one of the Joyent outages: ▶ this video

I'll give it a watch and will summarize, because while i'm not always on board with the Rust hype (and therefore want to know more, to help eliminate my biases, one way or the other). That said, i rather enjoyed Bryan Cantrill's talk from 2017, "Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017": ▶ this video So i wouldn't necessarily turn away from any of his videos just because of the sometimes humorous or not awfully serious tone.

Not (I think) the exacttalk/blog post gp was thinking of - but worth watching IMNHO: "Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017" ▶ this video Ed: oh, here we go I think? > Running Aground: Debugging Docker in Production Bryan Cantrill19,102 views16 Jan 2018 Talk originally given at DockerCon '15, which (despite being a popular presentation and still broadly current) Docker Inc. has elected to delist. link

> Joyent famously took down their whole region by rebooting wrong nodes. In case anyone's interested, here's a pretty funny and educational talk by Bryan Cantrill about that particular incident: GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind ▶ this video

It reminds me of this: ▶ this video -- "GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill"

> I run a YouTube channel that is over 10 years old , has over 770,000 subscribers , almost 600 public videos , 120,000,000 views When writing software involved in managing a live, public, massively multi-user system, the traditional unix-style commands that are immediate, often silent, and capable of damaging effects become a really easy way to shoot yourself in the foot. Worse, some commands might let you accidentally shoot everyone's foot on a typo. The traditional example is accidentally typing something like "rm -f * .bak" (note the extra space after the star). For a good discussion of this type of problem, I highly recommend Bryan Cantrill's talk[1] about the time an operator accidentally rebooted an entire datacenter with a single miss-typed command. The general solution to this is building sanity checks into the software. The user just asked to reformat 500 hosts, but almost all previous uses olf the 'reformat' command affected less than 10 hosts. Maybe we should ask for verification from an actual human if they really intended to run this unusually destructive command. Why doesn't YouTube have this kind of sanity check in their automated takedown/strike/channel-deletion tools? Google wrote automation that can decide to delete a channel with a long history and many successful videos. Why doesn't that automation have basic sanity checks that ask for operator input when asked to do an unusually destructive action like deleting a 10 year old channel with a huge history?

> In my experience customers deeply detest the idea of waiting around for a failure case to re-occur so that you can understand it better. Bryan Cantrill has a great talk[0] about dealing with fires where he says something to the effect of: > Now you will find out if you are more operations or development - developers will want to leave things be to gather data and understand, while operations will want to rollback and fix things as quickly as possible [0] Debugging Under Fire: Keep your Head when Systems have Lost their Mind - Bryan Cantrill: ▶ this video

> I'm not sure you do, since you admitted to really being at least part sysadmin, earlier :) Hah! I wish I could remember which talk it was. Bryan Cantrill had a good line about DevOps in one of his talks (I think it was this one: ▶ this video ). The gist was "you can say you're DevOps, but when the shit hits the fan, you're either going to be Dev or Ops. If you're a Dev, you're going to want to debug the problem before rebooting the failed machine. If you're Ops, you're going to want to reboot the machine as fast as you can to get it back up." Through that lens, I'm definitely pretty far over on the Dev spectrum; when something goes catastrophically wrong and someone reboots a box to "solve the problem", my first reaction is "YOU FUCKER YOU BURNED THE CORPSE" It's such a tricky thing all around. Looking back at what I wrote earlier, I also realize that in some ways I'm facilitating the ignorance. I've got a really nice Consul and Nomad setup for the team, so they can pretty much just toss .wars and Docker containers at the cluster and they'll automatically get scheduled somewhere with spare capacity. The load balancer, the database cluster, all of the service discovery and job scheduling stuff... they've never had to get in and set any of that up. Maybe it's time to do more mentoring... Anyway, thanks for getting me thinking. I've been a little bit grouchy lately about all of the recent experiences of people not knowing how the stuff they build actually runs. You've been a great