devops

Howard Abrams demonstrates, in an EmacsConf 2015 talk, how he uses org-mode's Babel and Tramp to configure and operate remote servers from a single document. Code blocks, their results and the explanatory text all live together.

A podcast episode of Arrested DevOps with Andrew Clay Shafer and Bryan Cantrill, featuring Cantrill's rant on integrity in companies and technology.

Bryan Cantrill's GOTO Chicago 2017 talk on debugging production systems under pressure, centered on the incident where an operator's mistyped command rebooted an entire Joyent datacenter. It covers human factors, automation and architecture.

Pixar staff tell the story of how Toy Story 2 was almost deleted from the studio's computers and was saved by a home computer backup.

A KRAZAM comedy sketch in which a character recites reassuring affirmations to site reliability engineers, satirizing on-call and outage culture.

Kevin Fang comically retells GitLab's 2017 incident in which an engineer deleted the production database, based on the company's postmortems. It walks through the confusion and failed backups.

Richard Cook's short Velocity 2013 talk on resilience in complex adaptive systems, arguing systems operate near the edge of failure and that the surprise is how rarely they fail. He presents a model of how pressures push systems toward failure boundaries.

Rick Houlihan's AWS re:Invent 2018 deep dive on advanced DynamoDB design patterns. It covers index overloading, GSI sharding, composite keys, graph processing and transactions.

Liz Rice live-codes a minimal container runtime in a few lines of Go at GOTO Amsterdam 2018. She explains namespaces, control groups and what a host knows about its containers.

Bryan Cantrill's Uptime 2017 talk on the engineering challenges of the stateful data path, including storage, firmware and hardware bugs. It uses the 'zebras vs. horses' diagnostic heuristic to explain why rare failures keep appearing.

Bryan Cantrill of Joyent presents Manta, an internet-facing object store that lets you run compute directly on the data. The talk is peppered with ZFS history and Cantrill's trademark commentary on the industry.

Bryan Cantrill's 2007 Google Tech Talk on DTrace, showing how dynamic tracing improves debugging of both development and live production systems. It includes a vivid war story about tracking down a problem on a Sun Ray server.

Liz Rice builds a container from scratch in about 100 lines of Go, showing that a container is not a lightweight VM but a set of Linux kernel features and flags.

Nickolas Means recounts the 1979 Three Mile Island accident minute by minute, from the control rod insertion to the operator's decision to switch off emergency cooling. He argues that finding systemic causes is more useful than blaming individuals.

Dr. Richard Cook, a physician and patient safety researcher, presents his ideas on how complex systems fail, at Velocity 2012. He argues for designing for resiliency rather than trying to exhaustively handle every failure mode.

Jon Moore's Strange Loop 2017 talk argues that rate limiting is the wrong tool for protecting services from noisy clients. He proposes concurrency-based capacity management and quotas instead.

Jérôme Petazzoni explains how Linux containers are built from kernel features like cgroups, namespaces and SELinux. He demonstrates putting the pieces together by hand to make a container and compares runtimes and approaches like jails and zones.

Dave Plummer tours the IBM z16 mainframe from design through factory assembly, test lab, accelerators and water cooling. He explains why mainframes still exist and what makes them reliable.

Kyle Drake of Neocities explains the bandwidth pricing pitfalls of the big cloud providers. He describes how to run high-volume, bandwidth-heavy infrastructure far more cheaply, including through co-location.

Jared Mauch tells how he got fed up with poor internet options in rural Michigan and built his own fiber network to his town. The talk was given at NLNOG Live in September 2020.

Mikey Dickerson's keynote at Velocity NY 2014, one year after the healthcare.gov launch failure. He recounts how a small team rescued the site and what the episode showed about government IT contracting.

Michael Bryzek of Flow gives a QCon London 2018 talk on key decisions for building maintainable microservices. It covers infrastructure, continuous deployment, communication, event streaming and language choice.

A dotScale 2014 talk by Robert Kennedy, a former healthcare.gov SRE who later worked at Google, about the engineering effort to rescue the troubled site. It explains the technical and organizational problems behind the failed launch.

Matt Godbolt tells the story of Compiler Explorer, from a hacky basement server to a cloud-scaled service handling tens of thousands of daily requests. He also shows its features and how it is built.

An AWS re:Invent 2018 talk on how AWS limits the blast radius of failures. It covers design techniques such as cell-based architecture and shuffle sharding.

The official Kubernetes documentary, part 1, tells how a handful of Google engineers, inspired by Docker's success, built a container orchestrator. It covers the internal resistance to open source and the competition of the container orchestrator wars.

A short satirical video, originally titled Broccoli Man, in which an engineer who only wants to serve 5 terabytes gets tangled in Google's internal resource and production processes. It is the source of the phrase about forgetting how to count that low.

An AWS re:Invent 2019 session for advanced DynamoDB users on design patterns used by large customers, including Amazon retail. It covers GSI sharding, index overloading, graph processing, composite keys and transactional workflows.

An AWS re:Invent 2014 talk on the core distributed systems primitives behind AWS and Amazon.com, such as consensus, key-value stores and transactional logging. It describes how AWS builds large services from a few common building blocks.

Linus Tech Tips documents a major RAID failure on its Whonnock server and the attempt to recover the lost data with a data recovery firm.

A comedy video in which a salesperson demands that a web developer fix a down website, with the two talking past each other. It won a Webby award.

A 2012 Onion News Network satire in which HP announces a push into cloud computing and insists it knows what the cloud is.

A collection of lightning talks from the Surge 2015 conference on scalable systems, hosted by Geir Magnusson. HN mostly points to one segment, a Bryan Cantrill talk touching on madvise semantics.

A short video of five people carrying a live, running web server across the city by public transport to a new datacenter. It stayed powered and connected to the internet the whole way, so there was no downtime.