devops

A podcast episode of Arrested DevOps with Andrew Clay Shafer and Bryan Cantrill, featuring Cantrill's rant on integrity in companies and technology.

Jesse Newland describes how GitHub uses Hubot, its chat bot, to run operations such as deployment, monitoring, graphing and provisioning from chat rooms. He explains how this improves shared awareness and remote work culture.

Amazon's Jon Jenkins at Velocity 2011 on how operations can drive revenue and on Amazon's highly available, autonomous architectures. It includes details on deployment frequency and Amazon's move to AWS.

Richard Cook's short Velocity 2013 talk on resilience in complex adaptive systems, arguing systems operate near the edge of failure and that the surprise is how rarely they fail. He presents a model of how pressures push systems toward failure boundaries.

Bryan Cantrill's Uptime 2017 talk on the engineering challenges of the stateful data path, including storage, firmware and hardware bugs. It uses the 'zebras vs. horses' diagnostic heuristic to explain why rare failures keep appearing.

Bryan Cantrill of Oxide Computer Company gives a Stanford EE380 talk on rethinking the server. He argues that hyperscaler-style rack-scale machines with open firmware, RISC-V and Rust should be available to everyone, not just the giants.

Bryan Cantrill of Joyent presents Manta, an internet-facing object store that lets you run compute directly on the data. The talk is peppered with ZFS history and Cantrill's trademark commentary on the industry.

Daniel Azuma's ElixirConf 2018 talk on whether Docker and OTP conflict. He shows they can coexist and uses Horde and DeltaCrdt, with a live tank-game demo, to keep state alive across container kills.

Bryan Cantrill's 2007 Google Tech Talk on DTrace, showing how dynamic tracing improves debugging of both development and live production systems. It includes a vivid war story about tracking down a problem on a Sun Ray server.

Liz Rice builds a container from scratch in about 100 lines of Go, showing that a container is not a lightweight VM but a set of Linux kernel features and flags.

Dr. Richard Cook, a physician and patient safety researcher, presents his ideas on how complex systems fail, at Velocity 2012. He argues for designing for resiliency rather than trying to exhaustively handle every failure mode.

Jon Moore's Strange Loop 2017 talk argues that rate limiting is the wrong tool for protecting services from noisy clients. He proposes concurrency-based capacity management and quotas instead.

Eric Brandwine's AWS re:Invent 2015 session on how Amazon VPC networking works. It covers the problems the design solved, why traditional approaches do not scale, and the mechanisms used for customer isolation and VPC endpoints.

Bryan Cantrill's Velocity NYC 2015 talk argues for running containers directly on bare metal rather than layering them on virtual machines. He covers the history of containers and how SmartOS runs Linux workloads via syscall emulation.

Kyle Drake of Neocities explains the bandwidth pricing pitfalls of the big cloud providers. He describes how to run high-volume, bandwidth-heavy infrastructure far more cheaply, including through co-location.

Jared Mauch tells how he got fed up with poor internet options in rural Michigan and built his own fiber network to his town. The talk was given at NLNOG Live in September 2020.

Eager Space examines the practicality of putting data centers in orbit, walking through three scenarios with cost calculations and the launch cost reductions that would be required.

A walkthrough of installing Synapse, Riot and Jitsi from scratch on Debian to run your own Matrix communication service. The description warns it is now quite outdated and links to updated material.

Mikey Dickerson's keynote at Velocity NY 2014, one year after the healthcare.gov launch failure. He recounts how a small team rescued the site and what the episode showed about government IT contracting.

Rick Reed of WhatsApp speaks at MeetBSD about running a messaging service on FreeBSD and Erlang at huge scale. He covers the stack, tuning and the challenges of millions of connections per host.

Bryan Cantrill's illumos Day 2014 talk on running Linux containers on an illumos kernel via LX-branded zones. He covers emulating Linux syscalls on SmartOS and compares Linux and Solaris-derived systems.

A dotScale 2014 talk by Robert Kennedy, a former healthcare.gov SRE who later worked at Google, about the engineering effort to rescue the troubled site. It explains the technical and organizational problems behind the failed launch.

James Hamilton, AWS VP and distinguished engineer, gives an insider view of the innovations behind the AWS cloud at re:Invent 2014. He covers custom hardware, networking, regions and availability zones, and infrastructure at scale.

An AWS re:Invent 2018 session in which Holly Mesrobian and Marc Brooker explain how AWS Lambda is built, including its sandboxing, isolation and scaling architecture. It covers what happens under the hood when serverless code runs.

Der8auer tours Hetzner's test lab and shows a custom AM5 server motherboard with the CPU socket rotated 90 degrees, built for rack airflow in their AX52 servers.

Jessie Frazelle's DockerCon SF 2015 talk on running Docker containers for desktop applications, with fun container hacks and images. It shows how far containers can be pushed beyond servers.

Matt Godbolt tells the story of Compiler Explorer, from a hacky basement server to a cloud-scaled service handling tens of thousands of daily requests. He also shows its features and how it is built.

An AWS re:Invent 2018 talk on how AWS limits the blast radius of failures. It covers design techniques such as cell-based architecture and shuffle sharding.

The day one keynote of Google Cloud's 2016 GCP NEXT conference, with Sundar Pichai, Diane Greene, Eric Schmidt, Urs Holzle and customers. It includes a Cloud ML demo by Jeff Dean.

A PyCon 2013 block of Friday evening lightning talks. It includes the debut of Hy, a Lisp dialect that compiles to Python AST, and Solomon Hykes's first public talk on Docker.

A four-minute Velocity 2011 talk in which Artur Bergman, then at Wikia, argues bluntly and with profanity that SSDs should replace spinning disks in servers.

John Graham-Cumming of Cloudflare talks at dotScale 2015 about handling roughly 10 trillion log lines a month. The talk touches on Cloudflare's hardware, its logging stack and its use of Lua in Nginx.

An AWS re:Invent 2014 talk on the core distributed systems primitives behind AWS and Amazon.com, such as consensus, key-value stores and transactional logging. It describes how AWS builds large services from a few common building blocks.

Urs Hoelzle, Google's SVP of infrastructure, gives the 2012 Open Networking Summit keynote on how Google adopted OpenFlow and software-defined networking in its own networks. The slides are linked in the description.

An AWS re:Invent 2018 session diving into the Nitro system, a set of hardware offload and security building blocks behind next-generation EC2 instances. It covers the design and architecture and how Nitro enables new instance types.

A Google Tech Talk by Raymond Blum on how Google backs up massive amounts of data such as Gmail and Picasa. It covers scaling challenges, failure types, and approaches that did not work, including tape.

A QCon London 2020 talk by Tesla engineers on the architecture of Tesla's Virtual Power Plant. It covers aggregating distributed batteries and solar with vertically integrated hardware and software across cloud and edge.

James Hamilton of AWS gives a Velocity 2010 keynote on datacenter infrastructure innovation, focused on power and cooling costs and a breakdown of where datacenter spending goes.

Kelsey Hightower's opening keynote at KubeCon, delivered as a live-demo style talk about Kubernetes and cloud-native tooling.

A Netflix performance engineer presents how Netflix tunes EC2 instances, covering instance type choice, Xen modes, SR-IOV and kernel tuning. The goal is near bare-metal performance in the cloud.