file systems

Jeff Bonwick and Bill Moore, the creators of ZFS, present 'ZFS: The Last Word in File Systems' at the 2008 SNIA Software Developers' Conference. They explain the design, including copy-on-write, checksumming and avoiding the write hole.

A Microsoft Research talk on BetrFS, a file system built on B-epsilon trees, a write-optimized data structure. It covers WODS such as LSM-trees, how to design a file system around them, and benchmark wins over ext4 and xfs.

Dave Chinner's linux.conf.au 2012 talk on XFS scalability, covering pain points with large numbers of files and complex metadata, and the work to fix XFS metadata performance.

Marshall Kirk McKusick gives a talk at EuroBSDCon tracing filesystem and storage development from 1979 to today, using the BSD Fast Filesystem as the focus. It covers early disk-geometry optimization through later contiguous layout and small-file performance work.

The author of copyparty introduces the FOSS file server he has built over five years, demoing its features such as resumable uploads, WebDAV and deduplication. The video itself is shown running on a demo server.

Kent Overstreet's Linux Foundation talk about writing a filesystem in Rust. Its Q&A segment, where Rust-for-Linux developers ask about interface semantics, became widely cited.

Jeff Dean of Google describes his experiences building Google's systems and how its infrastructure evolved over time. The Stanford talk covers search, scaling tipping points and lessons from building large-scale software.

An AWS re:Invent 2014 talk on the core distributed systems primitives behind AWS and Amazon.com, such as consensus, key-value stores and transactional logging. It describes how AWS builds large services from a few common building blocks.

A panel from the Lang NEXT 2014 conference on systems programming in 2014 and beyond. Language designers including Bjarne Stroustrup and Rob Pike discuss the topic, including C++ header compile times.

The first lecture of Andy Pavlo's CMU 15-445 Intro to Database Systems (Fall 2019), covering course introduction and the relational model.

Joran Dirk Greef's Systems Distributed '23 talk on TigerStyle, the engineering methodology behind TigerBeetle. It covers designing safer systems in less time through up-front design, assertions, and deterministic simulation.

Timothy Roscoe of ETH Zurich delivers the joint USENIX ATC/OSDI 2021 keynote arguing that operating systems must reckon with modern hardware, which is a heterogeneous collection of processors, firmware and coprocessors rather than a simple machine. It shows how much runs below and beside Linux.

The first lecture in Martin Kleppmann's 8-lecture distributed systems series from the University of Cambridge undergraduate course. It introduces the subject, and accompanying lecture notes are available.

Aditya Siram introduces ATS, a statically typed ML-like language that compiles to C and offers type-safe pointer arithmetic and a full proof language. The Strange Loop talk shows off C interop, concurrency and verification features with non-trivial examples.

Joe Duffy's RustConf 2017 closing keynote on his experience at Microsoft building Midori, an operating system written in a Rust-like safe systems language. He covers the technical and cultural challenges of applying those lessons to Windows.

Tom Scott explains why Windows does not let you name a file CON, tracing the reserved device names back to backwards compatibility with DOS.

Will Wilson of FoundationDB explains how the team tested their distributed database using deterministic simulation. By running the whole system in a simulator with injected faults and reproducible seeds, they could find rare concurrency and network bugs.

Allan Jude of Klara presents work on making ZFS scale on fast NVMe storage, covering fixes to the TaskQueue interface and removal of bottlenecks. The talk was given at EuroBSDcon 2022 and describes raising write throughput from about 3 to over 7 GB/sec.

Richard Cook's short Velocity 2013 talk on resilience in complex adaptive systems, arguing systems operate near the edge of failure and that the surprise is how rarely they fail. He presents a model of how pressures push systems toward failure boundaries.

Dr. Richard Cook, a physician and patient safety researcher, presents his ideas on how complex systems fail, at Velocity 2012. He argues for designing for resiliency rather than trying to exhaustively handle every failure mode.

Primer uses simulations to compare voting systems such as first-past-the-post, approval, ranked choice and others. It shows how each system behaves and can be manipulated, with links to further voting theory resources.

A short screen capture of SGI's File System Navigator (FSN, also called Fusion) running on IRIX 6.5 on an Octane. It shows the 3D file browser seen in Jurassic Park.

Peter Alvaro's Strange Loop 2015 talk argues that query languages let you state what you mean, and applies that to distributed programming. It covers Datalog, Dedalus and the Bloom lineage for reasoning about distributed systems.

Tom Caputi presents the design of native encryption for ZFS at the OpenZFS Developer Summit, including design decisions and how authentication uses checksum fields.

An OpenZFS office hours session with ZFS co-creator Matt Ahrens answering questions on topics such as hole birth time, send/receive buffers, device removal, block pointer rewrite and performance with many snapshots.

Optiver's Carl Cook explains how to build high-frequency trading systems in C++, where the critical path runs rarely but must be extremely fast. He covers techniques for low and predictable latency, including avoiding main memory and cache misses.

Benno Rice's linux.conf.au 2020 talk asks whether UNIX was the best design we could have had. He looks at what alternatives existed, what was gained and lost, and what could be re-learned, including the limits of treating everything as a file.

A 1994 talk by systems thinker Russ Ackoff, who knew W. Edwards Deming, on continuous versus discontinuous improvement. It ends with an unexpected lesson.

Brian Warner, author of magic-wormhole, presents the tool at PyCon 2016. He explains how a few secret words let two computers securely transfer files, covering the cryptography (NaCl, SPAKE2) and library use.

Robert Collins explains why rustup installs were painfully slow on Windows and how changes to rustup and Windows cut install time from 3m30s to 14s. The talk covers NTFS, small files and the cost of antivirus hooks on file close.

Joe Armstrong's Strange Loop 2013 talk on building large scalable systems from small isolated parallel components. It covers detecting and correcting errors at run time, changing code while it runs, and how Erlang embodies these principles.

An Empire Files episode giving a timeline of Zionist settlement, the Nakba and Israel's expansion over Palestinian land, presented from a critical, anti-colonial viewpoint. It previews Abby Martin's reporting in Palestine.

Bryan Cantrill's QCon SF 2019 talk asking whether operating systems should be rewritten in Rust. He draws on his systems experience to discuss C, memory safety and Rust's fit for system software.

Matthias Felleisen's keynote uses weather prediction as an analogy for type systems, arguing they forecast problems so you can act on them. It draws on his experience developing Typed Racket.

Chris Houser's Clojure/conj talk on why Clojure's exceptions destroy information as the stack unwinds, unlike Common Lisp's condition system, which can resume from the error site. He surveys the existing libraries that approximate conditions, explains why they don't compose with each other, and considers what a better design would look like.

Martin Kleppmann's Strange Loop talk on database transactions, covering why they were dropped by NoSQL, their reputation for being slow, and newer algorithms that make serializable transactions perform and scale. It touches on consistency, linearizability and the CAP theorem.

A guest lecture Jon Gjengset gave at Two Sigma on using Rust for large, high-performance systems. It covers why Rust suits such projects, the Noria research database, a concurrent data structure used in it, and practical experience.

Martin Kleppmann's Kafka Summit 2018 keynote asks whether Kafka can be considered a database. He covers event logs and related data-systems ideas.

The opening lecture of MIT 6.824 Distributed Systems (Spring 2020), introducing the course and core challenges of distributed systems. It begins a playlist that goes on to cover Raft and other consensus papers in Go.

Rich Hickey's 2013 talk applies language-design thinking to whole systems, covering names, queues, data, failure, and the interfaces between programs. It ends with a call to action for building systems holistically.