filesystems

Dave Chinner's linux.conf.au 2012 talk on XFS scalability, covering pain points with large numbers of files and complex metadata, and the work to fix XFS metadata performance.

Tom Caputi presents the design of native encryption for ZFS at the OpenZFS Developer Summit, including design decisions and how authentication uses checksum fields.

Marshall Kirk McKusick gives a talk at EuroBSDCon tracing filesystem and storage development from 1979 to today, using the BSD Fast Filesystem as the focus. It covers early disk-geometry optimization through later contiguous layout and small-file performance work.

Allan Jude of Klara presents work on making ZFS scale on fast NVMe storage, covering fixes to the TaskQueue interface and removal of bottlenecks. The talk was given at EuroBSDcon 2022 and describes raising write throughput from about 3 to over 7 GB/sec.

An OpenZFS office hours session with ZFS co-creator Matt Ahrens answering questions on topics such as hole birth time, send/receive buffers, device removal, block pointer rewrite and performance with many snapshots.

Kent Overstreet's Linux Foundation talk about writing a filesystem in Rust. Its Q&A segment, where Rust-for-Linux developers ask about interface semantics, became widely cited.

Will Wilson of FoundationDB explains how the team tested their distributed database using deterministic simulation. By running the whole system in a simulator with injected faults and reproducible seeds, they could find rare concurrency and network bugs.

Rich Hickey's JaxConf 2012 keynote examines what values are and why immutable values matter as information systems grow more complex and data gets bigger. It contrasts places with values and builds the case for immutability.

Rick Houlihan's AWS re:Invent 2018 deep dive on advanced DynamoDB design patterns. It covers index overloading, GSI sharding, composite keys, graph processing and transactions.

Martin Kleppmann argues that databases are global shared mutable state and proposes treating them as an ever-growing log of immutable facts. Derived data such as caches, indexes and views is then built by stream processors like Apache Samza. The talk draws on Kafka and LinkedIn's experience with commit logs.

Martin Kleppmann's GOTO Berlin 2016 talk on how collaborative editors, offline apps, and multi-datacenter databases handle concurrent updates. It covers conflict resolution approaches including operational transformation and CRDTs, with a worked look at text CRDTs.

The first lecture of Andy Pavlo's CMU 15-445 Intro to Database Systems (Fall 2019), covering course introduction and the relational model.

Martin Fowler's GOTO 2012 introduction to NoSQL databases: where they came from, their data models, how consistency thinking differs, and when to use them. He argues they won't make relational databases obsolete and introduces polyglot persistence.

Joran Dirk Greef's Systems Distributed '23 talk on TigerStyle, the engineering methodology behind TigerBeetle. It covers designing safer systems in less time through up-front design, assertions, and deterministic simulation.

A five-minute excerpt of Rich Hickey's Clojure Made Simple talk, with new slides, ranting about HttpServletRequest and specificity. He argues for using generic maps instead of many specific classes and getters.

Bryan Cantrill's Uptime 2017 talk on the engineering challenges of the stateful data path, including storage, firmware and hardware bugs. It uses the 'zebras vs. horses' diagnostic heuristic to explain why rare failures keep appearing.

Bryan Cantrill of Joyent presents Manta, an internet-facing object store that lets you run compute directly on the data. The talk is peppered with ZFS history and Cantrill's trademark commentary on the industry.

Database pioneer Michael Stonebraker argues that 'big data' is really at least four different problems: big volumes with small analytics, big analytics, velocity, and variety. He comments on where each market is heading.

Peter Alvaro's Strange Loop 2015 talk argues that query languages let you state what you mean, and applies that to distributed programming. It covers Datalog, Dedalus and the Bloom lineage for reasoning about distributed systems.

Dan Robinson, then CTO of Heap, explains how Heap runs its product analytics backend on PostgreSQL and CitusDB. It covers conversion funnels, retention and cohort queries across tens of billions of events and 50 TB of data, plus Postgres and Kafka together and the indexing strategy.

Dropbox engineer Kevin Modzelewski gives a Stanford colloquium on Dropbox's history and the technical problems it faced scaling, including detailed database schema decisions.

Jeff Bonwick and Bill Moore, the creators of ZFS, present 'ZFS: The Last Word in File Systems' at the 2008 SNIA Software Developers' Conference. They explain the design, including copy-on-write, checksumming and avoiding the write hole.

Greg Young's Code on the Beach 2014 talk on CQRS and event sourcing. He argues mature domains are built on facts that occurred rather than current state, and covers the business and technical benefits.

SQLite creator Richard Hipp gives a history of SQLite and explains where it shines as a database at the edge of the network with no DBA. He covers long-term support, testing and how it is used in products.

Rich Hickey, creator of Clojure and designer of Datomic, gives a JaxConf 2012 talk that takes apart traditional database architecture. He presents a different way to think about databases and explains the reasoning behind Datomic's fact-based design.

The author of copyparty introduces the FOSS file server he has built over five years, demoing its features such as resumable uploads, WebDAV and deduplication. The video itself is shown running on a demo server.

A three-hour PyCon 2016 tutorial by Christophe Pettus that orients Python developers to PostgreSQL. It covers installation, administration, and use, including vacuuming, MVCC, replication and backups.

Redis Labs and Tim Davis present RedisGraph, which uses GraphBLAS to treat graph queries in Cypher as sparse linear algebra for lower latency.

Turing Award winner Michael Stonebraker gives the 2018 Donald B. Gillies Memorial Lecture at Illinois. He discusses the gap between academic research and industry problems, publishing incentives, and the influence of large industry users on research.

Mark Raasveldt of CWI presents DuckDB, an embeddable analytical database described as the SQLite for analytics, in a CMU Database Group talk. He explains why it was built, what it is for, and how it works.

Elasticsearch engineer and Lucene committer Adrien Grand explains the design decisions inside a Lucene index and what problems it can and cannot solve. The talk goes into how Lucene stores data for fast search and analytics.

Bryan Cantrill presents the ARC (Adaptive Replacement Cache) paper by Megiddo and Modha at Papers We Love. He opens with a survey of caching strategies before digging into the self-tuning LRU/MFU design.

Avi Bryant demos Dabble DB, a web-based spreadsheet-like database for building apps without code. It shows how users can model and relate data through a smooth interface.

Konrad Malawski's Strange Loop 2023 talk shows how the FoundationDB team adopted Swift incrementally in a C++ codebase. It covers Swift's C++ interoperability, which let them move single functions or entire types over one at a time.

SQLite author Richard Hipp's talk at SouthEast LinuxFest criticizing Git's workflow and introducing Fossil as an alternative. He lists Git enhancements and argues for keeping history intact.

Lukas Eder's Devoxx talk on how modern cost-based SQL optimizers rewrite and plan queries in ways developers would not expect. It argues that declarative SQL, used well, can run complex queries very fast.

Scott Wlaschin's NDC Oslo talk on four 1970s languages, SQL, Prolog, ML and Smalltalk. It explains their philosophies and what practical principles they still offer modern developers.

Markus Triska tours Prolog's most distinctive features, including its use as a database and query language, rule processing, constraint solving and compile-time rewriting. It also covers concise meta-interpreters.

Bryan Cantrill presents Manta, an object store with in-situ compute that lets tools like grep, awk, and sed run map-reduce style over huge data. He frames it as bringing the Unix philosophy to big data.

Martin Kleppmann's Strange Loop talk on database transactions, covering why they were dropped by NoSQL, their reputation for being slow, and newer algorithms that make serializable transactions perform and scale. It touches on consistency, linearizability and the CAP theorem.