database engines

Rich Hickey, creator of Clojure and designer of Datomic, gives a JaxConf 2012 talk that takes apart traditional database architecture. He presents a different way to think about databases and explains the reasoning behind Datomic's fact-based design.

Mark Raasveldt of CWI presents DuckDB, an embeddable analytical database described as the SQLite for analytics, in a CMU Database Group talk. He explains why it was built, what it is for, and how it works.

SQLite creator Richard Hipp gives a history of SQLite and explains where it shines as a database at the edge of the network with no DBA. He covers long-term support, testing and how it is used in products.

Lukas Eder's Devoxx talk on how modern cost-based SQL optimizers rewrite and plan queries in ways developers would not expect. It argues that declarative SQL, used well, can run complex queries very fast.

A 2006 Google TechTalk in which SQLite's author Richard Hipp introduces the embeddable, zero-configuration C database library. He covers its history, strengths and weaknesses, and where it beats client/server databases.

Martin Kleppmann's Kafka Summit 2018 keynote asks whether Kafka can be considered a database. He covers event logs and related data-systems ideas.

Martin Odersky's JVM Language Summit 2015 talk argues that compilers can be thought of as databases. It describes design ideas for the Scala compiler and for developer tools.

Martin Kleppmann argues that databases are global shared mutable state and proposes treating them as an ever-growing log of immutable facts. Derived data such as caches, indexes and views is then built by stream processors like Apache Samza. The talk draws on Kafka and LinkedIn's experience with commit logs.

An AWS re:Invent 2018 session on how DynamoDB was built as a hyper-scale NoSQL database. It discloses underpinnings of the service and how it is run for over 100,000 customers.

Jon Gjengset's 2020 PhD dissertation presentation at MIT on partial state in dataflow-based materialized views, the idea behind the Noria database. The Q&A is cut out.

Brian Bucklew of Freehold Games discusses design decisions behind Caves of Qud and Sproggiwood at IRDC 2015. The talk centers on data-driven, event-driven engine design.

Martin Fowler's GOTO 2012 introduction to NoSQL databases: where they came from, their data models, how consistency thinking differs, and when to use them. He argues they won't make relational databases obsolete and introduces polyglot persistence.

Dropbox engineer Kevin Modzelewski gives a Stanford colloquium on Dropbox's history and the technical problems it faced scaling, including detailed database schema decisions.

Redis Labs and Tim Davis present RedisGraph, which uses GraphBLAS to treat graph queries in Cypher as sparse linear algebra for lower latency.

Martin Kleppmann's Strange Loop talk on database transactions, covering why they were dropped by NoSQL, their reputation for being slow, and newer algorithms that make serializable transactions perform and scale. It touches on consistency, linearizability and the CAP theorem.

Cuong Do of YouTube describes how a handful of engineers scaled the site, covering hardware, software and database scalability challenges during its rapid growth. The talk was given at the 2007 Seattle Conference on Scalability.

Michael Arntzenius's Strange Loop talk on Datafun, a functional query language that brings database-style querying into a general-purpose language. It discusses bridging database theory and programming languages.

Aaron Boodman of Replicache asks why sync engines have not yet fixed web app complexity and what is missing for them to go mainstream. The talk lays out the ideas that led to the Zero sync engine.

A Clojure/conj 2014 talk by Lucas Cavalcanti and Edward Wible on four uses of Datomic. They cover HTTP caching, audit trails, mobile database sync and authorization, drawing on the immutable transaction log.

Ahmon Dancy of Franz Inc explains low-level programming in Common Lisp, including off-heap data structures used in AllegroGraph to manage memory outside the garbage collector.

D. Richard Hipp's PGCon 2014 keynote on SQLite's relationship to PostgreSQL. He explains that SQLite was originally written from PostgreSQL 6.5 documentation and that the two engines serve different classes of problems.

Elasticsearch engineer and Lucene committer Adrien Grand explains the design decisions inside a Lucene index and what problems it can and cannot solve. The talk goes into how Lucene stores data for fast search and analytics.

Markus Winand shows how SQL has evolved since SQL-92, comparing old-style solutions to modern features such as window functions and system versioning, and notes that many developers never learn them.

Martin Kleppmann's Local-First Conf talk reviewing where the local-first idea came from, where it stands five years after the essay, and what the community should work toward. He argues for generic sync infrastructure.

Miguel de Icaza's GodotCon 2023 talk on Swift bindings for Godot. He argues that garbage-collector pauses in C# are a persistent problem and that Swift's reference counting avoids them.

Will Wilson of FoundationDB explains how the team tested their distributed database using deterministic simulation. By running the whole system in a simulator with injected faults and reproducible seeds, they could find rare concurrency and network bugs.

Martin Kleppmann's GOTO Berlin 2016 talk on how collaborative editors, offline apps, and multi-datacenter databases handle concurrent updates. It covers conflict resolution approaches including operational transformation and CRDTs, with a worked look at text CRDTs.

In this 2019 GDC session, Petri Purho of Nolla Games explains the technical side of Noita's physics engine, including scaling a falling-sand simulation to large continuous worlds and adding destructible rigid bodies. It also shows how the emergent physics drives the gameplay.

A Linear engineer talks about scaling the company's sync engine, which keeps client data local and synchronized. It follows an earlier talk on the same system.

Casey Muratori's remastered BICFest 2018 talk on how the collision system of The Witness was engineered to prevent whole classes of bugs found in game engines. It covers character movement and collision handling.

Rich Hickey's JaxConf 2012 keynote examines what values are and why immutable values matter as information systems grow more complex and data gets bigger. It contrasts places with values and builds the case for immutability.

Rick Houlihan's AWS re:Invent 2018 deep dive on advanced DynamoDB design patterns. It covers index overloading, GSI sharding, composite keys, graph processing and transactions.

Joran Dirk Greef's Systems Distributed '23 talk on TigerStyle, the engineering methodology behind TigerBeetle. It covers designing safer systems in less time through up-front design, assertions, and deterministic simulation.

A five-minute excerpt of Rich Hickey's Clojure Made Simple talk, with new slides, ranting about HttpServletRequest and specificity. He argues for using generic maps instead of many specific classes and getters.

Bryan Cantrill's Uptime 2017 talk on the engineering challenges of the stateful data path, including storage, firmware and hardware bugs. It uses the 'zebras vs. horses' diagnostic heuristic to explain why rare failures keep appearing.

Bryan Cantrill of Joyent presents Manta, an internet-facing object store that lets you run compute directly on the data. The talk is peppered with ZFS history and Cantrill's trademark commentary on the industry.

Peter Alvaro's Strange Loop 2015 talk argues that query languages let you state what you mean, and applies that to distributed programming. It covers Datalog, Dedalus and the Bloom lineage for reasoning about distributed systems.

Dan Robinson, then CTO of Heap, explains how Heap runs its product analytics backend on PostgreSQL and CitusDB. It covers conversion funnels, retention and cohort queries across tens of billions of events and 50 TB of data, plus Postgres and Kafka together and the indexing strategy.

Jeff Bonwick and Bill Moore, the creators of ZFS, present 'ZFS: The Last Word in File Systems' at the 2008 SNIA Software Developers' Conference. They explain the design, including copy-on-write, checksumming and avoiding the write hole.

Greg Young's Code on the Beach 2014 talk on CQRS and event sourcing. He argues mature domains are built on facts that occurred rather than current state, and covers the business and technical benefits.