data modeling

Rich Hickey, creator of Clojure and designer of Datomic, gives a JaxConf 2012 talk that takes apart traditional database architecture. He presents a different way to think about databases and explains the reasoning behind Datomic's fact-based design.

Jon Gjengset's 2020 PhD dissertation presentation at MIT on partial state in dataflow-based materialized views, the idea behind the Noria database. The Q&A is cut out.

A five-minute excerpt of Rich Hickey's Clojure Made Simple talk, with new slides, ranting about HttpServletRequest and specificity. He argues for using generic maps instead of many specific classes and getters.

Martin Kleppmann argues that databases are global shared mutable state and proposes treating them as an ever-growing log of immutable facts. Derived data such as caches, indexes and views is then built by stream processors like Apache Samza. The talk draws on Kafka and LinkedIn's experience with commit logs.

Michael Arntzenius's Strange Loop talk on Datafun, a functional query language that brings database-style querying into a general-purpose language. It discusses bridging database theory and programming languages.

Martin Kleppmann's Kafka Summit 2018 keynote asks whether Kafka can be considered a database. He covers event logs and related data-systems ideas.

James Powell's nearly two-hour PyData talk builds up Python's data model, decorators, generators and context managers, coding live in Vim. It aims to take intermediate programmers toward real Python expertise.

Richard Feldman's elm-conf talk on designing models so that bugs from impossible application states cannot occur, using Elm's compiler and types. It demonstrates techniques for making invalid states unrepresentable.

Mike Acton of Insomniac Games argues that a program's purpose is transforming data, and shows how common C++ approaches work against that in performance-critical game development. He covers cache behavior, data layout and the limits of compilers.

Juan Pedro Bolivar Puente explains immutable and persistent data structures in C++ and how they support value semantics. He presents the Immer library, rrb-trees, and a prototype text editor that opens a huge file instantly.

Lee Byron of Facebook explains how persistent immutable data structures work and how to use them in React apps with Immutable.js. He covers memoization benefits and avoiding coupling through shared mutable state.

Peter Norvig's 2007 Google talk on theorizing from data, covering what huge datasets can do for language understanding, translation, information extraction and inference. It argues that more data often beats cleverer models.

John Ousterhout's Netdev 0x16 keynote arguing that nearly every aspect of TCP is wrong for the datacenter, from connections and streams to reliable delivery and congestion control. He presents the case for a replacement protocol.

Raymond Hettinger explains Python's dataclasses module, showing the problem it solves and its key design decisions. He places it in a progression from tuples and dicts through named tuples and attrs.

Edward Kmett's 2013 talk at Prezi in Budapest on succinct data structures, which store data in close to the information-theoretic minimum space while supporting fast queries. It ties together the research and walks through his Haskell succinct library.

James Hamilton of AWS gives a Velocity 2010 keynote on datacenter infrastructure innovation, focused on power and cooling costs and a breakdown of where datacenter spending goes.

Mark Raasveldt of CWI presents DuckDB, an embeddable analytical database described as the SQLite for analytics, in a CMU Database Group talk. He explains why it was built, what it is for, and how it works.

Rick Houlihan's AWS re:Invent 2018 deep dive on advanced DynamoDB design patterns. It covers index overloading, GSI sharding, composite keys, graph processing and transactions.

SQLite creator Richard Hipp gives a history of SQLite and explains where it shines as a database at the edge of the network with no DBA. He covers long-term support, testing and how it is used in products.

Lukas Eder's Devoxx talk on how modern cost-based SQL optimizers rewrite and plan queries in ways developers would not expect. It argues that declarative SQL, used well, can run complex queries very fast.

Bryan Cantrill presents Manta, an object store with in-situ compute that lets tools like grep, awk, and sed run map-reduce style over huge data. He frames it as bringing the Unix philosophy to big data.

A Clojure/conj 2019 talk by Wilker Silva on how namespace-qualified keywords and context-free federated names can be used pragmatically for data modeling, drawn from his experience at Nubank. It builds toward a graph-based view of connecting data from many sources.

An AWS re:Invent 2019 session for advanced DynamoDB users on design patterns used by large customers, including Amazon retail. It covers GSI sharding, index overloading, graph processing, composite keys and transactional workflows.

Simon Peyton Jones explains nested data parallelism and Data Parallel Haskell, an approach to exploiting multicore and distributed hardware while keeping a simple single-control-flow programming model.

Zig creator Andrew Kelley's RedisConf 2020 talk on modeling data flow with asynchronous I/O in Zig. It explains how Zig's async/await avoids function coloring, and Loris Cro demonstrates OkRedis, a zero-allocation Redis client.

Martin Fowler's GOTO 2012 introduction to NoSQL databases: where they came from, their data models, how consistency thinking differs, and when to use them. He argues they won't make relational databases obsolete and introduces polyglot persistence.

Peter Alvaro's Strange Loop 2015 talk argues that query languages let you state what you mean, and applies that to distributed programming. It covers Datalog, Dedalus and the Bloom lineage for reasoning about distributed systems.

Martin Odersky's JVM Language Summit 2015 talk argues that compilers can be thought of as databases. It describes design ideas for the Scala compiler and for developer tools.

Andrew Kelley, creator of Zig, presents practical data-oriented design techniques for shrinking memory footprint and making programs cache friendly. He ends with a case study of how these ideas sped up the Zig compiler.

Alexander Nix of Cambridge Analytica presents at the 2016 Concordia Summit on using big data, psychographic profiling and behavioral microtargeting in elections.

A CppCon 2018 talk arguing that OOP has failed to deliver on reuse and performance, and presenting data-oriented design for C++ with practical worked examples from real systems.

Zoz's DEF CON 23 talk explores practical, often explosive ways to destroy data and hard drives quickly. It tests methods such as thermite and other physical techniques for rapid destruction.

Chandler Carruth's CppCon 2016 talk on designing hybrid data structures for performance. He covers cache locality, working-set size and allocation overhead, and why standard C++ containers fall short.

Steve Yegge's OSCON Data 2011 talk imagines what you could do with Google-scale storage and compute a decade later. He argues smart people should work on hard problems such as biology rather than cat pictures, and the talk ends with a moment many took as him quitting Google.

Bryan Cantrill of Joyent presents Manta, an internet-facing object store that lets you run compute directly on the data. The talk is peppered with ZFS history and Cantrill's trademark commentary on the industry.

Martin Kleppmann's Strange Loop talk on database transactions, covering why they were dropped by NoSQL, their reputation for being slow, and newer algorithms that make serializable transactions perform and scale. It touches on consistency, linearizability and the CAP theorem.

Dave Chinner's linux.conf.au 2012 talk on XFS scalability, covering pain points with large numbers of files and complex metadata, and the work to fix XFS metadata performance.

An AWS re:Invent 2018 session on how DynamoDB was built as a hyper-scale NoSQL database. It discloses underpinnings of the service and how it is run for over 100,000 customers.

Ahmon Dancy of Franz Inc explains low-level programming in Common Lisp, including off-heap data structures used in AllegroGraph to manage memory outside the garbage collector.

A Google Tech Talk by Raymond Blum on how Google backs up massive amounts of data such as Gmail and Picasa. It covers scaling challenges, failure types, and approaches that did not work, including tape.