data

Juan Pedro Bolivar Puente explains immutable and persistent data structures in C++ and how they support value semantics. He presents the Immer library, rrb-trees, and a prototype text editor that opens a huge file instantly.

A five-minute lightning demo of VisiData, a terminal tool for exploring and manipulating data. It uses two months of NYC 311 complaint data to drill down, plot points on a virtual map and save selected rows.

Database pioneer Michael Stonebraker argues that 'big data' is really at least four different problems: big volumes with small analytics, big analytics, velocity, and variety. He comments on where each market is heading.

Peter Norvig's 2007 Google talk on theorizing from data, covering what huge datasets can do for language understanding, translation, information extraction and inference. It argues that more data often beats cleverer models.

Peter Norvig's UBC distinguished lecture on how massive amounts of simple data, such as for language modeling, can yield understanding that hand-built models could not.

Edward Kmett's 2013 talk at Prezi in Budapest on succinct data structures, which store data in close to the information-theoretic minimum space while supporting fast queries. It ties together the research and walks through his Haskell succinct library.

Jon Gjengset's 2020 PhD dissertation presentation at MIT on partial state in dataflow-based materialized views, the idea behind the Noria database. The Q&A is cut out.

Eager Space examines the practicality of putting data centers in orbit, walking through three scenarios with cost calculations and the launch cost reductions that would be required.

James Hamilton of AWS gives a Velocity 2010 keynote on datacenter infrastructure innovation, focused on power and cooling costs and a breakdown of where datacenter spending goes.

Michael Arntzenius's Strange Loop talk on Datafun, a functional query language that brings database-style querying into a general-purpose language. It discusses bridging database theory and programming languages.

Martin Kleppmann's Kafka Summit 2018 keynote asks whether Kafka can be considered a database. He covers event logs and related data-systems ideas.

SQLite creator Richard Hipp gives a history of SQLite and explains where it shines as a database at the edge of the network with no DBA. He covers long-term support, testing and how it is used in products.

Lukas Eder's Devoxx talk on how modern cost-based SQL optimizers rewrite and plan queries in ways developers would not expect. It argues that declarative SQL, used well, can run complex queries very fast.

Bryan Cantrill presents Manta, an object store with in-situ compute that lets tools like grep, awk, and sed run map-reduce style over huge data. He frames it as bringing the Unix philosophy to big data.

John Ousterhout's Netdev 0x16 keynote arguing that nearly every aspect of TCP is wrong for the datacenter, from connections and streams to reliable delivery and congestion control. He presents the case for a replacement protocol.

Martin Odersky's JVM Language Summit 2015 talk argues that compilers can be thought of as databases. It describes design ideas for the Scala compiler and for developer tools.

Raph Levien presents his new Rust GUI framework, built on data-oriented design and ideas from entity component systems, for the Xi editor. He weighs the approach against alternatives and discusses pros and cons.

Brian Bucklew of Freehold Games discusses design decisions behind Caves of Qud and Sproggiwood at IRDC 2015. The talk centers on data-driven, event-driven engine design.

Simon Peyton Jones explains nested data parallelism and Data Parallel Haskell, an approach to exploiting multicore and distributed hardware while keeping a simple single-control-flow programming model.

Steve Yegge's OSCON Data 2011 talk imagines what you could do with Google-scale storage and compute a decade later. He argues smart people should work on hard problems such as biology rather than cat pictures, and the talk ends with a moment many took as him quitting Google.

Yaron Minsky of Jane Street shows how the OCaml Incremental library lets trading UIs express data transformations that are easy to read and efficient on streaming data. It includes a live table demo.

A London Haskell talk on the algebra of algebraic data types, showing how sum and product types behave like arithmetic. It frames types as numbers obeying algebraic laws.

Zig creator Andrew Kelley's RedisConf 2020 talk on modeling data flow with asynchronous I/O in Zig. It explains how Zig's async/await avoids function coloring, and Loris Cro demonstrates OkRedis, a zero-allocation Redis client.

Jeff Hawkins presents his view that the neocortex works on principles different from traditional computers. He describes sparse distributed representations and how they are learned, then shows how Numenta applies them to predict from high-velocity data streams.

Mark Raasveldt of CWI presents DuckDB, an embeddable analytical database described as the SQLite for analytics, in a CMU Database Group talk. He explains why it was built, what it is for, and how it works.

Erik Demaine's MIT 6.851 Advanced Data Structures lecture introducing succinct data structures, covering rank, select and tries. It is part of the Spring 2012 course.

A short product demo of Dabble DB, a web tool for importing, entering and manipulating structured data with little effort. It was built in Smalltalk and later acquired by Twitter.

Avi Bryant demos Dabble DB, a web-based spreadsheet-like database for building apps without code. It shows how users can model and relate data through a smooth interface.

Journalist Matthew Cole's Black Hat 2013 talk on the CIA's operational security failures in the 2003 Milan kidnapping. Phone and location metadata exposed the team and led to arrest warrants.

Bryan Cantrill of Joyent presents Manta, an internet-facing object store that lets you run compute directly on the data. The talk is peppered with ZFS history and Cantrill's trademark commentary on the industry.

Peter Alvaro's Strange Loop 2015 talk argues that query languages let you state what you mean, and applies that to distributed programming. It covers Datalog, Dedalus and the Bloom lineage for reasoning about distributed systems.

David Kriesel's 36C3 talk, in an English translation, analyzes a year of scraped Deutsche Bahn data covering every stop of every long-distance train in Germany. It explores delays and how punctuality is actually measured.

Martin Kleppmann's Strange Loop talk on database transactions, covering why they were dropped by NoSQL, their reputation for being slow, and newer algorithms that make serializable transactions perform and scale. It touches on consistency, linearizability and the CAP theorem.

Dave Chinner's linux.conf.au 2012 talk on XFS scalability, covering pain points with large numbers of files and complex metadata, and the work to fix XFS metadata performance.

Ahmon Dancy of Franz Inc explains low-level programming in Common Lisp, including off-heap data structures used in AllegroGraph to manage memory outside the garbage collector.

A 2006 Google TechTalk in which SQLite's author Richard Hipp introduces the embeddable, zero-configuration C database library. He covers its history, strengths and weaknesses, and where it beats client/server databases.

Richard Feldman's elm-conf talk on designing models so that bugs from impossible application states cannot occur, using Elm's compiler and types. It demonstrates techniques for making invalid states unrepresentable.

A five-minute excerpt of Rich Hickey's Clojure Made Simple talk, with new slides, ranting about HttpServletRequest and specificity. He argues for using generic maps instead of many specific classes and getters.

Turing Award winner Michael Stonebraker gives the 2018 Donald B. Gillies Memorial Lecture at Illinois. He discusses the gap between academic research and industry problems, publishing incentives, and the influence of large industry users on research.

Bryan Cantrill's Uptime 2017 talk on the engineering challenges of the stateful data path, including storage, firmware and hardware bugs. It uses the 'zebras vs. horses' diagnostic heuristic to explain why rare failures keep appearing.