HackerTube
HomeLongsLikedHistoryWatch laterCategoriesPlaylistsSettings
Playlists

Systems and databases

Curated by Claude · 12 videos · 11 hr 32 min

These are the talks Hacker News points to when someone wants to know how real systems work underneath. It opens with a friendly story about scaling Dropbox. Then it moves through databases and storage, consensus and distributed data, running at scale, and performance. It ends with what happens when systems fail and how the people running them cope.

Play all on YouTube
  1. 1
    How We've Scaled Dropbox1:08:17

    How We've Scaled Dropbox

    372K views • 14 years ago
    Stanford
    Dropbox engineer Kevin Modzelewski gives a Stanford colloquium on Dropbox's history and the technical problems it faced scaling, including detailed database schema decisions.

    A good way in: a Dropbox engineer frankly walks through growing the service from nothing to millions of users, schema decisions included, and one commenter found it one of the most riveting things ever.

  2. 2
    SQLite: The Database at the Edge of the Network with Dr. Richard Hipp59:58

    SQLite: The Database at the Edge of the Network with Dr. Richard Hipp

    82K views • 11 years ago
    Method
    SQLite creator Richard Hipp gives a history of SQLite and explains where it shines as a database at the edge of the network with no DBA. He covers long-term support, testing and how it is used in products.

    The creator of the world's most widely used database explains where SQLite fits, how it is tested and why it gets long-term support, with an energy commenters enjoy as much as the content.

  3. 3
    AWS re:Invent 2018: Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB (DAT401)59:56

    AWS re:Invent 2018: Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB (DAT401)

    468K views • 7 years ago
    Amazon Web Services
    Rick Houlihan's AWS re:Invent 2018 deep dive on advanced DynamoDB design patterns. It covers index overloading, GSI sharding, composite keys, graph processing and transactions.

    Rick Houlihan's dense session is HN's go-to for learning to design data around access patterns, and a clear overview of how NoSQL thinking differs from SQL.

  4. 4
    "Turning the database inside out with Apache Samza" by Martin Kleppmann47:43

    "Turning the database inside out with Apache Samza" by Martin Kleppmann

    205K views • 12 years ago
    Strange Loop Conference
    Martin Kleppmann argues that databases are global shared mutable state and proposes treating them as an ever-growing log of immutable facts. Derived data such as caches, indexes and views is then built by stream processors like Apache Samza. The talk draws on Kafka and LinkedIn's experience with commit logs.

    Kleppmann reframes a database as a log of immutable facts with caches and indexes derived from it, the talk commenters say changed how they think about derived data.

  5. 5
    Paxos lecture (Raft user study)1:06:34

    Paxos lecture (Raft user study)

    114K views • 13 years ago
    Diego Ongaro
    John Ousterhout's slides-and-voiceover lecture on Paxos, made for the Raft user study comparing how students learn Raft and Paxos. It is a full introduction to the Paxos consensus algorithm.

    On to consensus: John Ousterhout's clear walkthrough of Paxos and Multi-Paxos, recorded for the Raft study and best watched alongside its Raft companion lecture.

  6. 6
    Conflict Resolution for Eventual Consistency • Martin Kleppmann • GOTO 201651:36

    Conflict Resolution for Eventual Consistency • Martin Kleppmann • GOTO 2016

    46K views • 9 years ago
    GOTO Conferences
    Martin Kleppmann's GOTO Berlin 2016 talk on how collaborative editors, offline apps, and multi-datacenter databases handle concurrent updates. It covers conflict resolution approaches including operational transformation and CRDTs, with a worked look at text CRDTs.

    When replicas disagree instead of agreeing on one leader, this approachable primer on operational transformation and CRDTs is what HN keeps linking in local-first threads.

  7. 7
    "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson40:20

    "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson

    55K views • 12 years ago
    Strange Loop Conference
    Will Wilson of FoundationDB explains how the team tested their distributed database using deterministic simulation. By running the whole system in a simulator with injected faults and reproducible seeds, they could find rare concurrency and network bugs.

    FoundationDB ran its whole database inside a simulator with injected faults to find rare bugs, and HN treats this as the defining talk on simulation testing.

  8. 8
    Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp49:15

    Erlang Factory 2014 - That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp

    50K views • 12 years ago
    Erlang Solutions
    Rick Reed of WhatsApp explains at Erlang Factory 2014 how the service scaled to billions of messages with a very small engineering team. He covers the FreeBSD and Erlang stack, Mnesia usage, server counts and bottleneck work.

    Rick Reed shows how a handful of engineers ran WhatsApp's message load on FreeBSD and Erlang, with real server counts and the bottlenecks they fixed.

  9. 9
    AWS re:Invent 2016: Tuesday Night Live with James Hamilton1:33:27

    AWS re:Invent 2016: Tuesday Night Live with James Hamilton

    133K views • 9 years ago
    Amazon Web Services
    James Hamilton's Tuesday Night Live keynote at AWS re:Invent 2016, touring AWS infrastructure at scale including networking, power and custom silicon. It also features NASA JPL's Tom Soderstrom.

    Moving down to the physical layer, Hamilton tours how AWS builds its networking, power and custom silicon, which HN considers the best look inside a cloud provider.

  10. 10
    CppCon 2017: Matt Kulukundis “Designing a Fast, Efficient, Cache-friendly Hash Table, Step by Step”59:34

    CppCon 2017: Matt Kulukundis “Designing a Fast, Efficient, Cache-friendly Hash Table, Step by Step”

    137K views • 9 years ago
    CppCon
    Matt Kulukundis explains how Google designed SwissTable, a fast, cache-friendly hash table, step by step from std::unordered_map. The talk covers open addressing, SIMD probing and the tradeoffs of changing a fundamental data structure.

    Performance at the scale of one data structure: the step-by-step story of SwissTable, the design that now sits behind Abseil, Rust's HashMap and Go's maps.

  11. 11
    "How NOT to Measure Latency" by Gil Tene42:59

    "How NOT to Measure Latency" by Gil Tene

    119K views • 11 years ago
    Strange Loop Conference
    Gil Tene of Azul Systems explains common pitfalls in measuring latency and response time, including why averages mislead and the problem of coordinated omission. He introduces open source tools like HdrHistogram for better measurement and reporting.

    Before trusting any benchmark, learn why averages mislead and what coordinated omission is; some commenters call this talk life-changing.

  12. 12
    Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 201752:35

    Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017

    101K views • 9 years ago
    GOTO Conferences
    Bryan Cantrill's GOTO Chicago 2017 talk on debugging production systems under pressure, centered on the incident where an operator's mistyped command rebooted an entire Joyent datacenter. It covers human factors, automation and architecture.

    Cantrill recounts the day a mistyped command rebooted a whole Joyent datacenter, which HN remembers as the best postmortem it has seen, and a funny one.