All of our data is GONE!
Linus Tech Tips1 HN recommender · recommended in 1 month, 2019-2021
9.4M views · 10 years ago · commentary · evergreen
Linus Tech Tips documents a major RAID failure on its Whonnock server and the attempt to recover the lost data with a data recovery firm.
HN refers to it as an honest, candid account of the common story of losing precious data to storage failure.
1:14:19NYC Tech Talk Series: How Google Backs Up the Internet
Google TechTalks
41K views • 12 years ago
17:37My Framework Investment Should NOT Have Worked Out
Linus Tech Tips
2.4M views • 1 year ago
12:39I Hope Google Doesn’t Ban Us... - Abusing Unlimited Google Drive
Linus Tech Tips
5.8M views • 8 years ago
16:56Fixing Apple's Engineering in an Hour
Linus Tech Tips
3M views • 6 years ago
10:20Dev Deletes Entire Production Database, Chaos Ensues
Kevin Fang
3.6M views • 3 years ago
3:55Velocity 2011: Artur Bergman, "Artur on SSD's"
O'Reilly
30K views • 15 years ago
22:51Dell SCAMMED Me - $1500 PC Secret Shopper 2 Part 4
Linus Tech Tips
3.8M views • 5 years ago
5:22Lest We Remember: Cold Boot Attacks on Encryption Keys
pucitp
749K views • 18 years ago
14:19This WAS under NDA... Until I Called in a Favor
Linus Tech Tips
1.4M views • 2 years ago
53:09DEF CON 23 - Zoz - And That's How I Lost My Other Eye...Explorations in Data Destruction (Fixed)
DEFCONConference
533K views • 11 years ago
15:59I’m Legally Obligated to Disclose This - Framework investment
Linus Tech Tips
3.3M views • 5 years ago
28:21Velocity 2012: Richard Cook, "How Complex Systems Fail"
O'Reilly
60K views • 14 years ago
4:13Samsung SSD Awesomeness
ssdawesome
4.9M views • 17 years ago
1:17SSI's Shred of the Month: E-Scrap - Hard Drives Shredding (Q)
bigshredder
966K views • 20 years ago
32:30Benno Rice: The Tragedy of systemd -- BSDCan 2018
BSDCan
25K views • 8 years ago
17:39Linus builds Linus’ new PC!
Linus Tech Tips
3.7M views • 6 years ago
3:39AWS re:Invent 2016: Move Exabyte-Scale Data Sets with AWS Snowmobile
Amazon Web Services
238K views • 9 years ago
46:37Zebras All the Way Down - Bryan Cantrill, Uptime 2017
Code & Supply
49K views • 9 years ago
36:47Harder Drive: Hard drives we didn't want or need
suckerpinch
2.1M views • 4 years ago
53:43AWS re:Invent 2018: Amazon DynamoDB Under the Hood: How We Built a Hyper-Scale Database (DAT321)
Amazon Web Services
112K views • 7 years ago
4 comments from Hacker News
Sebastian and Linus Media Group seem to be pretty frank and open about a lot of this: see for instance link on putting up gurning thumbnails and navigating the YT algorithm (and that whole playlist, including a revenue breakdown link ), the Gamers Nexus video on LTT's production process link and the coverage LTT regularly lavishes on its own production setup and process, from a nearly-12m video about the awkwardness of ingesting video from their new 12K cameras link to seemingly quite genuine fly-on-the-wall coverage of temporarily losing their data to a RAID failure ▶ this video . Obviously you always have to take anyone's self-presentation with a grain of salt. But they're certainly not hiding the fact that there's a more grounded reality behind the "on" personas.
Video for those interested: ▶ this video Actually a pretty good telling of a story all too common for those tasked with handling precious data.
I'm not sure if you're referring to ▶ this video ("All of our data is gone"), but in any case, I've heard references to ZFS in later videos so yeah, the situation's probably not quite that insane now. That video is technically hilarious though, it TL;DRs as "download Ubuntu-based USB image provided random RAID recovery company, boot USB and provide said company headless SSH to broken system"... at one point in the video I remember Linus talks to another recovery company about the process and the guy pretty much went "...O.o..." which IMO sums my own opinion up, lol
I really like the idea behind storing error-correction codes at the application level. Disks should just store and retrieve a chunk of bits and it should be pushed up to the application level to decide whether or not those bits were what we wrote. From there, we can make the right decisions about how durable we really want a file to be. Instead, people seem to have invested a lot of effort in going the other direction; let the application have less control and let many magic layers in the middle figure out the details. We have had filesystems for decades, which pretend to map names to a list of durable block numbers. Of course, the blocks are not so durable (nor is the database), so people lose their data all the time and act like it's normal. People have a solution to that problem, though, make the block devices more durable! We can just add software or hardware RAID on top of a bunch of disks, and now suddenly what the filesystem thinks is just one block is now 3 blocks sharded across 3 disks. Some magical curator process then tells me when disks have failed (as though it can detect that with 100% reliability), and I pop in a new drive and magically everything has been repaired. Of course, in reality, the entire disk doesn't just disappear all at once; some blocks just return data with a flipped bit, or all 0s, or an occasional error. My experience in running RAID-1 arrays under Linux (for my workstation) is that I notice some errors in the logs and have to manually figure