Shouting in the Datacenter

i just deleted a many-paragraph response that was getting long-winded(incoherent) and started over -- yes, i concur. like many, i dislike the "i want to help save the world" generic posts, but i think most here are being quick to dismiss TFA. to others (not the parent): just want to help place a human face onto this guy, who has (not known to me until now) not only published a bunch of works (whose titles i find very relevant to me as a sysadmin), but was the guy in front of the camera in the Shouting in the Datacenter clip[0] !! things are difficult, but i hang on to hope. and undoubtedly, i have tons to learn from this guy. [0]: ▶ this video

I believe it because it's a plausible variant of what I call the "Fus Ro Data Loss" vulnerability: shouting at hard drives causes them to resonate in a way that affects their ability to access data. ▶ this video

In the photo of him on his last day [0], there's a cassette deck on his desk. That could be something mundane, but I'd like to believe something crazy happens if you yell at it [1]... [0] link [1] ▶ this video

I recall a video of a guy temporarily reducing hard drive performance by shouting at it edit: here it is! ▶ this video

You clearly haven't read the full article as Jim Salter writes about the mechanical stuff at the end of the article. Also, you want to reduce vibrations because of this: ▶ this video (Shouting in the datacenter)

Didn't know Brendan Gregg was at Intel now. link He is also the star of the Shouting in the Data Center viral video ▶ this video True genius!

> For example, imagine an evil SSD which had a 1% chance of rolling a sector back to a previous version, a 1% chance of saying a write failed when it didn't, a 1% chance of writing data to the wrong sector number, a 1% chance of flipping some bits in the written data, and a 1% chance of disconnecting and reconnecting a few seconds later. There are stories from the ZFS folks of dealing with these issues and things ran just fine. While not directly involved with ZFS development (IIRC), Bryan Cantrill was very 'ZFS-adjacent' since he used Solaris-based systems for a lot of his career, and he has several rants about firmware that you can find online. A video that went viral many years ago, with Cantrill and Brendan Gregg, is "Shouting in the Datacenter": * ▶ this video * link (making of)

I had the "vibration" problem a few times in me IT career, train, construction work and once elevators. It is the kind of issue that are really hard to diagnose the first time but after the first time it's part of the standard "diagnose suite". I love to show this video about the effect of noise on HDD (be sure to put the volume down) ▶ this video

As long as you don't traumatize the hard drives by yelling at them, feat. brendangregg & bcantrill: ▶ this video

One of the little gems I found while I was putting together the video for this post was "Shouting in the datacenter", an older clip (from the pre-HD era) demonstrating the consequence of doing exactly what the title says: ▶ this video Warning: loud!

You might not notice a drive being a tiny bit louder/hotter, but you might notice if you've got an entire NAS rack full of 'em. Don't shout at your JBODs: ▶ this video

It also works the other way around: sound will make harddrives malfunction. This means that one could theoretically use a harddrive as a microphone! See ▶ this video for a demo, yelling at harddrives increases their latency.

There is a bit of precedent in Bryan Cantrill's 2008 video of Brendan Gregg shouting at some disks and inducing latency: ▶ this video (HN discussion last year: link ) But yeah, a) turning it from simply detecting that loud sound is happening to getting a meaningful audio channel out of the sound and b) treating this as a security concern instead of just a performance one (and flipping the direction of the attack, from sound attacking drive performance to drives attacking sound privacy) is novel, AFAIK.

Interesting. I just watched a video yesterday about how you could see screaming on hard drive stats in a data center: ▶ this video

That's a reposted video, the original is from Bryan Cantrill (formerly of Sun, now CTO of Joyent): ▶ this video

The paper links to this entertaining YouTube video from 2009, in which someone demonstrates the effect by literally shouting at a rack of servers: ▶ this video (EDIT: replaced the link I found via YouTube's search with the original, thanks robotmlg!)

One thing that I learned from him and have never forgotten is that you can reduce a harddisk (the spindle kind) performance just by screaming at it ▶ this video

This didn't surprise me, only because the video of the guy screaming at the hard drive has been stuck in my head for years -- ▶ this video

+1, in general. To adda bit of content: 1) Power Hungry, the total power envelope of a rack position is the limiting issue. You can fit 16-20 disks per RU no problem. The constraint on total density is your 5kVa or 10kVa power budget. Watts matter. 3) sensitive to vibration, be very very quiet ▶ this video 4) Density, actually 4TB is hitting the sweet spot for $ per byte last time I looked. If you need absolute density look for 5 & 6TB Real Soon Now. If you can tolerate some loss variable density disks around 5TB look quite a bit more cost effective. 5) Read ahead, you only need to read ahead by a couple of chunks. For video ball park it at 2MB. You don't really need lots of RAM, think 32-64GB per chassis. Additionally each 8GB dimm costs about the same power as a disk. By going with 32GB instead of 64 I can fit another 2-3 disks in the power budget per chassis.

Also interesting: a Sun engineer uses dtrace to show HDD latency spiking after he yells at his disks ▶ this video