NYCBSDCon 2014: Serving one-third of the Internet via FreeBSD

Pretty sure it's still UFS+J. They've mentioned in the past that the benefits of ZFS don't really apply to the CDN boxes: if a drive dies, it's simply removed from the local cache, and files are re-downloaded/redistributed as necessary across the remaining cache drives the next time the appliance refreshes from its upstream origin. Additionally, ZFS prefetching was actually less performant in some cases from what I remember -- or at least providing no benefit -- because of how it couldn't really anticipate the I/O patterns of video segment requests from users. Additional reference: - link ("…but caching is futile.") - ▶ this video

Netflix delivers all video streaming content via FreeBSD which makes up 37% of all internet traffic. link They also contribute to FreeBSD core. It is not just an "appliance" as some people seem to want to point to. link link ▶ this video In addition, a guy I met, who worked at Yahoo, told me about 25% of Yahoo still runs FreeBSD as of two years ago.

Scott Long has summarized it a few times, eg, from link : • Availability of expertise, outstanding community • Works well, good vendor support • No GPL • Features used: – SUJ – gmirror – boot drive only – AIO – Dtrace, HWPMC – TCP Stack, modular CC It's missing "Integrates tightly with nginx: Async I/O and sendfile()" which Scott included in ▶ this video . In the video he elaborates a bit more about the GPL... edit: formatting

Actually it turns out they don't really have a need for most of the features of ZFS. So they don't use it. ▶ this video

> I'm assuming whatever file system is on the disks... Netflix uses UFS+J on FreeBSD 10. Here are some notes from their talk at NYCBSDCon 2014: - 400,000 stream files per appliance. - 5,000 - 25,000 client streams per appliance. - 300 - 500 streams coming off each disk all the time. - Attempt to buffer 1MB ahead, but caching is futile. - Result is completely random disk workload. - System becomes limited by disk latency and CPU load. Video here: ▶ this video