FreeBSD optimizations used by Netflix serving at 800Gb/s - Drew Gallatin - EuroBSDcon 2022

Netflix discusses their reasons starting at 18:20: link tl;dw: the performance, the efficiency of development, the community, FreeBSD is a complete operating system, the code base is smaller, the ports system, and the license. and this video covers the optimizations Netflix has made to FreeBSD: ▶ this video Also potentially a reason: According to drewg123, Linux's kTLS was broken. Which I see drewg123 also commenting in this thread. Is he the "Drew on my team" mentioned in the first video? Is he the speaker in the 2nd video? Idk link

OP mentions that "I would love to be a little mouse and peek at YouTube’s complete architecture to see how far we are from them." You can occasionally find posts -often linked here- from another player in streaming video which you might have heard of, discussing technical architecture. For example, this might be a little lower level that you may be interested in as it relates to kernel optimizations to jack bit throughput rates, but I dig this sort of thing - ▶ this video

> Ethernet is stuck on 1500 byte frames, with the result of limiting TCP as well. That's just nuts on 100G or 400G networks, even on 10G it's a bit of a stretch already. How limiting is 1500 for the end-nodes for most people? For access-, distribution-, and core-layer switches the ASICs process things at wire speed, so does it matter what the frame size is? I can see some server applications, but we've had offloading for years now: * link Netflix is hitting (encrypted) 800 Gb/s: * ▶ this video * link > TCP is also sticking around for too long. Almost nothing wants a stream these days, lots and lots of applications reinvent datagrams on top of it. Tallk to the vendors of middleware boxes (e.g., (CPE) firewalls): we've had (say) SCTP and DCCP for twenty years that offer variations, but they're never allowed to pass through. No new Layer 4 can really be allowed through because of ossification, and so kludging everything to TCP/UDP port 80/443 is what we're left with. NAT and a lack of end-to-end also doesn't help: one has to futz around with things like STUN and TURN.

This talk was given at this years EuroBSDcon in Vienna, recording is up on YouTube. link ▶ this video Some really great talks this year from all the *BSDs, highly recommend checking a look: link

> How did you generate those flamegraphs and what other tools did you use to measure performance? We have an internal shell script that takes hwpmc output and generates flamegraphs from the stacks. It also works with dtrace. I'm a huge fan of dtrace. I also make heavy use of lockstat, AMD uProf, and Intel Vtune. > Did the graph show the bottleneck contention on aio queue? Did the graph show that "a lot of time was spent accessing memory"? See the graph on page 32 or so of the presentation. It shows huge plateaus in lock_delay called out of the aio code. Its also obvious from lockstat stacks (run as lockstat -x aggsize=4m -s 10 sleep 10 > results.txt) See the graph on page 38 or so. The plateaus are mostly memory copy functions (memcpy, copyin, copyout). We already use FreeBSD on our CDN, so it just made sense to do the work in FreeBSD. The talk is on Youtube ▶ this video