Why Snow and Confetti Ruin YouTube Video Quality

To this day when I have buffering problems with online videos I immediately think about his explainer-demo with confetti [1]. Love his work! [1] ▶ this video

Yeah. And that's why TV stores really like slow-motion shots or static landscapes to show off the TV. Any motion will cause "HDTV blur" as the encoder struggles to describe complex motion with the limited number of bits it's allowed to use. Stuff like static, film grain, particles like snow or rain, those all suck up bits from the same encoding budget. "Why Snow and Confetti Ruin YouTube Video Quality" by Tom Scott probably explains it nicer than I can ▶ this video This could be a problem for video game streaming, and it could affect the artistic decisions a game studio makes - Drawing a billion tiny particles on a local GPU will look crisp and cool, but asking a hardware encoder to encode those for consumer Internet (or phone Internet) might be too much. I think streamers have run into this problem already.

Some of it is people ask the wrong questions. On a loudness-war-wrecked pop song I may not be able to tell 128Kbps from the original, but on specific content I have been able to tell. I'm not even claiming golden ears or anything; some specific audio content is the audio equivalent of visual confetti [1], and anyone can hear the difference, because the codec isn't even close. And let me underline, I mean, anyone . No special claims being made here. But all in all, that content is relatively rare, and generally transient even in the music they appear in. [1]: ▶ this video

It may be easier to think about high-quality swathes of pixels, because obviously any one pixel can't be wrong in isolation, and even to the extent it is, who cares. However, if you look at a swathe of pixels, say a 16x16 chunk, you can reasonably talk about with what fidelity that set of pixels represents the original signal, whatever that original signal was. You get into the whole perceptual coding thing which tells us that determining the human-perceived accuracy of a chunk of pixels is more than just adding up the differences, but you can take that as a suitable first approximation for thought. High-quality pixels have minimal deviation from the source material. Low-quality pixels have significant deviation. Tom Scott has a good video where turns up and down the quality live so you can see it in a single video: ▶ this video The video's resolution never changes, and you could say the video's bit rate never changes (it probably does because adaptive coding saves YouTube a fortune, but conceptually you could say it doesn't), but the quality of the pixels sure does.

YouTube's 1080p60 is already at a decimation ratio of about 200:1, then you have to consider how efficient P and B frames are with motion/differences. if your data looks like noise you're gonna be completely screwed since the P and B frames will absolutely destroy the quality. There's a bunch of other things too, like YUV420p and TV colour range: 16-235, so you only get 7.7bits / pixel. If anything you would want to encode your data in some way that abuses the P and B frames, and the macro block size of 16x16. Coding theory for the data output at your end is only one side of the coin, the VP9 codec stupidly good compression is a completely different game to wrangle. And I kinda doubt you'll get much better than your estimate of 1% from the original scheme. ▶ this video

That's the thing though, "looks like" breaks down pretty quickly with things that aren't real images. It even breaks down pretty quickly with things that are real, but maybe not so common, images: ▶ this video So one question would be: Does your image generation approach preserve a higher information density than big enough pixels?

I do not know the answer, but I would absolutely shocked if it was streamed as a regular video. To the best of my knowledge, there are many, many optimizations which go into video. This is a pretty good high-level video I came across - ▶ this video

Quite a few. Though to be honest, 4k is currently good enough for me, resolution wise. But: even the highest bitrates you get on eg YouTube ain't good enough to show certain scenes nicely. Especially when snow is involved. That's because the video compression algorithms we have don't cope well with that. See ▶ this video So there's a clear and simple use case for more bandwidth, even if we don't increase resolution.

For 4k video, H.265/VP9 have better quality at the same file size, or equivalent quality at a smaller file size. Bandwidth usage matters for keeping people on the website and viewing Google's ads, so Google wants to ensure a smooth streaming experience. And since YT compresses so heavily, uploading in 4k increase the quality at lower resolutions (it seems weird but a 4k video uploaded to YT and streamed at 1080p will look better than a 1080p video uploaded to YT and streamed at 1080p). And as xnyan said, high motion video really destroys video quality on YT, this video really shows how bad it can get: ▶ this video

You can make educated guesses about the future, which is how Oculus's Asynchronous Spacewarp works. Rendering a whole frame is slow, but warping a pre-rendered frame is fast. If the next frame is taking too long to render, you can warp the last frame to roughly match the perspective that corresponds with the current head tracking data. You get some artefacts, but they're not as noticeable as the judder caused by a missed frame. Prediction can also be used to estimate the head-tracking data at the time the frame is drawn to the display, rather than at the time the frame starts to render. Similar techniques are used in video compression - encoding the exact value of every pixel is expensive, but you can trade bandwidth for processing by encoding transformations of a previous frame. A modern compressed video consists mainly of these interpolated frames, with only a minority of frames containing a full image. This interpolation can use data from both past and future frames (B frames) but can also use just the data in previous frames (P frames). This works extremely well most of the time, but there are some edge cases: ▶ this video link