Enhancing Photorealism Enhancement

Researchers trained a deep learning network on photos to make GTA5 more realistic-looking: ▶ this video I was surprised that most of it is is just tone mapping. It looks too green-ish to me, needing some white balancing, but realistic. I think Video games are intentionally over-saturated and contrasty to make them look better, just as paintings are[1]. I took many screenshots in Horizon Zero Dawn that would be look bland if it was realistic like the modfified GTA5 footage above[2]. You also don't want animation movies[3] with realistic tone mapping. But for a horror game you would do exactly that, it's about immersion (or you would never be frightened), not beauty. [1] link [2] link [3] link

I think for the near future we’ll see something like this: ▶ this video From a few years ago, where the game is rendered traditionally and used as a ground truth, with a model on top of it that enhances the graphics. After maybe 10-15 years we will be past the point where the entire game can be generated without obvious mistakes in consistency. Realtime AI dialogue is already possible but still a bit primitive, I wrote a blog post about it here: link

> AI will be doing graphics programming? I doubt that. I think the bigger concern (if you could call it that) is that certain AI techniques may be able to replace entire stacks once they get advanced/stable enough. Something along the lines of this perhaps, where you render simplified geometry and let the AI fill in the blanks: ▶ this video Even without experimental technologies, you can get a glimpse of how shipped tools like ray reconstruction can morph the field in the future, forever entrenching themselves into graphics programming one way or another. As far as AI writing graphics code? No way, at least not in the next couple of decades. Button snippets are a far cry from rendering virtualized geometry 60 times a second.

Video is really a series of frames, the framerate for film/human can get away with 24 frames/second-- so maybe ~40ms/image for real-time at least? What's cool about the era in which we live is if you look at high-performance graphics for games or simulations, for instance, it may in fact be faster to a the model to "enhance" a low-resolution frame rather than trying to render it fully on the machine. ex. AMD's FSR vs NVIDIA DLSS - AMD FSR (Fidelity FX Super Resolution): link - NVIDIA DLSS (Deep Learning Super Sampling): j link AMD's approach renders the game at a crummy, low-detail resolution then each frame uses "upscales" Both FSR and DLSS aim to improve frames-per-second in games by rendering them below your monitor’s native resolution, then upscaling them to make up the difference in sharpness. Currently, FSR uses spatial upscaling, meaning it only applies its upscaling algorithm to one frame at a time. Temporal upscalers, like DLSS, can compare multiple frames at once, to reconstruct a more finely-detailed image that both more closely resembles native res and can better handle motion. DLSS specifically uses the machine learning capabilities of GeForce RTX graphics cards to process all that data in (more or less) real time. Video is really a series of frames, the framerate for film/human could get away with 24 frames/second-- ~40ms/image for real-time. What's cool about the era in which we live is if you look at high-performance graphics for games or simulations, it m

DLSS & ideas like it outside an idea are really interesting: link Ex. run the base game at an insanely crummy/low resolution. On low(er) it’s possible to run a model to kinda “upscale” in each frame in a performant way. Or even this CNN initiative from Intel where they took Grand Theft Auto & “painted” it with dashcam footage from rural Germany: ▶ this video The results are pretty interesting— all “brown” southern CA mountains were green like in Germany

Likely not, but that was not my point either, it was just an non-stretchable analogy. AI is not a single network usually, it is a pipeline of networks of different kinds, see e.g. ▶ this video (seek to 1:35 for beginning of an architecture overview). You can’t call it “just pattern matching” without fetching it too far.

The video linked of applying a neural net to GTAV for photorealism is really impressive if you haven't seen it. ▶ this video

I noticed some strange (unrealistic) effects on the back of the red car at the 8:19 mark in this video: ▶ this video

There's also a version trained on Mapillary Vistas, which has images from all around the world. Results from that are at the bottom of the page and at the end of the video ( ▶ this video )

From the "Enhancing Photorealism Enhancement" video [1]: "It is a convolutional network which produces images frame by frame and can be run at interactive rates.". [1]: ▶ this video (0:19)