Face2Face: Real-time Face Capture and Reenactment of RGB Videos (CVPR 2016 Oral)

Founder here. AMA :) To answer a few recurring questions in the thread ---> Use case. Video is a way more effective way to communicate than text. Not for the HN crowd, but if you're a blue collar worker a 2 minute video in your native language is much preferred to a 5 page pdf for training. Anyone who has tried to record a simple corporate video know the pain of cameras, film crews, 25 takes to get one that works and post production. Cumbersome, slow and multidisciplinary. By the time the video is done the content is out of date. Synthetic video is not yet at the quality of real video. Eventually it will be. But the mistake many are making here is comparing it to real video; it should be compared with text. In X years we'll be able to make Hollywood films on a laptop without needing anything but time and imagination. Just like we can digitally compose music in Ableton, create images in Photoshop and type novels on keyboards rather than with pen and paper. My (obviously biased;)) belief is that synthetic media will eventually become foundational technology that will move media production from cameras/microphones to API's. We'll be able to do all kind of things we couldn't do before. Eg. personalized and interactive rich media, video-driven chatbots and eventually Hollywood blockbusters made by your favourite YouTuber from his or her bedroom. ---> Uncanny valley Simulating real video is incredibly hard. We're constantly improving and launching more expressive synthesis soon. Fr

Founder here Maybe – one of my co-founders is Prof Matthias Niessner who's been behind a large chunk of the seminal and widespread research in this space. [1] ▶ this video [2] link

▶ this video Face2Face would defeat this: anyone with enough images of the victim (and copy of ID) could impersonate them in this scheme.

Photo, audio, and video evidence should already be dismissed until one is able to verify the integrity and source. All of these can already be believably faked - it's just a matter of educating people that a layperson can easily create fake things by using tools developed by research teams. Fake text is the easiest to fake if you can identify the font used - any image editor will work. HN uses 9pt Verdana, even without using dev tools I could fake your post to say anything I wanted it to say since it would just be 9pt Verdana on a solid background set to text wrap every 1050px. See: ▶ this video & link

Here's another scary GAN proof of concept [0]. In this case, researchers transferred someone's face in real time to facial expression and mouth movements of public figures. Combined with DeepMind's new tech that seems to be able to produce human voice with believable candor and inflection [1], you could make some very convincing fake footage. [0] ▶ this video [1] link

We're already at the stage where one can't trust the video evidence, unless it's backed by the camera's signature/encryption. Creating a very realistic fake is now trivial: ▶ this video link Google is very close to synthesizing realistic voice. It's game over, as far as I can see. It's a matter of time someone creates a fake video of someone famous saying something very outrageous, like nazi propaganda, and it will result in the destruction of that person's career and life. We really need something like Secure Enclave in every camera. EDIT: Another related video: link

> Seems from right out of the gate, they are breaking their own ethical guidelines as a cheap promotional tactic. If they care that little about themselves and a former president of the United States, what do they care about your likeness. We state in our blogpost that we make an exception for Obama/Trump in order to raise public awareness. Both of them are regularly used in Machine Learning benchmarks (for example [0] [1]). Note that we don't allow users to generate from Trump/Obama's voice. Once again, we care a lot about these issues and that's why we only allow users to copy their own voice. [0] link [1] ▶ this video These issues are challenging and suggestions about how you think the technology should be introduced/regulated are very welcome.

Face2Face: Real-time Face Capture and Reenactment of RGB Videos (CVPR 2016 Oral) ▶ this video And I agree, I find this much more verosimil.

This article title is a bit off. The audio is not generated, the video is not convincing, and the original conversation is not fake. This is an art project which highlights the implications of research like Face2Face: ▶ this video

Combined with Face2Face[1] live video impersonation, it is truly time to be very careful verifying videos or even live streams. ▶ this video

While the technology is amazing, I am a bit bothered by all these picture and video modifying algorithms. The issue is that we can't know what's real any more. It used to be if you saw a video or a photo depicting an event you could be pretty sure that what you're looking at actually happened. Now, if you see a video of a prominent politician saying something awful in your twitter timeline (or whatever), they may have actually never said anything remotely close. It could be a completely fictional video that looks perfectly realistic[0], made by some teen in Macedonia.[1] I realize photography and video have always been used to trick people into thinking things that aren't true, but this technology enables nuclear-grade deception. I am wondering: is there a use-case for such an algorithm that is practical and good for the world? PS: I know an eye-rolling algo is quite innocuous but I've had this thought on my mind about these in general and needed to air it out. [0] ▶ this video [1] link

> "Fake News" is about to get a lot more compelling hen you can make anyone say anything as long as you have some previous recordings of their voice. Adobe has already developed that technology: link Now imagine combining it with this: Face2Face: Real-time Face Capture and Reenactment of RGB Videos ▶ this video Perhaps using the intonation from the face-actor's voice to guide the speech synthesis.

I still think that it is incredibly important that we make tools for validation of primary source material easy to use and friendly for non-technical people. Tech for faking video is getting more powerful day by day. link ▶ this video

Maybe CG will be good enough to keep casting dead actors? The face2face[1] tech looks pretty convincing to me and it already works in real-time (which isn't necessary for films) and you have plenty of old footage of them. [1] ▶ this video

It gets even better when combined with Face2Face, a live-editing software that can transfer facial expressions including lip movements from actors to e.g. politicians on TV: ▶ this video

OK, throw this into the mix: many of you reading this have smartphones which are voice controlled, and for which voice control is activated at all times. In the case of Google, that processing must take place on Google's centralised servers. Siri may or may not do centralised processing (and can operate in standalone modes). Microsoft's Cortana, Facebook's "M" (IIRC) and Amazon's Aero are all various stylings of "Stasi in a Glade form factor", as Maciej Czeglowski so memorably put it.[1] Voice stores distressingly cheaply in terms of space, and with the Internet of (broken) Things (that spy on you), odds of finding yourself surrounded by microphones in the most unexpected locations,[2] controlled by a wide variety of quite probably competing interests.[3] And if they cannot find what they're looking for in the surveillance tape itself, they'll simply manufacture their own evidence using your own phonemes[4] and video.[5] ________________________________ Notes: 1. link 2. link 3. link 4. link 5. ▶ this video

The consumer use cases are interesting but the propagandist use cases are terrifying. Along with this (Real-time Face Capture and Reenactment): ▶ this video

I find it amusing that they're touting it for an application where I find it quite likely that similar NN algorithms will excel at generating speech in someone else's voice. (c.f., "neural style transfer" in images, but applied to speech.) We're already getting pretty decent at this for video -- see, for example, ▶ this video (Face2Face: Real-time Face Capture and Reenactment)

Yea, perhaps. Like in the recent "face2face: realtime face capture and re-enactment video"[1]. and your reference to "your avatar" reminds me of the scene from the matrix where Morpheus describes "residual self-image"[2] [1] ▶ this video [2] link