Movie reconstruction from human brain activity

I imagine the best case outcome would look something like Jack Gallant's 2011 work on visual mind reading, where they trained a model on brain activity watching hundreds of hours of YouTube and then attempted to reconstruct a view of video not in the training set by correlating the new brain activity with frames from the input... Sorry I'm not explaining very clearly but there's a YouTube video of the result naturally :) ▶ this video Learned about this from Mary Lou Jepsens 2013 TED talk, what a throwback link

Reminds me of this paper [1] from 2011. See it in action in [2] 1. link 2. ▶ this video Edit: Just realized the paper above is also from Shinji Nishimoto

There was a video many years ago (early 2010s?) demoing a similar technology, which would overlay and blend many images on top of each other to make a fuzzy image approximating what was actually being viewed. Edit: found it! ▶ this video

I mean in 2011, they were already able to do some basic reconstruction: see ▶ this video I could imagine improvements since then,especially with advances in image networks.

I wasn't able to realize the scale of the shift that's about to happen back then. --- I just remembered another example of hearing about something that seemed like pure magic: Reconstructing what a person is seeing out of their fMRI scan (2011): ▶ this video

The errors and bias in these systems add to the ethical concerns. While it is arguably a good thing to substitute torture with brain image interrogation, there is a risk of putting too much trust in these systems, subjecting innocents to days of leading investigation, just because the computer said there is something to be found there. Israeli airport security (arguably the best in the world) deploys derivatives of these systems, that look at micro-gestures, elevation of heart rates, pupil dilation, and temperature changes, to see if passengers respond with familiarity to terrorist imagery flashed on a screen as they walk by it. If that already works in practice, imagine the same, but being strapped with hundreds of sensors. See also the 2010 research on image reconstruction from brain activity, and extrapolate that 10 years in the future and applied to military interrogation: ▶ this video

I feel like we are so close to wirelessly monitoring all brain waves (next 50 years) we will probably be able to, in real time, recognize and diagnose abnormal brain patterns. I think it would be promising if we get this sensor tech up and then perhaps deploy nanotech to interfere with maladaptive brain patterns. The ethics of such a device is considerably sticky. TED talk by Jack Gallant link recreating vision: ▶ this video another ted talk by John-Dylan Haynes link paraphrased: "wireless non-contact mind reading tech may be cheap and mass producible within the next 50 years." the only rate limiting step with his method of decoding thoughts is that it takes time under a high energy device to pair associations in the brain. Think of how much screen time we currently use. If we could pair a wireless 'headset' like headphones while looking at pictures on a screen it would be possible to build a brain bank of children->tenenagers. Currently we just need a lot of processing power and reserved bandwidth on the internet to do some cool stuff.

Reminds me of this project from 2011: ▶ this video If I were a billionaire, one of the projects I'd do would be to apply this sort of technology at scale, upon volunteers who were witness to poorly recorded historic events. WWII, Beatles concerts, JFK assassination, 9/11... all of them have been recorded from angles/perspectives that exist only in peoples' memories, and the longer we wait, the more of those memories will be lost.

Extremely interesting to compare with Jack Gallant's reconstruction of video from the brain MRI of a viewer: ▶ this video

MRI technology is improving over time, but I don't know how practical it would be in everyday usage. fNIR can potentially do the same thing much cheaper. I believe it's restricted to only the first few cm of brain tissue but that might be plenty. With fMRI you can even reconstruct entire videos of what a person is seeing (e.g. ▶ this video ). Controlling something simpler should be easier.

Select the 100 clips whose predicted activity is most similar to the observed brain activity. Average these clips together. This is the reconstruction. Mind to explain to me why the right clip is inconsistent with the left clip? ▶ this video Starting at 20-second I feel like the right clip is out of touch with the left clip. between 20th and 22nd second I see at least three individuals rendered from the reconstruction. From 26th to the end of the clip I also see multiple individuals. The names also look different from one another... When you say find the closest is that an expected result?

Although it's hard to tell from the images presented with the article, the face generation looks like it could be similar to the techniques used in Nishimoto et al., 2011, which used a similar library of learned brain responses, though for movie trailers: ▶ this video Their particular process is described in the YouTube caption: The left clip is a segment of a Hollywood movie trailer that the subject viewed while in the magnet. The right clip shows the reconstruction of this segment from brain activity measured using fMRI. The procedure is as follows: [1] Record brain activity while the subject watches several hours of movie trailers. [2] Build dictionaries (i.e., regression models) that translate between the shapes, edges and motion in the movies and measured brain activity. A separate dictionary is constructed for each of several thousand points at which brain activity was measured. (For experts: The real advance of this study was the construction of a movie-to-brain activity encoding model that accurately predicts brain activity evoked by arbitrary novel movies.) [3] Record brain activity to a new set of movie trailers that will be used to test the quality of the dictionaries and reconstructions. [4] Build a random library of ~18,000,000 seconds (5000 hours) of video downloaded at random from YouTube. (Note these videos have no overlap with the movies that subjects saw in the magnet). Put each of these clips through the dictionaries to generate predictions of brain activity. Se

This is a much cooler example ▶ this video If I'm understanding the description correctly, they are just training it to recognize what the image is closest to and taking an slice of a youtube video that most closely matches it. I imagine if they used a more efficient method or trained it more, they could do way better. It seems like most of the data to build an accurate picture of what they are seeing is already there.

According to the abstract: Here, we present a neural decoding approach in which machine learning models predict the contents of visual imagery during the sleep onset period given measured brain activity, by discovering links between human fMRI patterns and verbal reports with the assistance of lexical and image databases. Decoding models trained on stimulus-induced brain activity in visual cortical areas showed accurate classification, detection, and identification of contents. So it sounds like the fMRI patterns are associated with words via verbal reports from humans, and the video is simply generated from image databases by using those words. In other words, they classify the fMRI patterns based on how well they correspond to various words (measured while the subjects are awake and giving verbal descriptions of images they're looking at), and then the program measures the subjects' fMRI patterns while they're dreaming, determines the strongest 'word signals' in the patterns, and selects images from a photo database corresponding to those words and assembles them into a video based on how strong the similarity score for each word is at each moment in time. It's interesting, but the video seems like more of a "reenactment" of a dream than a prediction. It doesn't look like you can infer actions and occurrences within a dream from this. But this is all just my interpretation. edit: This is a fundamentally different approach from the one used in other research (e.g. ▶ this video me

This is one year and something old and returns better results, in my opinion. ▶ this video On the other hand, they presented only the best results in that video.