Head Tracking for Desktop VR Displays using the WiiRemote

Wouldn't it be better to use head tracking to get the position of the head relative to the monitor, so the monitor behaves like a window? Like in Johnny Lee's classic Wii demo [1]. The way it currently works (rotating the view upon head rotation) doesn't really make sense because a monitor is not a head mounted display. 1: ▶ this video

You're probably recalling OG Johnny Lee video: ▶ this video That's cool, what were the benefits/drawbacks of that compared to other VR headsets would you say?

I tried to implement Johnny Lee's amazing idea ( ▶ this video ) using mediapipe face tracking. I could not move much far using simple webcams since it was getting difficult to determine the distance of the face from the camera when the face was turned. I had an Inel RealSense 415 depth tracking camera from a different project and it took care of the distance thing at least. But the jitter thing had me stumped for a long time and I put the project away. With your ideas, I get the strength to revisit it. Thanks!

... or the various projects by Johnny Lee [1], using the then new wiimote in completely novel ways [2][3]. 15 years later, I'm still surprised these proof-of-concepts didn't evolve into something more commercial. [1] link [2] ▶ this video [3] link

You need to use the Wii remote in reverse so you can make a fake window. ▶ this video (can't believe this was 15 years ago now..)

I absolutely agree. I'm very much a whiteboard hand-wavier guy. I explain with gestures as much as I do with the actual drawing (might be an Italian thing too). That said, this "pancake" problem reminded me of the old Wii head-tracking project.[1] There might be an interesting hybrid opportunity with that idea and 3d avatars. I feel like the head-tracking hack never really took off because it only worked as a solo experience. But since remote work is mostly us all individually sitting at a computer I could see it working better. Combine this idea with a large dedicated monitor/tv and now you have something that would literally just feel like a window/portal to the person you're talking to. [1] ▶ this video

> ▶ this video I did this at the time. I downloaded his Wii app that shows the little red/white discs, and bought a battery powered Wii-bar thing that had the IR leds in it and strapped it to my head. Can confirm, the 3D effect is 100% spot on, even the smallest move of your head caused the display to adjust perspective and your brain is totally fooled into thinking the Wii display is a just a window into a 3D scene. tbh, it was the only cool thing I actually did with my Wii!

Cool, but, it looks exactly like watching a display with a Wii remote on your head [0, 2007]. What’s so special? Is it actually 3D this time? [0] ▶ this video

"We offer an experience of 3D on a single 2D image using the parallax effect, i.e, the user is able move his real-time tracked face to visualize the depth effect." Since there are no examples I can't be sure if this is is what I think it is, but IF it is: I want this on huge monitor for any 3D game instead of clunky headgear VR or tiny smartphone AR. Months ago I also tested this with a small 3D visualization and very crude head tracking. The effect is damn awesome! To be able to move around in real space with the rendering adapting to it, makes it so immersive, even for my very crude tests. In my opinion the resulting 3D effect is MUCH better than viewing in stereo with one picture per eye. Here is an example from someone else from 2007: ▶ this video From 2012: link Obviously this works for only one person viewing it, but does that really matter? There are a LOT of use cases where only a single person uses a single monitor for viewing, especially in these times. In fact it is the standard.

What's interesting about this is that the tech has been proven dating back all the way to 2007 [1] and also re-implemented by enthusiasts many times over the years with better eye tracking via CV. What Sony has done here is refine it and package it in a way that's nice for end users. Personally, I'd like to see this implemented more cheaply in larger displays to act as virtual windows for indoor environments, like [2] from 2010, but with 3D video/rendering. [1] ▶ this video [2] link

This reminds me of ▶ this video (2007) Johnny Lee's demonstration using head tracking with infrared led's and a wii remote to implement the same thing. Obviously that limits the effect to the individual's head being tracked, but its amazing its been 12 years since that demo and we're still trying to commercialize the same affect.

Yo mark willson of co.design. Were you around 10 years ago? Let's tame the article title a bit huh? ▶ this video

Check out Johnny Lee's awesome "Head Tracking for Desktop VR Displays using the WiiRemote"! ▶ this video More great stuff: link

Nice job! If you have a camera that tracks the viewer, you can also make the image it displays 3D as seen in this video from 2007: ▶ this video (back then it used a Wiimote to track the head position).

"- Johnny Lee and the ATAP-Project Tango Team" Johnny Lee was the guy with the awesome Wii Controller demos back in 2007 (can't believe it's been that long). ▶ this video edit: here's the full set of demos: link (also, I'm assuming it's the same guy, but his site says he's at Google now)

Actually, stereoscopic vision is overrated. There are many hits our brain uses to make the 3D model of the world we have on our minds. Stereoscopic vision is one of them, but really just kick in for objects nearer than you arm cam hold. For distant objects, our brain uses other hints - that's why people aiming at far objects close one of their eyes. One of the strongest is the parallax effect, specially for medium distance and relative distance between objects. If you doubt it, look at this video: ▶ this video Relative size and color fading works well for big distances - this is the prominent effect in parachuting videos. These are only two mecanisms, and immersion is not only about 3D sensation. Full periphery vision is of great value for this (IMAX, anyone?). Sound hints, brigthness, contrast, and a lot of other hints also help our brain making the 3D model.

Johnny Lee's work with the WiiMote uses the same technique, and his demo is much more visually impressive. :-) ▶ this video

So damn clever... releasing an SDK for the Kinect[1] was one of the smartest things Microsoft could have ever done and I think has gone a long way to contribute to it being the fastest selling piece of hardware in history[2] (did anyone know that? I sure didn't, and barely believed it reading it). This reminds me a lot of what Johnny Lee did with the Wii soon after it was released with head-tracking 3D[3] Part of me wants to stop using a computer and take up beet-farming because the cool factor seems so much less than what these guys are doing. Anyone on HN actively toying with a Kinect and want to share some video? [1] link [2] link [3] ▶ this video

This particular implementation will probably never work, but that doesn't mean "3D movies" in some form will never work. This guy built a stunning demonstration giving a 3d effect with a wiimote and a standard TV: ▶ this video

Specifically: ▶ this video Check out his entire youtube channel, it is full of interesting software/hardware hacks.