Topic: NX Revealed in Gaze Tracking Patent?

In short, Nintendo filed a patent last May for a pair of glasses that combine head tracking and gaze tracking to turn your television screen into a homebrew hologram box at low cost.

The NX controller will likely be a glove partnered with a 3D tracking wand with an analog stick (or simply two 3D tracking wands with dual analog), but no matter the design of their controllers, it is highly likely that the NX will feature the tech displayed above in the patent that Nintendo filed May 2014, showing a player wearing glasses and experiencing a 3D effect on a 2D display.

At first glance, one my say, “We’ve seen games using 3D glasses,” but there is more than meets the eye in the patent content. The tech combines eye-tracking and head-tracking to create a 3D effect on a 2D display. The headtracking would work exactly like the video below by Johnny Lee.

The glasses in the illustration could also utilize a shutter effect to simulate a convex 3D illusion with a 2D display. Combining both shutter glasses with head and eye tracking could create both convex and concave 3D images; essentially, turning your television into a hologram box (below).

It would not be the first time that Nintendo has released a pair of shutter glasses. Nintendo launched the Famicon 3D System in 1987 for the NES in Japan only (below). Today a shutter system would likely utilize translucent LCD screens to perform the shutter rather than a physical mechanism. If the glasses do utilize a shutter, then games will have to run at 60fps in order to display a proper 30fps to each eye or developers will have to opt to deactivate the shutters.

The advantage of this technology is that it presents a 3D experience at a relatively low cost, does not require heavy wearables, and most importantly to Nintendo; it does not isolate the user from their surroundings and friends the way that head mounted displays do. In an interview with TIME, Nintendo’s Shigeru Miyamoto talked about Nintendo’s mission statement to bring people together in the living room and his skepticism of virtual reality:

“We want… a game system that brings video gamers into the living room.…When you think about what virtual reality is, which is one person putting on some goggles and playing by themselves kind of over in a corner, or maybe they go into a separate room and they spend all their time alone playing in that virtual reality, that’s in direct contrast with what it is we’re trying to achieve”

The danger of this concept is that people might find it to be superficial over time, like with the 3DS’ 3D effect. There is the risk that gamers and the press will immediately point to the struggles of the 3DS and question the market demand for more 3D illusionary tech. There would undoubtedly be arguments put forth that the 3D trend is over and that Nintendo is again behind the times—chasing 3D television displays while the industry is pivoting to visors that immerse the player within the game world.

The criticism that Nintendo needs to be making; if they want to sell this as an alternative to head mounted displays, is that VR just is not practical with current technology. When gamers actually use VR, some are going to find they get a little dizzy and ultimately will find that playing a game for even an hour with a visor the size of Zack Morris' brick-phone strapped to their head is going to hurt their neck. Nintendo needs to get out in front of their competition and journalists, and they need to communicate how VR is the technology that is superficial.

This was already possible on the Wii. It didn't catch on as an idea because it was a single-player effect and more of a visual novelty than something that was actually usable. I believe Boom Blox had it as a feature but you had to make your own glasses. I could be wrong on that though.

But eventually the dude behind the idea ended up working for Microsoft. Where the same tech ended up being used in some form on the Kinect. Because the Kinect does the same thing except it projects the dots. I susspect the patent filing is more a case of Nintendo using the scattergun approach. They have all sorts of crazy patents, even more patent applications. Doesn't mean that anything will come of it.

The patent was filed in the midst of NX R&D. Combine the timing with the fact they have no signs of any other tech that they are working on for NX that would fit the bill for a console that will be revolutionary while responding to the VR trend, and I think it's highly likely. The signs for what Nintendo was working on for Wii came out long before the controllers were shown. Nintendo invested in Gyration-- their plan for one-handed controllers was clear. The NX will be no different. The concepts behind the NX will be things that already exist (but havn't been applied to a game console from day one platform-wide).

The difference between this and previous attempts at head tracking is that it will feature gaze tracking (maintaining the 3D effect no matter where you look), and you will have the majority of titles for the console supporting the tech automatically. The key to making an innovation work is having support in the majority of games and having it there day-one (look at how MS and Sony have failed with add-ons, look at how Motion+ failed for Nintendo because it too was an add-on).

The issue that Nintendo has with local multiplayer already sabotaged the WiiU's design. They shouldn't let it stop them from using this design. The 3D effects would only be experienced by one player in the room but unlike VR visors, these glasses would not cut you off from the other people in the room and others would still be able to see the action on screen (just not in 3D).

The reality of how the majority of the industry plays multiplayer games today is they play them online. Nintendo can't design this console around imaginary consumers from 1996. Online gaming drives the industry now.

You would no doubt be able to turn the head tracking off.

That would make it about as useful as the 3D on the 3DS, can't remember the last time I turned that on.

kinect 2.0 problems

I pointed out the dangers in my OP with comparisons to the 3DS. It is upon Nintendo to properly demo the system and get gamers and the mass market excited. They have to pitch it as holographic gaming and not simply a 3D effect like the 3DS. It's an effect which the player can alter with their body movement-- it wouldn't be static.

It's no where near as superfluous or as expensive as the WiiU's tablet controller so it's not going to create a barrier at the register (where gamers are thinking, 'damn, this adds $100 to the price').

Iwata has stated that they intend to 'change our gaming lives' with NX. That is a pretty bold claim if they are going to return to a dual analog controller.
Miyamoto has also confirmed, in reference to the next Mario and F-zero, that they are working on a new controller that will do something new that the WiiU doesn't permit.

the problem with the GamePad was the tech was expensive while the R&D was probably pretty cheap, they already had the numbers from the DS. This seems to be the opposite. The R&D is what will be expensive while the tech, seems relatively cheap. That expense is still transferred to the consumer in the price of the machine. I don't think it will be successful as an optional accessory, and if they make it mandatory then I could see issues. I don't think this is what the NX will be but it is an interesting idea.

A glove partnered with a wand. You know, I can actually see Nintendo trying to push it as the future of gaming. They seem so adamant at trying to reinvent the wheel every generation now.

So basically what Microsoft is already working on?

@Bolt_Strike
Microsoft's Hololens is going to be much larger, heavier, and between $500-2000. Microsoft has an interesting idea but it's way too expensive for the mass market right now. Nintendo would be piggy-backing on technology that you already have in your house-- the television. In this case, it's Microsoft that is trying to reinvent the wheel while Nintendo is working on new ways to display games on the television.

@DefHalan
I think that the tech should be inexpensive on both ends. I have no doubt that they've just been taking two or three Wii-mote cameras (each one can track 4 IR points), mounting them on a bar over the television, and plugging in IR lights to the glasses like Johnny Lee. The eye tracking requires two cameras mounted in the glasses and would be the most expensive piece of the puzzle as they would be higher def.

What I am hoping is that Nintendo is investing in a tracking system, whether optical or auditory, that will be for tracking the controllers in 3D space. We're seeing this tech on the PC already-- it just needs to be adopted for a console and be in the package at launch. Given that the console is coming with a tracking system for the controllers, that same tech can double for the tracking of the glasses.

@Operative
I think that will be the pairing. A glove offers something wildly new while the wand (featuring an analog stick) would leave the door open for players to buy a second wand and have dual analog/dual wand control.

Obviously the NX is R.O.B. 2.0, and it will come with a dating simulator where you must stare into his lovely robot eyes of love:
http://www.thecreepiest.net/wp-content/uploads/2014/06/Creepy...

You know he just wants to be loved.

2007....

"Johnny Chung Lee is a Human-Computer Interaction researcher currently working at Google. Lee is best known for his work on Kinect development, extending the functionality of the Wii Remote controller of the Wii video game console, most notably by taking advantage of its high resolution IR camera."

JESUS! Could have done without looking at that pic...

Anyways, how do we know that this didn't come about because of the head tracking function in the N3DS?

@shaneoh
This design/patent uses multiple pieces of hardware, is patented to work with the television, and utilizes gaze tracking via glasses.

As cool as it sounds, companies make patents that they don't use (or shelf for a long time) all the time.

Exactly, which is why it could have been patented because of the N3DS head tracking function. Of course the OP is assuming the NX is going to be a console...

I think it's plausible and a well written opinion. Seems to make sense when it comes to what they've said about VR. Platforms for Nintendo isn't an idea exclusive to consoles, they consider Amiibo a platform for example, so i could see NX being an add on for the Wii U and their next gen console. With the gamepad i wonder if they could implement 3D on multiple displays to create some cool effects. Nintendo loves to use well known/cheap technology in new ways, it's been an important design philosophy for them for a long time. I seem to recall they have a patent on hologram projections as well so perhaps they'll include that as well.

I'm glad we have nintendo that pushing the boundaries of the game industry, otherwise we will had a quite boring game industry.

Not exactly, i don't know if you read the patent but it's a whole new concept. First of all it's not like the 3ds, it doesn't cross your eyes, that why you don't have the feeling to turn it off. Also it does allot more than headtracking.
The Big N hints that taking advantage of smell could come to fruition through this concept. There was also mention made to virtual wind or air that might spray on the player's face when appropriate. Also it can fadeout the environment around you with some kind of light, that way the focus is only on your television.

• Still everyone can look at the television
• Your eye's doesn't hurt from the 3d effect.
• you don't need to have a heavy headset to do this
• You still can play the games without it.

I still don't think nintendo nx wouldn't be anything like that, it will be a console + handheld that using the same programcode for the games, it will show better visuals on the console than the handheld. Developers could only program for one game that can be played on both systems.

Still i want them to push the boundaries.

whats that?

What is your question Ella?