News Article

Nintendo Files Patent for "Eye Tracking" Device to Enable 3D Experiences on 2D Displays

Posted by Thomas Whitehead

Also proposes enhanced experiences through smell, temperature and light

Nintendo may have submitted some rather peculiar patent applications in the past, but the most recent may have some foundation in reality, indicating a potential move into Virtual Reality for the company. It proposes a wearable device — potentially glasses — that is detected by a camera, and adjusts your viewpoint to give an illusion of 3D on a 2D display, which would then also be used in various ways to impact interactive game experiences.

You can see a typically crude illustration above, though the explanation of how the system will work places this technology somewhere between Microsoft's Kinect and VR headsets such as Oculus Rift and Sony's Morpheus. The technology to give an impression of 3D without an autostereoscopic screen (like that on the 3DS) or 3D glasses is not new, as there are already some smart devices that can track your eyes to achieve similar results. Nintendo's proposal appears to aim to take this technology much further in terms of interaction — examples given include dodging objects by moving your head, adjusting your view to see something on screen that others can't (an idea suggested as potentially suited to multiplayer experiences) and also the ability for in-game characters to react to where you're looking on screen.

There are more ambitious ideas for immersion, too. Additional devices could expand the experience further, while emotional state could also be measured.

[0099] Enhancing the VR experience Additional output devices that enhance the experience can be provided. For example, we can put light out that is correlated to the image to provide "ultra wide field of view correlated lighting." Given that your eye does not see clearly in the periphery, this could still be useful and interesting.

[0100] Additionally, smell is a very strong sense. There may be some ways to produce aromas for a very strong experience.

[0101] Virtual wind could enhance the experience.

[0102] Temperature: blowing cool air on your face.

[0103] Physiologically comfortable stereo viewing is a way to prevent headaches. If you find a little spec on your windshield, focus on that and then far field and then back again. Eye strain happens quite quickly. Lots of folks in the past require the users to focus far field and close up, but this can cause headaches. We can stay on one side of the focal point cone, to provide higher level of comfort.

[0104] Detect Emotions via monitoring mental state. Brain wave detection, detect eye movement, heart rate monitor or the like can be used. If we provide goggles, we can also provide detectors (electrodes) fairly easily.

It's certainly intriguing, and if it becomes a reality — which is far from certain, as is always the case with patent applications — would continue Nintendo's policy of adopting existing technology for new, potentially innovative experiences. It also fits into the rhetoric of Nintendo's senior figures that it's currently unsure of the reclusive nature of headsets such as Oculus Rift — this patent's setup could be more social in the living room — while maintaining recently that it'll continue to seek innovation as an entertainment business.

What do you think of this concept? Does it look exciting to you, or do you feel it won't work well or ever see the light of day? Let us know in the comments below.

Thanks to Otto Petersen for the heads up.


From the web

User Comments (70)



Zyph said:

Aside from Johnny Lee's Wiimote concept, the idea was already implemented on the DSi with Looksley's Line Up using the inner cameras for motion tracking. Remember that game?
The concept of the tech is really amazing if executed properly. I prefer this method for creating 3D parallax "illusions" rather than having to sit directly in front of a 3D TV. I read the whole patent application and there's definitely a lot of potential.



shigulicious said:

Nintendo can't even implement the gamepad properly and they are drawing up this goofy stuff?



ThomasBW84 said:

@Zyph That is true, as I say in the article, it'd play into Nintendo's tactic of expanding and improving existing ideas.



Zyph said:

@ThomasBW84 Yes definitely. After reading the patent filed, I have good hopes for this technology. But I don't see it being used on Wii U though. Maybe on their next hardware perhaps to add to it's feature set.



Artwark said:

Why are they reworking on stuff that they already did? Can't they use the 3DS with some add-ons that aren't worse than Genesis?



MAB said:

More lame gimmicks to try and sell me another boring Mario game



Pod said:

Nintendo's "basics" was always presenting something not seen before.

Gotta love all these goofy ideas. Only a fraction of them ever make it to market, so don't be scared you'll be living in Gimmickville, though.



Zyph said:

@daveh30 Yeah but they're not the first one to use it. And besides the patent focuses on the parallax 3D effect on 2D displays. In Sony's case, they're using Project Morpheus for that. Spectators will still see the game being played normally on the TV. As for the patent, even if you're not the one wearing the glasses or the head mount, you can still see the effect on the 2D display.



Spoony_Tech said:

I enjoy every gimmick they've done so far so I can't wait.

All you people saying focus on the games what do you think they're doing. Do we really need the same stale old games in hi res graphics? No! Nintendo is the only company always trying to give us different gaming experiences and all they get is flack for it. If most of you haven't noticed gaming is starting to take a hit and by that I mean console and handheld gaming will become less and less because of the smart devices. Nintendo is trying to stay relevant in these increasingly harder times. I love them for that.

Wiimote, 3ds, and gamepad ftw!!!

Oh and ds!



Link506 said:

@Artwark Yes, that is if they don't overdo it. Remember the Virtual boy? It focused on 3d so muxh, that it had to be downgraded in many areas to just make it affordable. And since it was downgraded, no one wanted to buy it. If Nintendo put all of this into their next console, many things would have to be downgraded to be affordable, and many people would not like that at all.



Figeluren said:

Don't forget Nintendos main motto, play together. This tech is only usable in single games for the lonely gamer. The depth effect will be very weird for other viewers and can very likely cause nausea for many that is watching but not wearing the glasses.

Still, I hope they go forward with this tech.



Spoony_Tech said:

@Link506 That was probably Nintendo's biggest mistake obviously. The world wasn't ready for a 3d system and it wasn't implemented right. To most it didn't look fun and only really in one color kinda killed it before it ever hit the market. I applaud them for the effort though but that was never going to take off.



Jazzer94 said:

Could be cool but this looks like it could also be expensive. Honestly even though this and Oculus rift are really cool ultimately to they are gimmicks that won't have lasting appeal.



Shambo said:

Oh I can feel the tension rising... Companies across the globe are starting to reay themselves to sue Nintendo for whatever patent they have that is even remotely linked to this...

@shigulicious Nintendo may not be the best at using the features of the gamepad (although I found the first-person twin stick/gyro aiming in Wind Waker to be something shooters could really benefit from), ever since DS it's all been 'developer systems' in my eyes. They provide different possibilities to experiment with for creative, daring developers.

And at least Nintendo knows how to move on without abandoning all, unlike Sony. My Vita is lying there with almost no games, and next to none to look forward to. "Vita means life"... Well, my Vita is dying. And don't get me wrong, I love the system. Not just another fanboy. But 3ds looked like it would "lose", yet it clearly bombs the Vita any day of the week.

Also, this could be the ultimate step towards perfect 3D without having to adjust your position constantly, or having to get a new TV while the 'old' one still works perfectly well. I don't know how they'll really do it, but the possibilities might go beyond regular 3D TV.



IxnayontheCK said:

I may be a minority, but I love 3D! Its a shame TV in the US never came up with a standardized 3D usage and its now faded away...



lerobadey said:

Gimmick, gimmick, gimmick. Woopsiedoopsiedo. Let's just sit in front of our TVs and play one determined game because everything is gimmick.



TheWPCTraveler said:

@cfgk24 They did that with six cameras and no other sensors. Nintendo is trying to do this with a single camera and multiple sensors, as well as with glasses.



Sparx said:

Just realized, imagine if they implemented the eye tracking into a game's dialogue and cut scenes? I imagine a few hilarious quotes due to where eyes are drawn to...



nesvc said:

All Nintendo needs to do is make a more powerful console with normal controls and a reasonable price. Their games will take care of the rest.



Pod said:

They have a department for coming up with new poopoodoodoocacapoopledoopledoggiepoopledoodoocacapoop. And they're only gimmicks until they prove themselves.

Meanwhile they have over 20 departments that focus on games.



JaxonH said:


They already did. It was called the Gamecube. And while I liked the Gamecube- it had some of my all time favorite games, I was pretty let-down at the lack of innovation. Every new generation I get excited. Not to find out how powerful the next Playstation will be, or how powerful the next Xbox will be, but to find out what Nintendo is going to do next.

Normal controls and higher power is fine for other consoles, but when it comes to Nintendo, I want to see more. I want to see that new innovation they're bringing each generation. Ya, you're right, they could just do like everyone else and make new Gamecubes every gen, and I'm sure they would chug along just fine- like you said, the games sell the system. BUT, it would definitely be disappointing to never see innovation, or new ways to play those games. Power is becoming irrelevant anyways. Especially for Nintendo. I look at their games and I think they're finally hitting that visual plateau.



The_Ninja said:

@MAB You kind of remind of memoryman3....

Anyways this sounds intresting..... does sound like something Nintendo would do, actually. They love trying out new gimmicks, and I always end up liking them.



ModernMARVEL said:

I would love the smell of a grassy field every time I turned on a Zelda. And having wind and temperature would be awesome. I'm all for it!



Darknyht said:

Camera on the gamepad? Check
Stated ability to see things others cannot? Check
This feature added to existing Gamepad? Possibility.



BlackHearted1 said:

This would be pretty cool, especially "the ability for in-game characters to react to where you're looking on screen." that would be really cool like your looking up in the sky and other NPC's stop and look to, if you do this IRL it can be pretty funny just stop and gasp while pointing in the sky usually people stop for a sec and, than i burst out laughing, its pretty fun.



Takerkaneanite6 said:

To be honest with you, this seems more like a potential perperial for their next console, rather than a gimmick, but that's just what I think........



xerneas said:


No but if they'd implement it well and made the potential glasses or whatever light and comfortable (oculus rift is really far from comfortable...) this could be pretty cool for a next console.



Savino said:

@Darknyht I was thinking the same thing.

This could be a tech that would make that WiiU more unique.
Forget about all those brain wave readers, smell and other crazy stuff in the file and you can have a cheap and imersive device that creates a great 3D experience.

That Johnny Lee guy´s video is amazing, you can see the effect even on youtube!



TheRealThanos said:

@Zyph I do remember that game, but didn't have a DSi back then. Is it still available? Couldn't find it in the eShop at first glance...
@Montegarde and smell garlic all the time? No thanks...

And to all the people screaming gimmick: I think you all need to understand that a gimmick is something that is implemented/used just for the sake of and really adds nothing other than being a trick and attracting attention. Taking that into account, you will find that none of the things that Nintendo have done such as the DS, the Wiimote, the 3DS and the Wii U GamePad are gimmicks.
Sure, all of these did and still do attract attention, but they're not there just for the sake of. Or at the very least let me say that it wasn't Nintendo's intention to do so. They implemented all of these features to try and truly add something to the gameplay instead of just tacking it on and have an attitude of "let's wait and see what happens".
It is of course perfectly debatable, especially in the Wii U's case, if they have already done enough to please the public, but the final verdict seems to be firmly in favor of "not a gimmick".



Hy8ogen said:

Nope. This is stupid. Imaging playing Deadspace, Resident Evil etc with this TV. I'll vomit along the way.



Zyph said:

@TheRealThanos Can't find it either. I think it wasn't a DSiWare game but a packaged software. Not sure though. And all of the things you said is true. Thanks for that. It seems these people just takes things for granted. I'm not biased or trying to defend Nintendo in any way, but those "anti-gimmick" guys doesn't even try to see the value in what Nintendo is trying to do (gameplay-wise) even if they won't use them at all. I always thought, what if these "gimmicks" were actually the standard of gaming since long ago and the "traditional" method of playing were the gimmicks. Now the original PS Eye for the PS2 was a gimmick. Not sure for the PS4 though. The Kinect imo isn't 'cause it adds value to games in some ways, but you could argue that most of the games using it says otherwise.



Varoennauraa said:

I'm loving this! The 3d effect can be really effective, and I have been dreaming, and actually waiting to Nintendo to use this technique with the Zapper add on.



Zyph said:

@Varoennauraa That's a nice idea. Imagine Link's Crossbow Training with the addition of parallax 3D tracking. That game was okay, but with this it can add more variety.



DinoFett said:

I want this system to be Nintendo's version of 'Google Glass' connected to a future version of 3ds! Going about our world Nintendo style.



Action51 said:

Phillips is drawing up plans and filing a similar patent as we speak for their eventual lawsuit.



IceClimbers said:

It should be pointed out that this is different from the 3D that we're used to on things like the 3DS and 3D TVs. This also goes around the limitations of some people getting headaches or not being able to see normal stereoscopic 3D, as well as the motion sickness of VR tech like Occulus and Morpheus. There's also no sweet spot, so you can view it at multiple angles.

This essentially can take any normal TV and make it a 3D TV. Johnny Lee's video is the best way to see it in action. Combine this tech with touchable holograms (which are possible, and have been done), and you have some amazing potential (Senran Kagura Burst, anyone?). Could be really immersive in Zelda or even Metroid. Oh, and Pokemon Snap with this tech would be godly.



Imagamerboi said:

Looks like Nintendo wants to get some money back after losing from patent troll Phllips in the UK.



RawkHawk013 said:

sounds REALLY awesome (no guarantees it happens let alone any time soon but still). Whether or not it really works out well I do not know, but there is SO MUCH potential with it!



kenzo said:

This two camera tracking idea for a future Nintendo console application is a great idea.

However I have seen something very patently similar described by a person called 'budda' on the mtbs3D virtual reality forums.

see this:

And I quote at the end: -

"As I noted previously, this device could be used as a head tracker, or gun aiming device for a first person shooter.

The essential elements are the IR LEDS and IR Camera, which could be configured in various ways to acheive the same results.

For instance, I would like to see a wireless version of the Wii Camera (without the Wii Remote body) attached to some glasses.

In principle this should be easy to do, but no-one has apparently done it.

Another idea would be to mount a miniature fish-eye lens on the Wii Camera to get wide angle tracking. And so on. "



FriedSquid said:

It's interesting, to say the least. You guys have to remember that this doesn't mean that this is Ninty's newest console. It's just an idea, nothing more and nothing less.

What intrigues me most is how they plan on creating aromas and artificial wind!



TheRealThanos said:

@Zyph Yeah... some people just seem to need to take a step back and think instead of continuously pulling some card that says "gimmick" or "not innovative" or whichever negative comment they can come up with next. Sometimes it is true, but more often times than not a bit of calm breathing and clear thinking will do the trick for you and will let you come to the only correct conclusion. Oh well...
@Squid Man, that's the easy part. Way back in the sixties and seventies they were already making stuff like that:
There were even companies experimenting with bringing set top boxes into homes, but, obviously, it never took off. Right now, the Japanese are experimenting with it once again, so maybe that is what Nintendo is aiming for:
The wind part could be done by installing 4 wireless mini fans around your room, couple them with receivers and the console would then send a signal to one or several of them whenever the game requires it.



kyuubikid213 said:

If this were to work...and work well, I'd probably buy it. Having that much more immersion in a game would be pretty great.

On the other hand, I don't want to have to run out and buy scents for every game I own and I don't want to clutter my entertainment center with fans and such. It's a neat idea, but I think it would need some more time. (I haven't read the entire patent.)

I see this as a proper first step to Virtual Reality, though. Oculus Rift and Project Morpheus look cool, but it's just a gyro sensor attatched to small screens placed in front of your eyes. That's not so much Virtual Reality as it is a Virtual Boy with color and a gyro sensor.



nungi said:

Guess some people don't want Nintendo to try different things.Thats what makes Nintendo the best.You don't like the way they taking their time advancing technology,think you should stay on the Xbox and ps.This is a Nintendo site and Nintendo don't compete with the competition,they do there own thing and been at it before there was competition in today's world and full of immature people just can't let them do their thing who you to say,are you working on anything revolutionary.ok.time will tell on who's got the brighter ideas and who's stuck in the past three gens with the same innovation



nungi said:

Things about to change for U.and if you don't have 1 yet something is wrong with you.Xbox and Ps have nothing to offer besides power and not anything new,all that power for the same past experience



FJOJR said:

That looks too far fetched. I mean come on, a new Madden on Wii U? But that eye tracker could make 3D gaming in the sense that you could get a different angle depending on where you're sitting or standing.



kenzo said:

i am no patent attorney but I have noticed something wrong with Nintendos patent.

This application claims the benefit of priority from provisional application No. 61/378,921 filed Aug. 31, 2010, the contents of which are incorporated herein by reference.

The priority date for this US patent 8704879 (granted April 22, 2014) and new patent application US 2014/0184588 (filed March 5, 2014) - is August 31, 2010.

However, very similar subject matter to Nintendos claimed invention was published by budda on the mtbs3D forums on September 4, 2009 - as described earlier in this thread. This is 12 months before the priority date of Nintendos claimed invention.

It seems budda's publication of this prior art could substantially invalidate Nintendos patent, because of the loss of novelty, loss of originality and because of obviousness to those skilled in the art. It strikes at the heart of Nintendos patent claims.

I call OBJECTION ! Your Honor.

Leave A Comment

Hold on there, you need to login to post a comment...