Introduction
With the ingress in the market of products like Nintendo Wii, Apple iPhone and Microsoft Kinect, developers finally started realizing that there are several ways a person can control a computer besides keyboards, mice and touch screens. These days there are many alternatives, obviously based on hardware sensors, and the main difference is the dependency on software computation. In facts solutions based on computer-vision (like Microsoft Kinect) rely on state of the art software to analyze pictures captured by one or more cameras.
If you are interested on the technical side of it, I recommend to have a look to the following projects: Arduino, Processing and OpenFrameworks.
Usage with Ubuntu
During a small exploration we did internally few months ago, we thought about how Ubuntu could behave if it was more aware of its physical context. Not only detecting the tilt of the device (like iPhone apps) but also analysing the user’s presence.
This wasn’t really a new concept for me, in 2006 I experimented with a user proximity sensitive billboard idea. I reckon there is a value on adapting the content of the screen based on the distance with who is watching it.
We came up with few scenarios which are far to be developed and specified, hopefully will just open some discussions or, even better, help to start some initiatives.
Lean back fullscreen
If the user moves further from the screen while a video is playing on the focused window, the video will go automatically to fullscreen.
Fullscreen notifications
If the user is not in front of the screen, the notifications could be shown at fullscreen so the user can still read them from a different location.
Windows parallax
Since this is the year of 3D screens, we couldn’t omit a parallax effect with the windows. A gesture of the user could also trigger the launcher appearance (see prototype below).
Prototype
With few hours available, I mock up something very quickly in Processing using a face recognition library (computer-vision).
Despite it could be hard to detect the horizontal position of the user’s head without a camera, we are in no way defining the technology required. The proximity could be in-facts detected with infra-red or ultra-sound sensors.
Parallax and fullscreen interaction via webcam from Canonical Design on Vimeo.
Wow! This stuff looks amazing! Good to see the design team are thinking towards the future :)
This looks awesome! Sounds like a lot of work, but its something not found in any other system either. Very innovative!
AWESOME!!!
Another usage to this can be to change window focus based on what the user is looking at. On dual screen mode, that can be very useful
Simply awesome Ubuntu is made to be the perfect O.s. of the future. We need pacience and money!
So is this something that Canonical actively plans to develop and put in Ubuntu in the near future, or just a “It would be cool if…” kinda thing?
@dieki, at the moment we just did this little exploration and it’s not planned to be developed in Ubuntu any time soon. As I mentioned in the post, it would be great to see what the community thinks and if they can bring this further. Said that, this doesn’t mean the Canonical Design team won’t come back to it at some point in the future, we’ll try to keep you posted if that happens! ;)
Interesting but it would be a lot cooler if you could use the software to unlock your computer by looking at the webcam and stuff. Moving the windows by moving yourself is great though.
This is clever and exciting. On some systems, the overhead from running the camera constantly might be an issue.
Another feature might be to pause a video if you are distracted (the tracker loses your face for 10-15 seconds).
well done, ciaparat!
I would encourage you to seek out accessibility experts who are working on computer systems for individuals with limited mobility and can not work with traditional input devices. Head movement and eye tracking input hooks need to well integrated into the existing accessibility frameworks. It would be best if you treated this exploration as a special case of accessibility more generally.
-jef
I’ve always wanted this. Looks brilliant. :)
I would probably have lean-back to Exposé if you’re not watching a video. Though I guess you don’t want to be rocking back and forth all the time.
@kennet, if you lean back the eventually scaled down windows will be less recognizable ;)
That makes me think of other uses I would see for it (other than the obvious accessibility as Jeff points out):
- Switching from headphones to speakers
- Lowering volume if I leave the room
- Pausing the movie if I raise my hand
- replaying the last 10 seconds if I raise my eyebrows :)
- forcing me to stretch if I haven’t moved much in 2 hrs :)
tons of fun coming up !
Clippy 2.0, eh? Generally I’ve had only bad experiences with computers guessing what I want to do. Explicit hand/motion gestures are one thing, but interpreting passive actions as commands is a recipe for disaster.
This is like the face gestures first of April joke opera did.
great one, bro’ ;)
Nice scenarios!
Thanks for sharing.
Head tracking is a well known topic for flight simulator players.
Track IR is the best known of the bunch:
http://www.naturalpoint.com/trackir/
Compiz already has a headtracking framework to allow a lot of this……
What I (and others) have always wanted is focus follows eyes. Now, I think we are still a way away from getting eye tracking. But, in a multimonitor situation, I’d love to have my mouse pointer jump to the screen that I’m looking at, and the focus go to the control that last had focus on that screen.
So often I find myself typing on a window on the wrong screen and wondering why it isn’t appearing.
I posted this on vimeo, but posting here too as it has wider audience.
There are more ideas here:
http://bit.ly/d5GlVI
Hi guy, I really your idea, I’m not a fluent programmer but I want to know if there is a mailing-list to share idea. I want to share my point of view about this great Idea.
Thanks for sharing
With Adobe’s ability to turn on your PC’s web Cam and Record you, via Flashplayer, I will never use anything like this.
Check out Adobe’s web site at:
http://www.macromedia.com/support/documentation/en/flashplayer/help/settings_manager02.html
@cyrildz, the best way to discuss things is joining Ayatana Mailing list: https://launchpad.net/~ayatana
@laxminarayan, thanks again for the link! ;)
Am really excited about the ideas posted by Anon on 14.09.10 .
Any source code available? Would be pretty neat to try out myself :)
wow, this is awesome!
will it come preinstalled with unity on supported netbooks in the future?
this could be an awesome revolution ^^
@Daan, I hope you know how to use Processing and third party libraries with it: http://nuthinking.com/goodies/FaceDetection_02_blog.zip ;)
Why don’t you fix what you already have before people starts to figure out the Emperor has no clothes?
Thanks Christian.
I know little about processing but the real expert in that area is a friend of mine Meredith L Patterson, she would love to experiment with this :)
Ubuntu is famous for making Linux accessible. What I would really like is to see is them bring hardware into the picture that benefits community organisations. For example, I’m president of several amateur sporting clubs, we would love to make use of things like RFID and/or Swipe cards but the hardware cost and implementation costs are way too high only commercial clubs can afford that technology.
A cheap off the shelf solution for managing memberships and access to venues based on low cost easily implemented hardware could have thousands of potential users if packaged correctly.
Really nice stuff guys! This are great ideas and I think it´s where the future goes. You should also have a look at our work and maybe get some inspiration.
http://www.youtube.com/watch?v=OHm9teVoNE8
(This is a prototype of a video player, that reacts to proxemics of people and devices in your living room)
@Till, definitely a more thought through solution… thanks for sharing! ;)
@Bruce, thanks for your opinion! I don’t think you will find many designers here in disagreement with you ;) But as said, no developer in Canonical is (and maybe will ever be) working on this.
Very cool work but when can I do something like this?
http://www.youtube.com/watch?v=b6YTQJVzwlI
This is the TED-talk of the guy who made the GUI to “Minority Report”. You can fastforward the first 5 minutes, then its getting interesting.
This could be interesting for “3D” Gaming without glasses.
Some of us who use open source software don’t use flash. Would it be possible for you to provide a link to the video in an open codec such as WebM or Theora?
I appreciate that this is asking you to do more work. However, I’d be happy to convert the (I assume) h264 video to one of these open codecs if you could give me access to it.
I’d really love to see this video, the write-ups I’ve read concerning it point to some interesting UI design ideas.
@Aidan, I am quite surprised Vimeo doesn’t offer you an html5 based version, at least through their website. Btw, here a Theora version: http://nuthinking.com/goodies/parallax.theora.ogv
Which is cheaper:
1. Gyroscope
2. Webcam
If its the we webcam, then it’ll be test to have tablets that have this feature in place to do most if not all gyroscope functions
@Shaun, it is not just a matter of costs but also battery usage (if we talk about tablets).
@Christian Giordano: nice demo and nice idea of course, I also used to think about user-awareness, but relating it to process scheduling rather than UIs; some more words on that here: http://ao2.it/en/blog/2010/09/21/user-aware-scheduling
Regards,
Antonio
Previous work:
http://www.reganmandryk.com/pubs/UIST08_epp_camera-ready.pdf
Wow, I just noticed this could be used to trigger actions depending on location. My personal use cases:
* I do gym exercises following P90X videos in the living room, and I always put the laptop in a specific place, then look the pdf p90x calendar, and then play the videos of the day.
* I play the drums, and always use tuxguitar for it. Also put the laptop in a specific place for that. Sometimes I go to the place I play with my band and use tuxguitar.
I think the laptop could notice these situations and automatically launch the appropiate activities. I could create “KDE Plasma Activities” (I’m a KDE developer, oh and looking for a parttime job btw :P ) for those things I do, and then let the ayatana plasmoid integrated with activities take some photos of the situation, then it might be able to recognize the place by the surroundings and change to that activity. wouldn’t it be nice? :D
GREAT!
Maybe it’s possible to scan the position of the eyes and to calculate with the angle of the cam, what the person sees.
So it should be possible to ignore the mouse (and the both functions a put-mousepointer-to-view sensor and the click-function could be shift on the keyboard).
So the (Ubuntu-)PC would be revolutionized. :-)
GREAT!
Maybe it’s possible to scan the position of the eyes and to calculate with the angle of the cam, what the person sees.
So it should be possible to ignore the mouse (and the both functions a put-mousepointer-to-view sensor and the click-function could be shift on the keyboard).
So the (Ubuntu-)PC would be revolutionized. :-)
What I’d really like to see is for the computer to see if there is someone in front of it or not; if not, then to not send out notifications until there is.
Face detection is extremely easy, face recognition is not; and at this time I would not advise it’s use. (Seriously!)
Another idea is also quite simple to recognize; gaze recognition. If the user isn’t focusing on the screen (and this is very easy to setup; the computer dosent need to know it’s screensize for this, it just looks for lack of gaze) then a video would pause; if there was a notification, it would make a sound, then wait till it got the user’s gaze before it displayed it.
Likewise, what about microphones? If there is a sudden jump in sound (for example, your watching a film, your friend comes in and dosent realize), then the computer could pause the music/movie until sound went down to it’s previous levels.
Another instance; gesture recognition; by this I mean recognition through, say, hand signs. Stop, forward, reverse, next, previous…Ubuntu could put that in very easily as I believe the OpenCV library makes gesture recog. very easy to do :)
I do think it’d be wonderful if the computers operated on our terms, rather then theirs. ;)
Very nice post. I just stopped by by your website and wanted to say that I have truly enjoyed checking your posts. Any way, I’ll be subscribing to your feed and also I’m hoping you post again soon!
Cool thing!
> Flash Player 10+ Required
Seriously? Give me a break.
It is amazing.
Folks will love it.
@Phil, I guess Flash Player is required if you are lazy enough not to go to Vimeo, and watching it there with HTML5, or to read the previous comment where I posted a link to an ogv file. Next time I will make sure this will be clearer.
hi this looks great, i was thinking why not let you use you fingertips as multi touch inputs. if you have a webcam, then you could wave your hands and fingers around the air, and use it just like a multi touch input device, but anybody could use it.
There is probably already something like my suggestion going on, but thought i might put the idea out there.
thanks Pancho
(My englisch is not so good, Dutch speaking)
The Unity launcher. I like this. Make it possible in 10.10. instead of the panel. And I mean in de desktop version. This is nice. Panel and dock in One.
Sounds as a nice experiment. As long as i keep the option to sudo aptitude purge idiocracy
Have to adore the effort you place into your blog :)
This could be great,
It looks like it could do what Johnny Chung Lee’s head tracking software did for the wii. That was some awesome stuff and I’d love to see that on a computer. Just google his name and watch some video’s. This could be a better value solution for 3d displays as having object move in perspective would make your mind think you were looking through a piece of glass. Boy I wanna play COD like that *sigh*.
Have a look:..
http://pam-face-authentication.org/
http://code.google.com/p/pam-face-authentication/
nice ))
There are so numerous large releases coming to the gaming world in the upcoming few years that several players are turning into confused and starting to ponder if they are heading to be able to purchase a console gaming system in the long run that has a remote control at all. The rapid solution to this is indeed, the real query is what that remote will glimpse like and how it will interact with the gaming console. Hardcore games will by no means give up the remote control totally since it provides way to considerably precision and manage that are unable to be in comparison to any movement based process controls at this position in time.
pE7i87 qprkugbfqnjy
This is the first time I visited here. I found a lot of interesting information on his blog. From the volume of comments on your articles, I’m not alone! keep up the good work.