Over the past few months we have been working on improving the multi-monitor experience in Ubuntu. We took the opportunity at UDS in November to get some feedback on a prototype, which shows how we are planning to develop the multi-monitor experience over the next few cycles:
Here is a short video of the prototype in action at UDS:
http://www.youtube.com/watch?v=lbwNMnNUGFA
We invested in a six monitor rig and the prototype to test a number of different display configurations and to ensure that our design ideas scale well. However, our main focus for Precise is to ensure that we deliver a reliable and supportive experience for the core use cases, such as connecting to a second display or projector, disconnecting displays and using a closed laptop with an external display.
So here is the Phase 1 specification, scoped for the next couple of cycles, incorporating the feedback we got from the prototype and sessions at UDS:
http://design.canonical.com/the-toolkit/unity-multi-monitor-interactions/
Work continues now on the prototype, which will be used to conduct usability testing on the launcher, spread, window management and workspace interactions for multiple monitor setups. We will be publishing the prototype on this site (the Ubuntu prototype application, along with the Qt C++ source code) in the near future, so keep tuned for more Multiple Monitor news.
Very impressive specification, looks really well thought out. Keep it up, guys!
Hi! I really hope MultiMonitorHandling will be improved cause at the moment, it has some parts that really don’t make any sense.
For example: myself and most other people i know working with multiple monitors is to have multiple windows open and viewable. NOT maximized. Since uity, all windows open maximized which defeats the purpose of running multiple screens anyway. if i have to ALT+Tab through windows anyways, theres no need to have multiple monitors.
Also 2.12 of the specification is – from my point of view and working experience – counterproductive. As i understand it, unity was developed the reduce clutter. Now i have TWO times my name shown, TWO clocks, TWO dropbox/u1 icons, TWo network indicators, TWO weather indicators, TWO skype indicators, TWO sound indicators, TWO mail indicators and worst of all, TWO system-monitor indicators.
And please don’t see this as a rant just for the sake of ranting! I really appreciate the work ypu’re all doing on FOSS and ubuntu!
Very inpressive…
You are working in new buttons too? I love the buttons in this mock-up http://goo.gl/aOaWP
Very soft and sutil.
I like this! I love that you’re approaching this in a dedicated way, and I really appreciate that you’re sharing the specification early.
A couple things I’m not so sure about:
Is anyone going to find the “Login on this display” option? In the specification, you can click the CoF icon to use the greeter on that display. Why not add a label beneath the CoF icon (saying what happens when you click) that appears when the mouse enters the display?
In the section titled Default Configuration Notification, you mention the Snap Decisions framework. Is there somewhere we can look for more information about that project?
One comment. In section 2.3, it says:
“If a window was fully contained within the removed display, then an attempt is made to place it in the same location (top-left coordinates relative to top-left of display) on an adjacent display”
Rather than using top-left coordinates, wouldn’t it be better to match the approximate position based on the dimensions of the screen? So rather than having it measure 200px from the top, 883px from the left, instead it could measure 1/8 from the top, 2/3 from the left. That way, if you have a small window in the bottom right corner of a small screen that is disconnected, it will show up in the bottom right corner of the larger screen it is placed into, rather than somewhere in the middle of the larger screen.
Seems like this would mirror the way users think about window positioning more closely. It’s more complicated though, I’m sure.
OK, one other comment, from section 2.5 regarding pillarboxing and letter boxing. The spec says that the black bands should be equal on both sides. If this is a secondary display, might it make sense to push the black bands together on the side opposite the adjacent monitor? So if I have two displays side by side, there won’t be a black band between them?
One of the most annoying things with virtual workspaces is for me that a program window can rise up into another workspace. If this is the case for multi-monitor too, please make an option to prevent it or add sticky borders to the screen to make it harder to happen accidentally.
Will you be able to support monitors of different sizes and rotations as well? This is an important part of multi-monitor support.
One multi-monitor thing I’d love is for the Gnome login prompt (username/password GUI) to appear on all the monitors, instead of just one of them.
Currently I have to search for the one monitor it’s on, and it changes on pressing Enter if I have moved the mouse around (this is with Nvidia’s TwinView in Ubuntu 11.04, would be surprised if it had changed in 11.10).
There’s no good reason for every monitor not to show the login prompt (the same in all, with password stars appearing on all as you type, for example), since until you’ve logged in, the remaining monitors are currently useless (they just show an empty screen with the login background).
Just replying to my comment above, after reading the “Greeter Screens” section in the spec.
Have to disagree completely with this: “The main Greeter screen is shown on the Primary display. A secondary screen is shown on all other displays.”
That’s what happens now, but for the reasons described above, it is suboptimal. IMHO, the greeter screen should be shown on all displays, with the same state/status in all displays. Essentially the spec is describing what happens now as regards the greeter, rather than having ambition and describing a more desirable destination. What happens now is not a good paradigm and it does not work well for logging in to multi-monitor setups.
“Alternatively, the user can simply click on the Circle of Friends icon in the middle of a display to summon the Greeter across onto that display.” –> no, again, display the greeter on all of them, and the user is sure to be able to see one of them! And if you really need to know which one is the primary for something that comes later, then have a radio-button on each display, to say “Make this my primary display”. But please, show the login prompt everywhere.
Interesting specification. I really love how workspaces and spread are supposed to work. But the specification should not say things like windows are opened on the screen where the mouse pointer is located. It should define display focus early and then only refer to that later in the spec.
For instance, I personally move the mouse pointer to a screen that I _don’t_ use, to get it out of the way so it doesn’t keep annoying me by hiding text, etc. This keeps confusing me when I use keyboard shortcuts, such as alt+tab. So, what screen gets focus should be its own topic. For instance, it would be very nice if one could use a webcam to decide display focus. Or based on the last activity. Or EEG, or anything else. I like the concept of a display focus, but it should be less narrowly defined and in one place, early in the spec.
This is Ubuntu. Making the future.
Great!
A question:
The document says:
——–
1.3 Competitor Products
* NVIDIA/ATI software extensions for graphics adapters
——–
Will this solution work with nvidia/ati hardware graphics acceleration?
I can imagine that it might cost some gpu computing power, so it would be nice to make this work with that extra hardware.
I was also surprised to see NVidia / ATI proprietary dirvers considered as competitor products. Let’s not forget that for anyone who considers gaming on Linux, these drivers are the only option.
The fact that the nvidia does not support xrandr has bugged me since I’ve started using Linux with a dual screen (2006). The default twinview setup only offers a single metamode and games can’t manage to switch resolutions.
My workaround for this problem was to manually add modelines in Xorg.conf for the most commonly used resolutions in games. But I always considered this to be a dirty hack.
I had noticed that it was still possible to change resolutions or turning off the secondary monitor with the nvidia-settings tool. With this tool being open source and as the author of the Lutris gaming platform, I wanted to integrate this functionnality into the platform.
After reading some source code for a while, I realised that there was a sample tool to do everything I wanted.
The tool is name nv-control-dpy and adds for Nvidia GPU all the missing xrandr fucntionnality. Getting it is quite simple:
– fetch the source for nvidia-settings
apt-get source nvidia-settings
– go in the samples folder
cd nvidia-settings-*/samples
– type make
– go in the build folder and you have the tool
cd _out/Linux_x86_64
./nv-control-dpy
My goal is to write a wrapper around xrandr using this tool which should not be hard to do, the hardest part would be to override the xrandr functionnality without the commande line tool.
I can have a use for this tool as long as I use it in my gaming platform but outside of it, it would not be usable out-of-the-box (if a SDL game wants to switch resolutions ) I doubt it will be making a call to the xrandr cli tool.
Anyway, all of Canonical’s efforts regarding multi-monitor support are great, but this TwinView issue remains the main problem. The nouveau driver has never been an option and never will be until it has the performance and features for 3D as the nvidia driver.
How about measuring windows from the nearest monitor edge and going from there? A 640×480 windows in the lower right on a small monitor would then be placed in the same spot on a large monitor. Coordinates are being used anyway, so why not measure from the edge rather than straight Cartesian?
Ulrich, try pressing +F5 in Unity if you want to reduce the size of your windows. It works perfectly in Unity. Also, if you mouse over the menu bar a the top left, the familiar window sizing icons appear. Click the square to unmaximize a window.
Edit: Alt+F5
That’s a very good and comprehensive spec. I would like to agree with a couple of previous comments though (and add a couple of my own):
* Put the greeter – and especially the lock screen – on all monitors. I sometimes turn my monitors off, but then want to turn one on to check the status of a running task. Depending on where I left the mouse, the monitor I turn on may not have the lock screen prompt.
* Re-position windows proportionally, rather than by absolute pixel coordinates. For windows whose width or height is “small” (e.g. less than 50% of the target screen size), position them relative to the closest edge(s), not just the top-left corner. This lets me have a progress dialogue in one corner and a chat window in another and have them sensibly placed when I disconnect a screen.
* It gets a brief mention, but is worth making clearer: the target for new windows should be the screen with the mouse pointer AT THE TIME OF THE ACTION that creates the window. If I click a launcher for something that takes a few seconds to start, I want it opened on the screen I clicked on, not one that I’ve meandered off to since then.
* With the workspace switcher a nice option would be to have a keyboard modifier to let me only switch the workspace on the current monitor, so I can “mix and match” apps that are in different workspaces on different monitors. If this is technically impossible, it could be emulated by automatically “rotating” the windows on the current monitor between workspaces to give a similar effect.
* I would suggest that notifications appear on the screen with the mouse pointer AND ALSO the screen with the focused window. Usually these will be the same, but if I’ve moved my mouse away to focus on some typing, they may not be.
YEAHHHHH !!!!! FINALLY !!!
you need to consider the case of a docked laptop with 2 external displays being ejected and those two 1080p resolutions going away and only a small 1400×900 display being what it on the laptop remaining.
This use case or ones VERY similar to it are what I hear from endless developers (personally at least 12 people have said this exact case to me) leaving windows to do rails development and trying ubuntu but giving up and buying a MacBookPro instead. launchpad bug #1 might be closed, not by ubuntu getting that marketshare, but rather ubuntu driving the marketshare to apple.
Hi Mike and OG (may I call you OG? :)
I agree that we can do better than using the top-left corner as our frame of reference for placing the windows.
Mike, your suggestion got me thinking along the same lines as OG, using the proximity of edges to determine new positions. I’ll post again when there is something new to see in the spec.
Two for two! That’s a great observation. I’ll need to tease out the edge cases (mixed resolutions/orientations, stacked arrangements), but we should get that in the spec. Thanks for your suggestions.
And youtube on dual monitor still not working properly…
Hi Jo,
Agreed. I will factor out the Display Focus rules into their own section.
On Display Focus itself, I did initially consider choosing the display which contains the focused window, and then the display which contains the cursor, if there was no focused window. However, in wanting to provide a predictable system response to input from the user, I wanted to avoid using too many ‘if…then…else’ rules, as they are likely to diminish the chances of the user building a good mental model of cause and effect in the OS.
As you say, going forwards there are some interesting ways by which we could determine display focus. Thanks very much for your comments.
Hi Mathieu,
On the ‘Competitor products’ point, good observation. I have changed this section to ‘Reference products’ instead.
Thanks.
Great! Now, maybe, multi-seat? (No, not term-server, but actually multi-seat, as in not only multiple monitors, but also multiple keyboards, mice, microphones, and speakers/headphones, each serving a different user. It would make the ultimate family/group/internet-cafe computer.)
Hi Dylan,
Thanks for the feedback and comments. The idea of drawing more attention to the COF icon (as the means to login on a secondary screen) when the cursor enters the display is nice. I’ll update the spec and let you know when there is something new to see.
I see this – the login prompt appearing on every screen – in 11.10.
Hi Nick,
Thanks for the feedback.
Actually, on my PCs the Greeter currently appears on every display (the display configuration is in Mirror mode by default on startup), and I personally don’t care very much for it.
I think it is more elegant (not to mention more desirable and ambitious :) ) to welcome the user onto the desktop through their display of choice. If they change their mind, they can choose a new display to Greet them.
There is always a tension to be managed, between establishing character and elegance, and optimising for maximum usability (eg. always visible everywhere).
I would like to think that we have struck the right balance with the design in our spec, especially if we hint at the purpose of the Circle of Friends more strongly, as suggested by Dylan in a post above.
Certainly we could tick the box for usability (visibility, discoverability, efficiency) with more confidence if we plastered the Greeter over every display, but I don’t believe that experience design should be ruled by usability hueristics alone.
I’d like to see Ubuntu using multiple screens on ISS. :-)
Thanks very much Bálint, and thanks to everyone who has posted their support, feedback, suggestions and constructive criticisms. Very much appreciated.
Hi MarcC,
Thanks for the feedback. You’ll see some responses on the ‘Greeter on each display’ and ‘Proportional window placement’ topics a bit higher up in the posts.
You know, I meant to add that ‘AT THE TIME OF THE ACTION’ clause, but I must have got distracted. I will add it now :)
Regarding the point about having different workspaces on different displays, you may be able to see that this feature is demonstrated in the YouTube video of the concept prototype we have created.
We will be working on a Phase 2 spec next, which will detail these behaviours. We will also be making the concept prototype and its source code (Qt C++) available at some point in the future.
Hi there,
In the post-12.04 concept prototype, we defined the behaviour that a window can belong to one workspace, and one workspace only (basically, when you organise workspaces, you do so in a Spread-type mode [Spread is the mode used to switch between the windows of an app]) You may even be able to make this out in the YouTube video.
This won’t be landing in P I’m afraid, but you will be able to check out the prototype once it has been made public.
great, you folks rockz!
Great work here! If all is implemented properly, Ubuntu will have the best multi-monitor handling in the world!
My reactions :
-As Dylan McCall said, i’m also not sure if it will be clear for all users that they can clic the CoF to get a login box. Maybe a small “Clic to log-in” can be useful.
-How do you think your implementation will work with proprietary drivers?
I disagree, the possibility to open a login box by clicking the logo is ok as long as this possibility is clear for the user. A multi-monitor setup with a login box on each screen will look retarded and will confuse the user who may think that the display is in mirror mode.
Personally I have a PC connected to a projector which is turned off most of the time. Currently with Ubuntu 11.10, the windows opens randomly on one of my two display, this is very annoying when it’s on the turned-off projector space. There is an option in compiz to open new windows on the screen where the mouse cursor is, but this does not work anyway.
I think using the mouse cursor to set the focus is a good choice.
Excited to see better multi-monitor support coming!
One multi-monitor feature that I’ve been waiting on for years now is the ability to change the workspace on one monitor only, NOT both at the same time. I like to have one monitor where I can flip around between workspaces depending on what I’m working on, and one to semi-permanently display the same window (e.g. a web browser). Right now I have to set that window to “Show on all workspaces”, which essentially means that workspaces for that monitor are useless.
It seems very natural to me that there should be a way to decouple workspace switching on the monitors. I’ve heard several people say that when they have multiple monitors, they have one dedicated to a single task (often email), and use the others for the rest of their work.
If designed correctly (so users don’t accidentally switch the workspaces on the wrong screen and wonder what happened), I’m sure this feature would be a big help.
Nikkita, you can do this already with compiz settings by setting the multi display option to the style you want (settings in desktop wall or desktop cube, depending what you use).
+1 for this. I use a laptop connected to an external display with laptop facing me and external display facing clients. It can be rather embarrassing when something I expect to open on my display comes up on the client display. LibreOffice is the worst offender, despite having set specific compiz rules for LibreOffice windows.
Hi Luke,
Absolutely. There is not a great deal of coverage on this in the spec, as the correct behaviour for dealing with heterogenous display arrangements and rotated displays should hopefully be well understood. Where we deviate from the correct behaviour, we will raise bugs and, where necessary, clarify contentious points in the specification.
My original comment was posted while I was at work, unable to see the YouTube video, so was based on how things sounded (to me) in the spec. Now that I’ve seen the video the option to have different workspaces on different displays is clear. If that is the default behaviour, then the opposite of my original comment might be a good idea: a modifier that lets me switch the workspaces across all monitors at once.
Regarding the Greeter and Lock Screen on each display, might there be some use in having a translucent or desaturated version of the username/password field on each of the secondary screens. This could be subtle enough to still make a clear distinction between the primary and secondary screens, but still allow for people to enter their credentials while looking at a non-primary screen without having to mouse around and click to activate a particular screen.
Very important to make sure Wine runs properly, and that after say running a game, that the resolution and fonts are returned to previous settings.
What I’m saying is if you make a multi monitor display API make sure it’s happy with resolution changes on single monitors, whilst maintaining others.
If each screen was a zone with its own zoom level for windows draw up then you could have an area encompassing all monitors at different resolutions.
You got it. Me, too. Monitors should be connected to workspaces. All major graphics cards should be supported, multiple of them in a same computer.
For example, I show slides, exercise paper and a timer in a projector in workspace 1 and in workspace 2 using my own screen I can do my own stuff like email, chat.
In a second case I run
. a code editor in workspace 1/ monitor 1, and
– my local tests in workspace 2/monitor 2 and
– my connection to Hudson continuous integration server in monitor 3 /workspace 3
– my video calls in monitor 4/ workspace 4.
Note that the monitors are not equal. Newer ones are 30″ 2560×1600. Turned 1080p is the norm and old monitors support 1600×1200. Analog is preferred and HDMI depreciated ;-)
Just please remember us poor (Lenovo/IBM T-series) laptop users that can support dual external monitors when docked. Every time enterprising individuals tweak this in the name of usability, they inevitably forget laptop docking & screw us up. It ends up being easier for me to fiddle with xrandr for several hours until I get it /mostly/ working:
$ cat `which dock`
#! /bin/sh -e
xrandr –output LVDS –off –fb 3200×1200 &&
xrandr –output DFP1 –auto –output CRT1 –auto –right-of DFP1
$ cat `which laptop`
#! /bin/sh
xrandr –output CRT1 –off 2> /dev/null
xrandr –output LVDS –auto –output DFP1 –off 2> /dev/null ||
xrandr –auto –fb 1440×900
$
Please make “discovery” work flawlessly first. Please let us store “configs” that we can cycle through and/or enable automatically when a connection event occurs.
“There is not a great deal of coverage on this in the spec, as the correct behaviour for dealing with heterogenous display arrangements and rotated displays should hopefully be well understood. ”
Not sure I get this — what is the correct behaviour? I don’t think there is a single “correct” answer. It is dictated by the layout the user wants to have for the screens.
I should be able to specify this — its important for a lot of embedded applications — rail stations and airports, for example (maybe not the best example), often use multiple portrait displays (4 to a deck) with two in portrait mode, showing the same thing, and 2 in landscape mode showing timetable data. Some traders do this too, with “wing” portrait setups, and central landscape setups.
Some may not even be oriented appropriately, so using tricks like querying the device may give you wrong answers (think the problem that wikimedia just had with their EXIF rotation idea that they rolled back.)
Adding a little rotate-clockwise/rotate-acw button on the monitor icon itself may help here.