In preparation for UDS, we conducted usability testing of Unity with general public users. We are now better informed as to where we should expend further efforts to enhance the experience for users working with Unity.
I summarize my findings below.
Participants Selection
We asked an external recruiting firm to find 15 participants who answered to the following criteria:
1. Mix of males and females: roughly equal
2. Age:
- 7 people between the ages of 18 and 30
- 6 people between the ages of 30 and 50
- 2 people 50 or over
3. Employment: employed full-time or to be full-time student
4. Involvement in marketing research: no such involvement in the last 6 months.
5. Basic technology: Each participant needed:
- To have a broadband connection at home
- To be a daily internet visitor, staying online for at least 2 hours every day for personal reasons
6. Online activities: Participants needed to have done all of the following while online in the last 2 months:
- Look for information
- Read comments written by others
- Read blogs written by others
In addition, they needed to have done at least 4 of the following in the last 2 months:
- Create or review a social networking profile
- Post comments or reviews
- Write a blog and conduct some activities on social networking sites
- Shop
- Share photos
- Play games
- Download music on a music player.
7. Participants needed to have a strong interest in technology.
8. Each participant needed to own at least one portable music device, one mobile phone, and one computer/laptop.
Of the 15 participants recruited, 13 were Windows users, 1 was a Mac user, and 1 used both Windows and Mac. None of the participants was familiar with Ubuntu.
Methodology
During one-on-one sessions, Unity was presented on a netbook. The goal of the sessions was to get participants to experience most of Unity’s features and functionalities. I introduced the session with the following instruction: “Imagine this is your new computer. What do you want to do with it?” I had songs and pictures available for them to import. During the 60-minute session, I asked them to do some of the activities they normally do, which included importing songs and photos, engaging in social networking, etc.. However, some of the tasks were tailored to fit the specific interests and knowledge of given participants. For example, if a participant was a professional musician, I proposed to him or her more tasks related to music management; similarly, if a participant was a student, more of the writing and reporting capabilities of Unity were explored.
Findings
What participants liked about Unity
The look and feel: First impressions of Unity were positive and many comments pertained to its elegance.
One participant said: “Simple and clear tabs on the side. I don’t like to be crowded with stuff on the screen. It looks quite approachable.”
The core concept of Unity: Many participants intuited the new concept underlying Unity’s look and feel and direction.
One participant said: “I like the application idea and that it is application-oriented and you can go there and have a quite user-friendly interactivity. I like the tabs at the top and when I hover it tells me what it is and it is big and clear, easy to see.”
Workspaces: Many participants were unfamiliar with the concept of workspaces and were intrigued by its overall potential, and its friendliness.
One participant said: “What you can do is that you can have different applications in different workspaces. This is a nice feature.”
Dash: Participants liked the simplicity of the dash and its look and feel.
Software Centre: Generally, the software centre was seen as impressive, particularly its large number of available free software.
Usability issues
Performance of Unity
Participants were challenged by various functionalities and conceptual design features of Unity. It is noteworthy that one factor stood out as principally responsible for many usability problems: this was Unity’s slowness and suboptimal responsiveness during testing. The level of performance in this regard significantly impaired the flow of use and the user experience. Testing was done on a mid-range netbook that users were likely to own, and consequently the user experience in this regard was similar to what users would have experienced outside of the usability lab.
Multitasking – Multitasking on Unity is disconnected and difficult at times
Task flow is interrupted: While working on a task, participants wanted to have all the documents and websites they were using easily available to them. For them, the task was the unit of organisation of all resources and tools. Thus, while working on a task, participants expected that Unity would provide them with a representation or visibility of what was available to them and how to easily access what they needed at any given point. Unity does not, however, make evident the resources and tools users have at their disposal — whether it be multiple documents, programmes or websites. In a word, while Unity relies on users to keep track of the resources they are currently using, users are habituated to relying on the software to “keep track for them,” by making the resources highly visible, for example, by means of tabs. With Unity, resources are hidden from view.
Overall, participants found the navigation between documents to be cumbersome. Often, they looked for a way, a back button or breadcrumb, that would navigate them through their open applications and documents. They didn’t find either. As a consequence, they ended up going through the files and folders section to access their open documents. Clicking on the OpenOffice icon to access a document already open was not immediately discoverable.
One participant noted: “[navigation] seem[s] awkward. I’m not getting around as quickly as I should. The icons are supposed to tell you but I don’t know what they are. That is a problem.
Another said: “It seems a bit pedantic to have to go through the application menu to navigate.”
Poor visibility of applications and of windows that are opened: Participants experienced difficulties with the following:
Minimising a document: When participants minimised a document, the document seemed to have disappeared when they expected the mininised document to be shown at the bottom of the screen. In Unity, there were no traces of the document on the screen and participants could not find it again.
“I’m not entirely sure how to get back into my new document [after minimising it]. [It is] not clear that it opened a new page for it. I don’t know how to get back to it. I don’t know, it’s weird. I expect it to be a tab and be able to switch between the two.”
Knowing which documents are currently opened: Participants made it clear that they not only wanted to know if their documents were opened, but also wanted to know how many documents or windows were open at any given time. But they did not always see the white triangle that indicates that a programme is running or that documents are opened. Consequently, they were not aware of what was available to them. In short, the general indicator was not enough for their needs for awareness.
Being able to distinguish between similar documents: In the exposé view, documents are very small and, when 2 documents are similar, participants could not tell which was the one they wanted without opening each one individually.
Identifying a document: When participants were in the process of writing a document, they looked for the name of the document they were working on in the top menu bar, but couldn’t find it. They were also looking for such information in the exposé view.
Being aware of multiple open documents or windows they needed to consult or modify: Almost all participants wanted to organise their work by opening several windows and physically positioning them in a way to maintain an awareness of all of them. For example, one participant opened Facebook and a word document. She wanted to move the word document so she could expose the top part of Facebook.
“Can you move the window down at all? I like to have a little bit of space for organisation. I want to be able to move the window down. It would be nice to [be able to do that], especially if you want to start working, you want to organise your environment. You want to feel the sense of relief and organisation before you start working and [the sense] that what will come to you will be manageable.”
Immediate access to documents they currently are working on: Exposé is not readily discoverable. The majority of participants could not find the exposé feature. As it is not a feature they are used to having, they need a prompt to help them explore it.
Exposé does not allow users to work on documents side-by-side. Participants expected to use the exposé feature to be able to cut and paste between documents directly without having to bring one document up in full screen and then having to navigate to the next document by means of the workspaces icon. As it is, exposé is useful for viewing only. Participants would have liked to have had the option to directly navigate and conduct operations (e.g., cut and paste) between documents.
Documents cannot be resized and placed side-by-side: No participant could find a way to resize his/her Open-Office documents in such a way that they could be placed side-by-side while working on both at once.
Overview of what’s going on: Many participants wished they could have a view of what resides in various parts of their computer, as is facilitated by Windows’s “my computer”.
Back button: Many participants needed to “go back” to a document or an application that was currently open. But the back button doesn’t work coherently and seemed to bring participants to arbitrary pages.
“The thing here, backward and forward buttons don’t work. Back button doesn’t get me back to where I was. I don’t know where it is taking me, I’m frustrated. The back button is taking me somewhere [else].”
“If you can’t navigate you feel like a fool and you feel ashamed even if you’re alone in your room.”
“I’m struggling to find short-cuts and a back button. Right now, to me, it feels very clunky to go from one place to another.”
Document management
Open documents: As already indicated briefly, participants didn’t know how many documents they had opened. The white triangle indicator, meaning that the programme is working or that at least one document is currently open, is too general for users’ needs.
Delete documents: Participants could not delete existing documents from their files and folders. They first tried to drag and drop documents in the waste basket; then they right-clicked to see if they had such an option.
Copy and paste: Copy and paste from one document to another didn’t always work for participants.
“Copy and paste is not going to work. I’m not impressed, it must work. I’m one for copying and pasting.”
Lack of feedback and guidance
Unity is often slow, and as a result participants tended to be confused about what was going on. They didn’t know:
1) if the system was slow and would eventually produce the desired effect, or
2) if the system had not registered the participant’s command and as a consequence, they should repeat their action, or even
3) if the system had crashed.
Unity needs to clearly show if it is engaged in a process or not.
“Has it crashed?”
“Let’s make sure I clicked it. [She opens a document. She’s getting confused navigating between documents and she closes the document without saving it.] When I close, it should ask me to save it. This is not good [that it doesn’t]. You rely on the computer to save the document or not.”
Search
When entering a search term in the search bar, there is no feedback to show that Unity is indeed engaged in a process. Many participants were confused by this.
“How do you know this is searching? Is there any sort of symbol to tell you if it is doing something? I would like to see if there is a symbol that says the computer is working so I am not clicking around.”
Other issues: Interactions with the launcher, finding the dash, lack of reference to understanding Ubuntu features
Interactions with the launcher
Adding an icon to the launcher: Many participants were not able to add a short-cut of an application to the launcher. They used various strategies but failed:
- They tried to drag and drop icons directly from the applications section to the launcher.
- They right-clicked, expecting an option to copy.
- A few participants noticed that when an application is open, it appears in the launcher, and when they close it, it disappears. They were confused by this feature, but at the same time perceived it as a clue. So they tried to drag and drop the icon from the applications section onto the similar icon visible in the launcher.
Changing the order of icons in the launcher: Most participants failed to reorganise the order of icons in the launcher. They selected an icon and tried to drag it upwards within the launcher; that is, they did not realise that they needed to remove the icon from the deck first before they could relocate it.
Deleting an icon from the launcher: Participants did not succeed in deleting an icon from the launcher. Most selected the icon and then dragged it into the waste basket at the bottom of the launcher. They did not see that they could remove an icon by dragging it horizontally away from the launcher.
Finding the dash: The majority of participants who found the dash found it by accident. They were not sure what it was, and didn’t know how they had gotten there if they accidentally had. As a consequence, they were not able to find it again. The problem for participants here was that the logo is not recognisable to them. Of course, they were not familiar with Ubuntu; but, equally important, the logo is less visible in size and colour than the icons contained in the launcher. It took participants a while to see it at all.
“The logo should stand out more. It would be nice if the icon itself [was] coming at you, more of a three-dimensional thing.”
Lack of reference to understanding Ubuntu unique features
Software centre: Almost all participants were surprised when I suggested that they install a game that was not currently installed on their computer. All of them immediately went to the internet. When I told them that such games could be found on Ubuntu, they were confused and many didn’t know where to find them. They went to the applications section, where there remembered a section called “installed” software. They expected either a section below this one with a list of uninstalled software or (at the very least) a link to the software centre.
Moreover, most participants expected to be able to click on an icon in the dash, for example ‘music,’ and directly access their albums and songs. They were a bit surprised to be taken to music applications instead, especially in light of the interactions of the other icons in the launcher, which are quite different.
“I expect to see music files, names of songs or albums, anything you’ve downloaded. Titles.”
Familiar short-cuts: Participants who normally used short-cuts to navigate didn’t find the familiar short-cuts and were frustrated.
“There are things that are nice to keep from one OS to another, like right click and short-cuts. It’s nice to have two possible solutions.”
Changing the wallpaper
Many participants did not succeed in changing their wallpaper because the default screen of appearance was open in full screen by default. Because of this, the action of immediate wallpaper change, as users select a wallpaper, was not visible to them. After participants had selected a wallpaper, they expected a step to confirm that they wanted their wallpaper to be changed. Instead they were given other options that were not relevant to what they were trying to do, and at which they had already (but unbeknownst to them) succeeded.
Search
The search functionalities felt limited to participants. When searching, they didn’t know what the field and scope were that were covered by the search engine they were using.
Grey icons in launcher
Participants thought that the grey icons in the launcher were inactive, especially when they clicked on them and the system was very slow in responding
This kind of usability testing is what the modern free software desktop needs, so well done Canonical! Hopefully the Unity-Compiz rewrite is going well and you won’t hit any limitations with being a compiz plugin and can soon focus on improving Unity itself to make it shine really bright for Ubuntu 11.04.
I love the usability testing done. It really gives developers some great ideas for what kinds of changes need to be made in their applications to make them super user friendly. Hope guys publish more stuff like this.
Also some great hints about what might be “fixed” in 11.04. =D
Ubuntu will need a hint’s and tip’s screen when first starting up. Just like windows95 had. And Windows vista also had something similar. Because they were a big change to what people are used to
Usability testing is what Linux has lacked all these years. Reading a post like this fills me with a lot of optimism, even if it full of epic fail with the usability of Ubuntu.
The best Linux usability article, ever. Unfortunately Unity itself is not yet there…
I agree with the previous comments, this is really what “Desktop Linux” needs right now. Kinda doubt even the most important issues will be thoroughly dealt with before 11.04 though. (because of the limited time)
This is great work. I’ve long wanted to colliade my interest in user-centered design and usability testing with Free software.
Would it be possible to release more details about your methodology so that other members of the Ubuntu community could repeat your methodology to run similar tests or to rerun the same test with a different group of subjects and tester.
Thanks for doing this work and publishing the results.
It would be wonderful if research like this could be conducted in a per-distro basis. Even better if such efforts could be conducted before launch, such as during beta. Obviously, this would cost a lot.
Does canonical consider this to be even an option?
This is the main thing that Linux was lacking. I’m more than pleased by such market research done by Canonical. This is the reason I’m glad Ubuntu changed to Unity.
Thanks for the usability testing and publishing the results. Reading their quotes was interesting.
This kind of testing exposes the very real issue that GNOME shell, Unity, and Windows Vista/7 all share: fancy menus hiding open documents. For all its sins, the current panel at least lets me see what windows I have open. It is far too easy to “lose” an application which is open under the current Unity/GNOME shell implementations.
With regards to fixing, I would rather go with a single, tabbed interface, somewhat similar to the old netbook-launcher, because it does not have the same issues with “hidden” windows.
I totally agree with you. Ubuntu its a hole new world to the most of the people. So they need some guidance to get confort with the system.
Great initiative by Ubuntu. I’d like to see the same test against experienced gnome, kde linux users.
What do you mean by “per-distro basis”?
Excellent initiative!!!
This is the right way if Ubuntu is to be made user friendly so it reaches the masses.
Correct me if I am wrong, but the focus of this research is unity right?
What I had in mind was conducting such research for each and every future ubuntu release.
It seems that my word “per-distro” caused confusion. That was not what I meant. (Of course if every other linux distro does this for each of their future releases, that would be good!)
Thank you for publishing the results here.
This shows one thing: More usability testing is needed. It’s critical.
Nice to see how you published all the criticism!! I am sure developers will do their best to make Unity shine!
As a side note: this is certainly not the most important issue, but regarding people expecting to be able to interact with window content in the “exposè” view, Input Redirection patches for X would allow that. It would be a great feature although probably not the most important one for this cycle…
Well done!
Now I hope that each of the usability issues gets it’s own “high priority” bug in Launchpad, that there is an open discussion about how the issues can be fixed and that the issues can be fixed within the framework of Unity.
Glad to see my own concerns reflected in the testing results.
I’m yet to see a better way to organise running apps then a simple window list, and I hope that whatever ubuntu and gnome evolve into, that I’ll always be able to have a window list applet on a panel somewhere.
I’d love to see someone come up with something better, but I think it’s clear that no one has yet.
Interesting stuff indeed. So there was basically the only one thing people liked about Unity. It’s an app launcher with big icons. So if you keep the current desktop environment and put a panel with big icons on the side (just with launchers, no taskbar feature) users will be happy the most. I’m afraid many issues the users were having are in the concept (design) of Unity and can’t be easily solved. Well, there is always a solution and you can find workarounds for every issue. For example, you can make a dash with a list of opened documents to solve the “document confusion”, but it’s not really a solution. And I would like to point out that this usability test was done on netbooks with a small screen, but Unity is going to be for every desktop user, even for users with screens 24″ and larger. On these computers, even more issues will be revealed. For example, the global menu. If you have a small window opened in the low right corner you have to move the cursor all the way to access the app menu. Try it out on a 30″ screen, it’s a usability nightmare.
Thanks for the results… I’m a Windows and Mac user (sometimes using Ubuntu). I used the Unity over Desktop Edition 10.04 and liked the fell of it, but with the same concerns of those found on the test and that I couldn’t see Youtube on full screen >.<
Software Centre is awesome and i hope more cool applications get there.
I completely agree on the big scree issue.
@mpt
He meant before every release. i.e every 6 months
As always, brilliant research findings.
I’m looking forward to helping Unity grow.
“But they did not always see the white triangle that indicates that a programme is running or that documents are opened.”
This dose not surprise me at all.
Good effort. We need more of this.
I am not familiar with the research that Canonical has done on Unity to date. Did you do something similar at the beginning of the development process, to check that you were on the right track? Some of those issues seem to come from the main ideas that shaped Unity, so it would have been nice to spot them as early as possible.
My respect for the transparency of communication. Although I’m not a friend of such per se terms like ‘unity’, which implies lack of individuality. As long as there is an opt-out option I’ll be happy with my gnome-panel, transparent cube and eight+ workspaces. However, for them who would like to use ‘unity’ hopefully it’ll be a joy. Thanks to all who developed Ubuntu the way it is today and getting better each release.
– Create or review a social networking profile ← no, online stalking sucks
– Post comments or reviews ← doing atm
– Write a blog and conduct some activities on social networking sites ← “social networking” is an euphemism for “data collection”
– Shop ← “secure online payment” is an oxymoron
– Share photos ← why would i want anyone to be able to see my photos?
– Play games ← Adobe Flash should definitely be replaced by something else
– Download music on a music player. ← i lost the cable a month ago, didnt show up on wikileaks yet :(
So i should probably stop using ubuntu.
When I installed ubuntu 10.10 netbook on my laptop, I was surprised and mesmerised by the new and elegant look of its desktop.I am glad that u kept such a trancparency in communication that u published all its tests and comments on it. I hope this brings a new era of Ubuntu desktop. I would like to thank Unity and Ubuntu teams for providing such a brilliant new look to the Ubuntu linux.
So, was anything done to fix the issues found in this
usability study? Judging from my experience, the
answer is ‘not much’. See my comments at
http://ubuntuforums.org/showthread.php?p=10663065
in which I describe problems finding application groups,
as well as groups that never show up even when you know how to look.
Hey MPT,
How’re you feeling about the results, was it as you expected, worse or better?
Is everyone at Canonical sighing and facepalming at all the work that needs to be done? :)
Dan,
Sorry for the delay. Your comment came in as I was doing Benchmarking testing of Unity. Now I have the results.
Many problems we found in the previous testing were resolved:
Performance issues have been resolved. The test went smoothly.
Multitasking: Participants could multitask, they were able to open several windows and navigate between them.
Awareness of running apps: Participants could tell which applications they had running during the test by looking at the white arrows.
Moving windows: Participants were able to easily move windows on their desktop.
Displaying documents side-by-side: Participants were able to do that.
Delete a document; Participants could delete unwanted documents.
Copy and paste: Participants copied and pasted text from one document to another while they had them opened side-by- side.
Adding icon to the Launcher: Participants dragged and dropped icons from Applications Lense into the Launcher easily.
Reordering icons in the Launcher: Participants were able to change the position of an icon in the Launcher.
Changing the wallpaper: Participants changed their wallpaper easily.
I agree with Mikko, ‘Unity is not yet there’, I tried to install Unity (Ubuntu 11.04) from Ubuntu 10.10 at home and it wiped out everything. Our IT Instructor installed it on the class computers and did his magic in order to get Ubuntu 11.04 to work on our computers in class.
H6XlID xzxeewmtfoql
I’m passionate about the work you guys are doing. Personally I found this page and this post because I want to delve into these issues. From my point of view there are some ways to fix or implement. I work with design software, Inkscape and Gimp specifically (among others), and what would be good would be to drag images from Nautilus to the Inkscape icon and it will open automatically and the image can be inserted into the software design. That’s something urgent. But in general terms, I believe that today, in 2013, Unity is one of the best-designed interfaces and provides an almost complete and comfortable experience. I respect your work and love what they do. Congratulations. Very good job have made, the developments that have taken Unity is incredible.