Top 10 Tools to Measure User Experience

October 27, 2009

Pragmatic Marketer Magazine Volume 7 Issue 5

Because there is a direct correlation between techniques used to measure an online customer’s experience and the ability to persuade him or her to act. As more of our daily activities, both personal and professional, are managed online, most organizations are responsible for multiple websites and web-based tools that serve a broad range of audiences: customers, prospects, employees, channel partners, regulatory agencies, etc. That means online success depends on the ability to accurately measure a website or web application’s performance for unique target audiences on multiple levels across multiple dimensions.

However, relying on assumptions drawn from rudimentary measurements of user-centered design can’t provide sufficient data to tailor a website or web application to your users’ unique needs.

What’s more, when determining whether a website or web-based application is meeting the needs of its audience(s), it’s important to separate aesthetics from function. For example, your product may be very well-designed and visually appealing to your users but at the same time, can be functionally “unusable.”

We’ve all experienced unusable sites and applications. Whether it’s the online help site provided by your cable or satellite provider, or the online banking system you use to pay bills. Beautiful design can be painfully frustrating for users. Alternatively, poorly designed sites and applications may be quite functional, but fall far short in reinforcing the brand. Finding the appropriate balance between aesthetics and function is a difficult task. Understanding and measuring the user experience is the first step to achieving that balance. To remain competitive and retain customers, product managers need tools to optimize both aesthetics and function.

This article provides a three-staged User Experience Measurement Hierarchy and also outlines a simple approach you can use to determine the best user experience measurement tools and techniques for your web product management needs.  

User Experience Measurement Hierarchy

The majority of organizations sit at the bottom of the pyramid, having only achieved Stage One: General Knowledge of their users’ online experience. Due to the perceived cost and complexity of measuring user engagement, the widespread availability of user-centered design best practices, and significant cost pressures, this basic level is understandable. However, leading organizations in industries such as insurance, manufacturing, healthcare, travel/hospitality and financial services, have made the migration to Stage Two: Understand User Behavior, and are seeing significant return on investment  by way of increased revenue, reduced support costs and improved customer satisfaction. Even rarer are firms that have achieved Stage Three: Influence Your Users. While we’ve yet to see these tools widely adopted, they represent the future of user experience measurement.

 

Stages Need Recommended Tools Ideally Suited For...
Stage One
General Knowledge
Provides a basic sense of site or web application performance
  • Hueristics
  • Expert Review
  • Web Hits/Usage Analysis
Getting a big picture sense of performance and major weaknesses
Stage Two
Understand User Behavior
Identify what users are doing and where problems exist
  • User Testing
  • Session Analysis
  • Online Surveys
  • A/B and Multivariate Testing
Documenting user behavior and understanding why users are not completing tasks
Stage Three
Influence
Your Users
Determine whether a website or application is compelling
  • Eye Tracking
  • Emotion/Trust Measurement
  • Neuro-Marketing
Measuring user thinking to compel and persuade users to act

Stage One

General Knowledge

A basic sense of site or web application performance

The first stage of user experience measurement involves very little measurement at all. At Stage One, organizations are typically working with “heuristic analyses” or best practices to create assumptions about users. Whether performed by an internal usability expert or a usability consultant, the result is an “informed guess” as to what users need.

Tools to find out general knowledge about the user include:

1. Heuristics

Heuristics are de facto web standards and research from  the cognitive sciences that reveal a number of best practices in interaction styles, page layout, and visual design. Usability problems found through a heuristic evaluation are typically linked to aspects of the interface that are reasonably easy to demonstrate. For example, use of colors, layout and information structure, consistency of the terminology, or consistency of the interaction mechanism.

2. Expert Review

An expert review is a rapid survey of a website/application from specialists in user-centered design. Expert reviews can take you to the next level of user experience measurement by systematically rating the performance of your product on several relevant dimensions: navigation effectiveness, content selection, visual presentation, branding, and interaction simplicity. An expert review can often be done with a short turnaround and can be used to develop a measure of all your web assets as well as identify areas that need the most attention.

3. Web Hits/Usage Analysis

Web hits/usage analysis uses tools like Google Analytics, Omniture and WebTrends to study the hit pattern of your site or application. They can give you a sense of the number of pages visited and viewed, the typical navigation flow, and key drop-off locations. You can also track statistics such as bounce rates and conversion metrics. When combined with customer relationship management  data, web hits/usage analysis can be highly effective in mapping and understanding user behavior.

It’s important to remember, web analytics/usage metrics rely on assumptions about user behavior and how those assumptions correspond to your web goals.  So while these tools can help you determine the relative “success” of your website in meeting business goals, it’s difficult to know whether you are truly capturing the needs of your key target audience(s). The knowledge you gain through web analytics tools may be too general to understand individual user’s thinking and ultimately persuade them to act or change their online behavior.

The clear advantages of Stage One user experience measures are they are inexpensive, easy-to-use, can be quickly deployed and applied to many sites and applications.    

 

Stage Two

Understand User Behavior

Identify what users are doing and where problems exist

The user-centered behavior metrics of Stage Two can provide an in-depth analysis of obstacles to accomplishing tasks on your site(s) and applications.

In order to truly understand what your target audience is doing online, you need to study a representative sample of individual users first hand. In-depth analysis typically is accomplished through user (usability) test sessions performed remotely or in person. These sessions will reveal much more about how users are anticipating the site or how the application works and what design issues might stop a user from accomplishing key tasks.

The best tools for understanding user behavior include:

4. User Testing

User testing is highly effective in designing for specific user groups. User testing can be done in person, in a lab or focus group or even remotely online, and involves interviewing individuals  who are representative of  your target audience. By asking them to accomplish specific, representative tasks to determine ease of use and recommend improvements, you can gather first-hand information about how users are behaving and what they  are saying about your site or application. User testing is often most effective when it’s done early in the design process, before development and visual design.

The advantage of user testing is you learn what the target audience can and can’t accomplish, understand why users are having trouble, how to correct it, and receive direct feedback. The disadvantages are obvious: it can be more expensive to test individuals, especially when you’re dealing with multiple user types and a large number of test participants. In addition to a testing facility and moderator costs, there are also costs to recruit and compensate test participants. Participants typically receive a small stipend and refreshments. Remote testing can serve as a more cost-effective alternative; however, additional web conferencing costs apply.

5. Session Analysis

Session Analysis involves studying multiple single-user transactions, live or replayed based on certain criteria (e.g., error message present, drop-off at buy button). They can reveal a great deal of information about the user’s path and how to correct flow problems. Session analysis tools like TeaLeaf® are good tools for transactional sites and web applications.

These tools provide detailed naturalistic web session analysis, monitoring and replaying individual sessions and capturing the user experience either live or replayed for specific individuals. Error messages, broken links, or  users clearly deviating from the expected path reveals critical system weaknesses. The ability to produce a recording of an individual user’s sessions—including the page sequence, form inputs, button selections and actual HTML page served to the customer’s browser—can help find and correlate an individual user’s behavior to application errors.

6. Online Surveys

Collecting larger population samples through online surveys is a quick and inexpensive way to capture likes and dislikes, most common feature requests, and summary information about a user persona. It is important to collect larger population samples to ensure that in-depth studies are representative of the target audience.

Companies like ForeSee Results provide a variety of intercept survey options integrating best practices from web analytics, market research and other tools, such as the University of Michigan’s American Customer Satisfaction Index (ACSI), to help companies convert satisfaction data into user-driven web development strategies.


7. A/B and Multivariate Testing

A/B and Multivariate Testing allows alternative designs to be tested and measured with a small segment of users. Nearly all high-transactional sites, such as, Yahoo, Google and Amazon.com, use this methodology which enables a comparison between the existing site and the prototype site for conversion optimization. Tools like SiteSpect and OnDialog are just a few of  the options available.

The beauty of A/B and Multivariate Testing is they offer the ability to isolate and assess the performance  of virtually every element of  a web site, landing page, or application. From page layout,  to headline text, to complete  color palette variations, A/B  and Multivariate Testing can be highly effective in identifying what works and what doesn’t from a user’s perspective. Plus, when managed through a marketing automation tool such as SiteSpect or OnDialog, it can also be easily scaled and deployed with no regard to internal technology infrastructure, a significant benefit for organizations with complex or legacy infrastructure issues.

Stage Two User Experience Measurement tools such as User Testing, Session Analysis, Online Surveys and A/B and Multivariate Testing are ideal for understanding what users are doing and where they are experiencing difficulty. Each has its own advantages and disadvantages. While all can be used, each solves a unique problem for marketers and product managers. Fortunately for product managers, each of these tools is also widely available at varying price points that can be customized to meet your specific requirements.

Stage Three

Influence Your Users

Determine whether a website  or application is compelling

It has never been enough to “put something out there” that basically works. In order to influence users, you must directly measure what users think. Websites and applications must be compelling to the users both to be successful and for key influencers to spread the word about their positive experience. Increasingly, sites are ranked on their persuasiveness and influence on user thinking rather than just the ability to accomplish a goal.

The following tools are ideal for understanding user experience  and satisfaction:

8. Eye Tracking

Eye tracking is a well-established vision science technology that can be readily applied to user testing. It can record where users are looking first, what is capturing their attention, and what they are choosing to act on. Eye tracking goes beyond measuring overt behavior and provides a window into the user’s thinking.

Eye tracking studies are particularly effective when applied in an online publishing environment; for example, news and entertainment sites. By testing a subset of a target audience, product managers can gain powerful insight into critical data unavailable through any other means:

  • What first captures readers’ attention when visiting the page
  • How often readers see the diversity of news and content the publication offers
  • Where readers are looking versus where they actually click

9. Emotion/Trust Measurement

Several early prototypes are using various measurements (e.g., skin conductance, heart rate, facial emotion detection, mood survey) to determine how users feel while using sites and tools. There is an increasing consensus that both explicit (survey) and implicit (emotion detection) tools are needed to really understand user motivations and experience.

10. Neuro-marketing

While only a few tools have made their way from neuroscience laboratories, there are some applications of Electroencephalograms (EEG) technology and functional Magnetic Resonance Imaging (fMRI) which are being used in the commercial arena to measure engagement and emotional response to user experience. These tools provide a window into user satisfaction.

The critical feature of these technologies is to measure the experience of your target audience and understand what they find compelling and appealing. Using these techniques, you can also identify which messages are most effective and what dictates trust and positive emotion from your website or web application.

The disadvantages of these “higher level” measurement techniques are many. They require more equipment and analysis. Given their lack of widespread adoption, it can be difficult to achieve buy-in within your organization to pursue such an initiative. Finally, due to the higher expense, product managers are cautioned to develop a realistic return on investment model before diving in.

Measuring and managing experience

Although time and cost limitations exist in measuring user experience, there is a clear progression in user experience measurement sophistication: from general knowledge to influencing user behavior. No matter what level of sophistication you choose, knowing as much information about your users as possible helps craft a compelling experience for your highest use sites and applications.


Looking for the latest in product management news, articles, webinars, podcasts and more?