We recommend that you view this web experience with JavaScipt enabled, and with the latest up to date browser.

I want to find out ...
Your stage of product development

Use the filters to find the most appropriate user research methods for your needs.

Filtered Methods

Remember these recommendations are only to be used as a guide to help inform your research plan.

Sorry, there are no research methods that match your query.

Try amending your query selections.

  • Usability-Lab Studies

    Participants are brought into a lab, one-on-one with a researcher, and given a set of scenarios that lead to tasks and usage of specific interest within a product or service.

    Key propertiesfor Usability-Lab Studies

    • Behavioural
    • Qualitative
    • scripted
  • Ethnographic Field Studies

    Researchers meet with and study participants in their natural environment, where they would most likely encounter the product or service in question.

    Key propertiesfor Ethnographic Field Studies

    • Attitudinal
    • Behavioural
    • Qualitative
    • natural
  • Participatory Design

    Participants are given design elements or creative materials in order to construct their ideal experience in a concrete way that expresses what matters to them most and why.

    Key propertiesfor Participatory Design

    • Attitudinal
    • Qualitative
    • hybrid
  • Focus Groups

    Groups of 6–12 participants are lead through a discussion about a set of topics, giving verbal and written feedback through discussion and exercises.

    Key propertiesfor Focus Groups

    • Attitudinal
    • Qualitative
    • de-contextualised
  • Interviews

    A researcher meets with participants one-on-one to discuss in depth what the participant thinks about the topic in question.

    Key propertiesfor Interviews

    • Attitudinal
    • Qualitative
    • de-contextualised
  • Eyetracking

    An eyetracking device is configured to precisely measure where participants look as they perform tasks or interact naturally with websites, applications, physical products, or environments.

    Key propertiesfor Eyetracking

    • Behavioural
    • Qualitative
    • natural
    • scripted
  • Usability Benchmarking

    Tightly scripted usability studies are performed with several participants, using precise and predetermined measures of performance.

    Key propertiesfor Usability Benchmarking

    • Behavioural
    • Qualitative
    • scripted
  • Moderated Remote Usability Studies

    Usability studies conducted remotely with the use of tools such as screen-sharing software and remote control capabilities.

    Key propertiesfor Moderated Remote Usability Studies

    • Behavioural
    • Qualitative
    • scripted
  • Unmoderated Remote Panel Studies

    A panel of trained participants who have video recording and data collection software installed on their own personal devices uses a website or product while thinking aloud, having their experience recorded for immediate playback and analysis by the researcher or company.

    Key propertiesfor Unmoderated Remote Panel Studies

    • Behavioural
    • Qualitative
    • scripted
  • Concept Testing

    A researcher shares an approximation of a product or service that captures the key essence (the value proposition) of a new concept or product in order to determine if it meets the needs of the target audience; it can be done one-on-one or with larger numbers of participants, and either in person or online.

    Key propertiesfor Concept Testing

    • Quantitative
    • Qualitative
    • hybrid
  • Diary/Camera Studies

    Participants are given a mechanism (diary or camera) to record and describe aspects of their lives that are relevant to a product or service, or simply core to the target audience; diary studies are typically longitudinal and can only be done for data that is easily recorded by participants.

    Key propertiesfor Diary/Camera Studies

    • Attitudinal
    • Qualitative
    • natural
  • Customer Feedback

    Open-ended and/or close-ended information provided by a self-selected sample of users, often through a feedback link, button, form, or email.

    Key propertiesfor Customer Feedback

    • Attitudinal
    • Qualitative
    • natural
  • Desirability Studies

    Participants are offered different visual-design alternatives and are expected to associate each alternative with a set of attributes selected from a closed list; these studies can be both qualitative and quantitative.

    Key propertiesfor Desirability Studies

    • Attitudinal
    • Quantitative
    • Qualitative
    • hybrid
  • Card Sorting

    A quantitative or qualitative method that asks users to organize items into groups and assign categories to each group. This method helps create or refine the information architecture of a site by exposing users’ mental models.

    Key propertiesfor Card Sorting

    • Attitudinal
    • Quantitative
    • Qualitative
    • de-contextualised
  • Clickstream Analysis

    Analyzing the record of screens or pages that users clicks on and sees, as they use a site or software product; it requires the site to be instrumented properly or the application to have telemetry data collection enabled.

    Key propertiesfor Clickstream Analysis

    • Behavioural
    • Quantitative
    • natural
  • A/B Testing

    Also known as “multivariate testing,” “live testing,” or “bucket testing”, a method of scientifically testing different designs on a site by randomly assigning groups of users to interact with each of the different designs and measuring the effect of these assignments on user behavior.

    Key propertiesfor A/B Testing

    • Behavioural
    • Quantitative
    • natural
  • Unmoderated UX Studies

    A quantitative or qualitative and automated method that uses a specialized research tool to captures participant behaviors (through software installed on participant computers/browsers) and attitudes (through embedded survey questions), usually by giving participants goals or scenarios to accomplish with a site or prototype.

    Key propertiesfor Unmoderated UX Studies

    • Behavioural
    • Quantitative
    • scripted
  • True-Intent Studies

    A method that asks random site visitors what their goal or intention is upon entering the site, measures their subsequent behavior, and asks whether they were successful in achieving their goal upon exiting the site.

    Key propertiesfor True-Intent Studies

    • Behavioural
    • Quantitative
    • natural
  • Intercept Surveys

    A method that asks random site visitors what their goal or intention is upon entering the site, measures their subsequent behavior, and asks whether they were successful in achieving their goal upon exiting the site.

    Key propertiesfor Intercept Surveys

    • Attitudinal
    • Quantitative
    • natural
  • Email Surveys

    A survey in which participants are recruited from an email message.

    Key propertiesfor Email Surveys

    • Attitudinal
    • Quantitative
    • de-contextualised

Usability-Lab Studies

Discover usability issues with your product, identify why a product feature may or may not be working.

Can help answer questions such as:

  • Can the product be used with ease?
  • What parts work or don't work for the user?
  • Do users understand the product?
  • Are they using it as intended?

In a Nutshell

Users are brought into a lab (a quiet private room) where a researcher will present them with task based scenarios that naturally lead to an interaction with the product or feature to be tested.

Users can be asked to verbalise their thoughts during the session which will be recorded along with their interaction behaviour by the researcher.

A cohort of 6-10 participants will usually be enough to identify the main usability issues.

When is it Typically Used?

Usability lab testing is usually done in the design process to identify usability issues with the product, from user flows through to micro interactions.

Prototypes can work just as well as the actual product, just make there is a clear hypothesis to test.

Measures

Generally a qualitative study, some metrics that can be captured include,

Measure Details
Task success Full, Partial, Incomplete successes
Error rate Number of errors per task
Time on task Time taken to complete task
Efficiency Task completion rate / task time
Learnability Improvement in time on task over time

Don't report numbers from small studies as it will not yield a statistically significant result that can be applied to the population.

Real World Tips

  • Setup a clear hypothesis with regard to what you want to test.
  • Define clear user goals, and couple these with realistic actionable tasks that a user can undertake.
  • High fidelity prototypes can provide better user feedback (again this is dependent on what you're testing)
  • Add in short interviews to capture contextual user information
  • Always have a stand-by for a no show
  • Mitigate user fatigue, avoid making sessions longer than an hour.

Run a pilot test with a team member, to give the script a go and test tasks and durations.

Positives

  • Test with prototypes early in the design phase to improve the product even before release.
  • Sessions are relatively cheap to run.
  • Capture rich user feedback.

Allows the entire team to get closer to the user.

Drawbacks

  • Remember small sample sizes provide indication of issues, and individual comments that don’t form a theme can easily mislead.
  • Lab environments don’t reflect real world contexts, so this needs to be considered especially for products like mobile apps.
  • Overheads of organising and recruiting users.
  • Identifies usability issues, but doesn't always give a true sense of severity of the issue for users.
  • Can take time to run sessions and consolidate feedback.

Example Scenario

Test a new checkout payment flow and page.

  • Setup a scenario - where by a user finds an item and pays for it.
  • Provide a fake credit card, address details and such they can enter.
  • Observe the paths they take and their thought process using a talk aloud protocol (i.e. making them speak their thoughts during decision making)
  • Record the session for further review.

Rinse and repeat for all participants.

Ethnographic Field Studies

Get real contextual feedback by observing product usage behaviour by users in their natural environment.

It helps answer questions such as:

  • When do users use the product?
  • Are they doing other things whilst using it?
  • What is the environment like?

In a Nutshell

Go out to where the study participants reside, and observe them using their product. For new products not yet deployed it might simply be to understand the potential usage environment.

Encourage users to perform tasks as they would and observe what they do, and how they do it. Ask questions and include short interviews, but the key here is to observe behaviour in context.

When is it Typically Used?

This is usually done early in the discovery and problem definition phase of the lifecycle to identify pain points and get to the core of what users actually want from a system.

Measures

Generally a qualitative study, mostly observational data taken in note form.

Real World Tips

  • Don’t ask leading questions.
  • Ask more open ended questions that fuel them to think more and provide details.
  • Don’t just focus too narrowly on specific issues, as it will cause users to filter our their responses and you can miss key information.
  • Consider the themes that come from results of the study and don't get hung up on anecdotal comments or pet hates of a single user.

Positives

  • Get another team member to come along to better understand the user's environment
  • First hand experience observing the user in their natural context allows for unprecedented data capture

Drawbacks

  • Going out to various user sites can be time consuming.
  • Can take time to analyse the results and produce a report.

Example Scenario

When designing a new insurance quoting platform for brokers, we may want to learn the way insurance brokers use the current insurance systems. You might visit the office of an insurance broker to see how they work with the system and their clients.

Participatory Design

A co-operative design strategy that bring customers into the design process.

It shifts the idea of designing for a user group, to with a user group - for a more user centric approach.

In a Nutshell

This is not a single activity but a strategic approach whereby users are brought into the discovery and design phase to contribute to product design.

This would typically involve workshops whereby users are brought into the design process to help in the design ideation process. User ideas would not necessarily be developed into products (they're not designers), but the process can inform the product designer with regards to the way in which users perceive the problems domain and their mental model.

When is it Typically Used?

Can run through the design process, but mainly in the execution part of the process.

Useful when you might have a disconnect with the product development team and the end user (this can be the case in specialist expert systems).

Measures

Users provide feedback, and contribute to design thinking methods.

Real World Tips

  • Users inform the design process, they don’t make product design decisions.
  • Keep activities engaging, simple and focused with users, remember they’re not designers.

Positives

  • Keep close with the voice of the user through the design process
  • Reduces risk of product failure when launching to a target market

Drawbacks

  • Not always easy to have users on hand to join workshop sessions.
  • Can be time consuming involving users in workshops and testing when timelines are short.

Example Scenario

When designing an internal online examination system for schools - teachers, and senior staff might be brought into workshops to show the types of things they would expect to see from a results dashboard.

Focus Groups

Get attitudinal user feedback via group discussion sessions early in the discovery process to shape the problem definition.

It helps to:

  • Define common issues and pain points
  • Get user attitudes and thoughts toward the product

In a Nutshell

Bring together roughly 6-12 participants to discuss issues and concerns with regards to the product and or it’s feature. Usually demo the product as a starting point before, moving into discussions.

Relatively free flow discussion is lead by a moderator that guides the group conversation and lasts about 2 hours.

Activities can also be added to the session to make it more interactive, such as role playing, card sorting or picture drawing.

You'll need to run 4-6 groups to identify themes across them, and remove biases.

When is it Typically Used?

These are usually done early in the discovery and problem definition process, to gauge attitudes and thoughts, before lots of design and development is undertaken.

Measures

Generally a qualitative study, mostly discussion notes taken by observers and moderators.

Real World Tips

  • Augment focus group data with other user research methods like interviews and usability or field studies to build a full picture.
  • Provide water and snacks to participants along with breaks.
  • Keep all participants involved and contributing and not purely dominated by a few voices.
  • Capture interesting quotes, that can be used to sum up themes.

Positives

  • Very low cost, as it can be run before having to create any artefacts
  • Identifies what user expectations and desires are, not their needs.

Drawbacks

  • Only captures what a user says they do, not what they actually do - as you're not observing their behaviour.
  • Can magnify non-issues if a discussion is not moderated well.
  • Biases can easily be introduced with focus groups, due to group dynamics, cultural norms and social etiquettes.

Example Scenario

When designing a new app to help parents set educational home activities for their children - a focus group might be used to discuss the appetite for such an app, when they would like to use it and how much they might pay for said product.

Interviews

Get direct user feedback from on-on-one interviews with regards to a specific topic.

It helps to answer questions such as:

  • How do they use the product?
  • How do they use technology?
  • What kind of pain points do they have?
  • What are their motivations and objectives with the product?

In a Nutshell

A researcher asks individual users questions with regards to a product/topic. Sessions shouldn't typically last longer than an hour.

Notes are taken and then later examined to identify common themes that emerge between groups of users.

Interviewees should be grouped by demographic or other relevant attributes to add more fidelity when it comes to identifying patterns in the result data.

Sample sizes for interviews can vary greatly with regards to demographic splits and other attributes, to get numbers.

When is it Typically Used?

Usually undertaken in the discovery and definition phase, when clarifying who the user really is, their needs and the problem domain.

Measures

Generally a qualitative study, mainly written answers to verbal questions.

Real World Tips

  • Useful as a the basis for building personas, and understand the user in more detail.
  • Keep the interview comfortable and flowing, avoid being head down writing notes.
  • Don’t make interviews too long, users get tired.
  • Avoid leading questions

Positives

  • Relatively fast and low cost, as it can be run before having to create any artefacts.
  • Can gather a lot of attitudinal information in a short space of time.
  • Unlike a survey, users give you their undivided attention during interviews.

Drawbacks

  • Can be time consuming especially if travelling over to client sites to conduct interviews.
  • Interviews are based on recall, and users aren't always good at doing this with clarity.

Example Scenario

Before launching a new faster broadband package, it would be useful who would pay for and use a super high speed service. Interviews could be a good way to identify these types of individuals, so products and services could be designed to meet their needs and expectations.

Eyetracking

Get behavioural information with regards to where a user's attention is during product usage or task execution.

It helps to answer questions such as:

  • Where are users looking, for how long?
  • Is a users attention focused when performing a particular task?
  • Are there any distractions that might reduce a users focused attention?
  • How should we design interfaces to maximise a user's attention with regards to the primary task?

In a Nutshell

Users have their eye movements tracked by a camera when using the product, and the resulting data is then overlaid onto the interface as a heat map, to give a visual representation of focus hotspots, and areas that don’t get attention.

Another visual representation that’s generated is a saccade pathways map, this outlines the eyes movement between areas of focus on the interface.

When is it Typically Used?

Generally used in the design process to assess design concepts and benchmark existing designs to determine where users are looking on the interface.

Measures

Primarily a qualitative study, even though distinct data points on eye tracking movements are gathered the resulting plot and maps generated are effectively descriptive and need to be analysed and interpreted.

Heat maps can generate quantitative data, but again this requires further analysis and interpretation.

Real World Tips

  • Clarify to users that you're capturing eye movements if its being undertaken during a usability study.
  • Test the eye tracking works in the lab lighting.
  • Keep the test area free from other distractions.
  • Avoid think aloud feedback protocols during aye tracking studies as this will influence the results.
  • Best used with actual designs and not just wireframes.
  • Run a pilot test to ensure the systems are calibrated and working correctly.

Positives

  • Shows the truth with regards to where a user is actually focusing, not just what the user says they’re focusing on.
  • Shows when is a user is lost and trying to locate an item on the interface.

Drawbacks

  • Can highlight false positive, users can unconsciously rest their eyes in random areas.
  • Only captures data on focused elements not items seen via peripheral vision.
  • Does not tell you why a user is looking at something.
  • Isn’t always accurate due to individual anomalies (spectacles, contact lenses, etc)

Example Scenario

When launching a new travel website homepage it would be useful to see how a user can navigate the content and begin tasks. Eye tracking studies can be useful in this regard to see if users can locate and execute tasks on visual designs.

Usability Benchmarking

Get ongoing feedback around the usability performance and satisfaction of a system relative to a baseline (previous version or competitor product) over time.

It helps to answer questions such as:

  • Has the usability of my product improved?
  • Which areas of my product need attention?
  • How does my product compare against a competitors'?

In a Nutshell

To usability benchmark the product, the researcher needs to create a clear testing script that has goal based tasks. Following each task is an evaluative question that can be used to measure task satisfaction for the user.

Benchmarking across product iterations, would mean initially testing on the existing product, then performing the same test again at a later time when the product had been updated with improvements, with the same size cohort.

Benchmarking would typically be done at regular intervals (i.e. quarterly) - data would be compared with previous iterations to identify trends.

Usability benchmarking can be done in a lab or remotely and unmoderated - to get a larger number of participants for a quantitative study, it's usually is easier to conduct testing remotely and unmoderated.

As we're trying to capture mote quantitative insights, cohort sizes will be larger (20 - 50 users) than a typical usability study. Cohort sizes need to be consistent across the tests.

When is it Typically Used?

This is a summative test, and typically run at regular intervals after improvements to the product have been made.

This could be on a fortnightly basis if improvements are more frequent or a quarterly cycle if releases are infrequent.

Measures

Generally a qualitative study, but with larger cohorts numerical data can be used to draw trends.

Performance based metrics to capture, include:

Measure Details
Task success Full, Partial, Incomplete successes
Time on task Time taken to complete task
Task failure If the task was unsuccessful

Behavioural metrics to elicit, include:

  • Ease/difficulty of task
  • If outcomes were as expected
  • Differential in expected time on task (should task have taken longer or less time)

Real World Tips

  • Run test remotely with a good script to capture more users quickly.
  • Analyse the data carefully to identify task completion and failure rates.
  • Identify the difference between the perception of expected task completion time vs actual completion time.
  • Use remote testing tools that capture rich data such as clickstreams and screen recordings.
  • Some tools can also record live video of the test - this can be useful to capture facial expressions.
  • For capturing behavioural feedback - use a Likert or semantic differential scale for capturing answers.

Positives

  • Shows if a product is improving over time, which helps to inform future product strategies.
  • Highlights if users are learning how to use the system.
  • Can compare benchmarks against industry standards and competitors.

Drawbacks

  • To capture comparable numeric data, cohort sizes need to be larger and can make testing time consuming and costly to run.
  • Shifts in the testing script need to be managed carefully so as to ensure comparisons can remain meaningful.

Example Scenario

Usability benchmarking a TV set-top-box interface would inform product developers about how much the product is improving over time, using a combination of performance and perception based metrics

Moderated Remote Usability Studies

Just like usability lab studies, they can uncover usability issues with your product, identify why a product feature may or may not be working - but all done remotely with video conferencing.

It helps to answer questions such as:

  • Can the product be used with ease?
  • What parts work or don't work for the user?
  • Do users understand the product?
  • Are they using it as intended?

In a Nutshell

User and researcher are located in disparate locations, sessions are run using video conferencing software with screen sharing and recording capabilities. Otherwise these are very similar in structure to a usability lab study.

Researcher presents the user with scenarios which naturally lead to tasks that interact with the product or feature.

Users can be asked to verbalise their thoughts during the session which will be recorded with their device camera - along with their screen interaction behaviour.

A cohort of 6-10 participants will usually be enough to identify the main usability issues.

When is it Typically Used?

Moderated remote usability testing is usually done in the design process to identify usability issues with the product from user flows through to micro interactions.

Prototypes can work just as well, just make sure you have a clear hypothesis to test.

Measures

Generally a qualitative study, some metrics that can be captured include,

Measure Details
Task success Full, Partial, Incomplete successes
Error rate Number of errors per task
Time on task Time taken to complete task
Efficiency Task completion rate / task time
Learnability Improvement in time on task over time

Don't report numbers from small studies as it will not yield a statistically significant result that can be applied to the population.

Real World Tips

  • Test the software you use for screen recording and video conferencing prior to the session. Become familiar with the participant interface as well, so you can diagnose and resolve any tech issues quickly.
  • If possible ask participants to install and test any specific software they need prior to the session.
  • Run a pilot session to identify and iron out any common pitfalls.
  • As a researcher turn off all notifications on your computer/phone to avoid any distractions.
  • Find a quiet place to run the session, participants should not hear distracting background noises.
  • If other participants are joining to listen in on the sessions, ensure they don’t turn their microphones on.
  • Keep sessions short (~30min) to avoid fatigue. Recruit more participants, as drop-out rates can be higher with remote studies.

Positives

  • Sessions can be cheap to run and easier to schedule as user's don’t need to come into a lab.
  • As a participant is using their own device, researchers can see a user's system setup.
  • Allows you to test with users anywhere in the world, and not just those located near your offices or other central areas.
  • Tests can be setup quickly, as there are large pools of users available and ready to test.

Drawbacks

  • Requires more planning to run remote sessions, to ensure the scripts and tech are working as expected.
  • Can’t see a user's body language via their device camera, which can reveal a lot about their behaviour.
  • Can be trickier to manage when to prompt and interject when a user is confused or just thinking and need more time. There are fewer cues to pick up on than when you're in both in the same room.

Example Scenario

During the design process for building a new client portfolio management system for financial advisers - the updated interface would need usability testing to identify any issues. A remote moderated usability study would be a quick way to test core tasks an adviser would need to do to help validate the new interface designs.

Unmoderated Remote Panel Studies

Remotely capture simple behavioural product interaction insights from a trained panel of users, all without moderation from a researcher. Software installed on the participant's device records the interaction - screen and audio/video.

It helps to answer more specific questions such as:

  • Is this particular function/feature usable?
  • Does the proposed interaction paradigm work for our users?
  • What parts work or don't work for the user?
  • Do users understand the feature/function?
  • Are they using it as intended?

In a Nutshell

A user panel is formed that will partake in the study, these users will be briefed about how to engage in testing remotely.

Each participant will have to install software on their device that captures screen interaction along with audio/video to record their reactions and expressions during usage. The captured data is then immediately available for the researcher to analyse and interpret.

Participants are sent an email to ask if they can partake in the study and given a window of time to complete the testing.

Unmoderated remote panel sessions usually only consist of a few tasks to complete and are kept short, as there is no way to moderate users that go off track.

Study sizes need to be larger for remote usability testing as there are higher drop-out rates and a greater chance for erroneous data due to questions being misunderstood and tasks not completed properly.

Once the window for testing is complete, the researcher can review and analyse the session data.

When is it Typically Used?

Unmoderated remote panel studies are usually done in the design process, usually to ratify and validate key design decisions.

Tests can be performed on live systems or prototypes.

Measures

Generally a qualitative study, some metrics that can be captured include,

Measure Details
Task success Full, Partial, Incomplete successes
Error rate Number of errors per task
Time on task Time taken to complete task
Efficiency Task completion rate / task time
Learnability Improvement in time on task over time

Don't report numbers from small studies as it will not yield a statistically significant result that can be applied to the population.

Real World Tips

  • There a number of commercially available tools, that can be used on desktop and mobile devices. See: Tools for Unmoderated Usability Testing
  • Keep tests specific and short, ensure scripts are self-explanatory and easy to understand.
  • Like any user study it is worth pilot testing to iron out any issues.

Positives

  • Sessions are relatively cheap to run and tests are completed by panel members on their own time within your allotted time frame.
  • Remote testing allows for the panel to be from a specific region or geographically dispersed.
  • Can pull in larger numbers of participants and generate quantitative feedback if desired.

Drawbacks

  • Users may not talk aloud even when your script has clear guidance. This can result in poor quality data being generated.
  • No moderation means scripts need to be very specific and clear, to prevent users misinterpreting what is asked of them.

Example Scenario

For an e-commerce retailer identify any usability issues for a newly updated search widget on the home page.

This would involve the user being asked to perform a number of searches from this starting point; also consider using the existing interface as a comparative benchmark.

Concept Testing

Gather attitudinal user feedback on ideas/concept early on in the design process. Concept testing covers a range of methods for doing just this.

It can be useful for getting feedback on a range of ideas and concepts, before they have actually been built or fully designed. It's also a way to gauge your product-market fit, and help steer design direction.

In a Nutshell

Concept testing can take a number of different forms, from one-on-one to large number of participants and these can be in person or remote.

The primary focus across all of these methods is to gauge user attitudinal feedback early in the design phase using lo-fi prototypes or even mock-ups.

A simple concept test could be to show users three different app design screens for a new service and asking for their feedback around each one. This test wouldn't provide any feedback with regards to how each of those designs may actually perform, but it does gauge the preference a user would have to a particular design. Remember this is not a usability test.

When is it Typically Used?

Typically used In the early design phase to help steer the product design direction and mitigate the risk of building a concept that user's don’t want or understand.

Measures

Can be either qualitative or quantitative in nature, depending on the methods employed.

In mainly qualitative concept tests the feedback would mainly be notes to the answers asked of the participants.

With larger numbers of participants the answers to questions may reside on a Likert or semantic differential scale which can be further analysed.

Real World Tips

  • Analyse the feedback carefully as it is primarily attitudinal and not rooted in usability.
  • Try to keep concept tests simple, too many variants to review and the user may get confused.

Positives

  • Concept tests don’t need complex prototypes as you're only gauging perceptive feedback.
  • In many instances concept testing can be run remotely.

Drawbacks

  • Feedback provided doesn’t give much depth to why a user may not like a particular concept.
  • Could provide improper feedback if the concept is not clearly defined, as concepts can easily be misinterpreted.

Example Scenario

For a music service, a researcher may want to investigate which new app design is best received by their target audience. A concept test in this instance would involve showing participants the new designs and asking them questions about their preference for each one.

Diary/Camera Studies

Get longitudinal (long-term) contextual behavioural information about how and when your users engage with your product/service or for a new product how potential users might do so.

This type of study helps to build up a wide range of self-reported contextual information, and can be useful in understanding user habits, behaviours, attitudes, scenarios, routines and pain points with regards to the product.

In a Nutshell

Identify the user behaviour you're looking to understand, and define the terms of the study. Diary studies can typically last from a week to a month but can be longer - it really depends on the product and the types of user habits and behaviours you've chosen to investigate.

Study participants (4-6 at a minimum) are briefed and given a diary or camera to record and describe aspects of their lives; with respect to a product/service your users are using or one a target user group would utilise.

Researchers should provide clear instructions (without being leading) to participants on how to make diary entries, so that they stay relevant and it reduces participant fatigue and frustration with the process.

Participants then log entries as frequently as agreed for the study. Logging can be done immediately after said activity/experience or a snippet is captured at the time and a larger expanded entry at the day end.

An alternative to the traditional pen and paper diary is an email diary, unlike the paper version it allows the researcher to see progress before the study is complete. Private twitter feeds can also be used as a mechanism to capture diary entries.

Once the study period is over, participants submit their diaries. After initial analysis, run a post-study interview/debrief with each participant to clarify points and ask questions about the study to build out the full picture.

Finally analyse and review the collected data, create a customer journey map and provide insights to the initial questions you were investigating.

When is it Typically Used?

Diary studies are an exploratory tool in your research arsenal, they are best used in the early discovery phase of a project to help shape problem definition and identify opportunities and user paint points.

Measures

Generally a qualitative study, mainly attitudinal feedback to tasks and activities in daily situational context.

Real World Tips

  • Run a pilot study even for a few days to iron out technical and contextual issues before you run the full study.
  • Audio/Video diary entries are easy for participants to create but can take longer to transcribe.
  • Drop out rates on diary studies can be high (~20%) as they are long term commitment - so recruit more users than you need.
  • Participant motivation can waiver during the study, regularly check in without being intrusive.
  • Provide a good user incentive for completing the study, more so than an hours user testing.
  • If you've got diary entries being submitted online daily, review and analyse them so you can track progress.
  • Data analysis can take time, it's recommended to factor this in so you have time to write the report.

Positives

  • Hands-off research for the most part
  • Relatively economical in gathering a lot of user research data.
  • Gives you a lot of contextual information about the user.
  • Allows you to see user habits and routines.

Drawbacks

  • Can be time consuming to analyse, review and transcribe a large number of diary entries, not to mention generate a report and identify findings.
  • Quality of entries can vary greatly between participants.
  • Diary studies can have high drop out rates.
  • User fatigue and boredom are common with studies over an extended period.
  • Data is all self-reported, this needs to be clearly highlighted in the report prior to distribution.

Example Scenario

For a video streaming service, a researcher might want to understand the viewing habits of it's users and ask them to log when and where they are watching any audio video media, and whom they might watch with.

They might ask them to record if they had conversations with regards to the show with friends and families in person and over social media.

Customer Feedback

Gather ongoing feedback from your users, usually from a simple widget readily available on your web/app experience.

This type of method is not so much a study as mechanism for collecting feedback from users who feel strongly enough about your service in order to give you feedback for better or worse.

In a Nutshell

A widget or similar mechanism is usually deployed within the product experience that allows the user to provide feedback in the form of a comment along with an NPS type question.

This is not a study as the feedback is provided by users sporadically and self motivated, usually highlighting a good or bad experience.

Feedback is subsequently reviewed and collated on a weekly or fortnightly basis to see if there are any trends or common themes.

This does make it a good barometer to see how users feel about the experience - especially to determine if new changes have a positive or negative impact. Similarly as this is all self-reported comments, it must be treated as anecdotal feedback.

When is it Typically Used?

Usually deployed once a product is live and used as a tool to capture the "voice" of the customer.

Measures

Primarily yields qualitative information anecdotally, structured questions can be included along with the free form comment.

Real World Tips

  • Easy to setup and gather customer feedback using a number of plug-ins and third party tools.
  • When sharing specific customer feedback with stakeholders pre-warn them that this is the comment of a single user and it's not necessarily representative of the user base.
  • Keep feedback buttons easily available so users don't need to hunt for them.
  • Don't make them obtrusive so that they obscure or inhibit the core experience.

Positives

  • Ongoing low cost way of letting your users provide feedback.
  • Can quickly action on negative feedback, and turn it into a positive outcome.

Drawbacks

  • Is not a reliable indicator with regards to assessing user satisfaction of your product.
  • Can only provide anecdotal feedback.
  • Usually only highlights cases of extremes, on the satisfaction scale i.e. those who've had a terrible or outstanding experience with the product.

Example Scenario

On a fashion retail site , a feedback button could be placed throughout the shopping experience (without being intrusive) so users could provide feedback at any time during their journey.

Desirability Studies

Gauge the user's emotional response to your product's visual aesthetics and brand; with a more comprehensive approach than just asking which design they prefer.

This research methods helps to better understand why a particular visual design or brand style resonates with your users. Most importantly it can stop stakeholders from pushing their design preferences over the users, and ensure your design communicates the intended brand values.

In a Nutshell

There are two approaches to running desirability studies, qualitative or quantitative.

Qualitative

Participants are bought into a lab individually and shown a series of different product designs (they don’t have to be UIs they can be mood boards representative of your visual aesthetic).

The researcher gives the participant index cards with adjectives they can use to describe each of the designs. The adjectives used need to encompass both positive and negative attributes - Microsoft created a set of 118 adjectives that are commonly used for this type of study.

Microsoft Desirability Toolkit Product Reaction Words (Opens in a new window)

Users then go on to select 3-5 cards for each design, the researcher then asks follow up questions to understand their reasoning.

Results are collated and analysed, Venn diagrams are useful ways to present the findings with respect to the types of adjectives commonly used to describe the designs.

Quantitative

This method simply scales the qualitative version with a larger sample to provide results with statistical significance. Instead of one-to-one sessions in a lab, to increase the cohort sizes - participants are sent a survey which asks the same question of the users using the cards to describe their choice, albeit without the follow-up probing by the researcher.

When is it Typically Used?

Normally used in the design process to help validate and refine the user's emotional response to a particular design style or direction.

Measures

In the qualitative lab variant, the follow up questioning that reveals the users reasoning is particularly valuable.

The quantitative version yields measures such as the frequency of selected adjectives, these can then be grouped by cohorts you've tested with (age groups, genders, variant user segments).

Real World Tips

  • All 118 adjectives do not need to be used, filter those most pertinent to your design - but ensure there is an even 60/40 split of positive/negative adjectives as per the original set.
  • When reporting metrics, avoid using absolute frequencies for adjectives and work with percentages (i.e. 80% of users use the word 'Innovative').
  • Use desirability studies as a way to prevent individual stakeholder opinions rail roading considered design choices.

Positives

  • Relatively low-cost to run, using either qualitative or quantitative approaches.
  • Provides genuine user feedback around aesthetics which can be difficult to extrapolate from users.

Drawbacks

  • The qualitative approach with small cohorts provides anecdotal insights, and can't be thought of as representative of the audience.
  • The quantitative approach via a survey does not allow for useful follow up questioning to really understand the "why" behind their choices.

Example Scenario

A bank wants to identify which interface design style to use for it's new customer portal. A desirability study in this instance would help determine if the designs echo the brand values such as trustworthiness, security, and professionalism.

For a deep dive - the following is a good example case study.

Young Adults Appreciate Flat Design More than Their Parents Do (Opens in a new window)

Card Sorting

Make sure users can find the information they're looking for - whether it's a webpage, a product or something else. Card sorting helps to identify, refine or validate your product's Information Architecture (the categorisations and taxonomy) with respect to your users' mental model.

It can assist in answering questions about,

  • Navigation structure
  • Browsing through data/product hierarchies
  • Search refinements
  • Labels and hierarchies All without actually using your product.

In a Nutshell

Card sorting exercises can be run in a number of ways. It really depends on what you're trying to determine and achieve.

For example as a telco you may provide customers with help & support articles - to better determine how to organise these you could run a card sorting exercise.

A typical card sorting cohort is about 15-20 participants.

Open card sort

The most common type of card sorting - used to discover categorisations and reveal a users mental model.

  1. Researcher selects a series of representative article titles (40-80) and prints them onto index cards for the session.
  2. The participant is then asked to review the cards and group similar articles together into piles. Piles can be big/small and there is no target number of piles.
  3. After grouping is complete - ask the participant to label the grouped piles.
  4. Debrief and discuss their choices and thinking in the creation of the groups and labels.

Closed card sort

Used to validate an existing categorisation. The key difference in a closed card sort, is that participants are asked to place cards into predefined categories provided by the researcher. This method is not as effective in understanding the users mental model; to better test and validate hierarches consider using tree testing.

Digital tools are available to run card sorts remotely, and without moderation.

When is it Typically Used?

Normally used in the design process to determine the user's mental model with respect to the organisation of data.

Measures

Card sorts yield rich data that can be analysed and represented visually to further help and interpret patterns and associations across the cohort.

A few of the tables/charts/graphics that can be generated are:

  • Dendrograms
  • Similarity matrices
  • Standardisation grids Some of these are time consuming and difficult to produce manually so using automated tools are useful

Real World Tips

  • Use talk aloud protocols during the session to better understand the participant's thought process.
  • User created label names aren't always very good, so it's best to reinterpret and create your own.
  • Digital card sorting tools, have built in analysis and data visualisation which make it faster to interpret findings.
  • Remember, that this type of testing works out of product context and the interaction design mechanisms deployed in the navigation will also influence the user's mental model.

Positives

  • Can be low cost to run these sessions digitally.
  • Provides a good way to begin understanding the users' mental model with regards to how they understand your categorisations.
  • Don’t need to have created the digital product to run these tests.

Drawbacks

  • Digital sorting sessions have one key drawback, in that you can’t observe the user's thought process as you would in a physical card session.
  • With physical card based sessions the results need to be put into a system and further analysed.

Example Scenario

See the example provided earlier - "In a nutshell" section.

Clickstream Analysis

Observe aggregated user activity across page/screen flows from various starting points. Useful in identifying actual common tasks performed and journeys taken.

It can help to answer questions such as:

  • What is the most common action taken on a screen?
  • Are user's getting lost?
  • Are user's performing the intended tasks from a particular screen?
  • How long users spend on a page and performing a task? This is extremely rich data and can be mined to answer many different questions.

In a Nutshell

In your product install a clickstream analysis tool such as Open Web Analytics, Google Analytics, Adobe Analytics, etc. that captures clicks, gestures, and other user activity.

Set some questions you'd like to get answers for with regards to user behaviour within your product.

Once installed, capture data over a set period, (a week at the least, typically a month) - start by classifying user groups, filtering the data (disregard small anomalies, and outliers as they skew results) and then create queries on the analytics dashboard to begin identifying patterns based around the questions you initially set.

It's imperative to remember that these tools tell you what users are doing, but the don’t tell you why they’re doing it.

When is it Typically Used?

This is used on live products, once they have actual real-world usage with your user population. You wouldn’t typically use this type of analysis on a small cohort of users.

Measures

The amount of data that can be captured from a clickstream tools is immense, here are a few:

  • Timestamp of each and every click/tap
  • Exact position of click/tap on screen
  • Type of browser/OS combo used
  • The amount scrolled on a screen
  • Amount of time spent on the screen
  • Next and previous screens From this raw data, there are a number of inferred metrics and ratios such as conversion rates that can be calculated and then further aggregated to identify trends.

Real World Tips

  • Clean your data, prior to analysis and perform rigorous statistical analysis on the data set.
  • Look at data across different intervals, time of day, day of week, across the month, week on week, month on month, seasonal shifts etc. if relevant to user behaviour.
  • Use the data to answer questions you set; blindly going through the data can lead to the drawing of false conclusions.
  • Goals and targets can be set and recorded by most systems, to help track and focus key tasks (i.e. purchasing).
  • Behavioural insights are only useful if you can action on them; tracking user behaviour at a macro level will in most cases yield a normal distribution - it becomes more powerful when you begin to segment users and identify behaviours within these cohorts, up on which you can begin to successfully influence user behaviour for specific groups.

Positives

  • Easy way to track user behaviour across your product.
  • Can provide invaluable real-world and real-time information with regards to this.
  • Helps to identify what users are doing and can be used to launch more in-depth usability studies to reveal the why behind behaviours.

Drawbacks

  • These tools do not help you determine why users are taking a particular path or performing a task. For this you need to run a usability study.
  • Clickstream tools can violate user privacy, as it can track their behaviour - ensure data becomes anonymised and usage is clearly identified to your users.
  • Don’t get hung up on tracking vanity metrics - metrics up on which you can't action upon and can't really influence easily.

Example Scenario

On a general store website I might want to see the most common inbound landing pages for those that finally convert into a purchase.

A clickstream analysis tool can identify this behaviour in detail and identify if users that checkout arrive from a search engine, directly via a bookmark, on the home page, onto a product page, or even a curated offer page.

This insight could then be used to maximise the conversion metrics for each of these different inbound routes using different strategies for each.

A/B Testing

Find our which designs interventions actually work, by testing their efficacy in a live environment using hypothesis testing methods.

It can help to answer:

  • Which design helps users reach their desired goals most efficiently?
  • Which design helps to generates the most profit?
  • Which design intervention engages the user for the longest? Effectively this research method can be used to compare the actual performance of one design over another using statistical methods to validate.

In a Nutshell

A/B tests are scientifically controlled experiments. Multivariate tests simply refers to there being more than one variation being tested.

A very simple A/B test could be to identify the most effective buy button design on a product page. The goal would be to identify whether the new button design prompted more users to click on it, than the existing one.

To ensure a fair test define the success criteria early on when designing the test.

An A/B testing tool such as Adobe Target, Google Optimize, Optimizely, Unbounce or Maxymiser, etc. would be used in web/app environments, where the alternate designs for the test could be setup.

The researcher then configures the test to allow a percentage of random traffic to reach the variant designs over an agreed period. This allows for enough data to be collected for the sample to yield a statistically significant result.

Once complete the data would be analysed and the appropriate model used to find a statistically significant result.

When is it Typically Used?

This is used on live products, once they have actual real-world usage with your user population.

Measures

Any number of measures can be used to in an A/B test, these could be as simple as click-throughs to a more complex combination of ratios. Measures along with confidence intervals and success/failure thresholds must be defined in the initial test design.

Real World Tips

  • Keep tests simple in design, where possible minimise test variables to understand what is/isn't working.
  • A/B tests can be run all the time, have a stack of tests going to keep improving the application performance.
  • Utilise the correct statistical methods to ensure test results are accurate.
  • Avoid testing too many variants with multiple variables, as it becomes more difficult to ascertain why a particular design is more successful over the other variants.

Positives

  • Cheap and effective to run.
  • Numerous tests can be run simultaneously along non-intersecting site/application journeys.
  • Allows for incremental improvements to be made on an experience, to improve it's throughput.
  • Very effective tool in refining goal based journeys i.e. purchase paths, shopping carts.
  • De-risks large changes by getting real insights from a sample group, before releasing to the entire audience.

Drawbacks

  • Provides shallow insights - does not tell you why users are performing said behaviours - a full suite of usability testing delivers deeper behavioural insight.
  • Only really useful when focusing on a single goal, and success metric
  • It's narrow focus can side-line other valuable non-quantifiable aspects of a design - brand, trust, security, perception and such at the cost of achieving a particular goal.
  • Does not give you insights into what aspects of your interventions are effective.
  • Just because an intervention succeeds does not mean it is actually effective, it's only better than the other tested version(s).
  • A/B tests are usually deliver short-term results - not longer term strategic goals.

Example Scenario

A/B testing can be used to determine which of a number of landing page designs for a digital magazine subscription, yields the highest conversion rate into actual sales.

Inbound visitors via search ads would be randomly directed to one of a series of landing pages, over a set period of time and benchmarked against an effective control (usually the existing version). Enough data would need to be collected to determine a statistically significant result.

The one which yields the highest conversion into sales would be deemed the most effective with regards to sale conversion.

Unmoderated UX Studies

Get focused user feedback on a particular site/application feature or journey, using automated remote testing tools.

It can help to answer:

  • Do users understand how to use the system and complete a specific task?
  • What parts work or don't work for the user?
  • Do user use the system as intended?

In a Nutshell

Unmoderated usability test sessions are completed by the participant on their own without any assistance from a researcher. They are best used to test the usability of specific task based journeys or features and would be far shorter in duration that a moderated UX study.

Researchers typically use a tool such as UserTesting, UserZoom or similar - to setup a survey. The survey contains a series of tasks which the participant then goes on to complete using the aforementioned product or a prototype provided within, whilst talking-aloud. Participants are also asked to install software that allows their reactions to be captured for the session duration.

These surveys are sent to participants, whom have to complete the test before a specified deadline. Results are subsequently analysed by the researcher, and insights drawn.

When is it Typically Used?

Used during the execution/build phase of the product lifecycle to quickly test the usability of a specific feature. Usually as an alternative to a moderated usability session.

Measures

Measures depend on the study goals, if determining the speed at which a user can complete a specified task - time on task, would be a key measure.

Conversely if the aim is to understand why a process is confusing users, the measures would be more qualitative and the A/V recordings from the test sessions would be more important.

Real World Tips

  • Keep tests simple, and focused. Long tests will cause fatigue, and participants may get confused.
  • Use clear concise unambiguous language in the survey questions and instructions - so they can be easily followed. But be careful not to drop hints on how to complete the task itself.
  • Run a pilot study to iron out the small practical elements that can easily get overlooked.
  • Unmoderated UX studies can be run quickly, and generate a lot of data points and artefacts, allocate sufficient time to analyse and organise the results.
  • Recruit more users than a typical usability study as drop out rates can be higher due to low commitment requirement of completing a survey.
  • Run statistical analysis tests to asses the significance of results.
  • Plan meticulously and pilot test to avoid a study full of poor test sessions.

Positives

  • Participants complete tests on their own device, so can better reflect how the users may actually perform, and provides context of their tech setup.
  • Good for getting insights quickly.
  • Relatively cheap to run, even with larger numbers.
  • If testing outside of a B2B context, tests can even be sent out on Friday and a completion deadline set for Monday morning.
  • Access to participants from anywhere - nationally or globally.

Drawbacks

  • Participants may forget to talk aloud and sessions can be quiet.
  • Lack of any intervention can mean participants can get lost during task completion.
  • Remote studies have high drop out rates.
  • It will take time to watch all the session videos and take notes.

Example Scenario

An online grocery store may want to see how users pick a delivery slot using a newly designed slot picker.

Participants would be provided with a short survey with tasks that are to be completed along with links to the picker prototype. They would be asked to describe their thought process out loud while completing the survey.

Once completed the results would be sent to the researcher who would review and collate them for analysis.

True-Intent Studies

Better understand if your users managed to accomplish their objectives upon using the site or app, with a simple post-task intercept survey.

It can help to answer:

  • What goal did the user want to achieve on the website/app?
  • Did the user achieve the intended goal?
  • Is there anything, that inhibits them from achieving their goal?
  • What does the user demographics look like?

In a Nutshell

Users are randomly selected whilst they are using the site/app and in real-time intercepted with a short two-part survey, firstly asking them what they are intending to achieve. This can be open-ended or have some options to help guide them, that reduce vague answers like "browsing" or "shopping".

Upon submitting the first part of the survey they are told to continue with their task and when they are done, to return to the survey to answer the last few questions.

The second part of the survey asks if they managed to achieve their goals, and elicits basic demographic information.

The survey results are then analysed by the researcher.

When is it Typically Used?

Used during the assessment phase of products, to understand real world usage and refine the product experience.

Measures

Measures vary and are dependent on the questions asked, scores such as NPS are typically gathered. A number of inferred statistics can also be generated using the survey answers, or a further analysis of open ended questions.

Real World Tips

  • Response rates can be low, especially if right at the beginning of their task, consider intercept surveys later on at the end of the goal.
  • Having a blank text box to capture intent, can yield low quality results which provide little insights, consider having some options alongside it.
  • Include a few demographic questions in the survey to better understand your users.
  • Keep surveys short, 3-5 questions is more than enough.
  • Collected demographics data can help inform personas.
  • Include the NPS question in the survey - "How likely are you to recommend this website to a friend?"

Positives

  • Cost effective to create and run on an on-going basis.
  • Can be quick to get results if you have high traffic.

Drawbacks

  • Open ended questions need to be categorised, into meaningful groups and can be time consuming.
  • Success of task completion is determined by the user - not if the task was actually completed.
  • Can be very slow to get results if you have low traffic.

Example Scenario

A library website might perform a true-intent study to see if users are completing the goals they set out to achieve.

A sample of users would be randomly intercepted on the home page when initially arriving on the site and prompted to complete the first part of the survey, then told to continue on to their task. Upon completing their task users would return to finish the remainder of the survey.

The researcher could analyse the results to infer statistics such as the number of users that managed to successfully complete their intended tasks.

Intercept Surveys

Content to be added

Email Surveys

Content to be added