Knowing Users via Customer Tickets
UX Research • Thematic Analysis
Overview
One of the key challenges of a B2B context is, often, the challenging access to end-users. Nevertheless, it is critical to be informed about who your users are and the quality of their experience with your service(s) - especially, when these can quickly shift, based on various trends and business opportunities.
The Customer Success Team is often the first line of contact with end-users and, ultimately, these will be the ones with the most contact with them. This communication is primarily achieved via customer support tickets. As a consequence, these support tickets are valuable sources of information regarding who uses the platform, how they do it, and how they (negatively) experience it.
From, late, Q4 2018 and for over 1 year, as part of my job functions, I conducted a large-scale continuous research of our customer tickets. This is the same analysis mentioned in my other project pages. This page, summarizes the overall goals, process, and outcomes of this initiative (*)
(*) Note: There are plenty of resources teaching how to create codebooks and conduct thematic analysis. This is not intended to be one of them, but rather to describe how I adapted these techniques to address our research goals. Moreover, due to NDA purposes, I am limited in the details that I can provide. Thank you for your understanding.
Team
• 1 UX Analyst (yours truly)
My Contributions
• UX Research
Timeline
• Q4 2018 - Q1 2020
Goals
• Identify the most common issues that impact the users' experience.
• Identify the most common causes and solutions for frequent issues.
• Identify in which stages of our user journeys users are the most negatively impacted.
• Learn more about what type of users are utilizing the platform(s), such as: their user groups, forms of communication, and actions or tasks that they are trying to achieve.
Methodology
For this initiative I conducted a large-scale Thematic Analysis over our customer tickets. This consists of a systematic classification of qualitative information (the tickets) through a series of codes/themes, used to identify common qualitative patterns across the data.
Procedure
• Each ticket was considered as potentially having one or more issues.
• Each identified issue was categorized based on their core themes.
• Themes were identified based on a sample of the tickets and, retroactively, applied to the set.
• Themes were grouped into different types.
• Themes and groups/types were regularly revisited to ensure consistency. If changes were to be applied, they were, retroactively, applied (yes, this required the creation and use of codes that, if deprecated, would allow to quickly and effectively be replaced).
• The categorization methodology and the tools used (e.g., scripts and query formulas) were also revised and iterated every month.
Groups
Although the groups/types did change across the various iterations of this project, the most common ones were:
• User: used to describe the user reporting the issue;
• Origin: used to describe location of the issue (e.g., which stage of a user journey);
• Solver: used to describe the level of support that helped to resolve the issue (e.g., customer support vs engineering);
• Actions: used to describe the type of actions that were done to solve the issue (e.g., a bug fix vs explaining what a certain feature does vs redirecting the user to a third party);
• Descriptive Themes: used to describe the issue itself and its context/characteristics.
Exclusion Criteria
Customer tickets under the following categories were excluded from the ongoing reports of this initiative:
• Expected communications: i.e., when contacting the customer success was a necessary part of the user journey;
• Automatic messages: e.g., shipping or invoice notifications;
• Revert communications: e.g., when the user requests something but, immediately after, cancels the request, since they found out the answer for their query on their own;
• Internal communications: e.g., Finance team requesting technical assistance, bug reports on private/test instances of the platform.
Reporting
The aforementioned process resulted in a large dataset from which various insights and conclusions were gathered. These were regularly reported in two main formats.
The first one, via company-shared spreadsheets, that allowed stakeholders to search and visualise for various statistics about this project - e.g., number of issues detected over time, most frequent issues, most frequent locations with issues, etc...
The second one, via company-shared confluence reports, that summarized and discussed the main insights gathered, based on the aforementioned statistics and, when applicable, their triangulation with other relevant initiatives and events.
Challenges and Opportunities
• It is important to acknowledge that the main source of information for this initiative has a mostly negative connotation. If users are contacting customer success, it is usually due to something that they cannot understand or do on their own - thus, these often reflect situations where the customer is having a bad user experience. As such, while this can help understanding the platforms' most serious issues, it should not be used to assess the overall health of the platform.
• Comparatively to the other entries in this portfolio, this is by far the one with the longest timeline. In fact, this initiative was originally planned to have a much shorter duration. However, due to the usefulness of its initial results, this ended up being continued for longer. However, customer tickets are an ever-increasing (and sometimes varying) source of information. While the analysis of this type of dataset, with such a long timeline, allow us to analyse the evolution of specific patterns over time, it also requires a strong discipline and frequent revisions over the methodology and its results.
Outcomes
The main outcomes of this project can be summarized by the following:
• Coverage of research goals: through the long-term exposure and analysis to various customer tickets, it was possible to identify the most common issues that impacted the users' experience and where they manifested, as well as to understand our customers better - from the types of tasks that they try to achieve, how they try to achieve it, and even how they communicate. Moreover, from a personal perspective, this has also significantly improved my own domain knowledge and provided me with a better understanding of how the company works.
• Triangulation with multiple initiatives: the various insights gathered through this analysis helped across various stages of different projects. These include the identification of potential requirements, in the early stages of a project, the design and/or review of multiple deliverables, like personas and user journey maps, or even in the post-release analysis, when comparing results to design changes and newly released features.
• Cross-function collaboration: this project was a particularly relevant opportunity to interact more closely with other stakeholders, particularly Customer Success. In this particular case, while this was one of the teams that I always had collaborated with, this initiative allowed us to interact based on a common artifact (the tickets) that they use on a daily basis and is strongly associated with their functions.
• UX Evangelization: customer tickets are a valuable source of (real) use cases and quotes, that can be used to help stakeholders understand better their users. In fact, more skeptical stakeholders may sympathize more with something that they identify as a real situation from a real user, than one described in a persona (even if the latter is similar, if not the same).
