Qualtrics Notifications

Illustration showing themes of notifications, settings, and action

Maturing principles for a new notification system through user research and iteration

Surfacing critical information users in any context

Before this project kicked off, customers had to scour dashboards and constantly check operational health and response rates to know what was going on, and determine what needed their attention. The impetus for building the notification system was that we realized that Qualtrics should be able to recognize patterns in the data and surface the things that need to be done to move the business forward.

Image showing two versions of the Qualtrics notification panel, one is the old version and the other is the new version after updates were made — `Before and after my work on the notification panel`

Creating relevant and actionable notifications

Our main goal, as a team in embarking on this journey, was to create a system of action in Qualtrics that could feed the user the things they needed to see at the most relevant time in order to improve productivity for the end user, thus improving the user experience and engagement. My role in this project was to take the existing principles recently put in place by my manager and expand and mature the system so that it was fully realized beyond the basics.

Balancing technical constraints and multiple stakeholder inputs

Some challenges I was facing were technical constraints, finding a solution that worked for all our stakeholders as a platform team, and implementing a system that would work with the mental models/ expectations of our users since the notification system was brand new. We also needed to get a better handle on what our content system would be in this context since we hadn’t had any precedent before this system in terms of persistent system messaging.

Researching user expectations and content needs

In order to understand our users’ needs better we did a couple of research studies to start out. First I did some market research to better understand the realm of notification systems, in general, and to see what the best practices were. I conducted this market research by reading articles by the likes of Nielsen and Norman and other UX experts in the field and by snooping around other notification systems and doing an audit of their features. It was helpful to see what other places were doing well - and also what I didn’t like. While doing this market research I audited different types of businesses (like LinkedIn vs Jira) and kept in mind what kind of platform I was using. If it was a social media product I was looking at, I kept that in mind to better understand what their motivations in sending a notification would be vs what we needed to be communicating as an enterprise business to achieve our goals.

Figma board showing screenshots of different notification systems and notes throughout — `Market research audit I did for better understanding of the space`

While I was doing my market research, we also worked with UX researchers to do a couple formalized studies to hone the content approach and to see how people might react to various options which would help us understand the users’ expectations of the system better. We led them through various interactive workshops where they answered questions and completed tasks live with our researchers. Through this research we learned some very important things! Some of these results included that users expected that they would receive a notification when there was a change in metrics, they wanted to receive notifications for trending topics, they expected that notification management should be able to give them more control and that brand admins could control the overarching topics while end users could control preferences within that, and that while using the notification system the user would be searching for a single source of truth in a notification that they can then investigate further after being notified - ideally with ease of navigation.

Image showing two screens with executive summary results from UXR — `Two of the UXR studies’ executive summary slides`

Maturing principles, guidelines, and touchpoints

With research in hand, we were able to get started iterating and maturing the existing designs.

Based on our findings we knew we needed to update the appearance to help our users have a consistent understanding of the structure of the information they were seeing and to understand what state the notification was in when they opened the panel and when looking at the bell icon and the new notification indicator. We also knew that we needed to provide more informative calls to action in the text itself. To update the bell icon and indicator we simply ran an A/B test that told us that we should be using an indicator without a number counter in it because it stressed people out. I also updated how the states updated when a notification was clicked on or the notification panel was opened and made an effort to make the visual design feel fresher. We also wanted to make sure that there would be the maximum amount of room possible for text so that long project names could be communicated effectively and without sacrificing understanding for our users. I played with different ways we could display the cards so that we had the max amount of room to display text and wound up with a design I thought effectively saved space to maximize information.

An image cycling through various versions of a notification card — `Here are a few of the ways that I formatted the notification card during my iterations`

For the navigation component, I wanted to dig deep into what the mental model should be for our users when interacting with various aspects of our notification CTAs. I created a series of diagrams to visualize how these systems should be working together to create a fully realized framework during interaction. These guidelines helped to inform when to use a particular type of CTA and what that would mean for the interaction. Ideally this would help set consistent expectations for the users. Developing this structure also helped us set apart the type of navigation we would use to get to the settings part within the notifications panel.

Image of a decision chart showing the viewer how the CTA would act in various situations — `One of the visualizations that I created to show what kind of CTA needs to be used in a particular situation`

When we were designing one of the constraints we were navigating was the fact that we had no ability to create true relevance for our users through machine learning in our product. This meant that we had to create relevance by asking directly what users liked and didn’t like themselves and based on the feedback from our research we knew we needed to add complexity to the settings. In the next version, we added the ability to have ‘Turn off’ controls from the notification card overflow menu and we added more categories to the list of possible types of notifications users could have control over, along with the ability to completely turn off notifications to specific channels. We decided that we wanted to have a better picture of the different types of notifications in order to start bucketing them into higher level sections and instead opted for a flat list until we had more data.

Solidifying guidance based on learnings

View full documentation prototype

Continuing the learnings through tracking MAU and MAB

In order to measure success for the notification system we tracked monthly active users and percentage of brands using the notification system and collected qualitative data. When I began working on the notification system, there was a legacy “feed” that technically had some monthly users so we had a number to start from. This feed wasn’t the same as a notification system but it was the closest we had to compare our growth rate from so it’s what we used to anchor our findings. At the beginning of my time on notifications the feed had about 2,000 monthly active users and 3,000 active brands using it. Through the experimentation and iteration mentioned above, when I moved on from the notification team, there were 133,000 (+48%) monthly active users and 10,500 (+14%) monthly active brands using the notification system. Alongside this, we had positive customer feedback such as Adidas noting that even upon the first launch, they were notified of an issue in a particular store and were able to act on it to resolve the issue!

A few reflections on the importance of triad collaboration

This project taught me how to scale a framework that had already been created through user research and a lot of iteration and collaboration with my cross functional partners while navigating technical constraints. I think one of the greatest lessons I learned from this process was how to work as a partner with my product manager. I became well acquainted with my PM and it served us incredibly well as we were able to not only deliver on business needs on the most basic level, but we were able to do a great deal of whiteboarding about ideas and I was able to help write the requirements alongside him and talk through these requirements with our engineering partners. It was an incredible learning experience!