Evaluating Reactions and Replies

theScore
Sept - Dec 2023
UX Research Intern

Summary

As a UX Research intern at theScore, I led a moderated usability and concept test to assess the usability and users' impressions of the new features reactions and threads in article comments. I found that although the majority of users had a positive impression and found them easy to use, there were concerns about negative reaction types and an expectation for sorting and non-disruptive notifications. This led to a change in the design of the reaction, and notifications to be added to the larger product and design roadmap.
Reach out to me if you want access to the full research plan and share out from this project!

Background

theScore is a sports entertainment company that provides sports content, scores and statistics to users primarily in Canada and the United States. In fall 2023, I worked as a UX researcher on the Community team, which was responsible for any features related to socialization -such as chats, comments, and account creation.
The Community Foundations Research project I led was an evaluative study to test the new features being rolled out by the Community team. Before development, the team wanted to concept test the reactions and thread designs to ensure that they were usable and received positively by users.
The current comment section and designs created by the designer to be tested are shown below:
Current comment
Proposed "Add Reactions"
Proposed "Reaction Breakdown"
Proposed User Account Info

Research Goals

I collaborated with the Product designer in a kickoff meeting, where we discussed the main goals and questions we wanted answers to from this research. From this discussion, two research goals came out:

1

To assess the usability of reactions and threads in comments

2

To assess users' impressions of reactions and replies in comments

Research Questions

Based on these goals and brainstorming of questions from the team, the following questions became the focus of the research:
What are users’ impressions of reactions and replies/threads?
1
What are the perceived use cases of each reaction?
2
What are users’ sentiments when viewing and receiving each type of reaction?
3
What are the perceived drawbacks or limitations?
4
Are users satisfied with the reaction options provided?
Are users able to use reactions and threads?
1
Can users figure out how to react to a comment?
2
Can users figure out how to view replies?
3
Can users figure out how to reply to a comment?
4
Are there any pain points or usability concerns?

Methodology and Participants

Based on the goals and research questions outlined, it was clear that there were two main objectives so a mixed-method approach would be best suited. I used a usability test to test the usability of the features, and conceptual analysis to learn about users' impressions. These were conducted using UserZoom.
Participants had to be theScore users who have commented in theScore app at least once in the past month. (n = 5-10, aiming for 10)

Conducting Research

The first half focused on reactions and the second half on threads.Participants were sent a link to a Figma prototype, and asked to share their screen to complete the following tasks:

1

  • React to a comment
  • React to a comment without any other reactions
  • Remove a reaction
  • See the breakdown of reactions

2

  • View replies
  • Reply to a message at the top level
  • Reply to a message within a thread (using @ instead of pressing “reply”)
  • Remove a reply
  • Open account sheet to mute/report/dm

Analysis

As I moderated the interviews, my designer or senior researcher would take notes in a shared Excel sheet. After all the interviews, I went through the sheet noting how many participants reached the expected outcome of each task, and any consistencies in how they interpreted each reaction. I put all my key learnings into a summary doc to quickly share with the team and prepared a more in-depth slide deck to share a week later.

Outcome and Deliverables

The findings of this research were shared at multiple stages:

1

Early discussions with Design

2

Summary Page with Core Community Team

3

Project shareout to Design, Product and Engineering

Recommendations and Findings

The following recommendations were brought forward based on the findings of this research:

1

Re-consider the money and thumbs-down reactions and replace them with supplementary reactions that more clearly represent users’ emotions in this context
Insights that supported this recommendation:
  •  All users understood the meaning and use thumbs-up, laughing, and thumbs-down reactions, but there were differing interpretations and confusion about the use of the money reaction
  • Most users would reconsider their opinion and feeling negative when receiving the thumbs down. A few users would not react either way but think others may be offended when receiving a thumbs down.

2

Consider ways of drawing more attention to the “+reaction” CTA
Insights that supported this recommendation:
  • Most users were able to add a reaction correctly on their first try, but a few users were drawn to the reaction breakdown first and clicked on that instead

3

Consider how to order or sort comments (i.e. by the most liked) and non-disruptive ways of notifying users of reactions/ replies
Insights that supported this recommendation:
  • Most users would expect the comments to be ordered by the most liked, followed by the newest and the total number of reactions with the option to choose how they order it
  • The majority of users do not like push notifications, and would rather in-app notifications or go back to the comment themselves to see reactions and replies

Results

I returned to theScore in June 2024 to see this feature begin development with the final results of my research project. Based on my recommendations, replies were able to be implemented and the  reaction types changed to the following:

Reflection and Learnings

1
The importance of being in-sync with design: This project really taught me how important it is to be on the same page as the product designer, and ensure they are always up to date on the research and findings. Building a strong relationship with my designer early on allowed me to ask questions in the research to directly influence designs and have meaningful discussions about what the interviews were teaching us. It also helped in being a strong team when presenting design decisions to product or senior management to benefit from our combined skills or design and research to pass decisions.
2
Not every use is a good use: In the world of sports and passionate fans, things can get tense, especially in the comment section of sports platform. Although the team wanted users to engage with replies and reactions, learning how some users negatively perceived the thumbs-down reaction made me realize there are poor ways to interact with this feature. Similarily with replies, there had to be guidelines and blocked words to keep the comment section a enjoyable place for all users.
Thanks for stopping by!
Know of any UXR or PM new grad opportunities, or just want to chat?
Reach out to me!
Made by Sydney LeBlanc, 2024  
Linkedinemail