By Emma Young
You want to choose a new vacuum cleaner, a new book, a new hotel, a new children’s toy or a new movie – so what do you do? No doubt, go online and check the star ratings for different options on sites like Amazon or TripAdvisor to benefit from the wisdom of the masses.
However, there are problems with this star based system, like a new paper in Nature human behavior makes clear. First, most of the reviews are positive. So how can you choose between two or possibly many more products with high ratings or even the same top rating? Second, star ratings are not a good predictor of success (and thus actual overall appeal and approval) of a movie, book, etc., note Matthew D. Rocklage of the University of Massachusetts and colleagues. The team presents an alternate method for selecting the best product and predicting success that focuses on the emotional responses of the reviewers.
In all four studies reported in the paper, the team used a text analysis tool called Evaluative Lexicon. This provided measures for the average emotionality and value (positivity) of a review. Emotionality refers to how deeply an attitude is rooted in emotions and not to how positive or negative it is (ratings that included many terms such as “impressive” or “adorable” achieved higher emotional scores than ratings with terms like “flawless” ..)
First, the team examined the earliest 30 ratings for all films featured on the metacritic.com website from 2005 to 2018. For each film, they collected star ratings (from 0 to 10), valence values and measures for the emotionality of texts.
Overall, 81% of these films received above-average star ratings. This underscores “the challenge of recognizing success and how people will behave in this sea of positive reviews,” which the team calls the “positivity problem”. They also found that star ratings weren’t a good predictor of box office revenue, and that text valence wasn’t a helpful predictor either. However, a higher level of emotionality was a positive predictor of future box office revenues. (This result was achieved by taking various factors into account, including the genre of the film, year of release, budget, etc.)
Next, Rocklage and colleagues tried the same way to predict the sales of all books listed on Amazon.com from 1995 to 2015. This time, star ratings forecast sales for some genres but not for others. However, greater emotionality was shown as a predictor for sales in 93 different genres. So it was consistently useful.
The researchers then turned to 187,206 real-time tweets posted in response to television commercials for 84 different companies played during the 2016 and 2017 US Super Bowls. The team found that the more emotional the tweets were about an ad, the more Facebook followers the company garnered over the next two weeks. The equivalent of star ratings for these ads was collected by USA Today newspaper, and these ratings were not a prediction for followers.
Finally, the team considered Chicago restaurant reviews on yelp.com and 1.3 million table reservations on a popular booking website. Unlike previous results, high star ratings predicted more table reservations. However, a higher level of emotionality was still shown to be a unique predictor for the number of bookings. As the team writes, “restaurants that evoked more emotions were associated with more table reservations”.
Overall, films, books and restaurants that aroused more emotions among consumers were more successful. Why could that be the case? Emotions mark memories as important and relatively easy to recall, and attitudes based on emotions tend to be more stable. This could clearly affect a person’s own behavior. “Additional work could investigate whether attitudes that are more emotional also influence success by increasing an individual’s propensity to spread word of mouth,” the team notes.
Overall, the new work questions the validity and usefulness of star ratings. This is not new in and of itself, but of course the researchers also describe a more useful system that could theoretically be widely adopted. “One possibility is that companies could consider aggregating the language of the reviewers and providing an ’emotional star rating’ to give more meaningful reviews to individuals,” they write.
– Massive emotionality shows human behavior and market success
Emma Young (@EmmaELYoung) works at BPS Biomedarticles