An In-Depth Look at Online Rating Systems and the Power of Bayesian Adjusted Ratings

In today‘s digital age, online ratings and reviews have become a crucial part of how we make decisions about what to buy, where to eat, what to watch, and more. A 2020 survey found that 91% of people regularly or occasionally read online reviews, and 84% trust online reviews as much as a personal recommendation (source).

But as much as we rely on these ratings, have you ever stopped to think about how they are calculated? It turns out, coming up with a fair and accurate rating system is far from straightforward. Many of the most popular online platforms today use advanced statistical techniques, drawing upon ideas from artificial intelligence, machine learning, and Bayesian inference.

In this article, we‘ll take a deep dive into the world of online rating systems. We‘ll explore why simple arithmetic averages often fail, how Bayesian adjusted ratings provide a more principled approach, and see real-world examples of these techniques at work in the platforms we use every day. Finally, we‘ll consider some of the open challenges and future directions in this important field.

The Perils and Pitfalls of Simple Averages

At first glance, calculating the average rating for an item seems straightforward. Just add up all the individual ratings and divide by the total number. This gives you the arithmetic mean, or what we‘ll call the "simple average".

For example, suppose we have an online product with the following 5-star ratings:

5 stars, 4 stars, 5 stars, 4 stars, 5 stars

The simple average is (5+4+5+4+5) / 5 = 4.6 stars.

However, there‘s a major issue with relying on simple averages like this, especially when there are only a small number of ratings. With just 5 ratings, how confident can we really be that 4.6 stars is a reliable measure of the item‘s true quality?

To illustrate the problem, let‘s consider a more extreme example. Suppose we have two restaurants on a review platform:

Restaurant A has 2 ratings:

• 5 stars
• 5 stars

Restaurant B has 100 ratings:

• Eighty 5-star ratings
• Twenty 4-star ratings

Using simple averages, Restaurant A would have a perfect 5-star rating, while Restaurant B would have a 4.8 star rating. So Restaurant A looks better, right?

Not so fast. With only 2 ratings, that 5-star average for Restaurant A is not very meaningful. It‘s entirely possible that those reviewers are not representative of the general population – perhaps they are friends of the owner. Meanwhile, with 100 ratings, we can be quite confident that Restaurant B is consistently delivering a very good (if not absolutely perfect) experience.

This is a classic example of the "law of small numbers" – the tendency to overestimate the significance of patterns in small samples. With a large sample size, an average can be quite meaningful. But with small samples, averages can be very noisy and misleading.

The small sample problem is especially prevalent for new items that have few ratings. This can make it difficult for high-quality new items to gain visibility, while allowing a few early positive ratings to propel mediocre items to the top.

So what can we do? Enter the Bayesian average.

The Bayesian Approach to Rating

Bayesian statistics is a branch of probability theory that provides a principled way to update our beliefs as we observe new data. It is named after the 18th century mathematician Thomas Bayes.

The core idea of Bayesian inference is to start with a prior belief about the probability of something, and then update that belief as we observe relevant data. Mathematically, this is expressed in Bayes‘ theorem:

P(A|B) = P(B|A) * P(A) / P(B)

where A and B are events, P(A) is the prior probability of A occurring, P(B|A) is the likelihood of observing B given that A is true, and P(A|B) is the posterior probability of A given that we have observed B.

So how does this apply to online ratings? The key insight is to treat the item‘s "true" rating as a probability distribution, rather than a single fixed number. Before observing any ratings, we start with a prior distribution that expresses our initial beliefs about what the rating might be. Then, as we observe individual rating data, we update our distribution to get a posterior distribution that integrates the new information.

For the prior, a common choice is the beta distribution. The beta distribution is defined over the interval [0, 1] and has two parameters, α and β, that control its shape.

Here are some examples of beta distributions with different parameter values:

(Image source)

By setting the prior α and β based on the overall average rating for all items in a category, we can express our initial belief that a new item is likely to be around average quality.

Then, as we observe new ratings for the item, each rating is treated as a Bernoulli trial (a success with probability equal to the rating value, where a 5-star rating equals 1, a 4-star equals 0.8, and so on). The α parameter of the beta distribution is updated by adding the sum of the rating values, while β is incremented by the number of ratings minus the sum of rating values.

After updating based on the observed ratings, the mean of the resulting posterior beta distribution gives us the Bayesian average rating.

Mathematically, the Bayesian average can be expressed as:

where is the Bayesian average rating, are the individual ratings, n is the number of ratings, and α and β are the prior beta distribution parameters.

Intuitively, the Bayesian average is a weighted average of the item‘s observed average rating and the prior average, with the weight determined by the number of ratings. If an item has very few ratings, the prior will dominate and pull the Bayesian average towards the overall mean. But as more ratings are observed, the item‘s own data will increasingly outweigh the prior.

So returning to our restaurant example, if we set a prior based on an category average of 4.2 stars, with α=40 and β=50, we get:

Restaurant A Bayesian average = (40 + 5 + 5) / (40 + 50 + 2) = 1.09
Restaurant B Bayesian average = (40 + 805 + 204) / (40 + 50 + 100) = 4.74

Even though Restaurant A has a perfect 5-star simple average, its Bayesian average of 1.09 is pulled sharply towards the prior. Meanwhile, Restaurant B‘s large number of ratings outweigh the prior, making its 4.74 Bayesian average very close to its 4.8 simple average.

This is exactly the behavior we want – Restaurant B‘s consistently high ratings make it more trustworthy than Restaurant A‘s small sample of perfect scores.

The Benefits of Bayesian Averages

The key benefit of Bayesian averages is that they provide a principled way to address the problem of small sample sizes. By incorporating prior information, Bayesian averages can provide meaningful ratings even for items with few reviews, while still allowing items with a large number of consistently high or low ratings to stand out from the pack.

This can help solve the "cold start" problem for new items. Without a Bayesian approach, items with no or very few ratings would either not be listed at all or be unduly influenced by a small number of potentially unrepresentative raters. The Bayesian average provides a more stable and broadly applicable approach.

Bayesian averages can be seen as a form of regularization – a technique used in machine learning to prevent overfitting to noise in small datasets. Just as regularization constrains model complexity to avoid chasing patterns in noise, the Bayesian prior constrains ratings towards the population mean until sufficient data is observed to justify deviating.

A nice side effect of the Bayesian approach is that it naturally incorporates uncertainty. Rather than just providing a point estimate, the posterior beta distribution captures the full range of plausible values for the "true" rating, given the data observed so far. This can be used to provide confidence intervals around the average rating or even full probability distributions in a rating UI.

While we‘ve focused on Bayesian averages for ratings, the same principles can be applied much more broadly. Bayesian techniques are widely used in AI and machine learning for tasks like classification, regression, and clustering. The core idea of starting with a prior belief distribution and updating it based on observed data is extremely powerful and generalizable.

Examples in the Wild

Once you start looking for them, Bayesian average ratings are everywhere in the online platforms we use every day.

IMDb, the popular movie database, uses a Bayesian formula to calculate its ranking scores. While the details of their formula are not public, they have revealed that it includes a prior based on the average rating of all films, the number of ratings a film has received, and a minimum number of votes required before a movie can appear in their Top 250 list (source).

Amazon uses machine learning throughout its recommendation systems, and Bayesian techniques are likely a core part of that. They have published research papers describing the use of Bayesian models for tasks like detecting fake reviews (source).

Yelp, Google Maps, and other local business review sites use variants of Bayesian averaging to calculate their rating scores. Yelp in particular has invested heavily in machine learning to combat fake reviews, with their systems automatically filtering out 25% of submitted reviews (source).

Reddit, the massive online discussion forum, uses a Bayesian average variant for ranking comments. Their "Best" comment sorting algorithm takes into account factors like the proportion of upvotes and downvotes and the "controversiality" of a comment (source).

And there are countless other examples across recommendation systems, spam filters, trend detectors, and more. Once you recognize the pattern, you start seeing Bayesian inference under the hood everywhere.

Challenges and Future Directions

As powerful as Bayesian averages are, online rating systems still face many open challenges.

One issue is the subjectivity and inconsistency inherent in individual ratings. My 4 stars might be your 5 stars. And ratings can often reflect factors outside an item‘s quality, like the reviewer‘s mood that day or their broader expectations. There‘s interesting work being done to try to account for these rater-specific biases (source).

Another challenge is the ever-present threat of fake or malicious ratings and reviews. Bad actors may try to artificially inflate their own products‘ ratings while tanking competitors. While Bayesian averages can help mitigate the impact of a small number of extreme ratings, they are not a complete solution. Platforms are in a constant arms race, building ever-more sophisticated ML models to detect fraud.

The rise of "review bombing" is a related problem, where groups of users coordinate to leave a large number of negative reviews for a product they dislike, often for reasons unrelated to its quality. Researchers are exploring techniques to identify and mitigate the effects of these campaigns (source).

There are also important questions about the fairness and transparency of online rating systems. Many of the major platforms keep their exact ranking formulas secret to prevent gaming. But this lack of transparency can also enable hidden biases and make it difficult for impacted parties to understand or challenge their ratings. As these systems exert growing influence over our online experiences, there are increasing calls to "open the black box" and give more insight into how they work.

Looking to the future, we can expect to see online rating systems become increasingly sophisticated. Some key areas of development:

• More personalized ratings and recommendations, taking into account not just overall item quality but also the tastes of individual users. Techniques like matrix factorization and deep learning are enabling powerful new approaches here.

• Increased use of natural language processing to extract insights from review text, going beyond just the numerical ratings. Reviews are a gold mine of valuable data on what specifically people like and dislike.

• Explainable AI techniques to provide more transparency and insight into rating systems, helping to build trust and catch potential issues.

• Ongoing refinement of algorithms to detect and mitigate spam, fraud, and other attacks on rating integrity.

• Creative incentive systems to encourage quality, honest reviews from real users. For example, Reevoo, a reviews platform used by many major brands, provides free products to users selected to write reviews.

• Decentralized, blockchain-based rating systems as an alternative to centralized platform control. Protocols like TruStory aim to use economic incentives and staking of tokens to reward honest reviews and penalize spam.

Ultimately, the goal is to continue honing online rating systems to maximize their value for users and businesses while minimizing potential harms and unintended consequences. Given how much we have come to rely on these systems to guide our online choices and interactions, getting them right is crucially important.

While a humble 5-star rating may seem a blunt instrument for capturing the nuances of quality, through the power of aggregation, statistics, and AI, it becomes a potent tool for distilling the wisdom of the crowd. The simple act of clicking a star has, in aggregate, woven a vast new feedback loop between the products we create and consume, shaping our culture in powerful ways. As we work to refine these systems, it‘s worth reflecting on what exactly we aim to measure, and how those metrics shape our incentives and behavior.