Challenges of recommender systems
Most articles about recommendation engines focus on all the bright sides of recommendations: personalized customer experience, lower churn, increase in sales, more revenue. While all of that is true, as we can see looking at the examples of numerous companies including Amazon, adopting a new technology requires a strategic approach – so you should be realistic and well-prepared, and not only optimistic about the future outcomes. There are some challenges that you have to be aware of.
Lack of data
Data is the key element of any predictive model, and recommendation systems are no exception. They generate accurate recommendations based on available information. It only makes sense that the recommender systems thought to be the best ones are those from companies who are in possession of vast amounts of data, such as Google, Amazon, Netflix, or Spotify. Good recommender systems analyze item data and customer behavioral data to find similarities and suggest items. Artificial intelligence thrives on data: the more data the system has to work with, the better the outcomes.
Everything is changing
Data is changing, user preferences are changing, your business is changing all the time. That’s a whole lot of updates. How well will your algorithm keep up with the changes? Of course, you can have real-time recommendations that take into account the most recent data, but they are also more difficult to maintain. On the other hand, batch processing is easier to maintain but does not reflect the recent changes in data.
Believe it or not, but people are the source of a large share of the problems with recommender systems. They may be unpredictable and yet expect technology to assist them no matter what. I may be browsing Amazon for a smartwatch today and leave the site, but tomorrow I won’t be interested in it anymore, now I need a present for my sister. If I want to buy a TV, and the system knows it, should it still recommend TVs or influence my behavior and recommend other items? There are a lot of difficult questions that may come to your mind when you focus on people – and focusing on people is still important because it’s them who buy your products.
Oh, and just one more small thing. As Steve Jobs said:
That’s why the “discovery” factor plays a significant role in recommendations. People may tend to watch similar shows and read similar books, let’s say sci-fi. Does that mean that all they want to see is sci-fi? They need diversity so they can discover things outside of what they already know too.
Additionally, sometimes ratings don’t reflect reality. When people watch a stupid comedy, they may rate it lower than an Oscar movie. Let’s say 2 stars vs 5 stars. This may teach the system that the Oscar movie is what should be suggested. However, the reality is not that simple. A stupid comedy may be worth 2 stars and still be a preferred choice on a Friday evening.
The recommender system should be getting better and better all the time. It’s not enough to just start it and let it run, machine learning algorithms help the system “learn” the patterns, but the system still needs some guidance to provide relevant results. You need to improve it and make sure that whatever changes are introduced, you’re still going towards your business objective. Have you heard of the Netflix prize competition? It was an open competition for the best collaborative filtering algorithm. The goal of the competition was to improve member retention. The winners won 1 million dollars. Netflix knows well that they have to improve, and the better they get, the more money they make. Recommendations contribute to lower churn, which in turn means higher monthly revenue.