This article is based on a presentation delivered during Neoteric AI Talks Berlin: Growing revenue with AI.
Have you ever heard of decision paralysis? It’s a strange phenomenon that happens when you have a lot of different choices and even though all of these choices are good for you, you just can’t make a decision – making the decision is extremely exhausting – and all of that is related to a huge overload of information that you receive. Even if you’re choosing between Mallorca or Ibiza, it’s just tiring, you hate it, and you don’t want to go anywhere anymore.
But there are ways to prevent it and to break through all the information overload around us. There are recommender systems – the systems that save us from drowning in the whole catalogue of Amazon products. That’s the thing that allows companies to give customers only the information that’s relevant and useful to them. For Amazon, it’s suggestions of products that we might find interesting, Netflix filters out and displays only the movies that we’ll be happy to watch, and Google has basically become the go-to search engine and it suggests what we might be looking for – and adjusts the ads that we see.
Recommender systems work much like our friends who give us recommendations – if we have common interests, we’re likely to enjoy the same things. So if your friend loved this new Netflix series and says “you have to see it too”, you go and see it, and you’ll most probably love it. Sure, recommender systems do it in a much more complex way, but the basic idea behind recommendations is the same: we’re served with what we’re known to like.
Types of recommender systems
There are 2 basic types of recommender systems: collaborative filtering and content-based filtering.
Collaborative filtering means that we ask a whole bunch of people what they like and what they don’t like, we segment them and get an idea of what profiles there are and what items we should be suggesting for them. The problem with it is that when there’s a new person that hasn’t browsed anything or hasn’t bought anything, we’re unable to give them good recommendations. There’s also a problem when your catalogue of items keeps growing – it’s pretty common for people to rate a fraction of the products only. Then, there’s the scalability issue when you have to take into account all of your users and all of your products, which is quite some data.
Let’s try to approach it differently: extract all the information about people and products and see what’s going on in there. So when you like movie A, you’ll also like movie B with similar features. There are some problems here, too. Extracting features can require a lot of work, it’s doable but quite difficult. Then, there’s over-specialization. We’re really attached to a specific domain, so we may be recommending the same things to the same person. Here’s an example: when we’re listening to music on Spotify, Spotify will keep recommending basically the same music genre over and over and over again. Finally, the objective features that we’re trying to extract are not always that objective. Humor or beauty are pretty subjective, so it’s difficult to say that something’s definitely good or bad.
Building a recommender system 101
When you’re building a recommender system, you have to start with business first. You need to define business metrics. You need to have a goal. Improving your revenue, reducing churn, getting more customers. Remember we’re working with data, so we may end up working for years – there’s never a perfect model. Define what has to be achieved, don’t aim for undefined “perfection”. So you choose a key metric, let’s say reducing churn, and you need to know how much better you need to get. What you need to know is the minimum level where you can state that you’re successful.
You also need to know your data well before you start building any model. We often come across cases when the client is certain that their customers do something in a specific way – like buy certain products – and that wasn’t always the case.
When you know what items you have and how often they’re purchased, it’s time to move on to users. There’s a set of information a company should have a look at to get on overview of a person: who that person is, how they interact with the service, what they bought and browsed, how they communicate with the business. With the essential information, you can build personas, then get trends and usage patterns for those personas.
Another important thing is to collect ratings. Not all products “ask” for ratings, and not all of them should. Sometimes it’s enough, or it’s even better, to get information on how much of the service the customers are using, how much they’ve purchased, how often they log in. All this information will give us a rating. This way, we’re able to still generate predictions, even when we’re not given explicit ratings. Implicit ratings can be more accurate than those explicit ones provided as an answer from the users. Implicit ratings are the ones used by Spotify or Netflix – that measure how much of a given genre, theme, artist you see.
Finally, we need to get a feedback loop to have people tell us whether something is good or not. That doesn’t always mean you have to ask how much they like something, that can also be checking how many times our recommendation was right (they watched the recommended video, bought the recommended item). When we have the feedback loop, we will be releasing new versions of the model. Your data and your model will change over time. A model that’s never updated just won’t work.
Not just recommended products
Recommendation engines are not all about recommending products that you might want to buy. They’re also there to optimize search, suggest the best action to take, display the most relevant ads, show new content to discover, or even match people.
There is an interesting part of a model that’s quite easy to miss – you need your users’ trust. The easiest way to gain it is to show people what you do with their data. Allow them to have control over their data, remove or change some pieces. Such a practice gives people a feeling of security and being in control of their personal information. What’s more, when people are able to remove the pieces of history that they don’t want us to see, they show us what they don’t want recommended, so they’re also improving our model, and that’s never a bad thing. That’s something that contributes to a better view of a person.
The process of building a recommender system
What does the process look like?
First, we collect data about our users and our products – which is what we’ll base the recommendations on. We aggregate and the data, create a huge excel-like table and feed it into a machine learning model. We get the recommendations, and then we can easily build not only recommendations for single products but also bundles of products that people are likely to buy together, which is an easy way to generate upsells.
Recommendations contribute to improved customer experience, higher customer satisfaction, lower churn – all that while generating more sales and increasing revenue. Recommender systems create a win-win situation, making both the customer and the company happy. And here we go, we’re making the world a better place.
