Recommendation systems are among the most familiar applications of machine learning and artificial intelligence. Not only are these systems valuable to consumers who may be looking for anything from new shows to watch or a better options for airfare, but they are also important to the producers and advertisers who benefit from matching content to its ideal audience. In his talk at ODSC Europe 2018, Dr. Mahdi Jalili discussed some of the different methods used to recommend new products and touched on some ways in which recommendations systems across the board might be improved.
[Related Article: What are Recommender Systems and Why Should I Care?]
Current Methods
Dr. Jalili first addressed the currently popular methods of recommendation, which are based on user data and rely on machine learning algorithms to rank items, i.e. Netflix movies, based on similar user profiles and past viewing history. Other data taken into account might be classified as “social data”—data gathered from preferences of the user’s friends, family members, and neighbors. Of similar importance is “contextual data,” which takes into account when and where a user might be most likely to watch a film. Busy weeknights, for example, are probably not the best time to start Lawrence of Arabia. Different systems weigh these data in different ways, but Dr. Jalili suggests that the most useful recommendations come from a confluence of diverse considerations.
Challenges
Dr. Jalili listed the following as challenges to the existing methods of ratings-based recommendations:
- Data Sparsity: There are, across all industries, more available products than potential users, and most products have no review data attached to them. The user-per-product and rating-per-product ratios lead to unconfident and ultimately unusable recommendations.
- Scalability: Running algorithms to recommend a wide array of items requires advanced IT infrastructure, which presents challenges to smaller and mid-sized firms.
- The “Cold Start” problem, which refers to new users, who have little or no data on which to base their preferences and are therefore impossible to recommend to confidently.
- Malicious users/hacks who want to unfairly alter rankings. The culprits could be hackers or simply spammers who are looking to boost their own product or act out a vendetta against some particular brand.
- Precision/Novelty Dilemma: Different users want different recommendations: some value being provided with options that they might not have otherwise considered, while some want suggestions that they are more likely to enjoy. Unfortunately, research shows that these two options are inimical: as the emphasis on novelty increases, the precision of alignment with consumer preferences decreases, and vice-versa. The vast majority of recommendation systems heavily favor precision, which shuts out potential consumers from a huge variety of content.
Experiment: Does this heavy bias toward precision reflect consumer preferences? Dr. Jalili, alongside a team of researchers from RMIT University, found that this assumption may not be justified. Their experiment provided three separate lists of movie recommendations to a selected group of users: one list that prioritized precision, one that prioritized diversity, and one that valued both equally. The study found that users selected the precision-based list about 42% of the time, which means that 58% of the surveyed population valued diversity as much or more than precision. The study also found that certain demographics were more likely to value novel suggestions.
[Related Article: New Approaches Apply Deep Learning to Recommender Systems]
Conclusion: The value of recommendation algorithms is well known, as is the fact that users will gravitate towards services that provide the best recommendations. What “best” means variable according to individual priorities, and more accurate recommendation systems will not only be better and smarter in their collection and use of data but will begin to take user preferences into account as well.
