The Recommender System Paradox
Are recommendation systems really what they purport to be? Or are they a manipulative use of Artificial Intelligence techniques that will lead humanity to dystopia?
Collaborative filtering, content filtering, matrix decomposition, clustering and deep learning techniques are among several methods being utilized to drive recommender systems around the world today and they are being widely explored by companies who have a large chunk of their customers utilizing their services via the Internet. These customers produce a large amount of data which these companies hope to capitalize on, not only to provide a better user experience, but consequently also to stay relevant among their competition. With all of that data, comes the ability to draw useful insights.
Despite the advanced approaches to design recommendation systems running real-time and operating on large datasets which have been made possible due to advancements in technology, their efficacy is constantly scrutinized.
We may not have explainable AI today, but we have a gamut of explorations in the field. These technological upgrades are not merely mathematical innovations for most people, since they are often directly linked to making businesses lucrative and marketing effective. Entrepreneurial advances in AI are a big contributor to this trend, but the true potential of AI is perhaps inexplicable at the moment. It is crucial to examine both sides of the coin and I am going to use recommender systems — a popular machine learning model that is proliferating today — to drive my point.
This blog can be read in three different sequences, each intended to shape your opinion on the subject in a different way.
(1) P1 > P2 > P3 > P4 > P5: The pros and cons of recommendation algorithms need to be fully understood to evaluate their potential.
(2) P1 > P2 > P4 > P5 > P3: Recommender systems are diminishing our ability to have self- defined experiences.
(3) P1 > P3 > P4 > P5 > P2: Recommendation algorithms are not taking over our lives and they are exactly what we need.
P1: How likely is a user of a webpage to hit ‘Read More’ to understand the ambiguous mention of ‘functional and analytical cookies’ which are being used to collect data to enhance the user’s personal experience? The probability of a user hitting ‘Read More’ is less than that of a user hitting ‘Accept’ and it is perhaps also lesser than the probability of a user ignoring both and continuing to view the webpage if he has the choice to, because — who wants to read that stuff? But what does this have to do with recommendations? Let me explain.
P2: The statistical data drawn from decision paths established from link-click data of a website can be used to analyze a user’s behavior and derive a recommendation algorithm that may or may not benefit the website but will certainly provide recommendations. These recommendations could be for a company on the internet, or for its users. The recommendation is made in the form of an algorithm, depending on the need for the recommendation and the owner of the need.
Humans will continue to explore the unknown as long as the will to do so exists, this is in our nature and the advancement in technology will not preclude the chances of us striking a period in time when we are isolated from the predicaments brought about by our own creation.
This is the crux of evolution and owing to the celerity of it (and its current exponential uptick), we tend to get ahead of ourselves and speculate the chances of things going wrong.
P3: Recommendation algorithms tend to diminish the probability of users viewing specific sections of a website, as long as their outputs are displayed effectively. This often happens when they are used as a marketing strategy. Do they take control over their users? Perhaps they do, in some sense. The pervasiveness of mobile devices which use the internet has led to an explosion of data and this data is what several industries currently thrive on. Facebook and Twitter display advertisements and promotions to users, which are generally recommendations based on their activity. Beside this, pages and friends are recommended based on existing networks and interactions with the website. This is done to such an extent that if a user so desires, decisions can be made solely based on recommendations. One may argue that this technique has proven useful by quoting the example of GooglePlus, a social media platform that Google closed down due to several factors, including a lack of users.
P4: Algorithms that run on websites usually don’t offer a guarantee, they provide a suggestion.
They are meant to go by their literal meaning of recommending and assisting, but sometimes they seem to be targeting users rather than assisting them.
Picture the User Interface (UI) of Netflix and Amazon Prime Video to understand this. Netflix has a section called ‘Top Picks’ and Prime Video has a similar one called ‘Recommended Movies/TV’. The suggestions are placed at different points of the homepage that appears on your screen after logging in. The UI of the perhaps more esoteric ‘ShowTime’, can be used to explain another point. A quick glance at the homepage will tell you that the company does not draw much of its revenue from recommendations made to users, and gives the user the power to decide what to watch. The popularity of shows is a parameter that is determined based on the number of viewers and the amount these shows / movies / broadcasts were watched. The less you use a website that thrives on recommendation algorithms, the less the algorithms seem to be taking over control. However, the control provided by the website must remain in the hands of the user. There is no guarantee that the more the possible decisions that a user could make would result in a quicker search, but the more the user is aware of what they are doing, the better it is for the owners of a webpage. One of my friends prefers Netflix ahead of Amazon Prime Video, and another friend prefers the latter instead. In my experience, the reason for such choices are generally biased and based on genuine recommendations from actual people. I like both equally, but I choose based on the availability of the show and often allow Google to decide which one I watch based on the search result. Clearly, I bank on Google’s search result which could be perceived as a recommendation in itself.
P5: Most recommendation algorithms aren’t invented, they are derived. Pragmatism is good as long as it is constructive, and this is especially true in the case of artificial intelligence applications. There exists an unquantifiable human element that remains amid all the recommendations made owing to the paradoxical nature of parameters defined by programmers. For example, a viewer may have viewed a show multiple times, but may have been away from the screen while the show was playing. In another instance, the viewer may actually watch the show but close the browser and continue on another device using a different account. These algorithms are momentarily inchoate, and we will reach a stage when they will no longer be, but they still would not assume absolute control over our online experiences.
Ask yourself again — ‘Can recommendation algorithms take away my free will?’
