If I like X, how did you know I like Y? (and would I have liked Z more)

If you have ever found a new film that you enjoyed via Netflix’s ‘If you enjoyed X try Y’ suggestions or connected with a colleague via LinkedIn’s ‘People you might know’ (PYMK) feature then you have experienced the power of Big Data to improve your online experience. These algorithmically generated suggestions essentially augment the users experience by offering a personalised navigation through an otherwise overwhelmingly large set of choices be they other users, films, books or musical artists.

What really appeals to me about these suggestions, which are now baked into 21st century cultural consumption, is their efficacy in spite of their simplicity. The PYMK feature was one of the LinkedIn data team’s first high profile features which is now ubiquitous. The idea is that if Alice and Bob are friends, and Alice and Charlie are friends then it is likely that Charlie and Bob also know each other. This is known as ‘transitivity’ in network science and it describes the tendency for people to form cliques and communities where everyone knows everyone else. If social connections were made between two people at random and independently of any third party then transitivity would be very low and the PYMK feature wouldn’t be of much use.

But, the world is not random. Things in the real world happen in bursts and display strong patterns even though they may be sometimes counter-intuitive; for example the link between high IQ and enjoying curly fries found in this infamous paper analysing Facebook ‘likes’. The human race is not immune from this predictability and information retrieval takes full advantage of this to offer personalised routes through otherwise intractably large datasets.

Filtering Content, Together

I find the other model of ‘If you like X try Y’ suggestions much more troubling than the innocuous PYMK model. Such features are examples of ‘collaborative filtering’ whereby a picture is built up of relationships between different items of interest by examining what others have previously expressed an interest in. By observing these past connections that others have made, these same connections can be suggested to other newly observed people about whom there is incomplete information. In the simplest terms, let’s suppose I observe a large number of people who enjoyed both Flight of the Concords and the Mighty Boosh (since they are both comedy shows with satirical musical interludes). If I find new people who liked the Mighty Boosh but hadn’t expressed an opinion on Flight of the Concords then I would bet that they would enjoy it and recommend it to them. This continues for large numbers of items so that full profiles of different kinds of people can be constructed describing all of the items they will, statistically, want to consume.

The Netflix Algorithm

Perhaps the most high profile example of a collaborative filtering system is the Netflix recommendation algorithm and associated challenge and reward. The Netflix challenge winning team did for Singular Value Decomposition (SVD) what Google’s PageRank did for eigenvector centrality; namely take an abstract piece of linear algebra and not only make it relevant but also deploy it extremely successfully. For simply bringing to life important topics taught to me badly 10 years ago, I applaud the Netflix challenge.

But perhaps I can explain my unease with collaborative filtering with an examination of the early applications of the SVD; image compression. An image is a matrix of pixels each with a colour. The most verbose way to describe an image is to provide a list of the form ”pixel at row 2 and column 3 is 10% red, 50% green and 0% blue”.

The idea of image compression is to find a way to describe the image more simply than the exact colour of each of its many pixels. The compressed image will most likely lose some detail compared to the original, but the trick is to make the lost detail as negligble as possible while making the compressed image as small as possible. There are many different ways to compress an image, JPEGs and GIFs each use different methods which suit different kinds of images. My personal favourite is fractal image compression which takes advantage of the fact that images of nature such as coastlines and river basins often resemble themselves on a smaller scale. (Technical difficulties in implementation did however lead to the following tongue in cheek algorithmic recipe for fractal image compression: (1) Lock graduate student in his office (2) don’t let him or her leave until they have arrived at the algorithm)

The SVD looks at the image and tries to find rows of pixels that best approximate many other rows in the image. As a simplified example, if the image was of a smooth sand on the bottom half and a plain sky on the top, the image could be compressed by storing a mostly blue row representing the sky and a mostly brown row for the sand. A crude approximation of the image could be formed by approximating every line in the original with one of these two rows. Using 2 different rows to approximate a detailed image would lead to an underwhelming approximation of the original, but as more rows are used to build up the image, more detail can be captured (the affect of including more rows or singular values can see in this demo). The process works by recognising structure in an image and then using a smaller number of ‘typical’ rows to describe all the rows in an approximate way.

Compressing Culture

You might have guessed what’s coming next; you are a row in an image to be compressed. The image is the mass of humanity with their individual preferences, quirks, unique nurture and tastes, and a recommendation engine somewhere wants to simplify this image to a few simplified rows. Once these representative rows have been derived, someone could look at the things that it is known that this person likes and fill in the blanks by comparison with the representative rows. Admittedly, the maths does make it possible to classify someone as ‘50% a sports person and 50% a comedy person’ and to make suggestions accordingly. Despite that, this process fundamentally proceeds by comparing each person to some smoothed and averaged vision of people who are ‘like’ them. The danger is that these handy representative vectors of taste that the SVD spits out become used to make suggestions that we cannot escape from. So these simplified caricatures of taste start to define our taste rather than describe it.

How often has it happened to you in a conversation about a band or film that you enjoy with a like-minded person, that the similarity of your tastes stifle any new avenues or discoveries? Shared enjoyment of one thing is enough to immediately and fully know the other person, for they are the same as you according to the algorithm. Of course there are real and organic relations; Democrats are truly more likely to enjoy Colbert than are Republicans and broadly speaking they would opt for more Colbert than Glenn Beck. But as the average characteristics of others are served up to us continually, something different happens. A vicious cycle ensues whereby our simplified classes of behaviour become more prescriptive than descriptive. Add in the peer pressure of those who have already imbibed the suggestions of the recommendation algorithm and the basin of attraction deepens.

This kind of phenomenon has not gone unnoticed. Ali Parisi has expounded on the Filter Bubble, the phenomenon whereby personalised web experiences limit the diversity of opinions and news that can be accessed stifling debate in the process. Zeynep Tufekci has also commented insightfully on the role of algorithmic filtering in news coverage.

It may seem churlish to complain that many platforms that simplify our lives dramatically, allow us to connect with friends and families and to easily enhance our career networks, are doing this in a slightly imperfect way. But the modern debate on how artificial intelligence can feasibly be integrated into society demands of us a thorough and honest dissection of all its implications.”

Data, science, data science and trace amounts of the Middle East and the UN

Love podcasts or audiobooks? Learn on the go with our new app.