This page has been migrated from an earlier version of this site. Links and images may be broken.
Pretentious enough title for you?
I
Recommender systems - those algorithms that guess what you may be interested in as you browse Amazon or listen to last.fm - are commercially important. Netflix claims that 60% of its rentals are driven by its Cinematch recommender system [link]; that’s over half a billion dollars of business in 2008. As online commerce continues to grow, recommender systems will only get more important.
II
Recommender systems are culturally important too. As more of our culture moves online, they will be responsible for more of our cultural experiences, and will play an important role in shaping the creative parts of our societies.
III
Recommender systems will get better. Ten years ago they were largely improvised. Now you can do a Ph.D. in recommender systems and there are international academic conferences all about them [link]. The subject is ideal for academics - it is algorithmic and yet open ended, with many different approaches and criteria for success. It’s an endless playground for exploration and simulation.
IV
Even though they will improve, there is no such thing as an optimal recommender system. Accuracy is insufficient. The interests of recommendees vary. Serendipity, intra-list variety, reliability and trust-generation are just a few other considerations [pdf link].
V
Don’t confuse the outcome of recommender systems with intrinsic merit. The recommendations are highly dependent on history and are the products of cumulative advantage. Many think that “if the experts could only figure out what it was about, say, the music, songwriting and packaging of Norah Jones that appealed to so many fans, they ought to be able to replicate it at will”. But hits cannot be reliably predicted because our choices and preferences are too inter-linked. Clive Thompson writes that companies with recommender systems can “track everything their customers do. Every page you visit, every purchase you make, every item you rate — it is all recorded.” [link] But other studies have shown the systems to be chaotic. Tiny, random fluctuations can lead to completely different outcomes. [link]
VI
Recommender systems can easily reinforce inequalities among recommended items. A system that recommends popular items will increase those items’ popularity. Unpopular items will be left in the dust. Such systems can make big hits even bigger, and can lead to an overall decrease in cultural diversity.
VII
Recommender systems can increase the experience of diversity. By drawing attention to items individuals have not found by themselves, they can lead to new experiences. But individual diversity is different from overall diversity. Some systems can increase both individual and overall diversity. Other systems increase individual diversity but, at the same time, prompt consumers to be increasingly similar to each other. Their selections then come from an increasingly narrow range of items [pdf link].
VIII
Ownership matters. Given the variety of approaches, outcomes, and absence of clear “best” alternatives, and given the ability of recommender systems to shape the experiences of their users, there is ample room for ulterior motives to become embodied in the system. The incentives for the recommender and the recommendee may be different. The incentives for Netflix in a regime where they deliver physical DVDs (of which they have limited stock) may be to promote the back catalogue. When they deliver movies digitally (as they are about to) there may be no such constraint and they may be more tempted to promote existing blockbusters. The most valuable recommender systems may be those that are independent of producers and vendors.
IX
Transparency matters. The unmarked presence of sponsored items in a recommendation list would be widely viewed as a corrupt set of recommendations, but just as some bookstores charge for premium display sites within the store, so sites on recommendation lists may be sold. Recommendees have a right to know if payola is part of the system.
X
Recommender systems will displace the filtering role of both reviewers and of publishers. But while bad reviewers and publishers would not be missed, good reviewers and publishers are not only filters; they are also an active part of cultural creation. The impact of recommender systems on these members of creative communities is important.
XI
The word “community” is widely used in conjunction with recommender systems, but they do little to build communities. Their use is essentially an individual, isolated act. Groups and networks are as important in the creation and experience of culture as individuals. Recommender systems will play a role in how culture is experienced, but they are not necessarily a strong force pushing us either towards or away from a healthy culture.
XII
Recommender systems only filter culture, in various ways; the point is to create environments in which artists can prosper.