Recommender systems have made waves in many industries, and technology-enabled businesses are adopting them in a big way. In this post, we will explore the enterprise needs that have led to this innovation, the issues of relevance and timeliness, technical challenges for advanced recommender systems, and the surprising ways in which they influence consumer behaviour.
Context and Benefits
Enterprises with diversified customer bases often face the challenge of delivering tailored experiences to their customers. Many such organizations leverage analytics to study and engage consumers on a large scale, and deploy intelligent systems of one kind or another, which take the customers’ behaviour into account. Recommendation engines are one such solution for businesses with diverse and broad based customer bases.
One factor that is also increasingly important in the context of recommendation engines, is the timeliness of recommendations. Customers who buy or receive products or services on different platforms such as mobile, web and in physical contexts – have become sources of data to organizations as well. Recommendation engines have therefore received significant attention – from the digital entertainment industry and the glut in e-commerce, among other domains. Some of the direct benefits of deploying recommendation engines are:
- Improved customer loyalty. Customers naturally respond better when we learn about their preferences and sell them products or experiences that they have a greater chance of liking.
- Improved revenue. Bottom line sales performance is directly influenced often by how well a retailer or service provider is able to upsell to a potential customer, or to an existing one. Naturally, profiling customers well helps organizations upsell better.
- Personalization and consumer delight. When consumers are exposed to an experience where their needs are understood, they are drawn to the brand for more than an opportunity to buy a specific product, or receive a specific service.
Relevance and Timeliness
The trend in the evolution of recommendation systems is in therefore in two directions – towards greater relevance of the recommendations and towards greater timeliness of the recommendations.
While improved relevance of recommendations is often leveraged in product marketing, timeliness is equally important, and therefore, unsupervised machine learning approaches and manifold learning approaches become a necessary capability. Traditional recommendation systems require scale to become effective and to begin to deliver value to consumers. Dimensionality reduction approaches have been used earlier to simplify large databases that are “wide” as well as “tall”, and this enabled companies to make sense of the key factors that impact consumer preference. However, this simplified customer bases into monolithic groups. Further, dimensionality reduction approaches required access to consumer behaviour data on a large scale to be effective, and with smaller amounts of data, customers may not realize effectiveness, or timeliness in the recommendations they receive. There’s also the need for feature engineering, which can vary on a case by case basis for effective recommendation with minimal misclassification. Collaborative filtering, specifically neighbour analysis algorithms, are one example of such an approach. Content based filtering approaches are less effective than collaborative filtering, because of the lack of a need to use content metadata, or consumer metadata. Because collaborative filtering uses consumer behavior rather than content analysis, it can be used in diverse settings. Collaborative recommendations systems are a natural fit for massive data sets, because they work better when large amounts of information is available to build models. However, they’re less able to handle situations where users are unique and fewer..
The timeliness of recommendations is often influenced by the network through which data is collected, and the infrastructure that companies have in place to address consumer interests dynamically. Advanced data processing platforms like Spark, which support streaming data pipelines, allow for analysis of data in motion, and interface this with data at rest. While network infrastructure and latency are a mostly solved problem, the ability to deliver rich recommendations as part of the consumer experience is still elusive for many companies. With the intelligence that can be built into modern recommender systems, and with the speed of distributed and parallel computing, more timely suggestions can be delivered to consumers than before.
The Data Team’s Cadence product is one such enabler of relevant and timely recommendations that integrates consumers’ digital and physical contexts.
Advanced Neighbor Analysis Methods
In the context of improved relevance for recommender systems, new data science and machine learning approaches are making their mark felt. Neighbour analysis based approaches are the most commonly used methods to build algorithms tailored for improved relevance. It is in this context that advanced neighbour analysis methods are adding value. Manifold learning approaches and dimensionality reduction approaches, such as Isomap and principal components analysis are now computationally scalable, and have made possible the description and classification of higher dimensional data. Methods like t-distributed stochastic neighbour embedding (t-SNE) are enabling visualization of higher dimensional data in new ways. When product families or customer families can thus be constructed in real time and in unsupervised learning paradigms, intelligent recommender systems can be made robust to product and service features or domains. When viewed in an industry setting featuring large data lakes or streaming data, smart recommendation engines also become timely.
Impact on Consumer Behavior
An emerging dynamic in the use of recommendation engines is that recommendations influence, and not only benefit from consumer behaviour. Recommendations could create powerful reinforcing loops in consumer preference patterns, and influence habit-forming behaviour in consumers. The long-tail nature of product sales has often influenced how managers take decisions about new products and services. Recommendation engines could provide them the context and the intelligence to make these decisions better. Content discovery platforms and the influence they have on the content itself is one example, with content curators in many cases transforming into well-informed content producers or enablers. Content filtering engines can therefore influence product development decisions, branding, sales and marketing decisions, and can have a large impact in the broader context of a content-driven organization, and by extension, drive customer focus. Recommendation engines that are effective will not only be timely and content-wise and context-wise relevant, but will also exhibit what is increasingly a useful characteristic – intelligent recommendation persistence. Recommendation persistence enables companies to test narratives that hold sway and consumer attention, in addition to just tailoring recommendations by customer behaviour or use cases.