Reference class forecasting using pluralism: Fighting single parameter obsessions

Traveling around prestigious Universities and Business Schools in the US this week on an institutional assignment (this post comes from Chapel Hill, NC), one thing struck me in this society, pluralism. I read with interest my friend Suresh Satyamurthy’s piece in (link here) that uses a hangman metaphor for an investor review in the start-up world. In Suresh’s start-up world, the investor is hung-up on a single parameter – scale (pun intended). It set me thinking – any evaluation of performance (more importantly, assessment of future performance) needs to be grounded in as many parameters as possible. In this post, I will introduce Reference Class Forecasting (RCF) as a technique for fighting such biases like single parameter obsession. Drawing on research on behavioural economics, I attempt to provide guidelines for entrepreneurs and investors to make better forecasts of future performance.

Intent-outcome relationship

This is possibly the first and the most obvious starting point of any assessment. Start with what was the intent in the first place. If the stated intent of the platform was to transform the industry, please define what is industry transformation and measure those, and not start harping on profitability. Not every business needs to show the same kind of performance on the same parameters. Take the example of baby products company, The founders’ motivation to start-up arose from the difficulty in finding products for their own children – availability, variety, poor quality, and certain international products/ brands not available in India (read their interview here). So, the best performance metric for assessing the performance of would be to see if they have been able to “make a wide variety of good quality international products and brands available to parents”. The performance metrics would therefore be (a) number of outlets – online and offline, (b) inventory size and variety, (c) number of brands, (d) number of products uniquely available at, at least in a specific geography, and (e) number of parents reached. Scale here would mean growth in number of customers, brands, products, and channels. Not GMV, not anything else. Yes, profitability is important, but not the first parameter of success.

Constructs, variables, and measures

Hmm, I may sound like a research methods teacher, but I think this is important to understand. Everyone (at least those reading this blog post) understands that everything could be measured in a variety of ways. A construct is an attribute of a person/ entity that cannot be observed or measured directly, but can be inferred using a number of indicators, known as manifest variables. For instance, entrepreneurial success is a construct that is measured by a variety of variables ranging from firm performance, firm growth, market power, firm’s influence in industry standard setting, pioneering innovation, to even investor wealth creation (or exit valuation) at sell-out to a large corporation. Each of these variables could be measured using different measures; see for instance, the number of measures we identified for firm growth in the context of in the last section. Can you see a decision–tree like structure here?


So, when I think of multiple parameters, I am reminded of indices. Indices like Human Development Index (HDI) as a measure of economic development, or a Consumer Price Index (CPI) as a measure of inflation. Each and every of these indices are prone to discussions and debates about what constitutes these indices and why; and in what proportion/ weights. Take for instance HDI that is a composite of life expectancy (personal well being), education (social well being), and income per capita (economic well being). Why only these? What about social and racial discrimination? What about ecological sustainability? Similar is the case with consumer price index (CPI), which is calculated using prices of a select basket of items, with price data collected weekly, monthly, or half-yearly for specific items. Again, why should tobacco products prices be included in CPI calculations? Or we could debate of how the housing price index is calculated for inclusion in the CPI. Does age composition of the household matter in calculating the CPI basket? For a relatively young family, would the basket of goods not be different than those families with more elders than children?

So, to cut my long argument short, please refrain from creating indices that just simply represent a mish-mash of parameters to evaluate a start-up.

My recommendation: Use reference class forecasting

Reference class forecasting (RCF), sometimes also referred to as comparison class forecasting is a method recommended to overcome cognitive biases and misplaced incentives. My favourite article on this appeared in The McKinsey Quarterly (see here). Let me elaborate the theory first.

Nobel laureate Daniel Kahneman and Amos Tversky’s work on theories of decision making under uncertainty is the starting point for understanding RCF. They described how people make decisions that are seemingly irrational while dealing with probabilities and forecasts using Prospect Theory (see an insightful class by Prof. Schiller, another Nobel Laureate, on YouTube here). Summary relevant to us: people are more concerned by smaller losses than equivalent gains; and people round off probabilities of occurrence to either zero or one, when it is close to either, and in between, exaggerate.

Let us understand how an entrepreneur could use this theory to manipulate his capital provider. She shows some initial success, and likens her business model to an already successful model somewhere else, in some other context; and gets the investor to exaggerate the probability of her success. For example, I know a friend wanted to build the Uber of toys in India. Why buy toys, just rent them, let the child play for a week, and return it back to the library next week to issue a new set of toys. Sounds exciting? Just that the economics did not work out the cost of damages to the toys small children could do, that would render it useless for the next borrower (like breaking one car wheel). The entrepreneur kept the rentals high enough to account for such losses, and soon her customers realised that the rentals were working out far more expensive than buying new toys, notwithstanding the child refusing to part with his toys at the end of the week. The entrepreneur continued to convince his investors to keep investing in her, luring them to wait for the economies of scale to kick-in and she could have enough bargaining power with toy manufacturers to directly import from the North of Himalayas, but that never happened and the investor exited the firm at its lowest valuation.

These biases manifest themselves in the form of delusional optimism, rather than a clear understanding and detailed evaluation of costs and benefits, even when hard data is available.

Steps in using RCF: A field guide

RCF helps forecasters and planners overcome these biases by situating the reference point outside of the subject being assessed. In order to forecast (or assess future performance) a business, investors need to identify a reference class of analogous businesses, estimate the distribution of the outcomes of those firms, and benchmark the enterprise at an appropriate point of the distribution. Firstly, the investors should identify appropriate reference class for the enterprise. These reference classes need to be identified using a variety of parameters that match the enterprise. The next step is to analyse the performance of the firms in the reference class and map them into a probability distribution. There may be clusters of firms that may emerge during this distribution-mapping exercise; there may be instances of only extremes of firm performance observed (say in winner-takes-all markets); or there could be continuous distributions.

The next task is to use pluralism in the parameters to position the enterprise in the distribution. Here is where multiple parameters would help in an reliable estimate of the position. For instance, an Uber for toys in India would only work when the marginal costs of renting out a car (wear and tear) is negligible compared to the fixed (sunk) costs of buying the car. Whereas in the toys market, the marginal costs of a child playing with the toy is a significant proportion of the market price of the toy, and therefore this enterprise would not be subject to the same evolutionary direction as Uber. However, if the enterprise was repositioned as a toy library (as my friend ultimately did), it would work – look at how the cost structures of library and toys work. It provided her a benchmark on only buying those toys that would be durable, held the customer’s attention for only short periods of time, and were very expensive to buy. Typical examples were multi-player games, which no child wanted to own independently (given the small size of families today), but would rent out during the weekends/ birthday parties for a small proportion of the cost of the game.

So, hers is calling entrepreneurs and investors to overcome such cognitive biases and forecast better.

Comments and feedback welcome.


Learning from the Network meeting of the Peter Pribilla Foundation

I had the privilege of attending the 10th networking meeting of the Peter Pribilla Foundation on the 5-6 May, 2016 at two wonderful villas around Rome, Italy. Thanks Kathrin Möslein for inviting me again to participate in this wonderful network meetings in picturesque villas. This is not intended to be a minutes of the meetings, but my own notes and learning.

Manfred Broy’s keynote on Digital transformation

Digital transformation today is being driven by multiple forces: technology push, infrastructure maturity, market pulls, and startups that can leverage these business model opportunities. As markets, technology, and competence come together to create new business models, the economy is flooded with startups that could disrupt our lives in more ways than what we can imagine.

The talk brought to the fore three observations in my discussion.

  1. Software is eating our lives

As the digital transformation evolves driving on increased computing power, trnasmission power (bandwidth), and programming; Governments are struggling to regulate these business models. For instance, Skype as a software disrupted the international telecommunication industry that relied on massive investments in hardware at the backend and the consumer end. Blockchain has created an entire monetary system with no involvement/ interference of the State.

  1. From Internet-of-things to Internet-of-systems

More and more devices are being connected to the internet, and more and more data is being collected about every part of our lives. The evolution of the Internet has followed the linear path from (a) http or internet 1.0 that connected computers in a network, to (b) web 2.0 that allowed for interactive content in the form of search and social media, to (c) a semantic web 3.0 that allows for semantic search, including images, videos and other references, to (d) the mobile internet, that focuses on the App Economy – hyperlocal and mass-customized content, to (e) integration of IoT devices and servitization applications that lead the Interactive Industry or what is called Industry 4.0.

  1. Moral questions on how these data is used

As more and more data is being collected and collated by corporations, that are mostly monopolies in their markets, questions remain on the nature of consumer choice on what and how their personal data is being used, definition of trust and transparency of these data banks, and how these changes are affecting our professional, personal, and social lives.

Four sub-groups deliberated on actions, competencies, infrastructure, and promises around digital transformation.

Peter McKiernan and Anne Huff summarized the discussion and left us thinking on two axes.

  1. Has all this digital transformation driven us towards so much personalization and customisation that we excelled in marketing to a segment of one; while we have ended up destroying the social processes that form the basis of creating vibrant communities?
  2. With all these investments in digital transformation, what social problems are we solving in the developed and emerging economies? What are our contributions to sustainable management of our ecological environment, alleviate poverty, and manage active and forced migration of people across national and continental borders? What can we contribute to the improvement of human development, fostering inclusive growth, and evolve meaningful networks of social and economic competencies?

Albert Heuberger talked about the need to integrate research on hardware, software, and open problems. He talked about the various projects that Fraunhofer IIS was working in collaboration with the FAU Erlangen-Nuremberg and the Bavarian Government. His view of the future was to sustain research on

  1. Power consumption economics, including battery technology, to power smart devices that need to be ‘always on’.
  2. Devices, software, and problems that help improve mobility through increasing the digital range of smart devices.
  3. Integration of data from intrusive and non-intrusive biological data like glucose levels, fatigue)
  4. Consumer applications of hyperlocal environmental data, like pollution parameters (COx and NOx)
  5. Long range imaging, including gesture control
  6. 3D displays for mobile phones (VR apps for end consumers)

Helmut Schönenberger and Dominic Böhler from the UnternehmerTUM briefed us about the TechTalents program where they have batches of students and entrepreneurs being mentored by experienced mentors.

Peter McKiernan summarized the two talks about the need for engaged scholarship in the context of business research losing practical relevance. I could summarize the day’s discussion and thoughts as an interaction of two triads.


Our second day began with Mitchell Tseng talking about his rich experience of how the world has evolved in his talk on leveraging individual expertise in the context of global cooperation. As the world moves from optimizing supply chains to global value chains, we need to build three related capabilities

  1. Actively manage the shift from reducing waste, focus on core competence, and being responsive to customer needs to increasing the customer willingness-to-pay, focus on the value communication and delivery, and be responsive to changes in customer value perceptions over time.
  2. In a world dominated by network effects, value providers could realize value from even customer indifference. The old chinese proverb says, “the wool grows on dogs, and the pigs pay for it”.
  3. Rapid prototyping in a globalized world requires organizations to embed the product concept into the prototype and be able to test it across different parts of the value chain and in different cultures.

Hans Koller commented that even traditional businesses like aviation (free flights for passengers paid for by advertisements/ shopping), renewable energy (freebies for consumers who allow for installation of solar panels on their rooftops), and healthcare (providing free healthcare advise/ services in exchange for data collected from patients through embedded devices) are embracing two-sided markets. He also added that such rapid prototyping may leverage modularity (as propounded by Prof. Charles Baldwin) in product design and development. Building modularity across global products and value chains requires well-defined international standards for interfaces.

Peter McKiernan commented that research on value creation from the eyes of the consumers (perceived value) could learn a lot from the research on cognitive psychology literature. The definition of business value creation has over the years evolved from (a) the traditional industrial economics SCP paradigm to (b) Porter’s industry attractiveness frameworks to (c) mass customization and value creation to (d) the experience economy of the 21st century.

Members and fellows of the Peter Pribilla Stiftung (PPS) shared their wonderful work, research, and experiences. Unfortunately, the notes are not part of this document.

The afternoon was centered around two sub-groups working on (a) how the research group could work together in joint projects and (b) designing formats for digital transformation. It was discussed that the network should be largely expanded to include people from outside Germany, maybe leveraging each others’ personal networks. The need to collaborate with each other in applying for joint projects from organizations like the EU was emphasized. The group on designing formats elaborated on the need for an agency that could act as a platform that would evangalize, educate, and build strong networks of organisations that enable digital transformation with those that need their services like the Government, Universities, Schools, non-proifts, and corporations.

The networking meeting ended with summaries by Anne Huff, Frank Piller, and Ralf Reichwald.

We have come a long way from when we started in the last ten meetings. Too much of our discussion was centered around white, middle-class caucasian world. We need to expand our focus to the globalized world that includes a lot of problem. The second problem is that we have been largely academic-centric. We are the product of a system that pushes us to be more theoretical, abstract, and less practical and working with the firms. It is imperative that we move more towards pragmatic application of our energies to solve the big bad world’s problems.

Dynamic capabilities is about how organization’s change and evolve over time. We need to adopt the same approach and ask ourselves, look at our own unconscious biases, shift from the technology level of analysis to the more micro-social levels, include people from more varied disciplines like Psychology and Sociology to educate us.

We have learnt a lot about technology, digital transformation, and new business models. We are so proud that we heard from our PPS Fellows. We have over 50 fellows right now working, and it is heartening to see them do so well in their research and careers.

Thanks to Claudia Lehmann and her team for the wonderful organization.

Comments, observations, edits, and additions welcome.


%d bloggers like this: