Recently, a large European bank decided to roll out a management innovation: Its operational excellence program to all its back office and IT operations, approximately 400 teams, executing on a detailed implementation plan. A multi-million euro program, that failed because of the inability to get early, clear, valuable information about unforeseen potential implementation issues. How about running a quick and cheap, albeit rigorous and thoroughly thought through, experiment on a smaller scale to reduce the uncertainty and improve the likelihood of successful program implementation? Similarly, a large New England seafood restaurant chain was trying to change strategy by market repositioning. It wanted to do so by emphasizing seafood freshness, betting on customers’ willingness to pay for a catch of the day plate that was really catch of the day. This was the innovation they wanted to introduce in all their 40 restaurants. They made some trials asking opinions to customers, they played with menus and lifted up prices, they tried to speed up operations reducing supply chain throughput time. They made a plan, allocated a budget, and executed on it. But remained deluded, unable to really tell what was working and why, and eventually abandoned the idea. How about conducting some small-scale parallel experiments in a few similar restaurants, testing for real customers preferences, dishes pricing, and the supply chain capable of bringing seafood from boat to belly in a few hours? Experiments seem the right approach to follow in the above two cases, as it would allow to get more and better information that could inform the decision-making process underlying innovation. But what makes experiments so valuable for innovation management? Why and when do managers need them? The ideal type of experiments are randomized controlled trials. Randomized controlled trials are experiments carried out on two or more groups where participants are randomly assigned to receive an intervention or not. In their simplest version, participants to the experiment, for example target customers, are randomly assigned to either an intervention group, also called a treatment group, who is given the intervention, or a control group who is not. The randomly assigned control group represents the counterfactual. It allows to compare the effectiveness of the new intervention against what would have happened if nothing had changed. Randomized controlled trials are considered the gold standard for establishing a causal link between an intervention, for example an innovation, and change. In randomized controlled trials, the outcome variables of interest, for example customer satisfaction, intention to buy, etc., are measured at the end of the trial, and the results from the groups are compared to see if the intervention has made a difference and achieved its desired outcome. If the size of the groups is large enough, managers can be confident that differences observed are due to the intervention, and not to some other factors. Randomized controlled trials are usually applied to determine whether some new technology, product, component, or policy will have the intended effect. They are often the best way to establish the effects of an intervention, because - thanks to the randomization - they help to eliminate selection bias and allow to identify cause-effect relationships. In the absence of randomization, other unobserved variables characterizing the two groups might generate defects that managers mistakenly attribute to the intervention. This is the problem when retrospective data and analyses, or other non-experimental analyses, are applied, as in these cases selection bias and omitted variable bias are almost always present. A well-conceived and well-executed randomized controlled trial can inform, enhance, and sometimes change innovation decisions. For example, regarding the introduction of a new product or a new process, or the development of new product features. For example, engineers might think that they can introduce a new algorithm or a new feature to a software application, and that this is great. But this is just their opinion/hypothesis that needs to be tested and validated by the market. A randomized controlled trial in which some customers try the new version of the software, while others stay with the older version, might tell - other things equal - if valuable outcomes, like intention to buy or customer excitement changes significantly between the two groups with the introduction of the innovation. Managers and entrepreneurs should carefully design and execute the randomized controlled trials, especially if the intervention might have unanticipated or potentially negative effects on participants. We recommend that the randomized controlled trial design is formalized into a protocol or plan that includes all the activities. The protocol or plan should explain the purpose and function of the trial, as well as how to carry it out, including the intervention, the systems that must be set up for recruitment of participants, randomization, data management and analysis. Conducting a randomized controlled trial typically requires the setup of a team which should be diverse enough to include all the necessary skills. Randomized controlled trials are not always the best method to understand if an innovation will work. Innovations aimed at solving complex, wicked, multifaceted problems can't be effectively evaluated with randomized controlled trials. But if a problem can be modularized, that is broken down in simpler sub-problems, then these can be translated into testable and falsifiable hypotheses corresponding to simpler, more linear, and well-defined interventions.
