In 2008, when Obama first began campaigning for the Presidency, early on, he went to Google, and spoke to all of their employees, had a question and answer session with them. Eric Schmidt who was the CEO, as a joke for the first question asked him what is the best way to sort a million 32 bit integers? Obama said I think bubble sort would be the wrong way to go. [LAUGH] Schmidt hit his forehead, the audience applauded. Obama went on to say, I'm a big believer in reason and facts and evidence and science and feedback. In the audience that day was a man named Dan Siroker, who decided on the spot to go to work for Obama. As he said, he had me at bubble sort. He had something from Google to offer Obama. It's called AB testing. When you don't know which of two courses of action is best, you flip a coin and you present one group with one and one group with the other. Google is doing hundreds of experiments any one time. What gets the best results for them? And you do what this experiment says is superior. They have a derisory expression at Google for how most companies make most of their decisions. They get the HPPO. That's the highest paid person's opinion. Here's how AB testing was carried out in the Obama campaign. Which combination of image and text do you think got the most clicks on the Obama website? One of the possible images was a turquoise portrait of Obama. Another was a black and white photo of his family. And another was a brief video of him giving a speech. And the text could be either learn more, join us now, or sign up now. Do you have intuitions about which of those are best? I actually find I don't, but it doesn't matter because I would probably be wrong. What's the most effective is the black and white photo, and learn more. As a matter of fact, it 40% more effective. That's huge, in the standpoint of campaign financing. But businesses of all kinds can do AB testing, as well. In grocery stores in the US, almost all have aisle four for sodas, aisle six for soups. Like kinds of foods are in like aisles. How else could you do it? The Japanese do it quite differently. They arrange things by type of meal. Japanese miso soup ingredients are all sitting in one place. Italian food ingredients are all sitting in another place. So, which is better? Aisles or this more holistic way of arranging food displays? You might think probably the aisles are better for Americans and maybe the holistic way is better for Japanese. That's actually not the case. The holistic way is much more effective for Americans as well because if they're going to make Italian food, they get home, they say my god, I forgot the Romano cheese. The supermarket takes care of that. Also by the way, this is not only good for the supermarket, they sell that Romano cheese, but it's good for you because you don't then have to say missing the Romano cheese, I'm going to have to have a TV dinner or eat a bowl of cereal. One very enterprising grocery store in El Paso, Texas decided to try to find out if they could sell more fruits and vegetables than they do. That's doing a good deed if they get people to do that because fruits and vegetables are better than most of the rest of the things they could be eating from the supermarket. But it also be doing well by the supermarket owner because the profit margin is higher on produce then it is on other things. One of the things he tried was signs saying that the average shopper buys x number of dollars of produce. That increases sales. Or you try putting a divider in the cart saying, place fruits and vegetables in front of cart. That actually doubles sales. Whatever business you're in, there are going to be opportunities to experiment instead of just getting the HPPO. You also need to be conducting AB testing on yourself. When I was in college, I had a roommate who when he had an exam the next day, he felt it was terribly important to get a good nights sleep. But he would be worked up so he'd lie in bed and he's having trouble getting to sleep so he would get out of bed and do 40 push ups to make himself tired and get back into bed and lie there for another 20 or 30 minutes without sleep. As I guessed at the time, it's a perfectly terrible idea to try to exercise and then try to go to sleep immediately after. Because your physiological arousal level after exercise lasts for 20 or 30 minutes and it's going to keep you awake. Should you work right up until bedtime? Or should you always quit well before bedtime to get relaxed? I know what it is for me because I've flipped the coin. If I work until I'm exhausted, I lie in bed and continue working as opposed to sleeping. Should you get out of bed when you're sleepless or should you stay in bed? It's so tempting to stay in bed. I don't want to get up out of bed, I need all the sleep I can get tonight. Flip a coin and try getting out of bed if it comes up heads, staying if comes up tails. You may find what I have, which is that you're much better off to get out of bed and read a book. Typically, within an hour, I'm tired, I go back to sleep. Whereas I can lie and toss and turn annoyed at myself for not being able to sleep if I'm going to stay in bed. Should you go to sleep in a warm room or a cold room? By the way, that's a very big what we call main effect there. That is most people sleep much better in a cold room than a warm room. Does coffee make you more efficient or does it just make you more jittery? You're probably not going to find out without AB testing. People's response to caffeine by the way is very different. It's actually a soporific for some people, makes people more relaxed [LAUGH] put's them to sleep, that's not common, but there are wide varieties of ways of responding to caffeine. Does yoga have good effects on you? Try the AB testing. Do yoga for a week, and no yoga for another week, or do it for six weeks, and no yoga for six weeks. Is meditation good for you? Does it make you a better person? Does make you more relaxed? Does it make you more efficient? Flip a coin before you decide whether to meditate today, or this week, or not. Are your papers better when you start with an outline or is it better to just jump into it? This can go on forever, of course. All kinds of things that make us more efficient, that make us happier, are subjects for experiments. They're possible topics for experiments and they're imminently worth doing. So if you want to know what's going on, you can't just observe the world, you can't even just be systematic and look at correlations. Usually, you have to do an experiment in order to be sure about what is causing what. In the next segment, we'll be talking briefly about some technical points about experiments and we'll be talking about two topics which are dear to my heart. One is the fact that natural experiments are occurring around us all the time, and we can make use of those. The other is how important it is for society to do more experiments than it does. We pay huge prices for the experiments that we don't do.
