Alright so big [inaudible] my name is Earl Lido. I actually took this class in 2004, I guess the predecessor to this class. I think, there were like five people in the class. So, it's great to see that it's grown over time, It's really a pleasure to come back here every year and talk to you guys, I actually worked in finance before this, and spoke about what I did then. And then, have since moved on to online advertising. So there are a lot of cool problems, in online advertising. And today, I just wanna back up and sort of tell you guys the, the story of how all this evolved. It's really easy to look at, you know, a textbook and see a, a really elegant theory that's been laid down. But, but the truth is most of this most of how online advertising came to use auctions was actually very organic. Sort of evolved naturally over time So I think to understand it you really have to flash back to 1994. I realized for the first time that many of you in this class may not actually remember 1994. But as hard as it is to believe, in 1994 people really didn't know what the internet was. It was just starting to enter the public consciousness. When people thought of online, they really thought of America Online. Like you'd sign on, you'd check your email. There are chat rooms. It was really cool back in the day. And people were just starting to hear about, about the world wide web. Companies were just starting to build websites. And most importantly, 1984 was the year that online advertising was invented. So this was the first online ad ever shown, actually. It was probably the most deceptive ad ever shown. It literally says, I can't read it. Have you ever clicked your mouse here? [laugh]. And it was it was an AT&T ad, as hard as it is to believe. And it did pretty well. Like, like, most people had no idea what this was. Seven out of ten people clicked on it. Which, by today's standards, is really, really good. It's, you're really lucky today if you get one out of every 1,000 people to click on an ad. So this is a very successful ad. And this really was the birth of a new industry. It's now, you know. $40 billion a year industry. It's hard to believe. Back, back then in early 90's, You know, how you got an ad on a website was a very simplistic model. You literally picked up a telephone and you called up the website, or sales team at a website, and you said, I want to put my ad here. And you haggle a little bit over price and you basically arrive at some CPM, a cost per thousand impression value, that you're going to pay. And your ad would go up for a certain amount of time. There wasn't a lot of sophistication to it. You know It'll be really interesting to see the equations for this because, this is very simple just haggle over price and decide. So not much actually changed until sort of late 90s. I think this was the first revolution in online advertising. This was the year that Britney Spears became famous. I don't know if you guys remember that. Do you guys remember. Like Britney Spears came out with, Baby One More Time? It was also the height of the dot.com bubble. Pets.com somehow raised 83 million dollars without a business model. It was amazing. And it was also the year that, Idea Lab, which was, like, the Y combinator of its day launched a company called goto.com. Here's the homepage of goto.com. It actually looks pretty similar, you know, despite the ugly logo here. It looks pretty similar to Google and their, their model there business model is actually very similar as well. They set out to disrupt the, the classifieds industry. So their, their business model was that they built a search engine. Actually, they licensed a search technology from Intomi, that allowed people to search for whatever they were looking for. And they'd allow advertisers to pay to have their advertisements actually interspersed with the actual search results. It'd be like, you know, getting Google search except the ad, you can't tell what's an ad, and what's an actual search result. And this model actually took off. Like, for a while people didn't really know whether GoTo would be the big search engine of the future or Google. So it's actually a fairly successful business, and here's how they sold their ad spots. So they, they built a self service platform, and advertisers could log in, and they could put in a bid value, an amount that they were willing to pay per click. And whoever bid the highest per click got placed up higher in the search results. Now, I think somebody over here asked a really good question about why that actually matters. Like, why is it better to be placed up higher? Look here it matters a lot you know, if you're, cause people are generally assumes unconsciously that whatever comes up first is more relevant, most relevant search result. So, you couldn't tell what was a search result and what was mad. Then whoever was at the very top got all the clicks. But their model was very simplistic. It was not a second price option. Whoever bid the highest per click got highest in the search results. This had some serious issues with it. So the biggest issue was that buyers figured out that they could game the system. So they log in, and people figure out what actually happens if I lower my bid, Could I get the same placement? And you know, the top buyer actually values you know, a click at $five. In this case, they can actually shave their bid, they can lower their bid to $four and a penny, and they can still get the top spot. So what happened was, literally, buyers would log in and they would change their bid thousands of times per day. This wreaked havoc on GoTo's infrastructure. It was very, very, like, difficult for them to maintain this auction system. And as a result, they also may have lost money. Over time everybody stayed at their bid, there's a lot of variation in that, but over time prices were actually lower than on Google, which we'll talk about. So, fast forward to 2000, 2001. So, a new company came along called Google. Which, you know I see a Google shirt over here. Okay. So, I guess people have heard of it. And, I think it was actually 2001 that they launched their self serve AdWords products. And this really changed everything. You're not only did Google have the best search engine out there, but they really innovated the way that ad space was sold. So they had two major innovations and both of these were completely organic. They, They weren't based on theory. Nobody read a textbook. According to In The Plex, like the book about Google was actually just the engineer working on this had these two absolutely brilliant ideas. So first was he implemented the option as second price option, generalize second price option. And as you guys know, what this means is whoever bids highest actually pays what the second highest bidder bid. So in this case if the top. Buyer. If the top bidder bids $five per click. And the second highest bidder bids $three per click, then the top bidder wins the top spot, but only pays $three. So this incentivizes, by, incentivizes advertisers to actually bid their fair value. So, in this case, if the top bidder bids $four they are still going to pay $three, if they bid at $five, they still pay $three. So they might as well bid their fair value. You don't see the same effect of what happened at goto.com where everybody would play around with their bids constantly and eventually it resulted in much lower bids. The second major innovation which actually seems very simple in retrospect. It's hard to believe that nobody else think about this. Was it google tied valuation to click through rate. What this means is that they, you know of course you don't realized that just because somebody bids more per click doesn't necessarily mean that Google is going to make more revenue or effectively more profit off of that. So if the top bidder here is willing to pay $five per click. And the second guy is willing to pay $three per click. But the top buyer get, top bidder gets one click every hour and the second bidder gets two clicks every hour. Google's going to make $six by putting the, the second highest bidder's bid up top. First it was $five. So this results in a very different ranking of, of how, how ads are actually shown on a page. And as a result, it maximizes revenue for the seller you know Google made a lot more money this way. It's actually kind of shocking to believe. So goto.com was later renamed as Overture, and sold to Yahoo for about $two billion. And Yahoo, for as long as they were running their own search engine, actually used a first, basically a first price auction model and didn't factor in actual click through rate into their, to their evaluation. So a lot has been, a lot academic research has been done on Google's auction model and people have, you know, rightfully discovered that it has a few inefficiencies net. So one of the, the biggest things and, by the way, it's worth noting that Google doesn't actually use a pure generalized second price auction anymore. There's a lot more that goes into it. So this is a bit of an over simplification. One of the, the big issues with generalized second price auctions in general is what happens if a bidder values two of the same goods or two ad spots, in this case, at effectively the same value. So, what if I'm a bidder and I realize that I get about the same click through rate. If I'm in the first placement or the second placement. So in this case I got one click every 60 minutes, if I'm not at the very top and they get about the same one click every 61 minutes if I'm in the second highest position. So in this case, if, if I value a click at $five and the second highest bidder values a click at $3.00, why would I pay $five to get a click? Why wouldn't I pay2.99 $2.99 just below the second bidder? In that case I'm going to pay a heck of a lot less for something that's worth just about the same amount as me. Because as an advertiser I really care about how often people are clicking on my ads. So this creates an issue. There's a lot of, you know, research into how to actually solve this problem. But, as you just learned Victory Clarke Groves or VCG is sort of the economically right way to solve this. I'm not going to go into a lot of depth about it. It actually gets kind of complicated. And the basic idea is this. So we have two bidders and we're auctioning off, essentially, two goods. So this is why I'm not in academia. I think in terms of apples and oranges, [laugh] not equations. But, but here is a very simple example. We have an apple and orange. We're auctioning off both the apple and the orange. We have two bidders. So bidder A says, I'll pay $ten for the apple but $nine for the orange. I've valued them, you know, roughly the same. Bidder B says I'll pay $nine for the apple and $eight for the orange. So in this case, you know, bidder A could just. Like bid $seven. Get roughly equivalent value for, for the orange but here's how to solve the problem. So in Victory Clarke Groves, VCG the bidder actually pay, the winning bidder actually pays the amount of harm they cause to the second highest bidder. So in this case, if bidder A wins the apple, say bid $ten they value at $ten they effectively cause a dollar of harm to the second bidder. The second bidder now has to live with this orange. They're are willing to value the apple. They, they would value the apple at $nine. So they are effectively living with a $one less of value that they could've had. So by, by charging bidder A just $one. There's no incentive for, for bidder A to actually bid less purposely to get a worse bid, but get something that is roughly equivalent in value. So again this, this actually gets kind of complicated. Won't go into it much. In practice it's worth noting that not too many people actually use VCG. The reason being is that sellers actually make less money and it's also very difficult to explain to buyers. Especially if you're running a self-service ad platform explaining something like VCG, which is very complicated, is just too difficult. So Google makes more money by not using VCG and it's easier to explain to buyers. It's also more computationally difficult to, to implement. So fast forward to current day. So Google has obviously been very successful with that model. They, they're on track to make about $40 billion in revenue off of advertising. Facebook is basically the new AOL. They, you know, a third of all page views on the Internet are through Facebook now. But shockingly most Internet ads are actually shown in a very inefficient way. Remember that first slide I showed, where somebody picks up the phone and calls up a website? That's effectively what happens for about 70% of all the money spent on banner advertising. It's banners, you see literally somebody at agency picks up a phone and calls up an ad network or a website and says, you know I'll pay you this much to show you my ad. Gets a little bit more sophisticated, but, but that's basically the general jist of it. It's absolutely shocking that this is the case, You know, 2012, we have all this really sophisticated technology, and yet most, most online advertising is still done by somebody haggling over a phone. So App Nexus was founded, 2007, based in New York City. And, we're basically trying to solve this problem. So what we do is, we, we replace that phone with an auction mechanism. We feel that this is more efficient at, at serving the right ad to the right user. What we do is, is, we, every time a user goes to the website, that site makes a request to our servers, and it says, Aoo Nexus go run an auction and figure out which ad we should show here. Then we reach out, We make a request to thousands of different advertisers all within about ten milliseconds. And we say, you know Amazon, how much are you willing to pay based on what you know about this user? Ebay, how much are you willing to pay? We get bids back from all these different bidders. And keep in mind, they're all bidding based on what they know about the user. So if you recently bought something from eBay, eBay might say, oop, this user has a higher propensity to buy so we're going to bid more on them. So everybody returns their bids, We select the highest bidder. So, this basically replaces the phone. It's like real time programmatic haggling essentially. And you know just like Google and AdWords, we used a second price auction as well. So, very simple example. Ebay is bidding two cents, GroupOn bids a penny. Ebay would win. They get to show their ad, but they would only pay a penny. Now, in some sense, this is a much simpler problem than Google has we're not auctioning off multiple ad spots on a page. We're auctioning off essentially, one at a time. But we do this 30 billion times a day. So, we run a lot of auctions [laugh]. But there's a lot of hidden complexity to this specifically, I mean, we know some major inefficiencies with a second price auction. So, here's, here's a good example of one. So, so in banner advertising how many times and what frequency you share your ads to, to user? Makes a lot of difference for an advertiser. So if I'm Ford and I show you an ad once, you know, that ad is probably going to be somewhat effective. If I showed it to you a second time, you've already seen the ad so it's going to be a little bit less effective, especially if I show it to you one right after the other. If I show you my ad 100 times, you're probably going to get annoyed. That might actually have negative value to me. So naturally, bidders tend to bid less after they've shown a user their ad. So the first time, you know, Ford sees a user they might bid $five. The second time they'll bid $2.50. Third time might bid $1.25 and so on. They're going to keep decreasing their bid usually some exponential function over time. So I modeled this act here. Here's a very simple example of this happening. So, here we have Ford, AT&T, and Kraft. And this is the number of times they have seen the same user, over and over again. So the first time they all see a user Ford bids $five. At&T bids $three, and Kraft bids $two, And so Ford is going to win that auction. They bid the most and they are going to pay the second price which is what AT&T bid $three. And the second time everybody sees, sees that user so now Ford is bidding less. They are only bidding $2.50 because they've already shown that to the user, to the user so the user is less valuable to them. At&t now has the highest bid. At&t is still bidding $three cuz they haven't had a chance to show that ad to the user yet. And Kraft bid $two So, AT&T is going to win that auction, and it's going to pay $2.50, the second highest price, and so on and so forth. So eventually, you know, the price naturally goes down the more frequently you see a user. Because everybody's had a chance to show their ads to that user. So, in this case, you know, the average bid price is $1.29. And the seller is making about $thirteen in total. Now, what happens if Ford, you know the highest bidder in this case says you know, we're gonna try something a little bit different. For the first five times we see this user, we're not going to submit a bid. We're not going to bid at all and the sixth time we see the user, we're going to start bidding then. But we're actually going to start bidding much less. So, what, what happens? So in this case, we're still running a second price auction. But here, you know, Ford isn't bidding for the first five auctions. In the first auction, AT&T bids $three, Kraft bids $two. Kraft ends up paying $two right? Less than what, what they would have made in the previous case. And so on and so forth, until the sixth time that we see that user. This time, Ford bids, and they bid $three but because AT&T and Kraft, the other two advertisers have already shown users their ads so many times, They're now bidding at 33 cents. So Ford wins the option as they bid $three but way less than they would have paid before but they're only paying 33 cents now whereas they paid $three before paying a tenth of that now. It's kind of crazy. So now all of a sudden the average price, the average bid price goes down to 33 cents and the seller only makes $seven almost half of what they made before. Yeah. Doesn't this strategy in which they bid over and over and over again hurt Ford? And what you said, the annoying of the user if, I don't know. If you have the, the ad show up multiple times? So, so you're, so you're saying, isn't it, so, it's less valuable the second time you show an ad to a user. Yeah. Yeah. So the answer is buyers will, advertisers will typically bid to show the user an ad a second time. They'll just bid less, because, as you say, it's less valuable. Yeah. In some cases there's a lot of very sophisticated stuff that goes on, so they may not bid at all. They may say, it's just not worth it for us to show an ad the second time or we're going to show our ads to the user every other time, something like that. This is a very simple example of that. Yeah? Isn't this from, [inaudible], that you have a [inaudible]? Yes, so as your bid density increases you have this problem much less. But also depends on, so when, when you take into account what people know, what different bidders know about the user, then it becomes very different because on any given option you may have, let's say, you know Ford, AT&T, and Kraft and the 100 other bidders may be participating but Ford, AT and T, Kraft may be the only advertisers that know something about that user. So Ford may know that you've been to their website recently, you know Kraft may know you've recently bought their Oreos or something. At and T may know that you're currently a Verizon subscriber. Because they now these things they're all bidding on a separate good. They're, they're actually bidding on much more than everybody else. So all of a sudden you have a situation where you basically, you know have a much lower bid density at a much higher price point. This actually happens quite a bit. Yeah? As a bidder, how am I choosing which sites I would want my banners to go onto? Like which sites [inaudible]? Yeah, so that's an excellent question. So there's actually, so, there's a lot of thought that goes into this. Typically, you, you serve a, an optimization process. So, over time, you basically run your ads everywhere, or where you think that they're going to work. And then you look at what sites are actually working. To where you get the most clicks, or where people sign up for AT&T subscriptions. And over time you start to spend your money in a more, like more concentrated way just on those sites. It's a very simplistic way of doing it but there is, it actually gets pretty complex. How you do at opposition. Sorry, you had a question before. I don't think it would make any difference in this option, but why did you lower Ford's first bid price? Would it make any difference to [inaudible]. Yeah, well I mean just to demonstrate they could actual bid less than their fair value and still win. Why would they want to do that? Why would they want to in this case? It actual doesn't make a difference in this case but I'm just pointing out that there's no, there's nothing preventing them from not telling the truth in this case. Yeah. If you have a small number of bidders, couldn't the bidders just like, collaborate and say, you're going to bid two cents, I'm going to bid one cent, and we're going to take turns? Yeah, collision its a big problem. [laugh]. [laugh]. We have like, its the result of. Like, [inaudible] way to get around that, or do you just have to, like, have [inaudible]? I mean, you, you can, you can look for patterns. So you can actually, you know, look to see what the average price of an ad placement is, for instance. And who's bidding on it and if you see sudden drops in the price or, you know, the price tends, trends down over time. Then you can kinda figure out what's going on. It's actually very difficult to, to figure out in practice. And it does happen. There, there are all sort of other tricks that, that bidders use. Say another one is called bid stuffing. So they'll, if, so each of these buyers probably has an, a budget. An amount of money that they're willing to spend to show ads. So, in this case, if AT&T knows that Ford's budget is $100, let's say. They could bid purposely higher to get Ford to spend more money. And, to spend out their budget faster so then, all of a sudden, AT&T can start winning at a lower price. Yeah. [inaudible] another answer [inaudible]. You can have like a minimum. The separation between [inaudible], continuity. If they did know each other, they can't influence the new web site. Yeah, right, so that, that's one, one methodology for doing that. There's, I read some papers on like, random enterprise auctions, so there's some, Some theory, I, I don't think it's actually been tried very well in practice. But what, what would happen if you actually ran a third price auction or a fourth price auction? And what happens if the buyers actually don't know which level they're being price reduced down to? Then it significantly changes that dynamics, and actually this incentivizes things like collision. Yeah. In this case, does each buyer know how much the other people are bidding? in this case they don't. Yep. What about a system where once people drop out of the auction, you're out of the auction for the entire cycle? Once people drop out of it? Yeah, so like. So, so you can never bid on the user again. Yeah, not that. That, that's. You can never bid on that user, but maybe for this specific time bracket. Like, can't bid for the next five minutes or. Yeah, exactly. Hour or something. Once you drop from the auction. It's an interesting idea, interesting idea. I don't know. I mean that, that would potentially be damaging to sellers because it decreases bid density on subsequent bids. But, but it's an interesting idea, and so we're trying. So, so here's what sellers typically do to prevent against this type of practice. So, so the best defense that they have today. Is called price floors. This is the equivalent of, of using a reserve price on Ebay. So sellers generally know, you know, what their ad placements are worth to buyers. They, they know roughly. So what they do is they say, you know we're not going to allow buyers to buy our ad placement for under this dollar value. So, in this case taking the exact same, exact same example as before but simulate it with $1.25 cent price floor. So what happens? So in this case, we run the first five auctions. In the AT&T and Kraft bid we notice that on auctions four and five and the seller actually chooses not to show an ad at all. So they don't accept anybody's bid. Cuz they, they know that they're probably not getting a fair value for, for their inventory. On the sixth option, you know, Ford comes in and they're going to get price reduced, but they're actually going to get price reduced down to the value of that price floor. So seller says, you know, I'll sell, I'll sell my ad placement, but for no less than $1.25. So this case, we're not price reducing down to the second bid, we're price reducing down to the value of the price floor. So Ford actually pays $1.25. On the seventh bid, they're also going to pay $1.25. So here the, the seller actually ends up making a little bit more money than they made before. So, seller made about $seven before. Now they're making $7.25. So seller's actually increased their revenue by putting in these price floors and actually not showing bid, not showing ads on some bids. But this also gets to be a very risky game for sellers. Because what happens if you set a price floor too high or too low? So here's a, the same example, but with $1.50 price floor on the right hand side. So see what happens here, is that the seller chooses not to show an ad more often, that, you know, Ford is gonna pay more when they actually win. But the seller actually makes less money than they would have made if they had no price wars. So a thing at $seven before, now they're only making about $6.50. So, this ends up being a pretty complicated game that buyers and sellers play. This actually happens in real life, like, it's not just theory, it actually happens. So here's an example of some real data. I can't tell you which website this is, but, but, but I promise you these are two different websites here. On the top, so on the Y axis on both these charts, is the win rate. So this is how often buyers are winning as a function of how much they're bidding. So on the, the top chart here, we see that it's, you know, roughly linear. It's basically what we'd expect. If buyers bid a higher price, they'll win more often. But on the slower chart here you see some interesting behavior. So. Buyers never win if they bid less than 90 cents, meaning the se, the seller here has probably put in a price floor at 90 cents. So nobody can, nobody can win my ad placement if they pay less than 90 cents. What happens is, okay, well, after 90 cents, then everybody's bidding just around 90 cents and is winning. So in, in this case, the seller's probably put in a price floor that's actually too, too high. Because everybody's winning after 90 cents, so there's no distri, there's not a linear distribution of bids after that 90 cent price floor. So this ends up being a pretty complicated game that we dealt with everyday at App Nexus.
