Google recently announced Google Campaign Insights: Better measurement for display advertising. I applaud their efforts because it is very difficult to measure display ad effectiveness, but Google's approach has two major flaws:
Problem #1: Sampling Method
You can't just have two groups: those who saw the ad and those who did not because the first group can be self-selecting. They may already be predisposed to taking action and put themselves in the position of seeing the ad by visiting relevant web pages. Therefore, they are naturally more responsive. Correlation is not the same as causality.
A better method is to set a control and sample group a priori. These two groups should be otherwise equal. Then expose the sample group to the ads and see how they perform.
Problem #2: Measuring Offline Behavior
If you limit your results to online behaviors (searches, clicks), you are missing much of the value of display advertising. Consider the scenario where consumer sees a display ad, but does not click and instead goes to the store and makes the purchase. Does that really happen? Yes, it has been proven by David Reiley, Principal Research Scientist at Yahoo!, who recently presented his findings at the DMEF Research Summit.
Mr. Reiley's tests conclude that offline value of the group of non-clickers exposed to advertisements exceeds the online value of the group of clickers. Keep in mind that in his study, there only ~5% were clickers over the lifetime of the campaign or, said another way, for every clicker there were 19 non-clickers.
Reiley says "understanding advertising today is the same as understanding physics in the 1500's" (and he is expert on both). You can read an abstract of the Advertising Works Project, but I highly recommend seeing his presentation if you can. It's a very well-constructed test ... not perfect (he identifies its flaws), but better than most I've seen.
Bringing offline methods to online
In the next couple of years, I predict a resurgence of display advertising as targeting and measurement become more precise - bringing methods used in the offline world with direct mail and response attribution (i.e. "match back") to the online world. It is an interesting time!
regarding #1, scientists do observational studies with matched controls all the time. it's how they do tests for pediatric treatments this way. random test-control is ideal but it's a bit much to say that's the only way.
Posted by: sir speed | October 30, 2009 at 06:22 PM
Sir speed - that's a good point. I did not say random test-control is the only way. I said it is a better way, which I am sure you would agree. I think it's quite feasible to implement this "ideal" solution that would provide a better test.
Posted by: Joseph Pych | October 30, 2009 at 06:55 PM