Question

CASE: Advertising Experiments at RestaurantGrades Shirley A. Critic is the founder of RestaurantGrades (RG), a restaurant review platform (similar to Yelp or TripAdvisor) with an impressive stock of online reviews written by ordinary restaurant-goers.Shirley is happy with the company’s growth and position in the industry, and has compelling evidence that RG reviews have an important influence on the restaurant choices people make. However, doubts have been raised about the efficacy of the company’s main source of revenue—selling ads to restaurants. To better understand this issue, Shirley decided to run a large-scale randomized controlled trial with a control group and two treatment groups: one treatment to test the impact of its current ads on restaurant sales, and the other treatment to test the impact of an alternative ad design that she is considering switching to. Her team has run the experiment and found the results in the attached spreadsheet. Being a data nerd (in the best possible sense), and knowing that the devil is often in the details, Shirley wants to look at the results herself. She wants to understand whether her company’sads really work, and whether they should stick with the current design or switch to the alternative design. Background on RG Advertisements On RG, each restaurant has a profile page with operating information including its hours, phone number, and location, where RG users who have visited the restaurant can leave reviews for other users. Users can also discover and search for restaurants on the platform using filters, and can make reservations and order food through a restaurant’s profile page. The majority of RG’s revenues stems from selling ads through its sales team, who cold-calls restaurants to try to convince them to advertise on the platform. Advertisements, labeled as sponsored search results, are placed in a separate section above the organic results for searches that users conduct. Packages of ads are purchased for about $300 per month, and advertisers are required to sign up for a one-year contract. While RG uses a search algorithm much like Google’s that determines when and which ads are shown given a user’s search for restaurants on the platform, restaurants have little say in what search terms will trigger their ads. However, they are guaranteed that their ads will be shown a minimum of 1,000 times per month to users. RG’s current search algorithm shows ads for restaurants triggered by type of cuisine within a 0.5-mile radius of a user’s search. For example, if a user searches for Italian restaurants in Harvard Square, the algorithm will choose two Italian restaurants within Harvard Square to advertise. The engineering team has run a variety of tests looking at how users respond to different types of ads in different searches, and has designed an alternative search algorithm. Rather than choosing two restaurants by cuisine, the alternative algorithm shows ads when a user searches for a specific restaurant and chooses two restaurants with similar ratings and hours. While Shirley’s team is reasonably satisfied with how the current algorithm shows ads, they are open to the possibility that the alternative design may be significantly better (or significantly worse) in providing benefit for their advertisers. Experiment For the experiment, Shirley’s team randomly selected 30,000 restaurants that were active on itsplatform but were not currently advertising, yielding a sample that is representative of its population of restaurants in the US.However, for the one-month duration of the experiment, 10,000 restaurants were randomly chosen to receive free ads using the current advertising approach, and another 10,000 restaurants were randomly selected to receive the alternatively designed ads. The main difference between these two treatment groups was that the alternative design used a very different algorithm to decide when to deliver ads and which ads to pair with each search, as described above. The remaining 10,000 restaurants received no advertisements. None of the restaurants were informed about the experiment or the advertisements. Spreadsheet The spreadsheet supplement for this exercise (HBS No. 916-702) contains the outcomes that Shirley’s team observed for the 30,000 restaurants during the one-month experiment. The unit of observation is a restaurant-month, so the data for each restaurant are in a single row. For example, in a row, pageviews refers to the number of unique visits to a restaurant’s RG page during that month. The outcome variables included are as follows: Variable Name Definition business_id The restaurant’s unique identifier treatment =0: in the control group = 1: in the first treatment group (ads using current algorithm) = 2: in the second treatment group (ads using alternative algorithm) restaurant type =chain: chain restaurants = independent: independent restaurants pageviews # of visits to the restaurant’s RG page per month calls # of phone calls made from the restaurant’s mobile RG page per month reservations # of reservations made from the restaurant’s RG page per month

QUESTIONS 1. Given that there is a set of restaurants on the platform that have bought ads, and another onethat have not bought ads, why is an A/B testing required. Can we not compare the outcomes for restaurants that do, and those that do not advertiseto find the effectiveness of advertising? 2. Which outcome variable is most useful to measure advertising effectiveness? How will you make this choice? Please explain your answer

Answer 1

Yes, we may find out the effectiveness of advertising by comparing the outcomes of the restaurants the do show ads on and those who dont. But this comparision will not be very accurate and approximate. Because sometimes some restaurents that doesnt show ads may recive more polularity and more users may have accessed them and at the some time some restaurents that show ads may not have got that polularity on the same website at the same time. So by just comparing both of them together it wont be sufficient to make a conclusion. Thats why we require a a/b testing approach.

The more effective outcome variable will be the algorithm to target correct audiences to correct advertisements. For this there must a strong and unique algorithm to study the behaviour of the user based on his activities and then showing proper add with a probabity of 90% and above that the user will see to it and like it. Also one algorithm is not sufficient for this we will require more than two algorithms and more than two approaches which will be experimented on selective amount of different audiences and by studying the result the approach should be decided on which approch to use based on which experiment has given a satisfied outcome.