Experiments for Fun and Profit: 3 Experiments that Changed Companies for the Better
Upom Malik • May 19
Experiments aren't just for laboratories. When used properly, businesses can improve customer experience, cut costs, and increase profits. Here are 3 stories that illustrate the power of business experiments, and what lessons we can learn from them.
1. Amazon's Shopping Cart Recommender System
Amazon was one of the earliest companies to use experimentation in an online setting. In a 2006 blog post, former Amazon engineer Greg Linden retells an experiment he ran by putting a recommender system in Amazon's online shopping cart. This recommender system would suggest items for a customer to purchase based on items they already had in their shopping cart. A senior executive at Amazon was strongly against this idea. Despite this objection, Linden went on to run an experiment to test the impact of the recommender system, an idea that was incredibly hard to object to in Amazon's culture. As it turns out, not running Linden's innovation was costing Amazon substantial amounts of revenue, and shopping cart recommender systems have been a mainstay of e-commerce ever since.
Lesson: Experimentation helps avoid the mistake of only listening to the Highest Paid Person's Opinion (HiPPO), and democratizing innovation across the company.
2. Bing and Ad Titles
As recollected in the book Trustworthy Online Controlled Experiments, engineers at Bing tested different ideas to try to increase revenue from ads on their site. Among the ideas proposed were to increase the title length of advertisements by moving some of the copy from the first line below the title into the title itself, as shown in the image below:
As you could imagine, Bing found it unlikely that this change would have any impact, and prioritized it accordingly. However when they tested this change, they saw an over 10% increase in their revenue, translating into over 100 million dollars for Bing.
Lesson: Most companies are actually not great at predicting which ideas will succeed and which ideas will fail. Testing ideas quickly and easily will help you find those changes that will make a difference to your bottom line.
3. Adda's Table 101
Adda is an Indian restaurant in New York City that serves delicious Indian fare like masala-fried chicken and shrimp curry. However what's lesser known is that Adda was the scene of a really great business experiment. For a restaurant, one of the most important metrics is the average revenue earned per minute by each table. By this metric, Adda's lowest earning area was a long window-side table known as table 101. This table earned substantially less than any other table in the restaurant. With the help of a restaurant consultant, Adda tested a non-intuitive idea: they added a small wall to the side of the table to convert their window-side seat into an alcove. While there were some tradeoffs (the table now seated one less person, and they had to pay for the cost of constructing the mini-wall), Adda was able to improve the revenue per minute of table 101 by 39% - an extra $18,000 per year for Adda's managers.
If you want to learn more about Adda and their table experiment, check out this great podcast from the folks at Sporkful.
Lesson: Experiments aren't just for online businesses. All sorts of companies, including restaurants, can benefit from running experiments.
Get Started Experimenting
These three examples go to show how experimentation can help your company become more successful. However, you may be unsure of how to get started with experimentation, or maybe have had some trouble with your own business experimentation efforts. If so, sign up for our mailing list to get started experimenting today. You can also reach out to me on Linkedin or Twitter. I'm excited to hear about all the experiments you want to run.