Explore Now!- Looking Back In 2023

[Additional reference - Link to a related artifact published on Reforge]

I recently saw this podcast where the host and guest were discussing the failure rate of tech products. It got me thinking about my own experience at the corporate innovation lab at Amazon. In the summer of 2021, I worked on an early-stage product called Amazon Explore. I was excited to work on this product as it empowered small-businesses, but the product got shut down in late 2022. Here I reflect on that experience. I then conclude with some learnings and reflections that can potentially help us reduce this inevitably high rate of failed tech products.

Amazon Explore was a marketplace for virtual tours and online experiences. It offered a wide range of experiences priced in the range of $10-$70 with an average of ~$40 per experience. Customers had to book the virtual experiences in advance- much like scheduling an appointment. Now, for context, Amazon Explore had a very high customer satisfaction score (CSAT)- customers who purhcased the product loved it. But, it wasn't growing as expected. I joined as Product Manager (Technical) for the summer as an MBA Intern. I was given the charter to identify the right growth rate so that the team could make an informed resourcing request to leadership during their semi-annual planning process (OP-1/OP-2).

To recommend the 'right' growth rate, I first had to identify appropriate benchmarks for this new type of marketplace. This was new category spawned by the pandemic and overlapped with many markets. There were a lot of niche players who offered virtual experiences in variety of formats. Given the breadth of experiences that Explore offered at the time, I treated it as an entertainment product for spending leisure time. With this framing, I then segmented the target market by identifying three dimensions that defined the unit economics of these online experience businesses. Here's what the competitive landscape looked like back then:

I could triangulate growth-rates of the above businesses based on certain proxies. Since these were private businesses, I had to use some hacks to piece together data from different sources. Using growth-rates as a proxy for product-market fit, I could identify two comparable products to serve as our primary benchmarks: Heygo (a seed-stage startup at that time) and Airbnb Online Experiences (Airbnb OE). Once I had the estimated growth-rates, I then had to ensure that my growth-numbers were grounded in reality. For this, I calibrated my CAGR numbers with the growth rate of mature products such as Twitch, Peleton Digital, etc. Finally, I ensured that my CAGR assumptions translated into a number of users that were in sync with the market size. With this, I had a defendable range of growth-rates that showed the upside potential of Explore. For context, Airbnb OE was the fastest growing product in its history. And so, the big question facing us was 'why wasn't Explore growing remotely as quickly even though it had very high customer satisfaction scores?' My project scope was subsequently expanded to also include how Explore can plan to achieve recommended growth rates.

To understand the customers and online-creators better, I had purchased a few experiences myself for all three products (Explore, Airbnb OE, and HeyGo) and noted my observations. Experiencing the product first-hand and talking to customers helped me discover certain insights. There were a combination of factors at play- so it took some time to validate the insights with data and figure out the reasoning behind the supposedly contradictory indicators. Ultimately, the proof was in the pudding. Here's a snapshot of the three products:

To understand the importance of these factors for Explore, it’s instructive to look at the famous Amazon Flywheel and understand the feedback loops that drove the growth of its retail business:

"Lower prices led to more customer visits. More customers increased the volume of sales and attracted more commission-paying third-party seller to the site. That allowed Amazon to get more out of fixed costs like the fulfillment centers and the servers needed to run the website. This greater efficiency then enabled it to lower prices further. Feed any part of this flywheel, they reasoned, and it should accelerate the loop."(Stratechery- The Relentless Jeff Bezos)

However, these loops always starts with exceptional ‘customer experience’. In Amazon Explore’s case, the high CSAT wasn’t translating into returning traffic and that was resulting in a cold-start problem. This, in turn, was a combination of 2 underlying problems:

1. Discovery - At the time, we were acquiring customers through Amazon.com. Given the resource constraints, this was done manually and hence there wasn’t dynamic matching between users and expriences. The team was trying to address this by integrating with internal personalization tool, and this was a medium term endeavor. But even after its implementation, this route meant competing with other Amazon Products for ad placements. Importantly, users coming to Explore would not be coming with an ‘Entertainment’ frame of reference, but with ‘Shopping’ frame of reference. As opposed to this, Airbnb was surfacing online experience on its home page to its ~100M monthly visitors who were frequenting their website with an exclusive intention to plan travel and experiences.

2. User Experience Our UX had friction beacuse of mode of delivery for an ‘entertainment’ product as well as the pricing. Users had to pre-book experiences which meant they had to anticipate when they wanted to be entertained. This meant they had to look for experiences that matched their schedule and book an 30-60 min experience for ~$40 without knowing if the experience was going to be worth it. Not having a lot of reviews for our inventory of experiences also wasn’t helping. This was leading to classic chicken-end-egg problem. In this respect, Airbnb OE had managed to get a lot of reviews. Delivering experiences via Zoom and allowing upto 50 participants per experience helped increase number of reviews as well as keep average prices down to about $15-20 per experience. HeyGo, on the other hand, had arguably the best UX. It was on-demand entertainment and free for users. They could tip the creators after they had chance to experience the product, if they liked. They were also free to drop off anytime.

Looking at the business with this lens helped explain what was going on. Amazon Explore experiences were more intimate since they were 1:1. Customers felt more of that human connection in Explore explaining the high CSAT. But its UX had friction on two critical areas. This meant existing customers weren't returning often to try out new experiences. This explained why the CSAT scores were high but the product wasn't growing. To address these gaps, I proposed an idea for an on-demand product called Explore Now. The main idea was to use free and instant entertainment to attract Prime Video users on a regular basis and then convert them into paying Explore users.

Explore Now was me trying to "Think Big". I was basically proposing to create a secondary Amazon flywheel that would create network effects and complement Explore's original flywheel. Not only would it solve our cold-start problem but also potentially make the business pandemic-proof! Preliminary research did suggest this was doable with existing in-house tech, but it required further assessment. My internship at Amazon was going to get over soon. I did not have time left to validate my hypothesis with experiements and scope out the technical details in great depth. So I scoped my project and concluded with certain actionable recommendations related to solving the 'Discovery' problem and the 'UX' problem.

In hindsight, I feel we did not course correct our strategy in time. Our stated goal was to unlock both 'online experiences' and 'virtual shopping' markets for Amazon. We'll leave out 'why-that-strategy' part since that's company's internal context, but this twin-goal made it difficult to optimize the product for either of the two markets. Airbnb Online Experiences and HeyGo did not have that problem. Airbnb Online Experiences is still around and part of Airbnb Experiences. HeyGo went on to raise $20M in Feb 2022 to keep growing after the pandemic. They ultimately closed shop in April 2023 after pandemic-related effects start to wear off. However, I think Explore Now had the traffic of Prime Video to sustain itself even after the pandemic. I, therefore, sometimes wonder if Explore could have been saved by Explore Now by leveraging Amazon's own strengths. But, we would never know! Looking back, the diagnosis of Explore's growth problem felt like a case of "reality is in the eyes of the beholder". One could potentially frame it as a Product problem vs Marketing problem (or pre-PM fit problem vs Growth problem). It is important to note this framing since our choice then governs our prioritization. That, in turn, determines the kind of motions the team will go through.

After my Amazon experience, I was curious to find how one could potentially identify this framing problem while it is happening. The goal was to proactively address this problem and steer product-strategy conversations so that the team chooses the 'right' framing. To recap, I'm referring to framing problems as those where one can find data to argue for either set of choices. In such cases, I think the following 2 techniques are useful: (1) Eigenquestions, and (2) Framing Diagrams.. In the same article, the authors also talk about 'Framing with Team'. I think it complicates the 'framing problem' with an additional layer of 'decision making in groups'. In this regard, I found this video by Shreyas Doshi very helpful. He talks about how smart people can end up shipping failed products when we don't watch out for our individual biases.

On a meta-level, I think the answer to making a better choice lies in getting the team on the same page and talking the same language. Usually when discussions are going in circles, that's because people are coming at a given problem with different frames, and those frames are implicit and colored by individual biases. Having a shared language makes those frames explicit and enables labeling of potential biases. This creates space for fruitful discussions and leads to better choices.



Do let me know if you find something else or can think of other ways to reduce the failure rate of tech products! As a Product Manager, the last thing you want is to have your team build something that doesn't get used.