Doubling new user activation with Heap's first LLM offering, interactive sign up, improved onboarding

Heap Analytics is the leader in self-serve product analytics. Most product analytics tools require manually tracking every click, tap, and pageview in code. With Heap, product managers and analysts can manage their own dataset without needing to pull additional resources from engineering.

When I joined, a completely self-serve setup process was possible, but it was technical and challenging. A self-serve purchase process was not possible at all.

During my time at Heap, I was responsible for building our Growth product area from the ground up. I (alongside one of our product directors) defined the strategy that grew Heap's self-serve revenue from $0 to >$1m ARR and nearly doubled our new user activation rate.

I designed the overall vision of what our onboarding and new user experience could look like, and partnered with our product and engineering teams to implement features and systems that moved us toward that vision.

As I (and the team) grew, I moved from my role as a Senior Product Designer into a Product Design Manager role overseeing multiple designers in both our Growth and Data Ecosystem product areas.

Leading analytics into the future with natural language analysis

LLMs are definitely having a moment right now, and Heap is no exception. However, we didn't want to spend a lot of time and energy unless it was something that could provide real value to our customers. To do that, this project started as a small working group focused on testing both the technology itself as well as the outcomes that could be generated with our users' datasets. The design portion was led by one of the ICs on my team, but we worked closely together at all levels of fidelity.

Discovering where there is opportunity

Even moreso than a typical design project, developing functionality with a new technology means starting from the very beginning. The team was already actively using machine learning for a number of features, and it seemed like LLMs could benefit Heap in some of the same ways.

We identified three potential areas of opportunity:

1. Data summaries: Starting with the raw data we were already capturing, the LLM could complete tasks such as summarizing the data, starting to interpret it, and presenting relevant segments
2. Natural language analysis: Rather than requiring users to navigate a sometimes complex set of steps to generate a chart or graph, they could type the question they have, and the LLM would generate the chart
3. Insight generation: Lots of customers already had 1000s of charts though. Instead of starting from scratch, the LLM could take a chart as an input and turn that into insights about key takeaways, inflection points, and other anomalies in the data

Although initially leaning toward option 1 (summarizing), we discovered that option 2 (natural language analysis) provided the most value for Heap and its customers by getting clearer on our target audience. Non-technical users joining existing accounts are often faced with an overwhelming amount of analysis tools and existing data. Allowing them to get started with a much simplier, chat-based interface could greatly decrease these barriers to success.

Training the model

With our audience in mind, we started to reach out to them to understand the exact questions they were likely to ask. This was possible through a combination of surveying and looking at existing charts and graphs.

Core experience

Although we explored experiences that would sit alongside or being integrated with our manual query creation process, it quickly became clear this would create an even more overwhelming experience than the one we were already trying to solve.

This led to a chat experience that was simplified and independent from the typical chart generation process. The initial empty states, especially, were able to be reduced to only a text input and a few other elements.

Relying on progressive disclosure as they interacted with the tool, we could slowly build out the chart they were looking for as well as introduce them to existing charts.

An imperfect science

With a limited training set, our LLM was bound to make mistakes. This definitely happened, but we were also pleasantly surprised at how often is was right. User research showed that users often put extreme trust in it though, even when the data was unreliable or innaccurate.

This led to us to increase the likelihood of it returning 'unable to generate chart' messages to avoid false positives. To get around that though, we plan to add followup questions that allow the user more direct interaction with their existing data set. This combines the best of both worlds. Easy-to-use, natural language chart generation with powerful data-driven fallbacks.

Entrypoints & education

The independent experience provided more advantages than drawbacks, but it did mean we had to dive deeper into how users would discover the functionality. This work is still ongoing, but we have a few very promising directions.

By including LLM processing with our existing search, it uses a surface that is already regularly use and introduces new functionality without any new interactions from users. Similarly, a simple callout on the typical chart generation page allows an easy shortcut to the natural language flow.