Chapter 06
Product outcomes
The visitor opens Flamingo's Envy from the trip planner and saves it. The recommendation fits four people and has the kitchen the visitor asked about.
But Twig's engineers also want to understand what happens after recommendations: do visitors open the stays, save them, or book anything?
The conversation alone won't tell them. They need to connect the answer to what the visitor does in Twig.
Follow the recommendation into the product
When the visitor opens Flamingo's Envy from the chat, Twig records the action with the stay's identifier and the recommendation's trace, generation, and session identifiers.
Those details let Twig's engineers distinguish opening this recommendation from finding the same listing through ordinary browsing.
Open the recommended stay, then save it and inspect the recorded actions:
Twig's engineers choose an attribution rule: the conditions under which they connect a later action to a recommendation. A save counts toward this recommendation when the visitor saves Flamingo's Envy within 30 minutes of opening it from that answer.
They record separate product events for opening a recommendation, viewing a stay, saving it, and completing a booking. These are events Twig's engineers define, not built-in AIO events. The separate events let them see how far the visitor gets.
Measure how often recommendations lead to saves
Twig's engineers build a funnel, which measures how often people complete a sequence of actions.
Their first funnel starts with opening a recommendation and ends with saving the recommended stay. For the beach conversation, the engineers can inspect the opening and save records to confirm that both refer to Flamingo's Envy and the same recommendation.
In this example, 120 visitors open a recommendation and 48 save the stay within 30 minutes: a 40% conversion rate. Select a step to inspect one matching event:
The engineers use the funnel to find where visitors stop. Someone who opens Flamingo's Envy but doesn't save it has taken a different path from someone who saves it and later completes a booking.
That doesn't explain why either visitor acted that way. And it doesn't prove the trip planner caused a booking.
To test whether offering the trip planner increases bookings, Twig's engineers need a controlled experiment: compare outcomes for visitors assigned to different product variants. Comparing people who chose to use the trip planner with people who didn't could reflect differences that already existed between those groups.
The support agent needs the same distinction between an answer and an outcome. The policy evaluation can establish that the heating reply offers a refund. It can't establish that the visitor actually received one. Twig's engineers need the refund record to answer that question.
Review an answer that passes but still isn't useful
Twig's engineers return to the city request for four and add its corrected answer to a review queue, a collection of AI responses set aside for human review. The answer avoids recommending an undersized stay, but Twig's engineers still need to judge whether it helps the visitor find a trip for four.
One of Twig's engineers reads the visitor's request, the empty search result, and the answer together. That engineer records two separate judgments:
- The answer avoids recommending an undersized stay.
- The answer doesn't ask a useful follow-up question.
The engineers define these questions as review scorers, reusable fields for recording a person's judgment. These aren't automatic evaluations; an engineer reads the trace and supplies the scores.
Here's how one of Twig's engineers reviews the answer:
Saving the review removes the trace from the pending queue. PostHog keeps the scorer version used for that review, so changing a review question later doesn't rewrite what the engineer originally assessed.
The improvement is specific: explain that no city stay fits four and ask whether the visitor wants a forest or beach stay instead. Twig's engineers save that expected behavior with the city request before testing a prompt change.
Decide what to fix next
Twig's engineers need to know which problems affect the most visitors – and which cause the most harm.
They start with the undersized recommendation: how many other groups received Le Nid Chic when they needed room for more than two? The engineers filter for those requests and inspect the search inputs to confirm the guest filter was missing.
For broader investigations, PostHog offers:
- Clusters, which group similar traces or generations. Twig's engineers inspect a cluster of city requests that found no matching stay to see how often the planner leaves visitors without a useful alternative.
- Scouts, which run scheduled AI investigations. The engineers ask a scout to find those no-match answers with negative feedback and return example traces for review.
A large cluster isn't necessarily a problem: successful beach recommendations can be common too. Twig's engineers prioritize by frequency, impact on visitors, and effort to fix. An incorrect refund refusal is more serious than an awkward follow-up.
Find the next investigation in your project
Copy this prompt into PostHog AI: