New arrivals: 14 Scout case studies

Internal exploration · Merchant agent

Investigating a sales day without jumping to conclusions

When a store's sales move away from their usual level, the first explanation that comes to mind is often the wrong one. For the merchant agent we are building, we wrote an investigation method: check the measurement, compare with history, follow several leads, deal with contradictions and conclude by naming what remains unknown. Today we apply it by hand on real days, and we put it to the test on a test bench that replays past days.

AIInvestigation methodAgent under construction

The merchant agent is an internal tool under construction. It has to connect an online store's advertising data, site traffic, orders and emails to explain each day to the merchant. A language model can produce a plausible explanation in a few seconds; that is precisely the risk. A useful investigation must be open to challenge, repeatable and correctable. So before asking the agent for any conclusion, we started by setting the order of checks, the levels of evidence and what the agent is not allowed to do.

Check the measurement before interpreting

The first question is not “why did sales drop?” but “are yesterday's figures complete and comparable?”. On one November day, data collection by our own traffic measurement tool stopped between the early evening of the day before and mid-morning. Without this check, the drop in morning visits would have been read as a fall in audience. With it, the drop is flagged as not comparable and taken out of the reasoning.

Other traps are less obvious. An indicator can be badly defined: we corrected a share of “one-page” sessions that had been calculated on the first recorded path rather than on the pages actually viewed. Measured against sessions whose page views could be observed, the correct figure was about 70%, not 88%. A one-page session is not a bounce anyway if the visitor interacted with the page. Likewise, unknown consent is not a refusal, and missing data is never replaced with a zero. Finally, visits from bots and from our technical tools are excluded.

Compare with the store's history

Once the measurement is validated, the agent places the day against the previous 7 and 30 days, leaving the day under study out of its own baseline. This comparison separates two questions that are often confused. First: is the day abnormal? Second: does the day reach the merchant's target? On the store we follow, the daily target had been met on only one day in thirty. A day at a little over a third of the target was therefore not necessarily an incident: it could reflect a usual level that was too low. The agent has to say so, rather than looking for a culprit at all costs.

The framework also limits unnecessary work. An in-depth investigation only starts on a new gap or on a topic left open on previous days. On days without a notable gap, the review stays short.

Follow the leads and the contradictions

To organise the leads, we follow the path of a lost sale: data collection, ad delivery, traffic quality, landing page, checkout, email customer relations, then the target itself. The agent keeps at most three hypotheses and proposes, for each one, a test that could refute it. If a campaign is suspected, it checks its clicks hour by hour and the account's change history. If one landing page accounts for a large share of the lost sessions, it may go and visit it. We set this threshold at 30% of the losses, to avoid piling up pointless visits.

Each finding gets a level of evidence: measured, cross-checked by several sources, hypothesis, or undecidable with the available data. On the November day mentioned above, the pause of a Google campaign around midday was measured: the account history dated it and the campaign's clicks fell to zero. The idea that it explained the drop in the account's clicks remained a hypothesis. Its effect on sales remained undecidable. The same rigour applies to a checkout fix made early in the afternoon: four of the five orders came afterwards, but nothing allows us to conclude that it had an effect.

The end of a lead is not the end of the investigation

We learned to separate two moments. A lead ends when its test has been done. An investigation is only finished when the day's orders have been matched to their journeys, the contacts have been put in order, the alternative explanations and contradictions have been examined, and each order is either explained or classified as a “qualified unknown”: we then know why we do not know.

The sequence we follow is therefore: collect, match, tell the story, critique, then go back to the sources to check what the critique raised. This last loop has corrected several of our own conclusions, such as the one-page session indicator or the origin of orders we thought were unexplained.

Keep a memory of previous days

A store cannot be understood one day at a time. An ad conversion can be revised several days after the sale; a paused campaign remains something to follow; a fixed fault has to be checked the next day. The agent must therefore distinguish settled facts, known context and problems that are still open.

Today, this memory is a dated log that we update by hand after each investigation: findings, corrections and questions still without an answer. We have designed a more rigorous way of working, but not built it yet. A day's facts would be frozen once final, and versioned if a definition changes. A list of open follow-ups would be reread every morning. Advertising figures that can still be revised, such as Meta's attribution over seven days, would be explicitly marked as provisional, with the date they were read.

Going back over a change or revising a conclusion would then become possible. If a campaign resumed or new conversions appeared, the agent could reopen the day concerned, compare the new reading with the old one and say what changed in its explanation. This automatic behaviour has yet to be built and then tested; for now, we are the ones who reopen these cases.

Replay past days to test the method

To test the method without the risk of misleading a merchant, we built a local test bench. It replays past days, giving the agent only what was known on the date under study. Later facts are blocked. A critical review by a model nevertheless found a leak: on two days, the agent's answer mentioned a fix made after the day under study. The bench now blocks this case automatically, and a test checks it; the two answers concerned were corrected by hand.

The bench now has about sixty tests. Its results are still partial. Of three replayed days, only one is validated as is. The other two were only validated after seven manual corrections in total, recorded in a log. The tests pass on our own machine, but running them in our isolated test environment has never succeeded. That is the real state of the method: written, applied by hand, partially tested. It is not yet run every morning by an autonomous agent.

Above all, this work taught us to make investigations verifiable. A conclusion that states its level of evidence, its refutation test and its unknowns can be discussed by the merchant. That is the quality we want to preserve when the agent takes over.

All case studies →

Something to dig into? Add this article to your office hours discussion.

See my discussion