New arrivals: 14 Scout case studies

Internal exploration · Merchant agent

Explaining a weak sales day by ranking the possible causes

A weak sales day quickly sets off a hunt for a culprit: a campaign, a page, a bug. For the merchant agent we are building, we investigated semi-manually, with a language model, a November day that ended well below target. We compared the drop with history, examined acquisition and traffic quality, followed the friction from the catalogue to checkout, then ranked the explanations by their level of evidence. The most important conclusion was not the one we expected.

AIDiagnosisE-commerce

That day, the online store of a brand we work with recorded five orders, or about 37% of its daily sales target. Several things had happened on the same day: an interruption in our own traffic collection, a Google campaign being paused, a fault on the order button, a Meta campaign bringing in a lot of clicks. Any of them could serve as the story. Our aim was to find out which ones really mattered, and to say so with the right degree of certainty.

First, is the drop really a drop?

Before looking for a cause, we compared the day with the previous seven and thirty days, without including it in its own baseline. Sales were about 19% below the average of the previous seven days, roughly one order. Over thirty days, the day ranked in the middle of the distribution, in eighteenth place, within the range of usual values.

The distance from the target, on the other hand, was much larger. It did not come from that day: the target had been met on only one of the previous thirty days, when the Black Friday promotions were launched by email. We drew a distinction from this that the agent will always have to make: the gap from normal and the gap from the target are two different questions. That day, the gap from normal was modest; the gap from the target was structural. No single incident that day could explain why a target that is rarely met was not met.

Check the measurement before reading the traffic

Our traffic measurement tool had stopped collecting visits from the early evening of the day before until mid-morning. Only part of the events were recovered. Morning traffic was therefore unknown, and all site measurements for the day were partial. Sales, recorded by the store itself, were not affected.

We also sorted the visits. Out of more than 1,600 sessions, about 620 were probably human; the rest came from bots or automation, or remained unresolved. All site measurements were calculated on human visits only. Finally, a key indicator turned out to be wrong on review: the share of sessions limited to one page was not 88% but about 70% of observable sessions. The first figure counted the first recorded path, not the pages actually viewed.

Acquisition and traffic quality

On the Meta side, the ads generated a lot of clicks, with a click-through rate above the average of the previous seven days. About 83% of spend went to campaigns optimised for purchases. Yet half of the clicks came from a campaign optimised for page views, which accounted for only about 17% of spend. Among the human visits observed after collection resumed, this campaign brought 117 sessions and not a single add-to-cart. Over the same period, Google brought 138 sessions and 8 carts, and email 63 sessions and 6 carts. More than 90% of the page-view campaign's sessions stopped after one page. Without it, the share of one-page sessions fell to about 65%.

On the Google side, an acquisition campaign was paused manually around midday from the account interface. Its clicks fell to zero in the afternoon, and the account's clicks were down about 40% on the average of the previous seven days. The number of conversions reported by Google, however, had not fallen. The reason for the pause was not established.

From the catalogue to checkout

We then followed the journey of human visits. Twenty-six sessions added a product to the cart, five started checkout. A one-page session is not necessarily a bounce: the vast majority of these sessions included an interaction with the page. The next day, our tests of the landing pages reproduced no general blocking issue. They found possible friction points on mobile: a first screen with no product on collection pages, a missing title, and a chat bubble overlapping the purchase bar. These are correlations, not proven causes.

At checkout, however, a real fault was identified. Until early afternoon, the order button waited for a check related to gift options, even for a customer who had not chosen any. The fault was fixed early in the afternoon; four of the five orders were placed afterwards. The number of customers held up before the fix remains unknown.

Ranking the explanations

We ranked the explanations while taking a position, in line with the rule we set for the agent:

The usual level of sales explains most of the gap from the target. The fact behind this judgement is measured: the target is met on only one day in thirty.

The Meta campaign optimised for page views brought half of the Meta clicks and 117 observed visits, without a single cart. This is measured on observable visits. Its effect on sales remains a hypothesis.

The order button fault may have cost sales before it was fixed early in the afternoon. It is established, but its effect is undecidable.

The pause of the Google campaign probably explains the drop in the account's clicks. This is a hypothesis, and its effect on sales remains undecidable.

The friction on mobile pages remains a set of leads to test.

We do not split the gap between these causes. Such a split would look precise, but it would be made up. What we can say is “probably an insufficient usual level, made worse by poorly qualified Meta traffic and a checkout that was held up until early afternoon”, along with what would make us change our minds: a Meta campaign that starts producing carts, or sales that stay low after the checkout fix.

What the investigation leaves open

Several questions were explicitly left open. Morning traffic could not be measured. Sales from email campaigns cannot be dated to the exact day; we left them empty rather than counting them as zero. Purchases attributed by Meta remained provisional for seven days. No margin data was available to judge whether a campaign was profitable.

This investigation was carried out semi-manually: a language model did the reading and the analysis under our direction, and critical reviews corrected several figures. It serves as a reference for the agent, which does not yet run any investigation on its own. Decisions, such as reactivating a campaign or cutting a budget, remain with the merchant and their team.

All case studies →

Something to dig into? Add this article to your office hours discussion.

See my discussion