Did the martech tool earn its fee? A four-line test-and-learn rule

The report from an email automation tool says: this month 280 orders were "driven by email", worth VND 50.4 million of gross profit, more than 12 times the VND 4 million fee. Do you renew? This post gives a four-line rule for answering from your own shop's numbers, then tries it on an invented example in which those 280 orders shrink to 36 incremental ones. It covers the testing step of the data-driven marketing loop.

What four-line rule decides whether a marketing tool earned its fee?

Use these four lines for email automation, retargeting or a loyalty app. I put them together myself; no book or company states exactly this procedure.

1. Write the baseline before switching on: the metric (say, the share of customers who buy within 30 days), its current value, how much it moves between periods, and the minimum uplift that pays the tool's fee.
2. Hold out a randomly chosen control group that the tool does not touch.
3. Fix the metric, duration, sample size and stopping rule in advance. Do not stop early because the numbers look good.
4. Judge on incremental orders: the group with the tool minus the control group, not the orders the tool credits to itself. Report a range, and report it in money.

Line 2 rests on what Kohavi, Tang and Xu stress: randomisation must be done carefully, and nothing may influence which variant a user lands in (Kohavi et al., 2020, p. 8). Line 3 rests on their point that an organisation needs a formal evaluation metric that can be measured easily over a short period, for example one to two weeks (Kohavi et al., 2020, p. 12); I use 30 days for a shop with a longer buying cycle. Line 1 is my addition.

Why do "orders driven by email" not prove the tool works?

Because the report counts orders that passed through the tool, including customers who would have bought anyway. Last-click attribution credits only the final touch, so it shows correlation, not cause.

Kohavi, Tang and Xu give an example from Microsoft Office 365: users who see error messages and crashes have lower churn, yet that does not mean Microsoft should show more errors. All three trace back to one factor, how much people use the product (Kohavi et al., 2020, p. 9). My reading: customers who click an email are usually already attached to the shop, like heavy users, so their orders do not show that the email caused the purchase.

Wedel and Kannan describe attribution as sharing credit among touchpoints, and say that even a complete solution is only an intermediate step; large-scale field experiments, they write, are a powerful way to answer causal questions about marketing actions (Wedel and Kannan, 2016).

What do a baseline and a control group look like in a worked example?

An invented example: every figure is mine, none belongs to a real business. An online shop selling household goods has 10,000 customers who bought in the past 12 months and agreed to receive email. The email automation tool costs VND 4 million a month. Average gross profit per order is VND 180,000. The tool's report credits email with 280 orders in the first month.

The baseline, written before switching on: last month's 30-day purchase rate was 3.5%. Break-even: 4,000,000 ÷ 180,000 ≈ 22 incremental orders a month, about 0.25 percentage points across the 9,000 customers who get email. Then draw 1,000 customers at random (10%) as the control group. After 30 days:

Three readings:

• By the tool's report: 280 orders × 180,000 = VND 50.4 million, 12.6 times the fee.
• Against last month (3.5%): up 0.9 points. The control group also rose to 4.0%, so more than half the rise came from the month itself.
• Against the control group: a 0.4-point gap, about 36 incremental orders, VND 6.48 million, 1.6 times the fee.

The last figure rests on only 1,000 customers. The 95% confidence interval for the difference runs from -0.89 to +1.69 points, that is -80 to +152 orders, or -14.4 to +27.3 million; the p-value is 0.56. The data fit the tool losing the shop money, and fit it earning a lot. Spreadsheet formulas (enter 396, 9000, 40, 1000 in B1 to B4):

B5  =B1/B2                                   email-group rate
B6  =B3/B4                                   control-group rate
B7  =B5-B6                                   difference
B8  =SQRT(B5*(1-B5)/B2+B6*(1-B6)/B4)         standard error
B9  =B7-NORM.S.INV(0.975)*B8                 lower 95% bound
B10 =B7+NORM.S.INV(0.975)*B8                 upper bound
B11 =(B1+B3)/(B2+B4)                         pooled rate
B12 =B7/SQRT(B11*(1-B11)*(1/B2+1/B4))        z statistic
B13 =2*(1-NORM.S.DIST(ABS(B12),TRUE))        two-sided p-value

I ran these formulas in LibreOffice Calc (not Excel); the results match my Python calculation. The test needs at least five successes and five failures in each group (Illowsky and Dean, 2023).

How many customers do you need, and why do small shops so often get "not known"?

Seeing a rise from 4.0% to 4.4% takes 39,473 customers per group, almost four times the shop's whole list (both groups need nearly 79,000 customers). The sample-size formula is standard statistics, not taken from Kohavi's book (I read only chapter 1): n per group = (z₀.₉₇₅ + z₀.₈₀)² × [p₁(1−p₁) + p₂(1−p₂)] ÷ (p₂ − p₁)², at a 5% two-sided significance level and 80% power. A 10% relative rise is only 0.4 percentage points (see relative and absolute risk).

With 5,000 per group the shop can only detect a rise from about 5.2%, while break-even is just 0.25 points. The 9,000/1,000 split in the example detects a 0.4-point rise only about 9% of the time. Greenland and colleagues warn that small samples drown even large effects in noise, so "not statistically significant" must not be read as "small effect" (Greenland et al., 2016). I read results by comparing the confidence interval with break-even:

The last case is common in small shops. The question becomes "what does being wrong cost": a monthly contract, a review date, no annual plan signed on the strength of the tool's own report.

Why not watch the numbers daily and stop when they look good?

Because each look is another chance to meet a random gap, and stopping the moment one appears turns noise into a "finding". I simulated two identical groups, both at 4.0%, 10,000 customers each, over 20,000 runs. Looking once at the end, 5.1% of runs showed a "significant difference", the 5% the test was designed for. Looking 10 times (every 1,000 customers) and stopping at the first apparent difference: 19.6% false positives. Looking 30 times: 28.2%.

Greenland and colleagues stress that the threshold must be set in advance and not changed in the light of the data (Greenland et al., 2016). My stopping rule: run the full 30 days and the planned sample, then read the result once. If the sample is short at day 30, record "no conclusion" instead of extending until it looks good. One exception, fixed from the start: stop early if a guardrail metric deteriorates badly, such as a spike in unsubscribes.

When should you not run a test like this?

When the scale or nature of the decision does not allow it:

• Too few people. Kohavi and colleagues recommend thousands of units (Kohavi et al., 2020, p. 10). If break-even is far smaller than what you can detect, as in the example, choose a contract you can leave rather than a test.
• A decision that is hard to reverse. They note you cannot run a controlled experiment on a merger or acquisition, because you cannot have the deal and its counterfactual at once (Kohavi et al., 2020, p. 10). Replacing a whole loyalty system on a three-year contract is close to that. Wedel and Kannan mention natural (quasi-) experiments, which often need more elaborate analysis to establish causality; their example, citing Ailawadi and colleagues (2010), a study I have not read, is the effect of Wal-Mart opening stores (Wedel and Kannan, 2016). Applying that to a marketing tool is my suggestion, and in my view it is less solid than random assignment.
• Customers who affect each other. Randomisation assumes users in one group do not influence the other (Kohavi et al., 2020, p. 10). Shared discount codes contaminate the control group.
• A promise already made. A loyalty scheme promised to every customer cannot withhold benefits from a group; in my view, test a new feature on one group instead of testing whether to have the scheme.

Going deeper: what did I read, and what did I not?

Kohavi, Tang and Xu (2020): I did not read the book. I read chapter 1 in full from the free sample on experimentguide.com; page numbers are those printed in the sample. I have not read the statistics chapters, so I do not cite the book for the sample-size formula or for checking results mid-run.

Wedel and Kannan (2016): the article is paywalled. I read the abstract and the authors' teaching slides on the American Marketing Association site, and an unedited preprint dated 13 May 2016 re-posted by a user on SlideShare, which may differ from the published version and has no printed page numbers. In it I did not find the four-step cycle that the library framework shows.

Illowsky and Dean (2023): I read section 10.3 on OpenStax. Greenland and colleagues (2016): I read the explanation of p-values and confidence intervals and the list of misinterpretations up to item 16; the article is open access. The sample sizes, intervals and simulations are my own calculations on invented data, checked with LibreOffice Calc and Python.