A data-driven decision loop for small businesses

The owner of a five-shop tea chain handed out loyalty cards six months ago and must decide next week whether to print more. Receipts are up, but plenty of other things rose at the same time. This post walks through one decision loop, from question to adjustment, on exactly that case. The general frame is the library's Data-driven decision cycle with the Theory of change and logic model. For what data can and cannot answer, see the post that reads two papers on Netflix.

What steps does the data-driven decision loop have?

Four, repeated: formulate key questions, collect and analyse data, communicate results to decision-makers, refine and start again (James Bell Associates, 2018, p. 2).

The source guide is for child welfare agencies, and says so (James Bell Associates, 2018, p. 1). It states its aim bluntly: decisions based on data, experimentation and evidence rather than opinion or intuition (James Bell Associates, 2018, p. 4). Everything about the tea chain below is my adaptation. The guide does not discuss loyalty cards or retail.

Step 1: how should the key question be written?

As one sentence tied to a decision with a deadline. The guide suggests four kinds of question: quantity, distribution, duration and variance (James Bell Associates, 2018, p. 6). The chain's sentence: "After six months, has the card pushed weekly receipts above the existing trend by enough to cover the free drinks? Decision: print more cards or change the reward, before month end."

What does a logic model for the loyalty card look like?

A theory of change says why we believe activities lead to the goal; a logic model is the visual tool that splits it into inputs, activities, outputs and outcomes, each output and outcome paired with an indicator (James Bell Associates, 2018, pp. 6, 10).

The guide advises listing the assumptions of the theory of change that need further evidence; its example is the assumption that the target group will enrol and attend, checkable from enrolment and attendance counts (James Bell Associates, 2018, pp. 7–8). The chain has three: customers agree to a card when asked; card holders visit more because of the card, not because they were regulars already; and the profit from extra purchases exceeds the cost of the free drink.

What data do we already have, and what is missing?

The guide asks for data that are accurate, complete, timely and actionable (James Bell Associates, 2018, p. 18). For the chain:

Analysis: how should the numbers be read?

The guide says descriptive analysis, meaning counts, percentages and averages, often supplies much of the information managers need (James Bell Associates, 2018, p. 26). The data here are generated by me in Python with a fixed seed: 52 weeks, 26 before and 26 after the card launch, 5 shops. I built in a growth trend of 0.3 per cent a week and a card effect of about 3 per cent once fully running, so the results illustrate how to read numbers and prove nothing about loyalty cards.

Most of the +10.5 per cent is the pre-existing trend. The range of 25 to 198 is a 95 per cent prediction interval from a regression on 26 weeks: the estimate ± 2.064 × the result of the fourth formula line (2.064 is TINV(0.05,24), 24 degrees of freedom). It assumes the linear trend continues; I have not modelled seasonality. The formulas (column A is the week number, B receipts, H days to the second scan; I ran them in LibreOffice Calc, not in Excel):

=SLOPE(B2:B27, A2:A27)
=INTERCEPT(B2:B27, A2:A27)
=AVERAGE(B28:B53) - (AVERAGE(A28:A53)*SLOPE(B2:B27,A2:A27) + INTERCEPT(B2:B27,A2:A27))
=STEYX(B2:B27,A2:A27)*SQRT(1/26+1/26+(AVERAGE(A28:A53)-AVERAGE(A2:A27))^2/DEVSQ(A2:A27))
=TINV(0.05,24)
=COUNTIF(H2:H651,"<=30")/ROWS(H2:H651)

The last line gives 67 per cent of cards with a second scan within 30 days. That does not show the card brings customers back, because there is nothing to compare it with among customers without a card; see selection bias.

On money, I assume a receipt of VND 48,000, a gross margin of 55 per cent before wages and rent, and a free drink costing VND 15,000 for every 10 scanned receipts. Reward cost is then about VND 1.89 million a week (an average of 1,261 scanned receipts a week). Gross profit from the extra receipts minus that cost leaves roughly +VND 1.05 million a week at the central estimate, and −1.23 to +3.33 million at the two ends of the range. Card printing and cashier time are not counted.

How should the one-page report for the owner read?

The guide says a 100-page report may not help legislators or department heads, while a one-page summary may be just right (James Bell Associates, 2018, p. 36). It keeps findings apart from recommendations (James Bell Associates, 2018, p. 38) and wants recommendations to offer options, with the risks and benefits of acting and of not acting (James Bell Associates, 2018, p. 39). A draft in my adaptation:

Loyalty card after 26 weeks. Decision needed: print more or change the reward, before month end.

Findings. Receipts are up 10.5 per cent on the previous 26 weeks, but the chain was already growing about 0.3 per cent a week; against that trend the card may account for about 2.6 per cent (0.6 to 4.7). Free drinks cost about VND 1.9 million a week; what remains is an estimated +1 million a week, from −1.2 to +3.3. Shop 4 scans only 12 per cent of receipts, the others 30 to 45 per cent.

Not known. Whether card holders return more because of the card or because they were regulars; seasonality.

Options. A: print more, no change; profit may be negative if the effect is at the low end. B: keep the card, fix how cards are offered at shop 4, trial a different reward in two shops for eight weeks. C: stop; lose the return-visit data.

Recommendation: B.

What should be adjusted, and how does the loop restart?

The guide says the process is not complete until the information leads to action (James Bell Associates, 2018, p. 44). For option B: retrain card offers at shop 4; ask why shop 1 scans 45 per cent, because the guide advises looking at what works, not only at weaknesses (James Bell Associates, 2018, p. 45); trial the new reward in shops 2 and 5, with the other three as comparison.

Then return to the logic model, which the guide treats as a dynamic tool, to be revisited to see whether the refinements require changes to it (James Bell Associates, 2018, p. 46). Add an indicator: visits per customer per month. The next question: does the new reward bring customers back faster?

When does this not fit, and where does the evidence stop?

It does not fit when the question is not tied to any decision, or when the sales records cannot be trusted; then fix the recording first. That is my conclusion, not something the guide says.

The real limit is the source. The guide was written for child welfare, where decisions affect children and families, and it reminds readers of that (James Bell Associates, 2018, p. 46); the data team it describes (analyst, systems specialist, evaluator) is far heavier than any shop chain needs (James Bell Associates, 2018, p. 19). The example here is hypothetical, one chain, one linear-trend model. Carrying it over to business is my inference.

Going deeper: what did I read, and what did I not?

James Bell Associates (2018): I read the main text of chapters 1 to 5 with the case vignettes, and appendices A and B; the text conversion cut some tables and exhibits and I do not rely on those parts. Appendix C, the sample data-sharing agreements, I read only at the start. Page numbers are the guide's printed pages. It is a guide commissioned by the US Children's Bureau, not an empirical study, and it gives me no evidence about small businesses. I have not read the works it cites, such as Funnell and Rogers (2011).

The example data and formulas are my own creation, from no source; I checked the formulas in LibreOffice Calc and against a Python calculation.