Evaluating change before, during and after

Mr Tuấn runs a 25-person packaging printer in Long An and changed how the firm takes orders three months ago. He asks his operations manager whether fewer orders now go wrong. "It feels like it," she says. Nobody recorded how many went wrong before, so nobody can prove it, and he cannot tell whether to roll the new routine out to a second line or drop it. The firm and every figure in this post are my invention. Three-phase evaluation of change answers what to measure and when; Scale–modify–stop gate for AI adoption adds a decision gate at the end.

When should you evaluate a change?

Three times: before it starts, while it runs, and once it is finished. The abstract of Park's (2017) chapter names the phases pre-intervention, intra-intervention and post-intervention, and says evaluation can occur several times through the process (Park, 2017). Rist et al. (2011) say much the same in the language of development programmes: baseline, midterm and final evaluation, plus impact evaluation (Rist et al., 2011, p. 173).

Monitoring and evaluation are not the same job. Monitoring reports progress at regular intervals; evaluation, in the OECD/DAC definition Rist et al. quote, is a systematic and objective assessment of a project that is ongoing or completed, covering its design, implementation and results (Rist et al., 2011, p. 172). My reading: the weekly figure is monitoring, and the sit-down at the halfway point to ask whether the design is still right is the evaluation.

What is a baseline, and why is it so often missing?

A baseline is the measurement taken before the change, with the same definition and the same method as the later measurements. Without it, every comparison afterwards rests on memory. It is often missing because measurement arrives late. Rist et al. describe programmes where monitoring and evaluation were not part of the original design but were added on later, creating gaps such as "the lack of a systematic baseline at the start" (Rist et al., 2011, p. 174).

Sri Lanka gives a concrete case. Three years into a government results-based initiative, the chapter's author recorded that many ministries had no baseline data and struggled to identify measurable indicators, because nobody collected data systematically against their targets (Rist et al., 2011, p. 19). That is the public sector, and I use it by analogy. The four rules below are mine:

1. Define each measure with a numerator and a denominator, such as "wrongly made orders per 100 orders".
2. Measure for at least one ordinary working cycle.
3. Take the figures from records that already exist (messages, job tickets, invoices) before asking anyone to write something extra down.
4. Write the decision thresholds before you look at the "after" figures, or the thresholds drift towards the result.

What should a manager's measurement board contain?

Five rows per measure are enough for a change of a small firm's size. I assembled the board from the framework's phases.

On the kind of measure, Rist et al. reproduce four levels of training evaluation from the World Bank (2007), from participant satisfaction up to impact on the organisation; the top level is compared with baseline measures of organisational performance (Rist et al., 2011, p. 182). That table is about training, not process change. What carries over is not to stop at "everyone uses it": keep at least one outcome measure compared with the baseline.

A 25-person printer: what do the before, midterm and after columns look like?

Suppose Mr Tuấn's firm had measured from the start. The invented change: customers send orders by Zalo, phone and email; salespeople write them in a notebook and ask the coordinator to retype them. The new way is one shared order form with five mandatory fields (size, quantity, material, colour, delivery date), and the coordinator checks that all five are filled before approving the job ticket.

Invented figures, not real data.

The midterm looks worse than the baseline: 26 hours instead of 20, and 30 hours of entry instead of 18. Suppose the cause is that salespeople keep the notebook and also fill the form, while the coordinator checks every form by hand. This is the performance dip in transition: the library notes that performance usually falls before it improves. The midterm fix: drop the notebook and check only the five mandatory fields. If staff keep the notebook anyway, ask why before you insist; the post on four theories of resistance to change has a diagnostic table for exactly this.

At the end, four measures meet their thresholds and order-entry time, at 22, misses 20. At about 400 orders a month, wrongly made orders fall from 28 to 12; if each costs VND 1.5 million (invented), the printer saves about VND 24 million a month.

When should you scale, modify or stop?

When you set the final board beside thresholds written in advance. The library has a scale–modify–stop gate: milestone M1 includes the baseline, M4 measures cycle time alongside quality, and M5 decides to scale, modify or stop each pilot. The framework was written for AI pilots and gives no numerical thresholds, so what follows is my adaptation to a non-AI example.

The example rule, set by me:

• Scale: every measure meets its threshold.
• Modify: the guard measure, here wrongly made orders, is no worse than baseline, and any measure that misses has moved the right way since midterm.
• Stop: wrongly made orders exceed 7 per 100 in any period, or fewer than 50 per cent of orders use the new form after the midterm fix.

On the board above the result is modify: entry time fell from 30 to 22 hours but has not reached 20. The action is to pre-fill details for regular customers, cut two optional fields and measure again for four weeks. If it passes, extend to the second line and update procedures, which is milestone M6.

The gate suits a change you can trial on a small part of the business, such as one line or one shift, and reverse. It does not suit a change you can only make once for the whole firm, such as switching accounting software at the start of the financial year: there is nothing to stop halfway. For AI pilots the library also records the ROI measurement gap: many organisations use AI without showing the benefit through metrics that have a baseline.

Is an "after" figure better than the "before" figure enough to say the change worked?

No. Park's abstract says the evaluation instrument and the evaluation process can potentially affect internal validity (Park, 2017). It does not mention season or group choice; the two risks below are my own addition for the printer.

Season: suppose the printer measured the baseline in a quiet month and the "after" in the year-end peak, and processing time was still 14 hours; the result is more believable for it. Reverse the order and 14 hours may be the season talking, so compare with the same period last year if the records exist. Choice of pilot group: if the trial line is staffed by volunteers, the result leans towards people already willing, which is selection bias. Draw lots for the trial line.

Going deeper: where does this come from, and what did I read?

The library's gate framework lists four other sources on organisational change. I have not read them to check the decision rule, so the gate section rests on the library framework page alone.

For Rist et al., I downloaded the World Bank's open PDF (302 PDF pages) and read the relevant passages of chapter 10 (printed pages 171 to 194), page 9 and page 19; in the other chapters I only searched for the word "baseline". The library treats the whole volume as one source although each chapter has its own authors, so the post cites it that way. For Park, I read only the chapter's abstract on the publisher's page, not the full text.

Limits: the examples in the sources come from the public sector and include no small firms. The sources describe what evaluation covers; they give no evidence that setting a baseline or using a decision gate improves results, and I have not read a study that tests it. In my reading, a three-phase frame suits a change with a clear start and finish. For a change that creeps in, there is no day called "before", and the board has to accept a reconstructed baseline.