Weighted scoring with a sensitivity check: pick a market, then change the weights

The scorecard gives market B 60 points and market C 57.5. Would you put money on B? If raising one weight by 2 points lifts C past it, a gap of 2.5 points tells you little. This post covers how weighted scoring with a sensitivity check works, an invented example worked number by number, and a public case from Britain in 1988 in which the weights changed the winner. It follows the post on choosing the next market: the CAGE distance framework screens candidates, a scorecard ranks them, and the sensitivity check says whether the ranking deserves trust.

How does weighted scoring work, and when is the addition valid?

Score each option on each criterion, multiply by the criterion's weight, add up, and the highest total wins. The addition is valid only when the criteria are preference independent: the score you give an option on one criterion must not change with its score on another (Dodgson et al., 2009, pp. 25, 65). I assembled these steps from the UK government manual (Dodgson et al., 2009):

1. List the options and criteria. Drop any dominated option first: one at least as good as another on every criterion and better on at least one (Dodgson et al., 2009, p. 24).
2. Score on a relative scale: best on a criterion gets 100, worst gets 0, the rest sit between in proportion to the spread (Dodgson et al., 2009, p. 60).
3. Set the weights and compute each option's weighted total.
4. Change scores and weights and see whether the order changes (Dodgson et al., 2009, p. 31).

Two traps. Two criteria that measure the same thing count it twice (Dodgson et al., 2009, p. 37). And addition lets a high score on one criterion make up for a low score on another; the manual says that if trade-offs are not acceptable, non-compensatory techniques are needed instead (Dodgson et al., 2009, p. 24). My own use of that: if a market must meet a legal requirement, use a gate first and score only what passes.

What does a weight of 35 mean?

Not "this criterion is 35 per cent important". In the Nirex case below, a weight of 100 on costs and 10 on safety meant that the difference in costs between sites was judged ten times as important as the difference in safety, not that cost mattered ten times more than safety; the sites had already been screened with safety as a main rejection criterion (Dodgson et al., 2009, p. 96). For an owner: if three candidate markets have all passed the compliance gate, "compliance" should carry a low weight in the scorecard even though it is a condition of survival.

Three invented markets: who wins when the weights are fixed in advance?

This is a hypothetical example; every number is invented by me and is not real data for any market. A small tea-packing company picks an export market to try first. Call the options A, B and C.

Convert to a linear 0 to 100 scale (sizes 40, 120 and 80 become 0, 100 and 50). Before looking at the scores, the company writes down its weights; its assumed goal is three-year sales, so size weighs most.

B: 100 × 0.35 + 60 × 0.25 + 0 × 0.20 + 50 × 0.20 = 60. C: 50 × 0.35 + 0 + 100 × 0.20 + 100 × 0.20 = 57.5. No option is dominated.

A rule stated in advance, mine and not from any source: choose the highest score, but call it a clear win only if the lead over second place is at least 5 points and the winner still leads when any one weight moves by 5 points. B leads by 2.5, under the threshold: B has not clearly won.

Does the winner survive changing the weights?

No. The first test swaps the whole weight set for different viewpoints, as the Nirex team did.

B wins three of five sets and C wins two. The second test turns one weight and rescales the others in proportion so the total stays at 100, as in figure 7.20 of the manual (Dodgson et al., 2009, p. 100). B and C tie when:

Raise the speed weight by about 2 points and C overtakes. The criterion with the largest weight (size) is not the one needing the fewest points to flip the result; speed is. Triantaphyllou and Sánchez (1997) define the "most critical" criterion as the one whose weight needs the smallest change to alter the ranking, and write that the intuition that the highest-weighted criterion is the most critical may not always hold (Triantaphyllou and Sánchez, 1997). Their reversal formula, as I read it, gives the same figure as my sweep: on a 0 to 1 scale, (0.575 − 0.600) / (1 − 0) = −0.025, a rise of 0.025 before renormalising, which is 21.95 per cent.

The third test changes a score. If B's true gross margin is 18 per cent rather than 21, its margin score falls to 40, its total to 55, and C leads with 57.5.

So the scorecard's answer is "B or C": moving any one of the four weights by 2 to 4 points, or B's gross margin coming in about 1.5 percentage points lower, changes the order. That is where information-gathering money should go: firm up the weights, re-measure B's margin and the days to first shipment and payment terms for B and C, or run a small trial in both. Triantaphyllou and Sánchez point the same way: measure the critical weights and values more accurately and leave the rest rough (Triantaphyllou and Sánchez, 1997).

A public case: how weights changed the winner at Nirex in 1988

Nirex was set up to build and run an underground repository for radioactive waste in the UK. In late summer 1988 it asked the Decision Analysis Unit at the London School of Economics to help cut 12 candidate sites (screened from over 500) to a few for its board (Dodgson et al., 2009, p. 90). Twelve people from six organisations met five times that autumn; the model had 30 criteria in four groups: costs, predictability (whether a site's expected performance holds as uncertainties are resolved), safety and environment (Dodgson et al., 2009, pp. 91, 93). The manual names the second group with a different word; I call it predictability. The base weights were 100, 20, 10 and 10 in that order (Dodgson et al., 2009, p. 96). The team then simulated other viewpoints (Dodgson et al., 2009, pp. 97-98):

Sellafield B came third (71) under the local view and fifth (72) under the national environmental view; the manual sums it up: "No site consistently comes out most preferred" (Dodgson et al., 2009, p. 98). Raising the weight on the predictability group alone, Sellafield B stayed first until that group held about 22 per cent of total weight (against about 14 per cent in the base case), when Site 6 overtook it (Dodgson et al., 2009, p. 100). Critics said the Nirex specialists' weights, in virtually all cases, sat where Sellafield B was most preferred, and called the pattern somewhat idiosyncratic (Dodgson et al., 2009, p. 100).

The team did not name one winner. The short list held Sellafield B, Site 6 or Site 7, and Dounreay or Site 2 (Dodgson et al., 2009, p. 101), with conclusions framed as conditions: if you believe this, Sellafield is best; if you believe that, Dounreay is (Dodgson et al., 2009, p. 111). In 1995, at the public inquiry into the proposed underground laboratory at Sellafield, the Inspector upheld Cumbria County Council's refusal of planning permission; one of his criticisms was that other sites should have been examined, specifically Site 6. The manual says he did not fault the scoring method but suggested more weight should have gone to safety (Dodgson et al., 2009, p. 101).

When does this method fit, and when does it not?

It fits, according to the manual, when the aim is to screen a short list with a simple, transparent model plus some sensitivity testing (Dodgson et al., 2009, p. 122). My added conditions: two to six options that have passed a screening gate, and criteria you can measure reasonably well.

It does not fit in three cases. One, when the quantities can be valued in money: the manual says multi-criteria analysis then does not replace cost-benefit analysis, only complements it (Dodgson et al., 2009, p. 5), so I treat cost-benefit as the main tool; for project cash flows see the post on NPV or IRR. Two, when you fix the weights after seeing who is ahead. Montibeller and von Winterfeldt (2015) write that when analysts elicit inputs from decision-makers or experts they meet the very biases they set out to overcome, some of them driven by a wish for a particular outcome (Montibeller and von Winterfeldt, 2015). My inference is to record the weights in a decision journal before scoring; the abstract does not say so. Three, when the criteria are really one.

Nor should you read a sensitivity check as if the result always flips. The manual says models of this kind often tolerate large errors in scores and weights, since scores on many criteria are correlated (Dodgson et al., 2009, pp. 70-71). My example flips because the top two are only 2.5 points apart.

Going deeper: origins, critiques and what I have not checked

According to the manual, the earliest linear additive model is perhaps the "Prudential Calculus" Benjamin Franklin proposed to Joseph Priestley in a 1772 letter (Dodgson et al., 2009, pp. 121-122). The two modern branches are Keeney and Raiffa (1976), who showed that in many circumstances the model is a simple, sturdy approximation to the more rigorous expected-utility model, and Edwards, who developed SMART (Simple Multi-Attribute Rating Technique), which later gained swing weights (Dodgson et al., 2009, pp. 25, 122-123). I have not read those three originals.

Critiques: the manual itself warns that the model's ease of use invites misuse, so a result can look clear while misleading if the logic of scales and weights is ignored (Dodgson et al., 2009, p. 121). Greenpeace criticised the Nirex exercise at the 1995 inquiry for involving only specialists (Dodgson et al., 2009, p. 91). Limits of this post: the tea example is wholly invented; the linear scale, four criteria and the 5-point threshold are my choices. Triantaphyllou and Sánchez tested random data, one weight at a time: measured in percentages the highest-weight criterion was most often the most critical, measured in absolute points the lowest-weight one was (Triantaphyllou and Sánchez, 1997). None of the sources I read measures the business results of people who used the method, and the Nirex case is a 1988 site choice with limited data, not a market choice.

What did I read? Dodgson et al. (2009): the 168-page PDF, through automatic text conversion. I read sections 4.5 to 4.8, the passages on double counting and sensitivity in chapter 5, section 6.2, the whole Nirex case (section 7.4), the lessons at the end of chapter 7 and the start of appendix 4; not the National Audit Office (NAO) case, the local-authority budget case or the appendices on AHP, outranking and fuzzy methods. Page numbers are the printed ones, and I checked the figure 7.17 table against an image of the page. It is government guidance, not a peer-reviewed paper; the GOV.UK page (viewed 3 November 2026) says the original was published in December 2000 and amended in January 2009 to update the software section, and I did not check whether other guidance has replaced it. Triantaphyllou and Sánchez (1997): the main part of the authors' posted manuscript, abstract to conclusion, except the proofs in the appendix; it has no journal page numbers, so I cite none. Montibeller and von Winterfeldt (2015): abstract only, so the post uses one idea from it. Keeney and Raiffa (1976), Edwards (1971), Nirex's own papers and the 1995 inquiry record: not read; the Nirex account is wholly the manual's.