How much does AI raise productivity? Read three studies closely first

A vendor's slide says "AI lifts productivity by 40%". You manage ten people in customer care and must decide whether to buy. What does that number stand on? This post reads three studies and asks four questions: what was measured, against whom, on what work, and who gained.

What did each study measure, and in whom?

Three different outcomes in three different groups of people, so the figures cannot be averaged side by side.

The Noy and Zhang figures come from the abstract of the Science article (Noy and Zhang, 2023); the occupations and task types come from the working paper. The Brynjolfsson and colleagues figures come from the abstract of the Quarterly Journal of Economics article (Brynjolfsson et al., 2025); the firm and chat details come from the NBER version revised in November 2023, which reports 5,179 agents and a 14% gain. The Dell'Acqua figures come from the abstract of the working paper (Dell'Acqua et al., 2023).

Each study also uses its own yardstick for "productivity". Noy and Zhang measure the time to finish a short piece and a grade given by people in the same occupation. Brynjolfsson and colleagues measure issues resolved per hour, built from chats per hour and the resolution rate the company already tracked. Dell'Acqua and colleagues use quality scores from two graders, and whether one recommendation was correct.

Which work does AI help, and which does it set back?

Dell'Acqua and colleagues call this the jagged technological frontier: AI does some tasks well and fails unexpectedly at others that look similar (Dell'Acqua et al., 2023).

In the first experiment, 385 consultants did 18 tasks for a new shoe. The control group's average score was 4.1; the group using GPT-4 with guidance scored 1.75 points higher (42.5%), and the GPT-4-only group 1.56 higher (38%). The control group completed 82% of tasks, the AI groups roughly 91% to 93% (Dell'Acqua et al., 2023, section 3.1). The AI users' ideas were better but more alike, according to the authors' semantic-similarity analysis.

In the second, 373 consultants did a business analysis. The authors built it so that the right answer depended on matching spreadsheet figures against interview notes, and GPT-4, if simply given the instructions, spreadsheet and interview notes, would reach a different conclusion. The control group made the correct recommendation about 84.5% of the time, the two AI groups 60% and 70% (Dell'Acqua et al., 2023, section 3.2). AI users' write-ups were still graded higher for quality, even when the answer was wrong. A good quality score can hide a wrong answer. The authors observe that those who did worse tended to adopt the AI's output without questioning it.

Why can't one study's number be used somewhere else?

Because four things can change: what is measured, the comparison, the kind of work, and who gains.

1. What is measured and who grades it. Time, issues per hour, a grader's score and correctness are four yardsticks. A score moving from 4.1 to 5.9 does not tell you whether the answer was right.
2. The comparison. Noy and Zhang, and Dell'Acqua and colleagues, assign people randomly. Brynjolfsson and colleagues use a staged rollout, so agents who got the tool differ from those who did not in ways the authors have to control for in their models.
3. The kind of work. A short self-contained piece written in 20 minutes is not a 40-minute chat, and neither is a task that needs figures matched with interviews. In the working paper, people who did not use ChatGPT in their real jobs said the chatbot lacked their company's specific knowledge (section 2.9) (Noy and Zhang, 2023).
4. Who gains. Your team's gain depends on who is in it.

With the same tool, how much would your team gain?

Less than the number on the cover, and it depends on the mix. This example is entirely hypothetical: the tickets per hour and the team mix are my inventions. Only the 34% and 0% gains come from Brynjolfsson and colleagues, for the lowest and highest skill groups in the NBER version. I apply 34% to "new hires" and 0% to "experienced agents", although the source measures skill groups rather than tenure in exactly this way; treat it as arithmetic, not a forecast.

A hypothetical example. The customer-care department of a distribution company in Can Tho has ten people. A new hire handles 1.7 tickets an hour and an experienced agent handles 2.8.

With four new hires and six experienced agents, the team total before is 4 × 1.7 + 6 × 2.8 = 23.6 tickets an hour. After: 4 × 1.7 × 1.34 + 6 × 2.8 = 25.9. The team gains 9.8%, less than a third of the headline 34%. Make it six new hires and four experienced agents and the team gains 16.2%. To know your own number, re-measure by tenure rather than by team average.

Going deeper: what do the three studies leave out?

A good deal, and the authors say so themselves.

Noy and Zhang write that their tasks are short, self-contained and lack company-specific knowledge, which may inflate how useful ChatGPT looks. Two weeks later, 33% of people who had been given ChatGPT had used it in their real job in the past week, against 18% of the control group; that follow-up was still running, with an 82% response rate among the 423 people invited so far. In the working paper, 68% of the people assigned ChatGPT reported submitting its first output unedited, which the authors read as AI substituting for effort rather than complementing skill (Noy and Zhang, 2023). The Science version says people given ChatGPT were twice as likely to use it in real work after two weeks and 1.6 times as likely after two months; I do not have the full text to check that.

Brynjolfsson and colleagues say plainly that the study was not designed to tell us about aggregate employment or wages. The attrition result deserves more caution than the main result, because the model cannot include agent fixed effects and the firm may have given the tool to people more likely to stay (Brynjolfsson et al., 2025). The firm outsources most of its customer support, so resolution rates and satisfaction scores exist for only some of the agents.

The Dell'Acqua team raise a long-run risk: if work inside the AI frontier stops being handed to junior staff, they lose the chance to learn the trade. That is speculation in the discussion section, not a measured result. The paper is a working paper; three of the authors are at BCG, and Lakhani states that he advises BCG on AI strategy (Dell'Acqua et al., 2023).

I read in full the March 2023 working paper by Noy and Zhang, the NBER version of Brynjolfsson and colleagues (revised November 2023) and the 22 September 2023 working paper by Dell'Acqua and colleagues. For the first two I read only the abstract of the journal version, via PubMed and Crossref; I did not read the full Science and Quarterly Journal of Economics texts, so any detail beyond the abstracts is from the working papers. Dell'Acqua and colleagues say the experiment's pre-registration will be published later, and I have not seen it.