Human-in-the-loop at two levels
When AI is added to a multi-step process, should the team lead put a human check on every prompt or only at a few checkpoints across the whole process?
- Origin
- Parasuraman et al., 2000; Bainbridge, 1983; Anthropic, 2024
- Diagram type
- Comparison table
- Level of analysis
- Team
Diagram
Human-in-the-loop at two levels
| Criterion | Prompt-level HITL (micro) | Workflow-level HITL (macro) |
|---|---|---|
| Scope | one step in a prompt chain | the whole multi-step task chain |
| Check question | right audience? off tone? genuinely usable? | if this point is wrong, how much damage results? |
| Action | accept, refine, discard | approve, reject, escalate |
| How to place the check | at every step | according to the cost of error at each step |
| Evidence kept | chain log | approval criteria and checkpoint log |
When to use it, and when not to
Use it when
Use it when a process chains several AI steps and you must decide where to put the reviewer. Prompt level (micro) checks each output in the chain; workflow level (macro) places a few checkpoints according to the consequences of error.
Avoid it when
The micro and macro split is the site’s label; the sources do not use these terms. (Parasuraman et al., 2000) discusses levels of automation across four stages, not prompt chains; applying it to AI workflows is the site’s extension.
How to apply it, step by step
Place checkpoints at the steps where an error does the most damage, using the cost-of-consequences criterion in (Parasuraman et al., 2000). Give the reviewer something they can actually check. (Bainbridge, 1983) points out that a person who only monitors an automated system struggles to stay alert to rare faults and cannot check the machine’s decisions in real time, so a checkpoint needs concrete accept-or-reject criteria rather than “have another look”. (Anthropic, 2024) describes agents pausing for human feedback at checkpoints or when blocked.
Pitfalls
Setting a checkpoint where the reviewer only clicks approve. According to (Bainbridge, 1983), a person who only monitors a system that usually works gradually loses both skill and attention.
Sources
References
Listed A–Z by author surname, Harvard style. A source with a DOI links to https://doi.org/…; one without links to its page or says plainly that it has no DOI.
- Anthropic (2024) Building effective AI agents. Anthropic Research.No DOIanthropic.comCited in 6 articles
- Bainbridge, L. (1983) ‘Ironies of automation’, Automatica, 19(6), pp. 775–779.doi:10.1016/0005-1098(83)90046-8Cited in 1 article
- Parasuraman, R., Sheridan, T.B. and Wickens, C.D. (2000) ‘A model for types and levels of human interaction with automation’, IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans, 30(3), pp. 286–297.doi:10.1109/3468.844354Cited in 2 articles