---
title: "Netflix: data or gut? What two papers say"
description: "Reading van Es (2023) and Gomez-Uribe and Hunt (2015): what Netflix's recommender does, what the 80% figure measures, and where data or people decide."
url: "https://staging.khoi.co.uk/en/blog/netflix-du-lieu-hay-truc-giac"
lang: en-GB
author: "Lê Quốc Khôi"
date: 2026-11-03
updated: 2026-11-03
topics: ["Data & analytics"]
tags: ["Netflix", "Recommendation system", "A/B testing", "Data and judgement", "The big data myth"]
---

# Netflix: data or gut? Reading the two original papers

## Key takeaways

- Van Es (2023) reads what Netflix itself said publicly about data between 2012 and early 2022, and how the press took it up. Her conclusion: the company keeps a deliberate ambivalence so it can be understood as both "data" and "gut". The paper analyses public statements and discourse, not internal decision-making (van Es, 2023, pp. 656–657).
- Gomez-Uribe and Hunt (2015), both Netflix employees, describe the recommender as a collection of algorithms. The 80% figure is the share of hours streamed where the system "influences choice"; the other 20% comes from search (Gomez-Uribe and Hunt, 2015, p. 5). Van Es words it more strongly: 80% of what people watch "originate in recommendations" (van Es, 2023, p. 656).
- In the technical paper, intuition loses to data on one narrow question, while people pick the yardstick and decide whether to trust a test result or rerun it (Gomez-Uribe and Hunt, 2015, pp. 7–11).
- The "do differently" section is my inference, not the authors' conclusion.

According to a source van Es cites, in 2015 Ted Sarandos, then co-CEO and chief content officer of Netflix, supposedly said that decisions at the company were 70 per cent data and 30 per cent human judgement. Three years later, in an interview, the proportions were reversed: 70 per cent gut, 30 per cent data (van Es, 2023, pp. 656–657). If you are thinking of copying Netflix, which is right? This post reads two primary papers: van Es's (van Es, 2023) and a description of the recommender by two Netflix employees (Gomez-Uribe and Hunt, 2015). See also [the Netflix case on data and judgement](https://staging.khoi.co.uk/en/case/netflix-data-and-judgement). How to run a four-step decision loop is covered in [A data-driven decision loop for small businesses](https://staging.khoi.co.uk/en/blog/vong-du-lieu-cho-doanh-nghiep-nho); this post only asks which questions the data can answer.

## What does it mean that the 70/30 split was reversed?

Van Es says both versions fit Netflix, because the company cultivates ambivalence about what it is; she calls it "strategic ambivalence" (van Es, 2023, p. 657). She analyses the Netflix technology blog, the Netflix Research site, LinkedIn posts and statements to the press (van Es, 2023, p. 657).

Until about 2017 the posts build an image of a technology company driven by data (van Es, 2023, p. 661); from 2018 they refer more often to human expertise and creativity (van Es, 2023, p. 663). In 2020 CEO Reed Hastings pointed out that the company has more employees in Hollywood than in Silicon Valley and spends two-thirds of its budget on content, and suggested it is neither a technology nor a media company but an entertainment company (van Es, 2023, p. 660). Van Es ties the change of tone to the backlash against data and algorithms and to the move into original programming (van Es, 2023, p. 656).

The 70/30 figures are second-hand: relayed through two press pieces I have not read, one of them flagged "supposedly" by van Es herself.

## What does Netflix's recommender actually do?

It is a collection of algorithms, each serving a different job, not a single algorithm (Gomez-Uribe and Hunt, 2015, pp. 1–2). A homepage typically carries about 40 rows of up to 75 titles each; in 2015 Netflix had more than 65 million members streaming more than 100 million hours a day (Gomez-Uribe and Hunt, 2015, pp. 1–2).

| Algorithm | What the paper says it does |
|:--|:--|
| Personalised video ranker | Orders the whole catalogue for each profile, blending personal signals with a fairly healthy dose of popularity (Gomez-Uribe and Hunt, 2015, p. 3) |
| Trending Now | Uses trends lasting minutes to days, such as Valentine's Day or an approaching hurricane (Gomez-Uribe and Hunt, 2015, p. 4) |
| Sims | Builds a list of similar titles for each title, not personalised (Gomez-Uribe and Hunt, 2015, p. 4) |
| Page generation | Chooses and orders the rows of the whole page; before 2015 a fixed rule-based template (Gomez-Uribe and Hunt, 2015, pp. 4–5) |
| Evidence and Search | Picks the images and synopses shown with a title; handles search, even when a title is not in the catalogue (Gomez-Uribe and Hunt, 2015, p. 5) |

## What does "80% of viewing comes from recommendations" measure?

The share of hours streamed in which the recommender was on the viewer's path, not the share of hours that would not have happened without it. The original sentence: the system "in total influences choice for about 80% of hours streamed at Netflix. The remaining 20% comes from search" (Gomez-Uribe and Hunt, 2015, p. 5).

Van Es cites the figure as 2016. She writes that 80 per cent of what people watch on Netflix "originate in recommendations" (page 656) and that 80 per cent of "consumption" comes from them (page 661) (van Es, 2023, pp. 656, 661).

| | Original paper | As cited later |
|:--|:--|:--|
| Unit | Hours streamed | "What people watch", "consumption" |
| Relationship | Influences choice | "Originate in", "comes from" |
| The rest | 20% from search, itself a recommendation problem | Not mentioned |

The paper also says the homepage alone is where 2 of every 3 hours are discovered (Gomez-Uribe and Hunt, 2015, p. 2), and admits that most of the company's statistical models ignore a feedback loop: titles with high engagement get recommended to many members, which raises their engagement further (Gomez-Uribe and Hunt, 2015, p. 14). My reading: those two points make 80% hard to read as a causal effect.

The value Netflix itself claims lies elsewhere: it says years of work on personalisation and recommendations have cut churn by "several percentage points", and, in a sentence that opens with "We think", that the combined effect saves "more than $1B per year" (Gomez-Uribe and Hunt, 2015, p. 7).

## Where does data win, and where do people decide?

Data wins on a narrow question: of two lists of titles similar to "House of Cards", which does better in an A/B test? People assumed the list that looked more relevant (it included the original version of the series) was better, yet A/B testing favoured the list leaning on popularity (Gomez-Uribe and Hunt, 2015, p. 8). The authors write that their own intuition, even collective intuition, often picks the wrong algorithm variant (Gomez-Uribe and Hunt, 2015, p. 7).

But people still decide three things:

- The yardstick. Revenue comes only from subscriptions, so they hold that maximising revenue is roughly the same as maximising the value members get; that is an assumption they state themselves (Gomez-Uribe and Hunt, 2015, p. 8). They measure retention, but because it takes a very meaningful improvement to move it even 0.1 point, they add medium-term engagement, and admit that counting hours "gives far too much credit to multiseason shows" (Gomez-Uribe and Hunt, 2015, pp. 9, 14).
- How to read a result. Interpreting A/B tests "remains partly art": when a test wins on retention but engagement does not rise with it, they assume random variation and rerun it, and usually find the win does not repeat (Gomez-Uribe and Hunt, 2015, p. 11).
- Sample size. Measuring a 0.1-point retention difference at 50% retention takes roughly 2 million members per cell (Gomez-Uribe and Hunt, 2015, p. 10). I checked it with the standard binomial formula: the standard error of the difference is √(2p(1−p)/n) ≈ 0.05 points, so 0.1 points is two standard errors, close to the usual 95% threshold. With 1,000 customers per group and 50% retention, the same formula gives a threshold of about 4.4 points. That 1,000-customer figure is my own calculation; the paper gives no such number.

## What does van Es say the "data or gut" framing hides?

That both poles are built with words, and that the system in between goes unexamined. She describes a "Moneyball mindset", following a book she cites, with four tenets: data is free of human bias, data beats intuitive knowledge, the two are usually in tension, and used wisely data gives a strategic advantage (van Es, 2023, p. 658). In her account Netflix fed that myth to promote itself (van Es, 2023, p. 658).

Netflix's later statements read very differently. A 2022 post says big decisions rest with "Informed Captains" who make judgement calls after reviewing data and hearing colleagues; a 2018 Netflix document says machine learning is there to support decisions, not to automate them (van Es, 2023, p. 661).

She does not treat that as the answer. In Netflix's language data is "accurate" and "objective" while people bring "emotion" and "expertise" (van Es, 2023, p. 665), and humans often stay outside the technical system, adding only a light "touch" (van Es, 2023, pp. 664, 668). Netflix is also a product of human practices, she writes, and we need open discussion of when, how and why to use data (van Es, 2023, p. 668).

She cites one example from The Wall Street Journal: image tests for "Grace and Frankie" found that artwork without Jane Fonda did better, but the Hollywood content team worried about relationships and contracts, and the article says the metrics were set aside to keep talent happy, not from distrust of the algorithm (van Es, 2023, p. 667). My reading: that was a judgement about an objective different from the one the number measured.

## What should a decision-maker do differently after reading both?

The four points below are my inference from the two papers, not the authors' conclusions.

1. Before saying "the data shows", write down the question the data answers. "Which list of similar titles does better on the metric we chose" has a data answer. "Should we keep a relationship with an actor" does not; a number is only one input.
2. Before believing a percentage, ask about the unit, the denominator and the verb. The 80% changes meaning when "influences choice" turns into "originate in recommendations".
3. Choosing a yardstick is a human decision. Record why you chose it and where it fails, as Netflix does about hours streamed.
4. A small business cannot copy a Netflix-scale A/B test. With a few thousand customers you only see differences of several percentage points or more; for smaller gaps, say plainly that you are using judgement. The reading habits in [Five questions before you trust a chart](https://staging.khoi.co.uk/en/blog/nam-cau-hoi-doc-bieu-do) apply to both.

## What are the limits of these two papers?

Gomez-Uribe and Hunt are Netflix employees writing about their own system, with 2015 figures, and several claims open with "we think" or "in our opinion" (Gomez-Uribe and Hunt, 2015, pp. 7, 14); I have no independent source to check them against.

Neither paper shows that "Netflix makes shows with data". Gomez-Uribe and Hunt use "House of Cards" only as an example of recommendations, a list of similar titles (Gomez-Uribe and Hunt, 2015, p. 8) and a subtitle-language case (Gomez-Uribe and Hunt, 2015, p. 14); the claim that the series was "generated by algorithm" is Sarandos in interviews, relayed by van Es (van Es, 2023, p. 659).

## Going deeper: what did I read, and what did I not?

Van Es (2023): I read the publisher's version in full from the Utrecht University repository. Page numbers are the journal's printed pages (656 to 672). I have not read the sources she cites: Wu (2015), the Vulture interview with Sarandos (2018), Netflix's technology blog posts, The Atlantic (2014), The Wall Street Journal (2018). Everything about them is van Es's account.

Gomez-Uribe and Hunt: I read the ACM PDF through an Internet Archive copy. The text conversion cut a few passages, including part of section 5.6, the conclusions and appendix B, and I do not rely on them. Page numbers are the paper's own (1 to 19). The reference line in the PDF says "December 2015" while van Es cites it as 2016; I follow the library and use 2015.

## Frequently asked questions

### What data does Netflix use about viewers to make recommendations?

According to Gomez-Uribe and Hunt: what each member watches, how they watch (device, time of day, day of week, intensity), where in the product each title was discovered, and even the recommendations shown but never played (Gomez-Uribe and Hunt, 2015, p. 2). Van Es reports Netflix saying that the system relies on subscribers' activity rather than demographic data, and that the company does not know individual users' race or gender (van Es, 2023, pp. 662, 667). That is the company's own statement, not something either paper tests.

### Are there humans inside Netflix's recommendation system?

Yes, according to van Es. She cites a 2014 article in The Atlantic about some 77,000 "altgenres" such as "Visually-striking Inspiring Documentaries", which took a great deal of human labour, and a Netflix presentation saying in-house experts tag more than 200 story data points (van Es, 2023, p. 664). I have not read either original, so this is van Es's account.

### How long does a Netflix A/B test run, and what does it compare?

Members are randomly assigned to "cells", each with a different experience, and left to use the product for typically 2 to 6 months; a test often carries 5 to 10 variants. The main yardsticks are medium-term engagement and cancellation rates (Gomez-Uribe and Hunt, 2015, pp. 9, 12). The authors prefer new members because they have not experienced an older version, but new members get a free month, so accurate retention cannot be measured until a month after the last new member in the test has joined (Gomez-Uribe and Hunt, 2015, p. 11).

### What does the Netflix Prize have to do with today's recommender?

The competition aimed to improve predictions of the star rating someone would give a title, the central problem in the DVD-by-mail era. The authors write that algorithms from the contest still run to predict ratings, but that the days when stars and DVDs were the focus have long passed (Gomez-Uribe and Hunt, 2015, p. 2). Van Es counts the Prize among the things that built the image of a data company between 2006 and 2013 (van Es, 2023, p. 659).

## References

- Gomez-Uribe, C.A. and Hunt, N. (2015) ‘The Netflix recommender system’, *ACM Transactions on Management Information Systems*, 6(4), pp. 1–19. https://doi.org/10.1145/2843948
- van Es, K. (2023) ‘Netflix & big data: the strategic ambivalence of an entertainment company’, *Television & New Media*, 24(6), pp. 656–672. https://doi.org/10.1177/15274764221125745

---

Canonical HTML version: https://staging.khoi.co.uk/en/blog/netflix-du-lieu-hay-truc-giac