Editors vs. Algorithms in Apple News

If you check Apple News right now (June 4th, 2020 as I write this), you will find a “Spotlight” about the U.S. protests that features rights for protestors, educational resources, mental health resources for BIPOC, and a list of organizations where you can donate to support racial justice:

• How to Support the Struggle Against Police Brutality
• If you're planning to take part in protests, know your rights. Read this.
• Here's How White People Can Support The Fight Against Police Brutality
• 44 Mental Health Resources for Black People Trying to Survive in This Country

In case it is not obvious, these stories are curated by a team of human editors at Apple News, and not an algorithm. They show further evidence of the value of human curation, in a time when news is especially important. This was the topic of my recently-published research study, which I will summarize in this blog post.

Background

Editors and algorithms at Apple News choose stories for millions of people every day, and that number is growing quickly. In fact, the app has climbed to over 125 million monthly active users. That number is on par with Flipboard, a similar app with 145 million monthly users, and is about half as many users as Google News (based on estimates).

So what is Apple News showing to these millions of people every day?

Is it showing a diverse set of sources, as it claims, or do most stories come from just a few select places? And what are the stories about? Last year, we tried to address these questions by collecting stories from Apple News for two months. I wrote a program that saved stories from the app every five minutes — here is what it looked like running on February 27th of this year:

In total, we collected 1,268 “Top Stories,” which are curated by human editors, and 3,144 “Trending Stories,” which are curated by an algorithm. Our analysis showed that human editors choose sources more evenly, and also prefer a more diverse set of sources compared to the trending algorithm. We also found the trending algorithm featured more “soft news” stories. Here are a few more details about the main findings.

1. Human Editors Chose Sources More Evenly

One of our main research questions was how news sources were concentrated in the app. A previous study in the United Kingdom showed that 75% of stories from Apple News editors were from just six news outlets, so we were interested if this concentrated pattern was happening in the United States.

Can you imagine if 50% of all stories in Apple News were from a single outlet like CNN or Fox? Not only could this skew the news for millions of readers, it could mean disproportionate payouts for some news outlets.

We found concentration patterns in both sections, but the editors’ Top Stories were distributed much more evenly. For example, the four most common sources chosen by the algorithm claimed more than 50% of all Trending Stories. In contrast, the four most common sources chosen by human editors only claimed 32% of all Top Stories (check out the paper for details about statistical significance).

2. Human Editors Chose More Diverse Sources

The four most common sources in the algorithmic Trending Stories — CNN, Fox News, People, and BuzzFeed — were chosen less frequently by the human editors at Apple News. Here is a comparison of sources:

What exactly does this mean? The algorithm is choosing more stories from BuzzFeed and People, while the editors are choosing more stories from The Washington Post and the New York Times. On the surface, this suggests that the algorithm could be choosing more stories about celebrities and other “soft news.” To find out if this was true, we analyzed every headline that appeared in the data.

3. Human Editors Chose Less “Soft News”

Our data showed that, as The New York Times summarized in their report about Apple News, the algorithmic Trending Stories “tend to focus on Mr. Trump or celebrities.” Here are some common phrases that appeared in the data:

Apple’s human editors actually never chose a story with Mr. Trump’s full name in the headline. Phrases such as “Trump Tax Returns” were common, but no headline used his first name. Common phrases in Top Stories headlines included international subjects (Brexit Deal, Kim Jong Un) and U.S. policies related to health (Affordable Care Act) and immigration (sanctuary cities).

Still, the editors balanced more substantive topics with some entertainment news (ex. Avengers: Endgame) and one story about a Northwestern University alumna having a baby — specifically, Meghan Markle giving birth to Archie. They also regularly curated collections of “The Week’s Good News.”

Takeaways

Our results highlight the trade-offs between human curation and algorithmic curation. While our study only looked at one platform, it shows that human editors were “much more subtly following the news cycle and what’s important,” as Lauren Kern (editor in chief at Apple News) put it.

For many readers and publishers, this is good news. The data shows that editors choose stories about important topics from a diverse set of sources, and choose those sources quite evenly. This is less true of the algorithmic Trending Stories, where readers will see more “soft news,” and just a few major publishers tend to make the cut.

Here is another anecdotal piece of evidence on the value of human curation. At the beginning of this post, we saw how human editors were valuable in the context of the U.S. protests. They were also valuable in the context of the COVID-19 pandemic. If you checked the Google News home page at the end of February, this is what you would have seen (based on the WayBack machine):

All of the stories mention two things: coronavirus and Donald Trump. If you read them, you may glimpse some information about the impending pandemic — a “severity warning” from the CDC, for example. The headlines probaly grab your attention, but they do not provide meaningful information.

Apple’s editors had a different approach that day, featuring an article with the headline “Coronavirus’s spread in U.S. is ‘inevitable,’ CDC warns.” It was a formal, descriptive piece from the Washington Post that quoted several officials at the Center for Disease Control. I scrolled back in the Apple News desktop app to find it:

During my commute on the CTA purple line, I remember one quote from the article that changed my expectations for the coming months:

“Disruptions to everyday life may be severe, but people might want to start thinking about that now”

I am grateful that humans chose my news that day.

If you are interested in learning more about our methods and results, you can read our full study here and another blog post by my advisor here. Finally, here is a link to the data and code from the paper.