A new report argues newsrooms need to do more experiments — like, the kind with randomization and results

Mississippi Today had a question. Did the stories around its Pulitzer Prize-winning investigative reporting affect the way those investigations were received?

It’s the kind of question local newsrooms can’t easily test on their own. But researchers from Local NExT Lab at Syracuse University could.

In a national survey, 1,768 adults were shown the same investigation surrounded by a different mix of stories. In one, the investigation was shown alongside positive community stories. (Think: “How a local fossil park changed science history.”) In another, the investigation was featured alongside other investigative stories.

The investigation was rated more accurate, important, and relevant when it was surrounded by community-focused stories, not other investigative stories. (Audience reaction to a third story mix, where the investigation was surrounded by stories about legislation, was not consistent.)

The result was described in the first report from the Local NExT Lab. Joshua Darr, director of the lab and associate professor at Syracuse University, believes that although dashboards and engagement analytics are “very important tools” for newsrooms, they aren’t enough.

“Engagement data can’t tell you what caused audiences to engage, settle a hunch, or isolate your contribution,” the report reads. “Randomization can.”

“The key word there is ’caused,'” Darr added in an email. “Without randomization, you can tell when engagement is up or down or if a particular story is doing well, but it’s hard to know exactly why that is happening.”

At Nieman Lab, we often hear from both sides of academic-practitioner partnerships. Researchers wonder why newsrooms don’t pay attention to their findings or fail to respond to their overtures about experiments. Publishers, editors, and journalists, meanwhile, tell me they don’t have time to read dense and jargon-filled academic papers. They can be annoyed by requests to spend time or resources they don’t have. And it’s not always fun to be someone else’s test subject.

When partnering with Local NExT Lab, Darr emphasized, the local newsrooms don’t have to commit money or manpower. They agree to two or three conversations. The researchers listen and then design, field, and analyze the experiments. In return, the newsrooms get their ideas tested and causal evidence.

In another experiment, the Local NExt Lab found that Instagram Reels including reporters’ faces drove the most interest and engagement, but when it came to clicking “follow,” profiles that looked like traditional news brands won out. Respondents chose to follow newsrooms (in this case New Orleans Public Radio) more often than reporter-named accounts.

The first batch of tests was funded by the W.K. Kellogg Foundation. Additional support from the American Journalism Project will allow for further tests on a larger cohort next year. The full report is here.