Thanks! Love this question.

Some initial findings are available in this series: https://knology.org/article/numbers-in-the-news/

The overarching finding is arguably to be clear about what we don't know. We keep seeing that people really want to make inferences about why, so you have to tell them why they can't. And similarly, articulating & visualizing the uncertainty inherent in probability is key.

We've also confirmed others' finding that the news is, in general, full of statistics -- and we found more in economic in health reporting than science or politics (which we included because we collected the data in a presidential election year). We're also currently analyzing a survey about the links between people's quantitative reasoning and news consumption.

I'm also hoping my colleague John Voiklis will comment in more detail on the findings here!

Hi Martin,

I too love your comment about expectations.

As a (social-)cognitive scientist, I came into this project expecting that news audiences would take away from the news what they expected was there to take away. In other words, I expected to see lots of reasoning based on prior knowledge about how the world operates--what has been called Bayesian rationality* (see the work of Nick Chater, maybe Gerd Gigerenzer, and their colleagues). What was true yesterday is true today will be true tomorrow. For most seemingly constant happenings in the world, that's a safe bet. Whether the "new" information is quantitative or not, people find the evidence that matches their expectations (sort of but not exactly what is called "confirmation bias").

As Jena pointed out, this kind of reasoning is rampant in the series of studies we're calling Numbers in the News. People draw inferences from the news that go well beyond the given (quantitative) information. One detail that is missing from the online reports (but will appear in fuller write-up) is how early this starts happening in the reasoning process. Our model of drawing inferences from the news follows a path that starts with people figuring out the topic of the story, then deciding if the perceived topic matters to them or those they care about, from which they judge the credibility of the story on the perceived topic, before drawing any inference. Prior experience, prior knowledge, prior commitments take effect immediately with topic perception. For example, mention the name of a politician in a paragraph that also reports the various ways to estimate the unemployment rate and some people will perceive criticism of the politician as the topic of the story rather than the headline topic about unemployment numbers. The reasoning process then proceeds from that perceived topic.

Jena also mentioned our study looking at the likely relationship between people's quantitative reasoning and their news habits. We are still in the thick of the analysis, so I can't really talk about findings, as yet. But, one interesting tidbit is that we can reliably segment participants based on their news habits. For example, there was a group of avid news users who rely primarily on the NY Times and Washington Post. While I find the notion of an "echo chamber" poorly specified, it could serve as a useful starting point for connecting these segments to people's quantitative reasoning. One source of prior expectations comes from one's reference group (aka social norms). In the so-called "echo chamber", news sources can serve as badges of group membership and amplify the norms of the reference group. In my prior work on analyzing the (moral) norms expressed in the news and opinion, I found that "straight news" (as opposed to opinion) didn't outright tell people what they should think. Perhaps, though, they limit what people can think--the available information and how deeply they should consider that information. The third study Jena mentioned is our text analysis of how a large sample of news stories from various sources use numbers and what quantitative reasoning skills they expect from their audiences. It might be that the amount of quantitative reasoning required to make sense of a story implicitly tells the reader, viewer, listener how deeply they need to reason about those quantities. But here I am getting into questions for a followup study. The existing data won't answer the question of whether and to what extent the quantitative demands placed on audiences might set expectations and delimit the depth of reasoning.

The problem with Bayesian rationality is that it sometimes (some might say often...though not I) leads people astray. How to fix this? There are many suggestions in the literature on probabilistic reasoning, but most of those suggestions require more time and resources than news providers can or should devote. For the news, we need a simpler and quicker mechanism. I want to reinforce what I said in the video and Jena repeated in her reply: "articulating & visualizing the uncertainty inherent in probability is key." Why? Because uncertainty is an invitation to pause and reflect. It is true that expressions of uncertainty can make people less certain about particular aspects of message credibility (a topic for another time and place). Nevertheless, pausing for a breath of time to reflect yields benefits to reasoning that would be a bargain at twice the price (for example see the work of Pennycook and Rand onthe role of reflection in detecting misinformation & so-called “bullsh!t”).

* For the technically inclined, I should clarify: I am not arguing that people actually perform formal Bayesian analyses in their heads. After all, it has been widely demonstrated that people are prone to ignore the base rate occurrence of a phenomenon--e.g. how often people in their 60s suffer strokes absent receiving any new vaccine. But at the conceptual level, prior experience, prior knowledge, prior commitments, etc. can easily take the place of quantitative base rates (aka prior probabilities) in everyday reasoning.

This is such an interesting and much needed project Jena. We emphasis in our Passion-Driven Statistics model about telling a story with the data to our students at all levels. I look forward to exploring your links to see how we can improve how our students tell their story to improve audience understanding. 

Are you finding that journalists not involved in the project/do you expect are interested in improving how the deliver the news to improve audience QR? Kristin

I absolutely love this question, and we are definitely hoping it's widely applicable.

There have been a lot of interventions for journalists that focus on journalists' own understanding & literacy but not on the skills to present information in a way that makes it easier for their audiences to draw the right inferences. I definitely think some of the ways we're looking at visualizing uncertainty are broadly applicable, for example.

A huge benefit of our project is that it's led by PBS NewsHour, which is the most trusted public news outlet we have in the U.S. I have to admit that don't know anything about the news landscape in Kenya - where do people get information from? Is there public media that's widely trusted? I'm hesitant to speak to the applicability anywhere that I don't have a good sense of the news.

I also have a big caveat: I'm always a little nervous about 'literacy' in and of itself as an intervention - since a lot of misinformation also comes from people who have the analysis skills but are skeptical of institutions. (Are you familiar with Crystal Lee's work on COVID-19 misinformation? https://news.mit.edu/2021/when-more-covid-data-... is a nice overview.) That is, we can only do the work we're doing because of NewsHour.

I found the link you provided absolutely fascinating. This gives me a deeper understanding of the "literacy is not enough" claim. The article you linked, links to their actual study, "The Data Visualizations Behind COVID-19 Skepticism."

Their work is so, so, so good. I always try to point to the structural issues, even though we have the incredible luxury of working with a news organization that's extremely well trusted across the ideological spectrum. It's not an either-or!

Another great set of questions! We've definitely been looking at the literature on formal learning as well, and always love recommendations. As I see it, this project sits at the intersection of a lot of bodies of literature:

• There's literature on QR/numeracy/statistical literacy that is heavily - but not exclusively - classroom focused
• Journalism studies as a field has looked at journalists' QR/literacy but hasn't always taken into account what their audiences need, although this is changing recently; they've also been looking at stats and numbers in the news
• Psychologists (Tversky & Kahneman, Gigerenzer, and their colleagues) have done tons of work on perception of probability and uncertainty
• STS also has a fair bit to say about how numbers circulate and why quantitative info tends to be perceived as 'better' or 'more rigorous' than qualitative info.

It's a lot to keep up on, and we're always trying to connect the dots.

In terms of interventions, we're working specifically with adults, and we're funded through the AISL program, which is specifically for informal learning. That said, there are a ton of opportunities for us to expand this work, and I love the idea of supporting formal educators too. The best people to weigh in on possibilities for the future are my PBS NewsHour colleagues - especially Patti Parson and Leah Clapman, who runs all of PBS NewsHour's work with formal education, including Student Reporting Labs.

I agree, wonderful project! Here's the direct link to our "COVID-Inspired Data Science for Youth" video if you'd like to check it out. Thanks! https://stemforall2021.videohall.com/presentations/1984 

Jacob, I'm so glad you posted here. I thought when viewing both of these project that there could be some overlap. Since I am interested in the education potential of this PBS project - not just through the articles themselves (which is the goal of this project) but through direct engagement with students - I thought it would be nice for your two projects to be connected.

Oh, wow, Jan and Jacob - speaking from the research side, your project looks amazing! I'm really looking forward to learning more, and I'm sure my colleagues on the PBS NewsHour side will have a lot more to say too.

I knew about Otto Neurath's work but could never remember his name. Thank you; knowing the name will help with the literature searches intended to inform ongoing studies in the Numbers in the News series of studies (https://knology.org/article/numbers-in-the-news/). That series has begun to systematically test several verbal and graphical manipulations of quantitative information that may improve quantitative reasoning.

I love your project Jacob and Jan...what a wonderful way to engage kids with data, while also teaching them about how to better understand what is going on in the world (and maybe teach their parents!). I'd love to see a copy of the book you wrote, Jan! The Meaningful Math part of our program at PBS NewsHour is aimed at those 18+. We know our audience is evenly split among those who identify as conservative, liberal and middle of the road, so hopefully what we learn from the research Jena and others at Knology are doing will help us better communicate to them. 

Yes, thank you for this Michael. Your questions here - "I wonder, how might your work inform K-12 education relating to building and communicating with data representations? I couldn't tell from the video, but is this part of your project? If so, what have you learned about students' use and understanding of data representations? If not, what suggestions do you have for teachers trying to support their students to develop this type of data literacy?" - are exactly mine.

This is such a focus in education right now and so would just love to see if there is outreach/overlap in that direction with this project.

Michael, I'm curious to hear which visualization in particular you find baffling! We started this project in summer 2019 and by, oh, January 2020 it started to feel like an epidemiology crash course. So I very much feel your pain.

We're not working directly with K-12 educators yet! But I think many of us would be interested in expanding - and PBS NewsHour already does a lot of formal education work too. There's so much room to build and I am very much looking forward to it.

Are you thinking about a particular age or particular topic area? Would love to continue thinking together!

Thank you for asking! The NY Times data representation is one they still have up near the top of their website. It shows the daily number of COVID cases and deaths, a percent change relative to a 14 day average, a trend line showing whether the numbers are increasing or decreasing, and a heat map showing changes by state. Since the start of the pandemic, it seemed like every time I looked at it, both cases and deaths were decreasing. I even made a spreadsheet to see if I could figure out how the numbers were calculated and I could not make my numbers match. I've been struck by just how important this representation is because it can send conflicting messages about safety depending on how you interpret it. Especially the seemingly always downward pointing trend line making it seem like we've always had things under control. Until you mentioned it below, I had never thought about the importance of getting people (me included) to slow down when looking at a representation - that's such a good point. From a news angle, how do you contend with a fast-scrolling approach to news? Do you think people will slow down or will they just ignore a representation if it is too complicated? These are the types of trade-offs that I think students should contend with as well. 

In terms of K-12 education, I would love to see this type your work in middle school where students start working with representing data variability. I love how this context introduces real stakes for representing data - it becomes more than just reporting information. The journalistic angle makes it also about effective communication. I also like that your project could expand students' toolbox of representations and open the space for them to invent new representations.

If I remember right, the NYT is looking at 14-day change in what is itself a rolling 7-day average? Something like that? And now the heat map has changed to vaccination status - but I have no idea if the denominator is total population or eligible population (which just jumped this week with the approval for adolescents).

Okay, I got curious, so I clicked through to https://www.nytimes.com/interactive/2021/us/cov... and found the little 'about this data' button, which says: "Sources: State and local health agencies (cases, deaths); U.S. Department of Health and Human Services (tests, hospitalizations). Tests, hospitalizations and deaths show seven-day averages. Hospitalization data may not yet be available for yesterday. Number of average tests may not include tests from some states for the most recent days. 14-day change is hidden if not enough data is available to make a comparison. Figures shown are the most recent data available." And for vaccines the denominator is total population, including children who are not yet eligible.

But to your question, I don't know how many people are going to go full 🤓 like me and click through until they find the explanation. And I've never tried to check it myself, so I love hearing that you did!

On the one hand, I'm so glad to see more and more news orgs are using techniques to account for natural variability and make numbers more comparable (e.g. rolling averages to account for reporting delays on weekends; population-adjusted numbers rather than absolutes).

On the other hand, these are mathematically sophisticated techniques, and I wish there were more (& more visible) explanation of both WHY and HOW.

Hello Michael: When I made my comment above, I was thinking about getting people to slow down their inferential process--to look before they leap to conclusions, so to speak.

How do you get people to slow down their fast scrolling? That's a good question. I would argue that scrolling through a news feed is itself an inferential process, with many rapid classifications: "know it, know it, don't care, don't care, boooring," and so on until you reach "hey, wait a minute, what's this about?" There are myriad design tricks for that, and some of them must use uncertainty of some form to prompt people to hover for a moment. Attention though is like a superbug and quickly becomes immune to overused tricks. The thing is, while news providers may want audiences to linger over everything they publish, people have limited time and cognitive resources. They cannot linger over every story. The real trick is to get them to reflect a little more and a little longer over what has caught their attention.

From a research perspective: we've been testing this very question, and here's what we're seeing so far. In general, showing ranges rather than point estimates helps people who don't have as much background - and doesn't hurt those who do. It lowers people's PERCEIVED understanding but improves the inferences they draw -- which might suggest that the additional visual complexity is getting people to slow down to process the numbers:

https://knology.org/article/numbers-in-the-news...

We also are seeing consistent evidence that at least some people interpret references to "official" or "government" statistics as being somehow better or more true, which is also a challenge:

https://knology.org/article/numbers-in-the-news...

We've also published a piece about navigating the epistemic uncertainty involved with all statistics, above and beyond the probabilistic uncertainty:

http://www.americananthropologist.org/2021/04/1...

Would you be interested in hearing about the news production side and the day-to-day conversations and discussions happening there? I know our PBS NewsHour colleagues have tons to say about that piece of the puzzle.

Thanks so much, George and Jena. The PBS NewsHour and Knology have been thinking a lot lately about how to accurately and effectively convey margin of error, particularly in polling results from the PBS NewsHour's partnership with NPR and Marist Poll. The video above was submitted before we had published our latest effort, but here is a link where we used shading to show the MOE range in a bar chart in an April 15 poll release.

https://www.pbs.org/newshour/politics/bidens-po...

We are also thinking about how to share MOE in different chart templates. This conversation about developing more creative yet streamlined ways to display MOE is on-going, and I'd love to see examples if you have some favorites in mind, George! Thank you again for your comment! 

Thank you so very much for the lovely feedback! It is certainly exciting to ask and answer questions that touch on those very real issues in statistical literacy and news consumption and comprehension. We look forward to being able to share more approaches as this project continues! 

Thanks so much for raising the question, Amy. When it comes to daily conversations that lead up to news production, we are increasingly working to factor in how to build statistical awareness, competency and comprehension in our coverage -- both in the newsroom, across beats and for our audience. News serves as a source of active learning throughout adulthood. It is an idea we want our work to serve, and this project provides such an opportunity.

To that end, editors at the PBS NewsHour ask these questions whenever they are working with a reporter who has submitted a draft for a story or graphics. We are working with Knology to develop a list of do's and dont's to offer quick-glance guidance for journalists on deadline but who still have to get the numbers (and related certainty/uncertainty) right in their reporting.

Thinking about how to show uncertainty in vaccination data when we know there are reporting gaps for race and ethnicity and concerns about equity in the U.S. vaccine rollout, we have launched a/b testing to gauge statistical understanding, different versions of graphics conveying these data. We began this test via the PBS NewsHour's science email newsletter, focusing on responses from people who are considered active users (they opened the last five newsletters) and offered a Google Form where those users could offer critiques about the graphics. We are working to develop alternative graphics that respond to strengths and challenges users raised, along with our own insights from existing and on-going research, and have mapped out plans to launch the next iteration of that data graphic series. The ultimate plan is to provide a resource for accountability that can also inform reporting as the nation works toward gaining control over the COVID-19 pandemic. 

The hope is that through these collective efforts, we improve our process in meaningful ways through these efforts so that we can build templates that can be quickly used by journalists on deadline but who also know that it is critical to share these data in ways that matter to our audience, whether we're reporting on public health, space exploration, economic renewal, social justice and more. 

Thank you for these responses Patti and Laura. All of this strikes me as so compassionate and also such a broad approach to the problem. You are not only trying to address the quality of the data itself (including issues of equity and inclusion) and how the data is presented, but thinking hard about the framework within which the data is received (what are the preconceived ideas about government, science, other sources of authority etc, that affect reception). All of this has really opened my eyes to the problem of quantitative literacy. It is not merely (or in some cases even primarily) a problem of education. It is also a problem of presentation and reception. By studying how people receive and interpret data, you are seeking meaningful ways to intervene in our the public's comprehension of quantitative information. And again I say this is compassionate because you are not blaming the viewing/reading public for failing to understand the data you offer, but rather you are trying to improve things from your end. I'm so impressed with this project and with the energy and sincerity of this conversation here.

Thanks so much, Amy. It really helps to hear such supportive words and give us new energy going forward.

I loved your video as well - getting students to tackle problems that they're interested in seems like the best possible way in. I'd be curious to learn more about how you're evaluating the students' critical reasoning skills, and would love to hear more about your findings as well! Let's try and keep in touch about what we're doing?

In our project we've been measuring growth in students' academic interests in statistics and data analysis with pre & post-tests adapted from existing validated scales and assessments. Because we're an early-stage project, we've also been collecting classroom observation, teacher and student survey and interview data, and student work. Yes, let's stay in touch about what our projects are doing and learning!