Yes!  Great question.  I have been playing with that question for awhile and have some rough scribbles that approximates a matrix.  Not much further that that but the dimensions probably relate to (1) goals for engaging people (understanding the problem, sharing experiences or thoughts about the problem, understanding different perspectives, exploring solutions, etc), (2) range of public/audience opinions (divisive, segmented, emerging, coalescing (3) types of experiences (exhibits, one time or long term , dialogue based), (4) nature of the problem.  The idea would be chart or set of charts that could help determine the pedagogy pedagogical approach and types of intervention most likely to work for specific types of audiences.  Is that what you're suggesting?  Of course it wouldn't be prescriptive but more in line with providing heuristics. 

Billy and Steve, (and thanks Kris).

Adding to Kris' comments, we think that the first issue is a definitional problem.  There is a tendency to jump right to the issue or problem and the debate, but seldom do we see a clear unpacking of how a social problem is constituted in general, how terms are defined structurally and for who.  There are often useful resources that characterize audiences, but it's important to structure the dimensions of the claim, and then the difference between who is experiencing a museum exhibit or program versus their social role in the problem.  ie: Are we preaching to the converted or the disengaged.  

We highlight Joel Best's work in our video and papers because that work can help structure how to set up reporting on an intervention in ways that contextualize the users who will experience an offering.

I advocate for using public data and measures as some form of indicator variable, whether a qualitative or quantitative study, as a way of describing the participant in their social context. 

As Billy knows, this was the approach we used for benchmarking the difference between aquarium and zoo visitors and the general population for the National Network for Ocean and Climate Change Interpretation (NNOCCI.org).  In that case, we discovered that these visitors were significantly (but magnitude was not massive) a bit more aware, interested, and more likely to vote and advocate for others to vote on environmental issues. ie: We had socially active visitors who would be willing to engage others in the problem.   So our data focused both on learning, and willingness to engage others in deliberating about the problem.  ie: activation of latent interest.  

I really do want to shout out to this citation in our video:

Ostman, R., Zirulnik, M. L., & McCullough Cosgrove, J. (2019). Storytelling, Science, and Religion: Promoting Reflection and Conversation about Societal Issues. Curator: The Museum Journal, 62(2), 117-134.

That team does a lovely job of describing a principle of social problems they were working on, and the need for more theoretical integration. They offer a set of principles that are worthy of testing by others. I believe four of the six are pretty portable. 

Thanks all for the great questions. 

Thanks, John and Kris, for this video summarizing your work. I'm looking forward to the final study -- I hope it encourages more integration and collaboration among people doing this work. I also thank you for calling out our article!

Great question Billy.  I think Johnny has more thoughts about this . We've had provocative discussions about the question of common measures as well as the need for developing alternative and flexible metrics for measuring more nuance and societal level changes.  I look forward to hearing others' thoughts.

Thanks Kris and Johnny for your thoughts on this question of common measures and methods, and on the importance of understanding the nature of the audience in relation to the social issue being studied. It seems like this domain of research could offer a whole new way to conceptualize what is meant by "broader impacts."

I can envision common methods, but interested to learn what you might have in mind for a benchmarked measure that would be both interesting and implementable across longitudinal studies in an ideal situation. Has this been done in the area of climate change or other forms of literacy at scale? So interesting. In classroom studies, researchers have studied knowledge integration and beliefs about science of middle school students into high school, but we know how to track and find these students (in the K12 same district.) 

I agree that this is the time to rethink our role within and as part of society! And if our work is truly integrated with other sectors of society, it may become more difficult to attribute change to the actions of any one sector or institution.  Which means measuring the success of our work may look different. In our review, we did see a number of creative and participatory evaluation efforts that might begin to speak to new approaches to measuring or documenting 'broader impacts" in the context of intersectionality.

Thanks so much for appreciating our work. Your parks work sounds familiar and I think we may have already been using some of your references in other work. I'd love to connect you with Rupu Gupta, a conservation psychologist on our team who leads our work on culturally responsive evaluation. She's done a number of projects in collaboration with groups like Outdoor Afro to think about perspectives and values related to park access and use.  Oh yes, and Joel was really generous with his time for his brief cameo. His book is definitely a must read for this work. Offline, I'll send an introduction to put you together with Rupu. 

John, I would very much appreciate the introduction to Rupu, thank you!  I am eager to be part of these conversations.

Nickolay, I sent you a connection request via LinkedIn because your email addresses on your website appear not to be working. I'd rather not share emails on this platform.

I just accepted your invitation request, John! Thank you! I am looking forward to continuing the conversations.

Sherry, Thanks for your thoughts too!  We used only published grey literature found on the InformalScience.org website. Most of the journals also publish qualitative research and small sample studies. For example, Curator: The Museum Journal, (I'm the editor) has no problem with qualitative data as long as the methods are well explained and the sample sizes match recommendations for those methods. In fact, we had many discussions about variables in the data and the value of case study. in fact, we don't think it's all about massive datasets, but rather, working to build connections across small sample projects. Like rare disease, we need common points of connection so we can make sense of many cases when they are all put together.  One point I'd hoped to see come through in this video is the idea that we can learn more transferrable information if the project teams work to link to principles of social problems, not just the specifics of the case.  Thanks so much for your comments.  

Patti,

We didn't find any substantial assessments of museum self-editing, but I do think it's fairly well known anecdotally.  The greatest challenge these institutions face is the politics of board and management's fealty to both civic and philanthropy.  Althought they are mission driven, we certainly witnessed this problem around climate change about a decade ago when museums and zoos were explicitly instructed by their boards to avoid the issue. 

There are three excellent references that are pretty much the go to illustration of this challenge:

The first, a sociological review of power dynamics that lead to self-censorship: 

Gray, G. C., & Bishop, K. V. (2009). Organizational Self‐Censorship: Corporate Sponsorship, Nonprofit Funding, and the Educational Experience. Canadian Review of Sociology/Revue canadienne de sociologie, 46(2), 161-177.

A historical review of a public museum whose director and leadership were ousted by the government when the work was supporting anti-government gentrification.  I think of this as the cautionary tale.

Mahony, E. (2016). The Uneasy Relationship of Self‐Critique in the Public Art Institution. Curator: The Museum Journal, 59(3), 219-238.

And lastly, a great book on curatorial activism.

Reilly, M., & Lippard, L. R. (2018). Curatorial activism: Towards an ethics of curating. London: Thames & Hudson.

I would say that this is a common concern in the practice area, but we seldom see it rise to the surface in the peer-reviewed literature. there may be more in the dissertation research since that fell outside the frame of our focus on actual engagement rather than the philosophy of decision-making.

James, 

Thanks for all the interesting stuff. Knology is working across the board on a lot of innovative ways that learning is happening in our world. We're dedicated to providing practical social science for a better world.  You'll see elsewhere in this showcase, our work with teens: https://bit.ly/3uM3SaY   Neurodiversity as an area for social justice in STEM learning https://bit.ly/3w1o890  and our meaningful math https://bit.ly/3eDUS22 with PBS NewsHour.  

We're also members of CLEAN, ESIP, and the AEA Environmental TIG where Rupu Gupta has been an active advocate for culturally responsive evaluation. 

We helped structure and administer the 5 year grant program that Underwriters Laboratories and NAAEE led to award excellence in demonstrated innovation with environment as a pathway to STEM https://ulinnovativeeducation.com/ 

We're the editorial office for Curator: The Museum Journal [curatorjournal.org]. And as a journal, we believe in an expansive definition including open air museums, parks and any structured experience can support musing on what we can do to help make our world better. 

And we're the researchers behind treating the city as a living museum where art and science co-mingle to reveal truths about our ecological lifeworlds https://www.cityaslivinglab.org/streamlinesorig...

Indeed, we'd love to do more with festivals.  We support Film Independent's Global Media Makers program but definitely would love to engage more with studying how the festival circuit is using time dependent events to drive ideas forward in society.

And to answer your first question last:

Beyond this film, we have three (maybe four) papers in various states of completion as the grant winds down this month, and preliminary results posters from AAM and ASTC.  Within the next month, all of our public resources and our final report will be available for download from our website and InformalScience.org, our NSDL pages will make data available for others to build on our work, and as the peer-reviewed papers are published, they'll be available through the NSF repository and by links to the publishers on our website. But that said, most folks that don't have access to the journals through their own subscriptions can often find them through public libraries if they reach out to their local librarian for help. 

Thanks for the shout out about all of our work. Indeed, we can only work for the public good if we share our toys.