This presentation summarizes results of a two-year effort to synthesize two decades of literature that addresses how informal learning institutions advancing the use of STEM knowledge and scientific reasoning to support engagement with the social issues of our time. It will present an overview of what the literature claims about these practices, and what has been demonstrated. To support users, it will illustrate the central definition of societal challenges based on research around the public understanding of social problems, the approach the team used to develop a data corpus for analysis, and a series of recommendations for informal learning researchers and practitioners to help move the field forward. The video will conclude with a set of links for further reading to support the migration of this knowledge into practice.
Research / Evaluation, Broadening Participation, Informal Learning
Knology
Advancing Informal STEM Learning (AISL)
Related Content for Research on Museums, STEM & Social Issues
Discussion from the 2021 STEM For All Video Showcase (28 posts)
Kris Morrissey
Co-Presenter Researcher
May 10, 2021 | 04:10 p.m.
Greetings and thanks for viewing our video. During this project, we had many provocative discussions about the unique opportunities for the ISL field to engage the public in understanding and addressing important social problems. We look forward to continuing those conversations with you.
I look forward to exploring the showcase and learning about all your projects and seeing the many connections in the work that excites and challenges us.
Stephen Uzzo
Facilitator Chief Scientist
May 11, 2021 | 08:46 a.m.
Thank you John and Team!
Kris This work is more relevant now than ever. I wonder if there has been thought about some kind of framework that can help stitch these kinds of studies into a more generalizable approach that can help ISLs find ways to help their visitors integrate them into their thinking and decision-making about social issues. I know its daunting to think about all the different issues, but perhaps starting with a small related group?
Kris Morrissey
Co-Presenter Researcher
May 11, 2021 | 09:19 a.m.
Yes! Great question. I have been playing with that question for awhile and have some rough scribbles that approximates a matrix. Not much further that that but the dimensions probably relate to (1) goals for engaging people (understanding the problem, sharing experiences or thoughts about the problem, understanding different perspectives, exploring solutions, etc), (2) range of public/audience opinions (divisive, segmented, emerging, coalescing (3) types of experiences (exhibits, one time or long term , dialogue based), (4) nature of the problem. The idea would be chart or set of charts that could help determine the pedagogy pedagogical approach and types of intervention most likely to work for specific types of audiences. Is that what you're suggesting? Of course it wouldn't be prescriptive but more in line with providing heuristics.
John Fraser
Lead Presenter President & CEO
May 11, 2021 | 09:40 a.m.
Billy and Steve, (and thanks Kris).
Adding to Kris' comments, we think that the first issue is a definitional problem. There is a tendency to jump right to the issue or problem and the debate, but seldom do we see a clear unpacking of how a social problem is constituted in general, how terms are defined structurally and for who. There are often useful resources that characterize audiences, but it's important to structure the dimensions of the claim, and then the difference between who is experiencing a museum exhibit or program versus their social role in the problem. ie: Are we preaching to the converted or the disengaged.
We highlight Joel Best's work in our video and papers because that work can help structure how to set up reporting on an intervention in ways that contextualize the users who will experience an offering.
I advocate for using public data and measures as some form of indicator variable, whether a qualitative or quantitative study, as a way of describing the participant in their social context.
As Billy knows, this was the approach we used for benchmarking the difference between aquarium and zoo visitors and the general population for the National Network for Ocean and Climate Change Interpretation (NNOCCI.org). In that case, we discovered that these visitors were significantly (but magnitude was not massive) a bit more aware, interested, and more likely to vote and advocate for others to vote on environmental issues. ie: We had socially active visitors who would be willing to engage others in the problem. So our data focused both on learning, and willingness to engage others in deliberating about the problem. ie: activation of latent interest.
I really do want to shout out to this citation in our video:
Ostman, R., Zirulnik, M. L., & McCullough Cosgrove, J. (2019). Storytelling, Science, and Religion: Promoting Reflection and Conversation about Societal Issues. Curator: The Museum Journal, 62(2), 117-134.
That team does a lovely job of describing a principle of social problems they were working on, and the need for more theoretical integration. They offer a set of principles that are worthy of testing by others. I believe four of the six are pretty portable.
Thanks all for the great questions.
1
Discussion is closed. Upvoting is no longer available
Found helpful:
Emily Yam
Rae Ostman
Higher Ed Faculty
May 14, 2021 | 07:30 a.m.
Thanks, John and Kris, for this video summarizing your work. I'm looking forward to the final study -- I hope it encourages more integration and collaboration among people doing this work. I also thank you for calling out our article!
Billy Spitzer
Facilitator PI
May 11, 2021 | 09:07 a.m.
Thanks Kris, Johnny and the Knology team for framing the research issues at the intersection of STEM and social issues. I was really intrigued by the range of social issues that have been explored, ranging from race to environment to health, and your efforts to think about how to create common methods and benchmarked measures across studies to assess the long-term impacts of these projects. Your analogy comparing the current array of disparate approaches to the study of "rare diseases" was so helpful in framing the challenge. I was wondering if you had any thoughts on what some of these common measures could be, or what a process might be for working to develop them?
2
Discussion is closed. Upvoting is no longer available
Found helpful:
Emily Yam
John Fraser
Kris Morrissey
Co-Presenter Researcher
May 11, 2021 | 09:23 a.m.
Great question Billy. I think Johnny has more thoughts about this . We've had provocative discussions about the question of common measures as well as the need for developing alternative and flexible metrics for measuring more nuance and societal level changes. I look forward to hearing others' thoughts.
Billy Spitzer
Facilitator PI
May 11, 2021 | 10:20 a.m.
Thanks Kris and Johnny for your thoughts on this question of common measures and methods, and on the importance of understanding the nature of the audience in relation to the social issue being studied. It seems like this domain of research could offer a whole new way to conceptualize what is meant by "broader impacts."
Sherry Hsi
Researcher
May 12, 2021 | 02:59 p.m.
I can envision common methods, but interested to learn what you might have in mind for a benchmarked measure that would be both interesting and implementable across longitudinal studies in an ideal situation. Has this been done in the area of climate change or other forms of literacy at scale? So interesting. In classroom studies, researchers have studied knowledge integration and beliefs about science of middle school students into high school, but we know how to track and find these students (in the K12 same district.)
Hi Johnny and team- thank you for a wonderful post and the recommendations. I think it's helpful to have some things to think about as we move forward from this moment- zoos, aquariums, and museums are really reconsidering our roles in our community, and acknowledging how intersectionality impacts us all. What responsibility do we have? And how do we measure our work? This is a great starting place to think about those common metrics.
Kris Morrissey
Co-Presenter Researcher
May 12, 2021 | 11:22 a.m.
I agree that this is the time to rethink our role within and as part of society! And if our work is truly integrated with other sectors of society, it may become more difficult to attribute change to the actions of any one sector or institution. Which means measuring the success of our work may look different. In our review, we did see a number of creative and participatory evaluation efforts that might begin to speak to new approaches to measuring or documenting 'broader impacts" in the context of intersectionality.
Nickolay Hristov
Facilitator Senior Scientist, Director, Associate Professor
May 12, 2021 | 11:58 a.m.
Greetings to the entire Knology Team! I am so pleased to see a larger, systems-thinking approach to your work that gives a practical meaning to the data-information-knowledge-action framework that so often gets fragmented. You might find additional inspiration and ideas from the iSWOOP initiative to bring more science interpretation to the nation’s national parks (you can see past project submission to the showcase in 2015-2019 and at www.iswoopparks.com). That team, spent a good part of a decade highlighting US National Parks as outdoor laboratories and terrific sites for STEM learning for the more than 300 million annual visitors. Much remains to be done to make these visitors representative of the larger society fabric and herein lies an opportunity that you are already thinking about in the context of zoos, aquariums and museums.
I am also eager to point out how lovely I found the visual presentation of your work. This is a dense and intense content, made playfully accessible by careful attention to visual design and multimodal communication. Anecdotes and metaphors help the interpretation and understanding further. I encourage your design and editing/storytelling team to continue in that direction and I am happy to share ideas. Also kudos for the Joel Best cameo. Clever! And effective!
John Fraser
Lead Presenter President & CEO
May 12, 2021 | 03:41 p.m.
Thanks so much for appreciating our work. Your parks work sounds familiar and I think we may have already been using some of your references in other work. I'd love to connect you with Rupu Gupta, a conservation psychologist on our team who leads our work on culturally responsive evaluation. She's done a number of projects in collaboration with groups like Outdoor Afro to think about perspectives and values related to park access and use. Oh yes, and Joel was really generous with his time for his brief cameo. His book is definitely a must read for this work. Offline, I'll send an introduction to put you together with Rupu.
Nickolay Hristov
Facilitator Senior Scientist, Director, Associate Professor
May 17, 2021 | 01:20 a.m.
John, I would very much appreciate the introduction to Rupu, thank you! I am eager to be part of these conversations.
John Fraser
Lead Presenter President & CEO
May 17, 2021 | 10:59 a.m.
Nickolay, I sent you a connection request via LinkedIn because your email addresses on your website appear not to be working. I'd rather not share emails on this platform.
Nickolay Hristov
Facilitator Senior Scientist, Director, Associate Professor
May 18, 2021 | 07:17 p.m.
I just accepted your invitation request, John! Thank you! I am looking forward to continuing the conversations.
Sherry Hsi
Researcher
May 12, 2021 | 02:51 p.m.
Thanks for sharing your video and work in this forum. I am curious how the grey literature such as unpublished yet high quality evaluation reports figured into the study and the 200 pieces of literature. It is hard to do empirical studies in museums and get them published given the wide variation of experiences, values, and cultural practices found in each place with different lengths of engagement and participation as others have mentioned already, so it was not surprising to see that there were more case studies that studies with quantitative measures. I share Kris's perspective that fostering creative and participatory evaluation efforts could be useful both for measuring and documenting changes.
1
Discussion is closed. Upvoting is no longer available
Found helpful:
Elysa Corin
John Fraser
Lead Presenter President & CEO
May 12, 2021 | 03:00 p.m.
Sherry, Thanks for your thoughts too! We used only published grey literature found on the InformalScience.org website. Most of the journals also publish qualitative research and small sample studies. For example, Curator: The Museum Journal, (I'm the editor) has no problem with qualitative data as long as the methods are well explained and the sample sizes match recommendations for those methods. In fact, we had many discussions about variables in the data and the value of case study. in fact, we don't think it's all about massive datasets, but rather, working to build connections across small sample projects. Like rare disease, we need common points of connection so we can make sense of many cases when they are all put together. One point I'd hoped to see come through in this video is the idea that we can learn more transferrable information if the project teams work to link to principles of social problems, not just the specifics of the case. Thanks so much for your comments.
Kris Morrissey
Co-Presenter Researcher
May 12, 2021 | 08:02 p.m.
Hi Sherry, Luckily, Doctoral Dissertations and Master's Thesis (which came from dozens of disciplines and universities) were more likely to look at institutions, exhibits, or practices that might not show up in other literature which helped bridge the gap somewhat. For example, one study looked at ways small museums are addressing social problems. Another studied dialogue programs across a number of institutions. One studied museums that had exhibits or programs about incarceration. And many of them included visitor studies or interviews with professionals at the more well-known large urban museums but came to the research with a different lens, such as a communication and rhetoric study and a child development study that both looked at the Race: Are We So Different? exhibit. I was also impressed with how many of those studies applied non-traditional methodologies. So we hope that the three sources of literature gave us a representative view of the types of activities taking place across the field, although we recognize the significant limitations of the fact that we weren't studying practices, we were studying what others said about practice. It was fun in a weird sort of way.
I'd love to hear more about your thoughts around innovative ways to document or measure change? I think with so much of the work tied to funding sources that look for evidence of impact, we need to find a ways to be accountable, organic, and sensitive.
Stephen Uzzo
Facilitator Chief Scientist
May 13, 2021 | 07:53 a.m.
Kris –
Some of the discussion reminds me of the problem of social capital I have been noodling about for a long time. Finding ways to measure and account for specific aspects of social capital in fostering retention or interest in STEM has been elusive. We employ a number of broad measures based primarily on SCCT (but other approaches as well) to see how student thinking changes over time. But in the “wild” it can be a very different story for each subject. Their ever-evolving ontologies seem to imply that at each point in time you can be measuring a different person and it is hard to track. We do a lot of grounded research as a result, which is really laborious, but at least gives us a sense of impact along with valuable information about each subject and how their thinking changes. Generalizing conclusions, that’s a whole other matter.
This helps bring clarity to the role STEM has/has not played in helping museum etc visitors understand social issues. While it seems there's a huge role science-based data could play in helping people understand such issues, I wonder how many museums/zoos feel constrained by the communities they are in and avoid anything that could be deemed controversial.
John Fraser
Lead Presenter President & CEO
May 15, 2021 | 01:22 p.m.
Patti,
We didn't find any substantial assessments of museum self-editing, but I do think it's fairly well known anecdotally. The greatest challenge these institutions face is the politics of board and management's fealty to both civic and philanthropy. Althought they are mission driven, we certainly witnessed this problem around climate change about a decade ago when museums and zoos were explicitly instructed by their boards to avoid the issue.
There are three excellent references that are pretty much the go to illustration of this challenge:
The first, a sociological review of power dynamics that lead to self-censorship:
Gray, G. C., & Bishop, K. V. (2009). Organizational Self‐Censorship: Corporate Sponsorship, Nonprofit Funding, and the Educational Experience. Canadian Review of Sociology/Revue canadienne de sociologie, 46(2), 161-177.
A historical review of a public museum whose director and leadership were ousted by the government when the work was supporting anti-government gentrification. I think of this as the cautionary tale.
Mahony, E. (2016). The Uneasy Relationship of Self‐Critique in the Public Art Institution. Curator: The Museum Journal, 59(3), 219-238.
And lastly, a great book on curatorial activism.
Reilly, M., & Lippard, L. R. (2018). Curatorial activism: Towards an ethics of curating. London: Thames & Hudson.
I would say that this is a common concern in the practice area, but we seldom see it rise to the surface in the peer-reviewed literature. there may be more in the dissertation research since that fell outside the frame of our focus on actual engagement rather than the philosophy of decision-making.
Thanks Patti for the comments and Johnny for the great references. There were indeed a few studies in the dissertation research that looked at visitor reactions when museums took a stand and a few of the evaluations looked at how visitors responded to messages, for example about conservation or other topics. While the results were mixed, in general visitors seemed OK and sometimes welcomed or expected strong messages that were aligned with the mission of the institutions (e.g. zoos talking about biodiversity, aquarium about oceans). The reactions weren't so positive when the messages were perceived as "preachy" or opportunistic.
I suspect the public is ready for museums to have a stronger voice but in the context of authenticity and transparency, topics for further conversations no doubt.
Laura Santhanam
Health Reporter & Coordinating Producer for Polling
This is such wonderful look at the literature landscape. Thank you! I was struck by how the premise responds to the urgency of this moment. It will be interesting to see how STEM studies are centered on social justice issues going forward.
Thank you, Knology team, for your important work and informative and enjoyable video.
Question: Has your team studied or published on this subject, or matters close to it?
There are now many well established informal learning genres and types of institutions. Museums and science centers are traditional of course. Now also, some newer forms: science festivals, communities STEM events, field trip hubs and a wide variety of creative approaches.
Part of the attractiveness of science festivals, for instance, is that they are free for the public to attend; and generally put a great deal of emphasis on drawing and engaging families from disadvantaged communities and communities of people of color.
While our program, now known as the Mobile Climate Science Labs, began in the science museum in the Bay Area twenty years ago, we find ourselves more at home now out in the communities. Especially that our team is quite ethnically diverse. The science festivals have been putting a strong emphasis on social justice and diversity for at least a decade. Very welcoming to educators who are people of color.
Have their been studies comparing the experiences of the different genres of informal education and their approach to social justice and ethnic diversity of educators?
In the CLEAN Network, in which we are members, discussion on social justice and how to advance in diversity has been a rich and productive topic. We all always learn a great deal form Knology studies. Would certainly want to recommend what you publish on matters such as these, as we recommend this video in the Showcase.
Thank you!
John Fraser
Lead Presenter President & CEO
May 18, 2021 | 09:57 a.m.
James,
Thanks for all the interesting stuff. Knology is working across the board on a lot of innovative ways that learning is happening in our world. We're dedicated to providing practical social science for a better world. You'll see elsewhere in this showcase, our work with teens: https://bit.ly/3uM3SaY Neurodiversity as an area for social justice in STEM learning https://bit.ly/3w1o890 and our meaningful math https://bit.ly/3eDUS22 with PBS NewsHour.
We're also members of CLEAN, ESIP, and the AEA Environmental TIG where Rupu Gupta has been an active advocate for culturally responsive evaluation.
We helped structure and administer the 5 year grant program that Underwriters Laboratories and NAAEE led to award excellence in demonstrated innovation with environment as a pathway to STEM https://ulinnovativeeducation.com/
We're the editorial office for Curator: The Museum Journal [curatorjournal.org]. And as a journal, we believe in an expansive definition including open air museums, parks and any structured experience can support musing on what we can do to help make our world better.
And we're the researchers behind treating the city as a living museum where art and science co-mingle to reveal truths about our ecological lifeworlds https://www.cityaslivinglab.org/streamlinesorig...
Indeed, we'd love to do more with festivals. We support Film Independent's Global Media Makers program but definitely would love to engage more with studying how the festival circuit is using time dependent events to drive ideas forward in society.
And to answer your first question last:
Beyond this film, we have three (maybe four) papers in various states of completion as the grant winds down this month, and preliminary results posters from AAM and ASTC. Within the next month, all of our public resources and our final report will be available for download from our website and InformalScience.org, our NSDL pages will make data available for others to build on our work, and as the peer-reviewed papers are published, they'll be available through the NSF repository and by links to the publishers on our website. But that said, most folks that don't have access to the journals through their own subscriptions can often find them through public libraries if they reach out to their local librarian for help.
Thanks for the shout out about all of our work. Indeed, we can only work for the public good if we share our toys.
1
Discussion is closed. Upvoting is no longer available
Kris Morrissey
Researcher
Greetings and thanks for viewing our video. During this project, we had many provocative discussions about the unique opportunities for the ISL field to engage the public in understanding and addressing important social problems. We look forward to continuing those conversations with you.
I look forward to exploring the showcase and learning about all your projects and seeing the many connections in the work that excites and challenges us.
Stephen Uzzo
Chief Scientist
Thank you John and Team!
Kris This work is more relevant now than ever. I wonder if there has been thought about some kind of framework that can help stitch these kinds of studies into a more generalizable approach that can help ISLs find ways to help their visitors integrate them into their thinking and decision-making about social issues. I know its daunting to think about all the different issues, but perhaps starting with a small related group?
Kris Morrissey
Researcher
Yes! Great question. I have been playing with that question for awhile and have some rough scribbles that approximates a matrix. Not much further that that but the dimensions probably relate to (1) goals for engaging people (understanding the problem, sharing experiences or thoughts about the problem, understanding different perspectives, exploring solutions, etc), (2) range of public/audience opinions (divisive, segmented, emerging, coalescing (3) types of experiences (exhibits, one time or long term , dialogue based), (4) nature of the problem. The idea would be chart or set of charts that could help determine the pedagogy pedagogical approach and types of intervention most likely to work for specific types of audiences. Is that what you're suggesting? Of course it wouldn't be prescriptive but more in line with providing heuristics.
John Fraser
President & CEO
Billy and Steve, (and thanks Kris).
Adding to Kris' comments, we think that the first issue is a definitional problem. There is a tendency to jump right to the issue or problem and the debate, but seldom do we see a clear unpacking of how a social problem is constituted in general, how terms are defined structurally and for who. There are often useful resources that characterize audiences, but it's important to structure the dimensions of the claim, and then the difference between who is experiencing a museum exhibit or program versus their social role in the problem. ie: Are we preaching to the converted or the disengaged.
We highlight Joel Best's work in our video and papers because that work can help structure how to set up reporting on an intervention in ways that contextualize the users who will experience an offering.
I advocate for using public data and measures as some form of indicator variable, whether a qualitative or quantitative study, as a way of describing the participant in their social context.
As Billy knows, this was the approach we used for benchmarking the difference between aquarium and zoo visitors and the general population for the National Network for Ocean and Climate Change Interpretation (NNOCCI.org). In that case, we discovered that these visitors were significantly (but magnitude was not massive) a bit more aware, interested, and more likely to vote and advocate for others to vote on environmental issues. ie: We had socially active visitors who would be willing to engage others in the problem. So our data focused both on learning, and willingness to engage others in deliberating about the problem. ie: activation of latent interest.
I really do want to shout out to this citation in our video:
Ostman, R., Zirulnik, M. L., & McCullough Cosgrove, J. (2019). Storytelling, Science, and Religion: Promoting Reflection and Conversation about Societal Issues. Curator: The Museum Journal, 62(2), 117-134.
That team does a lovely job of describing a principle of social problems they were working on, and the need for more theoretical integration. They offer a set of principles that are worthy of testing by others. I believe four of the six are pretty portable.
Thanks all for the great questions.
Emily Yam
Rae Ostman
Thanks, John and Kris, for this video summarizing your work. I'm looking forward to the final study -- I hope it encourages more integration and collaboration among people doing this work. I also thank you for calling out our article!
Billy Spitzer
PI
Thanks Kris, Johnny and the Knology team for framing the research issues at the intersection of STEM and social issues. I was really intrigued by the range of social issues that have been explored, ranging from race to environment to health, and your efforts to think about how to create common methods and benchmarked measures across studies to assess the long-term impacts of these projects. Your analogy comparing the current array of disparate approaches to the study of "rare diseases" was so helpful in framing the challenge. I was wondering if you had any thoughts on what some of these common measures could be, or what a process might be for working to develop them?
Emily Yam
John Fraser
Kris Morrissey
Researcher
Great question Billy. I think Johnny has more thoughts about this . We've had provocative discussions about the question of common measures as well as the need for developing alternative and flexible metrics for measuring more nuance and societal level changes. I look forward to hearing others' thoughts.
Billy Spitzer
PI
Thanks Kris and Johnny for your thoughts on this question of common measures and methods, and on the importance of understanding the nature of the audience in relation to the social issue being studied. It seems like this domain of research could offer a whole new way to conceptualize what is meant by "broader impacts."
Sherry Hsi
I can envision common methods, but interested to learn what you might have in mind for a benchmarked measure that would be both interesting and implementable across longitudinal studies in an ideal situation. Has this been done in the area of climate change or other forms of literacy at scale? So interesting. In classroom studies, researchers have studied knowledge integration and beliefs about science of middle school students into high school, but we know how to track and find these students (in the K12 same district.)
Emily Yam
Hi Johnny and team- thank you for a wonderful post and the recommendations. I think it's helpful to have some things to think about as we move forward from this moment- zoos, aquariums, and museums are really reconsidering our roles in our community, and acknowledging how intersectionality impacts us all. What responsibility do we have? And how do we measure our work? This is a great starting place to think about those common metrics.
Kris Morrissey
Researcher
I agree that this is the time to rethink our role within and as part of society! And if our work is truly integrated with other sectors of society, it may become more difficult to attribute change to the actions of any one sector or institution. Which means measuring the success of our work may look different. In our review, we did see a number of creative and participatory evaluation efforts that might begin to speak to new approaches to measuring or documenting 'broader impacts" in the context of intersectionality.
Nickolay Hristov
Senior Scientist, Director, Associate Professor
Greetings to the entire Knology Team! I am so pleased to see a larger, systems-thinking approach to your work that gives a practical meaning to the data-information-knowledge-action framework that so often gets fragmented. You might find additional inspiration and ideas from the iSWOOP initiative to bring more science interpretation to the nation’s national parks (you can see past project submission to the showcase in 2015-2019 and at www.iswoopparks.com). That team, spent a good part of a decade highlighting US National Parks as outdoor laboratories and terrific sites for STEM learning for the more than 300 million annual visitors. Much remains to be done to make these visitors representative of the larger society fabric and herein lies an opportunity that you are already thinking about in the context of zoos, aquariums and museums.
I am also eager to point out how lovely I found the visual presentation of your work. This is a dense and intense content, made playfully accessible by careful attention to visual design and multimodal communication. Anecdotes and metaphors help the interpretation and understanding further. I encourage your design and editing/storytelling team to continue in that direction and I am happy to share ideas. Also kudos for the Joel Best cameo. Clever! And effective!
John Fraser
President & CEO
Thanks so much for appreciating our work. Your parks work sounds familiar and I think we may have already been using some of your references in other work. I'd love to connect you with Rupu Gupta, a conservation psychologist on our team who leads our work on culturally responsive evaluation. She's done a number of projects in collaboration with groups like Outdoor Afro to think about perspectives and values related to park access and use. Oh yes, and Joel was really generous with his time for his brief cameo. His book is definitely a must read for this work. Offline, I'll send an introduction to put you together with Rupu.
Nickolay Hristov
Senior Scientist, Director, Associate Professor
John, I would very much appreciate the introduction to Rupu, thank you! I am eager to be part of these conversations.
John Fraser
President & CEO
Nickolay, I sent you a connection request via LinkedIn because your email addresses on your website appear not to be working. I'd rather not share emails on this platform.
Nickolay Hristov
Senior Scientist, Director, Associate Professor
I just accepted your invitation request, John! Thank you! I am looking forward to continuing the conversations.
Sherry Hsi
Thanks for sharing your video and work in this forum. I am curious how the grey literature such as unpublished yet high quality evaluation reports figured into the study and the 200 pieces of literature. It is hard to do empirical studies in museums and get them published given the wide variation of experiences, values, and cultural practices found in each place with different lengths of engagement and participation as others have mentioned already, so it was not surprising to see that there were more case studies that studies with quantitative measures. I share Kris's perspective that fostering creative and participatory evaluation efforts could be useful both for measuring and documenting changes.
Elysa Corin
John Fraser
President & CEO
Sherry, Thanks for your thoughts too! We used only published grey literature found on the InformalScience.org website. Most of the journals also publish qualitative research and small sample studies. For example, Curator: The Museum Journal, (I'm the editor) has no problem with qualitative data as long as the methods are well explained and the sample sizes match recommendations for those methods. In fact, we had many discussions about variables in the data and the value of case study. in fact, we don't think it's all about massive datasets, but rather, working to build connections across small sample projects. Like rare disease, we need common points of connection so we can make sense of many cases when they are all put together. One point I'd hoped to see come through in this video is the idea that we can learn more transferrable information if the project teams work to link to principles of social problems, not just the specifics of the case. Thanks so much for your comments.
Kris Morrissey
Researcher
Hi Sherry, Luckily, Doctoral Dissertations and Master's Thesis (which came from dozens of disciplines and universities) were more likely to look at institutions, exhibits, or practices that might not show up in other literature which helped bridge the gap somewhat. For example, one study looked at ways small museums are addressing social problems. Another studied dialogue programs across a number of institutions. One studied museums that had exhibits or programs about incarceration. And many of them included visitor studies or interviews with professionals at the more well-known large urban museums but came to the research with a different lens, such as a communication and rhetoric study and a child development study that both looked at the Race: Are We So Different? exhibit. I was also impressed with how many of those studies applied non-traditional methodologies. So we hope that the three sources of literature gave us a representative view of the types of activities taking place across the field, although we recognize the significant limitations of the fact that we weren't studying practices, we were studying what others said about practice. It was fun in a weird sort of way.
I'd love to hear more about your thoughts around innovative ways to document or measure change? I think with so much of the work tied to funding sources that look for evidence of impact, we need to find a ways to be accountable, organic, and sensitive.
Stephen Uzzo
Chief Scientist
Kris –
Some of the discussion reminds me of the problem of social capital I have been noodling about for a long time. Finding ways to measure and account for specific aspects of social capital in fostering retention or interest in STEM has been elusive. We employ a number of broad measures based primarily on SCCT (but other approaches as well) to see how student thinking changes over time. But in the “wild” it can be a very different story for each subject. Their ever-evolving ontologies seem to imply that at each point in time you can be measuring a different person and it is hard to track. We do a lot of grounded research as a result, which is really laborious, but at least gives us a sense of impact along with valuable information about each subject and how their thinking changes. Generalizing conclusions, that’s a whole other matter.
Patti Parson
This helps bring clarity to the role STEM has/has not played in helping museum etc visitors understand social issues. While it seems there's a huge role science-based data could play in helping people understand such issues, I wonder how many museums/zoos feel constrained by the communities they are in and avoid anything that could be deemed controversial.
John Fraser
President & CEO
Patti,
We didn't find any substantial assessments of museum self-editing, but I do think it's fairly well known anecdotally. The greatest challenge these institutions face is the politics of board and management's fealty to both civic and philanthropy. Althought they are mission driven, we certainly witnessed this problem around climate change about a decade ago when museums and zoos were explicitly instructed by their boards to avoid the issue.
There are three excellent references that are pretty much the go to illustration of this challenge:
The first, a sociological review of power dynamics that lead to self-censorship:
Gray, G. C., & Bishop, K. V. (2009). Organizational Self‐Censorship: Corporate Sponsorship, Nonprofit Funding, and the Educational Experience. Canadian Review of Sociology/Revue canadienne de sociologie, 46(2), 161-177.
A historical review of a public museum whose director and leadership were ousted by the government when the work was supporting anti-government gentrification. I think of this as the cautionary tale.
Mahony, E. (2016). The Uneasy Relationship of Self‐Critique in the Public Art Institution. Curator: The Museum Journal, 59(3), 219-238.
And lastly, a great book on curatorial activism.
Reilly, M., & Lippard, L. R. (2018). Curatorial activism: Towards an ethics of curating. London: Thames & Hudson.
I would say that this is a common concern in the practice area, but we seldom see it rise to the surface in the peer-reviewed literature. there may be more in the dissertation research since that fell outside the frame of our focus on actual engagement rather than the philosophy of decision-making.
Patti Parson
Thanks, John--very interesting!
Kris Morrissey
Researcher
Thanks Patti for the comments and Johnny for the great references. There were indeed a few studies in the dissertation research that looked at visitor reactions when museums took a stand and a few of the evaluations looked at how visitors responded to messages, for example about conservation or other topics. While the results were mixed, in general visitors seemed OK and sometimes welcomed or expected strong messages that were aligned with the mission of the institutions (e.g. zoos talking about biodiversity, aquarium about oceans). The reactions weren't so positive when the messages were perceived as "preachy" or opportunistic.
I suspect the public is ready for museums to have a stronger voice but in the context of authenticity and transparency, topics for further conversations no doubt.
Laura Santhanam
This is such wonderful look at the literature landscape. Thank you! I was struck by how the premise responds to the urgency of this moment. It will be interesting to see how STEM studies are centered on social justice issues going forward.
John Fraser
President & CEO
Thanks so much Laura
James Callahan
Thank you, Knology team, for your important work and informative and enjoyable video.
Question: Has your team studied or published on this subject, or matters close to it?
There are now many well established informal learning genres and types of institutions. Museums and science centers are traditional of course. Now also, some newer forms: science festivals, communities STEM events, field trip hubs and a wide variety of creative approaches.
Part of the attractiveness of science festivals, for instance, is that they are free for the public to attend; and generally put a great deal of emphasis on drawing and engaging families from disadvantaged communities and communities of people of color.
While our program, now known as the Mobile Climate Science Labs, began in the science museum in the Bay Area twenty years ago, we find ourselves more at home now out in the communities. Especially that our team is quite ethnically diverse. The science festivals have been putting a strong emphasis on social justice and diversity for at least a decade. Very welcoming to educators who are people of color.
Have their been studies comparing the experiences of the different genres of informal education and their approach to social justice and ethnic diversity of educators?
In the CLEAN Network, in which we are members, discussion on social justice and how to advance in diversity has been a rich and productive topic. We all always learn a great deal form Knology studies. Would certainly want to recommend what you publish on matters such as these, as we recommend this video in the Showcase.
Thank you!
John Fraser
President & CEO
James,
Thanks for all the interesting stuff. Knology is working across the board on a lot of innovative ways that learning is happening in our world. We're dedicated to providing practical social science for a better world. You'll see elsewhere in this showcase, our work with teens: https://bit.ly/3uM3SaY Neurodiversity as an area for social justice in STEM learning https://bit.ly/3w1o890 and our meaningful math https://bit.ly/3eDUS22 with PBS NewsHour.
We're also members of CLEAN, ESIP, and the AEA Environmental TIG where Rupu Gupta has been an active advocate for culturally responsive evaluation.
We helped structure and administer the 5 year grant program that Underwriters Laboratories and NAAEE led to award excellence in demonstrated innovation with environment as a pathway to STEM https://ulinnovativeeducation.com/
We're the editorial office for Curator: The Museum Journal [curatorjournal.org]. And as a journal, we believe in an expansive definition including open air museums, parks and any structured experience can support musing on what we can do to help make our world better.
And we're the researchers behind treating the city as a living museum where art and science co-mingle to reveal truths about our ecological lifeworlds https://www.cityaslivinglab.org/streamlinesorig...
Indeed, we'd love to do more with festivals. We support Film Independent's Global Media Makers program but definitely would love to engage more with studying how the festival circuit is using time dependent events to drive ideas forward in society.
And to answer your first question last:
Beyond this film, we have three (maybe four) papers in various states of completion as the grant winds down this month, and preliminary results posters from AAM and ASTC. Within the next month, all of our public resources and our final report will be available for download from our website and InformalScience.org, our NSDL pages will make data available for others to build on our work, and as the peer-reviewed papers are published, they'll be available through the NSF repository and by links to the publishers on our website. But that said, most folks that don't have access to the journals through their own subscriptions can often find them through public libraries if they reach out to their local librarian for help.
Thanks for the shout out about all of our work. Indeed, we can only work for the public good if we share our toys.
Nickolay Hristov
Further posting is closed as the event has ended.