Using web-based, personalized units, STRIDES is empowering teachers and students to take action to promote anti-racism. In a collaboration between UC Berkeley and ETS, STRIDES uses Natural Language Processing (NLP) to capture students’ evolving understanding of Next Generation Science Standards (NGSS) three-dimensional learning. STRIDES generates milestone reports that pair distributions of student responses with evidence based instructional opportunities. The reports support teachers to notice each student's ideas in their classroom. Teachers can use the report to customize instruction for their students. This video illustrates how teachers make customization decisions to enable each student to plan actions promoting anti-racism.
Science, Emerging Technologies, Addressing NGSS
UC Berkeley, ETS
Discovery Research PreK-12 (DRK-12)
Related Content for Web-based Personalized Science Units Empowering Anti-Racism
Hello Berkeley team! Thank you for your video, this is such a great way to customize a WISE unit. I was wondering if as part of this work you have tried to measure students' shifts related to their knowledge about (or positions on) racial equity, and if you have found any relationships between this and their science learning. One might hypothesize these learning outcomes might be synergistic but I imagine it would not be straightforward to measure this.
Sarah Bichler
Co-Presenter Post-Doctoral Researcher
May 11, 2021 | 04:13 p.m.
Dear Kevin, thank you for viewing our project! We have just started looking at students' responses and are in the process of refining our assessment items. Reviewing students' responses, we learned that many students appreciated learning that not all people are impacted by climate change in the same way or how redlining has led to fewer green spaces in black and brown neighborhoods. We noticed that students explain the relationship between how close someone lives to a freeway and the likelihood they get asthma, so they are linking the science to impacts on peoples health. We also noticed that most students do not yet include ideas about racial equity in their explanations. We are curious to see if our revised items will elicit these ideas as well so we can learn how students link science and racial equity ideas. We also discussed that students may not be used to discussing such questions/issues in science class and that they may link their ideas now that these questions are brought up more.
Thanks for the question!
Sarah
Brian Riordan
Researcher
May 13, 2021 | 07:47 a.m.
Hi Kevin, I'm the ETS PI on this collaborative NSF grant; I lead the development of NLP methods for analysis of student responses. In this work we've also developed new NLP methods to detect student ideas at multiple levels of granularity to help support teachers to understand the breadth of student ideas and facilitate discussion. One of the foci of future work on revised items will be better capturing the diversity of student ideas at the intersection of science and social justice issues. We described an early version of the idea detection approach in a short technical paper, "Identifying NGSS-Aligned Ideas in Student Science Explanations".
What incredibly important and thoughtful work! I really appreciate the explicit integration of systemic racism into an existing science learning platform in such an authentic way. And in a way that stems from what the educators were interested in bringing into their conversations in the classroom. Such a powerful example of making data human-centered and personal, understanding the historical implications of the world we see today -- emphasizing the need for change in the future! Did you get feedback from teachers about how things went? Including what conversations came up?
Sarah Bichler
Co-Presenter Post-Doctoral Researcher
May 11, 2021 | 04:22 p.m.
Dear Caitlin, thank you so much for the feedback! Teachers who are running the units participated in a workshop in which we discussed ways to support students during conversations about racial equity in science class. Looking at student data, we get the impression that students really appreciate learning about issues like urban heat islands. Many of our teachers are still teaching distance. We are currently interviewing teachers - my colleagues will share more details about what we learn from teachers!
Best,
Sarah
Marcia Linn
Lead Presenter Professor
May 11, 2021 | 04:51 p.m.
Thank you for your comments and interest in our efforts to welcome each student to science. We are investigating ways to support teaches to explore social justice issues, consistent with many students' concerns about the environment, BLM, and personal health. We seek to transform the goals of science courses from primarily preparing students for careers in science, to ensuring that each student, independent of race, culture, sexual orientation, or economic status is welcomed in science. This means developing the identity of each student as a science learner and respecting the science accomplishments of each person. Please tell us how your work can help us advance this goal.
Hi Marcia, Thanks for sharing such a great piece of work! I'd love to have a follow up discussion on applying NLP-supported innovations in classrooms to support teacher practices. Also quite interested in integrating social justice issues into student science learning. - Lei
Marcia Linn
Lead Presenter Professor
May 12, 2021 | 01:28 p.m.
Great. We will coordinate with you and Brian to set up a meeting.
I'd love to talk with you all more about how you are using NLP. It looks really powerful and I'd love to see more. Thanks for this work as a model for how to bring in new, important topics in science to make them more relevant for students.
Marcia Linn
Lead Presenter Professor
May 12, 2021 | 01:45 p.m.
Hi Judi,
Thank you for your comment. We are partnering with ETS to use their innovations. We can send you copies of our research papers. Here is one reference.
Brian Riordan
Researcher
May 12, 2021 | 03:38 p.m.
Here are some links to recent technical papers on our work:
Dear Lei and Judi, thanks so much for showing interest in our work! We'd love to share more about how we use NLP and implement NLP innovations in classrooms. Here is some more info:
1. We designed "milestone items" for several inquiry units. These items are complex and students have to link multiple ideas to explain a phenomenon. For example, in our Genetics of Extinction unit, students are asked "Why do siblings look similar but not the same?". Students can use ideas about genetic inheritance and genetic variation (gathered in the unit) to explain.
2. We develop a rubric to assess Knowledge Integration and also code for NGSS 3-dimensional understanding (DCI, CCC, or SEP dependent on the item). For example, another milestone in the Genetics of Extinction unit ask students to use evidence from a graph (SEP) to explain how species adapt to their environment.
3. Brian Riordan at ETS trains an automated scoring model and then we implement it in the units. Students' responses are scored in real time and teachers can view a report that is generated from the automated scores.
4. The report shows score distributions for KI and NGSS subscores and features example student responses. The report also includes recommended instructional actions that promote KI and/or are tailored to the ideas present in the respective classroom. For example, if students score really high on explaining natural selection but do not yet include evidence from the graph, recommended actions suggest ways to support students to use evidence.
Our reports now also show how students progress when they revised their responses after a teacher's intervention. In addition, teachers can view initial and revised responses (and KI scores) for each individual student. This way, teachers can more easily help those students who need the most help.
Looking forward to talking more and learning about your ideas how we can improve this work or what other work we can build on!
The use of data in studying social injustices is really important, and you clearly have some rigorous research going on! It may be beyond the scope of your project (or too controversial) but how you look at changes in students' attitudes about issues like anti-racism or climate change? It might be interesting to use some common instruments across projects, and our project is in the early stages of research about youth's understanding of social issues related to COVID.
Marcia Linn
Lead Presenter Professor
May 12, 2021 | 01:49 p.m.
Hi Jan,
That is an excellent issue. We have had good success with questions that get at the controversy and connect to the science. We will be reporting on this work shortly. A question that is interesting concerns whether a particular scientific phenomena (like pollution) has equal or unequal impacts on each family. We are seeing big changes in students thinking about this issue. Take care, Marcia
Sarah Bichler
Co-Presenter Post-Doctoral Researcher
May 12, 2021 | 08:06 a.m.
Hi Jan!
Thanks so much for your feedback. It would be great to share resources and learning from other groups how we can best approach measuring change in attitudes towards and knowledge of anti-racism and/or racial equity in relation to climate change!
We are curious to learn more about your project and what items to measure understanding of social issues you are developing. We're also curious about your COVID instructional unit - we have designed a WISE unit about COVID and social equity too!
Dear Berkeley team! Wow, what a great presentation. It is so interesting to hear about two really important topics: helping teachers to see components of student responses related to the three dimensions and the integration of issues of social justice in the classroom. I am curious to know more about the teachers and their views on the place of social justice in science. While it seems like the teachers in the video were very keen to integrate the two, many science teachers shy away from these topics because they are unsure how to handle the discussion or questions that students may bring up. Did you all experience any of this resistance? Or how might you all help a teacher see the importance of bringing anti-racism, social justice into science? This is something I am very interested in and would love the thoughts of others working in the area.
Marcia Linn
Lead Presenter Professor
May 12, 2021 | 01:53 p.m.
Dear Kirby,
Hi Kirby, Thank you for the kind words. The issues you raise are important. We will be reporting on the elements of our professional development approach that facilitated discussions around introducing social justice into science. The formation of a Research Practice Partnership was definitely a key element for addressing concerns of teachers. Best, Marcia
Marcia, what a rich set of tools you have created! I'll want to look at them more. You showed a child at home working; are these materials primarily for remote or online learning, or are they accompanied by hands-on investigations? What is the balance? And I see a lot of text; how dependent are the materials on children's reading levels?
Marcia Linn
Lead Presenter Professor
May 12, 2021 | 01:57 p.m.
Dear Penny,
How great to hear from you. The units were introduced in science classes using remote instruction. The students discussed the issues in class and, of course, this often meant that the discussion extended into the home. We took advantage of interactive visualizations, concept maps, and graph tools to make sure that the reading was not a primary channel for learning. We also supported text to speech so students could hear the material if they preferred--one slight advantage of working remotely!!
We would love to set up a meeting with you to get your insights into ways we can strengthen the work.
Nice work....I am wondering if there were issues that were studied that were not only "urban" issues but also dealt directly with race? The two seem to get conflated sometimes....any ideas? thanks for the great work!
Marcia Linn
Lead Presenter Professor
May 12, 2021 | 02:04 p.m.
Hi Jodi,
This is a good point. Our activities enable students to use the design of graphs to sort out race in exploring topics such as global climate change or the incidence of Asthma or COVID. We agree that these issues require a way to work out the separate contribution of race and class. We would love to meet with you to follow up on these issues. Thank you, Marcia
Heidi Carlone
Facilitator Distinguished Professor
May 12, 2021 | 11:38 a.m.
Great work, team! This environment is a rich resource for students' and teachers' learning. Building on Pendred's question above, I wondered about affordances and constraints of exploring issues that relate to anti-racism in a virtual environment. Are there opportunities built in to engage social learning among the youth participants?
On a different note, I wondered if you're finding certain topics in the science curriculum that lend themselves better to integration with justice-oriented issues than others.
Thanks for sharing! I'd love to explore this platform in more depth and plan to be delving into the website!
Marcia Linn
Lead Presenter Professor
May 12, 2021 | 02:10 p.m.
Hi Heidi, Thank you for your comments. Finding the most compelling topics is indeed important. We very much appreciate the thoughtful contributions of all the members of our RPP in figuring out promising topics. We would love to meet with you to expand our discussion of these topics. I hope you are doing well. Best, Marcia
1
Discussion is closed. Upvoting is no longer available
Found helpful:
Heidi Carlone
Sarah Bichler
Co-Presenter Post-Doctoral Researcher
May 12, 2021 | 11:49 a.m.
Dear Heidi, thanks so much for your comment and questions! I personally think that yes, there are certain topics in the science curriculum that lend themselves better to integration with justice-oriented issues than others. Climate change is one of them. I think this is because climate change impacts the world and us living in it "visibly" and students care about the environment. It is, in my opinion, easy to expand this and include social or political issues when teaching the science. I personally don't see how I can link social issues as easily with other science units I work on, for example Musical Instruments and the Physics of Sound Waves. However, now I am thinking about this and already have ideas about noise pollution... So I guess the more we think about how any science topic could be related, the more we learn about how it is related to social issues or how it can be taught and learned in relation to social issues. Thanks so much for sparking new ideas with this question!
Sarah
1
Discussion is closed. Upvoting is no longer available
What a great project! I agree it's wonderful to have an authentic connection between racial justice and climate change. I wonder if the students might consciously avoiding using racism in their explanations - it's such a brave topic to raise for them as well as the teachers... and probably particularly challenging in a virtual space where communication can feel so tenuous. Virtual bravery is the highest form! Do you know how the parents are perceiving this focus on racial justice in science? They may be listening/participating more than if their children were in an in-person class.
Allison Bradford
Co-Presenter Graduate Student Researcher
May 12, 2021 | 06:19 p.m.
Dear Sue,
Thanks for your comments and questions! We are definitely seeing some variation in how directly students talk about race during the activities in our units. We also have seen a lot of bravery to take on difficult conversations about racism from our students and teachers. From our observations and interviews with our partner teachers, it seems like it is much more challenging in the remote classroom to gauge what is coming up for students and how they are responding to the activities that raise the connections to racial justice. Most teachers are really excited to teach the units in person.
As far as the parent perspective, one of our teachers shared that received positive feedback from a parent about the importance of discussing racial justice in the science classroom. None have reported receiving any pushback, but several have voiced being aware that parents might be listening while they teach. We would love to learn more about how parents perceive this focus, particularly as we think about how to ensure our students and their families' perspectives are represented in our partnership. We would be interested to learn from any ideas or suggestions you have around this!
Allison
Daniel Damelin
Facilitator Senior Scientist
May 12, 2021 | 09:32 p.m.
There seem to be three key elements:
The curricular materials and how they frame the phenomenon students are exploring.
The teacher facilitation.
The NLP scoring of student writing.
Are there other components that you feel are critical in the success of the project?
It seems that helping teachers facilitate and navigate potentially difficult conversations must be challenging. It would be great to learn more about the professional learning program and how it supports teachers beyond the STEM aspects of the curricular units.
I'm also curious about the decision to pull out specific scores for each dimension of the NGSS, separately scoring DCI, SEP, and CCC. In some of the NGSS-aligned assessment projects I've been involved in we had some deep discussions about how to score for progress toward NGSS performance expectations. The standards are written to integrate these dimensions, so how do you think about teasing them apart for separate scores?
Sarah Bichler
Co-Presenter Post-Doctoral Researcher
May 13, 2021 | 07:42 a.m.
Dear Daniel,
thanks for raising such important questions! Separately scoring DCI, CCC, and SEP is something we also discuss a lot, within and beyond our research group. Here are some of my thoughts:
The teachers I have worked with highly appreciated seeing NGSS dimensions scored. They wanted ways to see if their students are meeting PEs. Scoring dimensions separately also has the advantage that teachers learn in which area students need more guidance. They can then target their instruction and supports they offer. I think it has great practical value to tease them apart.
The KI scoring we do/show in addition to the subscores is (this can vary from item to item or researcher to researcher) often designed to evaluate how well students integrate ideas from both subscores. For example, if all my students are scoring KI 5 (or if one is), it means they explain natural selection and use evidence from the graph. This is one way of approaching it. However, I think we do not yet have a good approach to assess integrated 3-dimensional learning, one that could be standardized and/or is used by multiple groups and across science domains. I think we don't yet have agreement (or know enough about it) how items measuring integrated 3-dimensional understanding look like. Is that one item or is it a set of items? Is there one score that indicates degree of "integration" or are there subscores for each dimension that are aggregated? Can we empirically separate the 3 dimensions? These questions are intriguing and I would love to hear your thoughts!
Because I have seen the practical value of scoring dimensions separately, I started thinking about the underlying ability we are interested in. Would we say a student who explains a DCI, transfers knowledge across domains, and e.g., uses experimentation and or evidence, i.e., whose work would receive a high score on each dimension, integrated the three dimensions? In other words, is a student who is proficient in all of the three dimensions a student who we say has integrated 3-dimensional understanding? Or is integrated 3-dimensional understanding something more than the sum of its parts? I am so curious to hear your thoughts here as well!!
Best, Sarah
Daniel Damelin
Facilitator Senior Scientist
May 14, 2021 | 03:04 p.m.
I think the argument that convinced me to try to keep the scoring integrated is the notion that, for example, proficiency in a scientific practice is going to be highly influenced by their disciplinary knowledge. I might be much better at investigating, modeling, arguing, or explaining in a discipline that I am more comfortable in. If that is the case then what does a score on a scientific practice mean divorced from the context in which it was measured?
One attempt to measure student outcomes in an integrated way can be found here where several projects have contributed to development of NGSS-aligned formative assessments. If one creates a free teacher account then you have access to the "teacher guide" for a task, available for many of the tasks. This guide has an integrated rubric for giving students feedback. For example, see this task and its associated teacher guide.
Sarah Bichler
Co-Presenter Post-Doctoral Researcher
May 14, 2021 | 03:24 p.m.
Daniel, this is so helpful! Thanks for sharing these wonderful resources. I agree with what you are saying.
Dr. Julia V. Clark
Facilitator Retired Federal Employee
May 13, 2021 | 10:19 a.m.
Addressing anti-racism is a thoughtful idea. It is not clear to me how the teachers used the web-based units to customize instruction for the students. Were the teachers and the students from diverse populations? How are the goals/objectives of the project evaluated? How is the Black Lives Matters concept incorporated into the project? It is stated that the project wants the students to connect their experiences in project activities to systemic reform. More clarity is needed to explain how this goal is accomplished. The concept of the project is commendable but not certain if the results intended will be accomplished
Sarah Bichler
Co-Presenter Post-Doctoral Researcher
May 13, 2021 | 12:15 p.m.
Dear Julia,
Thanks so much for you comment! You are raising really important issues. I can speak to how teachers customized instruction. During our workshop, teachers brainstorm and formulate their instructional goals. These differ between teachers. One of our teachers is eager to make units more accessible for their students who are all English Language Learners. Other teachers choose to design activities around local issues. Teachers are also using the NLP results and review their students work to see how their students express ideas. Ideally, instruction can then be build around specific ways in which students think about science concepts.
With respect to evaluating our goal to help students to connect their experiences in project activities to systemic reform, I think we are in a very early stage of exploring how to best support teachers, what teacher and students' needs are, and how we can design activities that empower students to take action. Currently, we are investigating how students bring up social justice and racial equity arguments and how they are connecting these arguments with the science they learn. I hope our units and activities raise awareness. I am eager to work with our team and to learn from others to find ways in which we can extend this beyond the classroom!
Please share your thoughts on how achievement of these aims could be evaluated. What outcomes would you like to see? I am very much looking forward to learning from your ideas!
Sarah
Marcia Linn
Lead Presenter Professor
May 13, 2021 | 01:45 p.m.
Dear Julia
We are excited to have your comments on our project. You raise important questions. As Sarah indicated we are considering all these issues and welcome input. Our customizations resulted from a Research Practice Partnership and the Anti-Racism units emerged from teacher desires to respond to the interests of their students in these issues. We were very gratified by the ways that participating teachers localized the units to their communities. For example, around the topic of Asthma, teachers guided students to analyze distributions of particulate matter, incidence of Asthma, and access to medical care in their communities. The teachers also engaged students in comparisons of their community to other local regions. We are using embedded assessments as initial evidence of student ideas and concerns. We are building on student responses to create new ways to measure overall progress as well as understanding of systemic issues. For example, we are building on our success with concept mapping activities that have automated guidance. These items have shown promise for valid assessment of each student, especially those who are English Learners. We are refining our measures while investigating their value for guiding customization of the units and improving student success. We will keep you informed as our papers report on this work. Take care, Marcia
Hi Marcia, Libby, and the Berkeley team! Thank you for the excellent video. I love the way of applying NLP to support responsive teaching. What an excellent idea! The Climate Change unit is a great example of infusing social justice into science learning. I’m wondering if you see any difference in terms of student learning (conceptual and affective) using the social justice based approach? Do students incorporate more social justice related issues in their explanations? I’m also curious how teachers and students respond to the NLP supported teaching? Do teachers feel more confident in responsive teaching?
Sarah Bichler
Co-Presenter Post-Doctoral Researcher
May 13, 2021 | 11:30 a.m.
Hi Helen!
Thanks so much for all these wonderful questions! The teachers I have worked with expressed that the report helped them understand a) the ideas students have or struggle with and b) the ways in which students think about taught concepts. One teacher I worked with taught the Musical Instruments unit. From the report generated from responses in their classroom, the teacher learned that students confused pitch and volume. The teacher expressed that they wouldn't have known if it is amplitude, pitch, etc. that trips kids up and because they saw the report, they designed a hands-on activity (this was right before shelter-in-place) supporting students to connect different properties of a sound wave and the wave's visual representation. In sum, this teacher felt the report helped to respond more to students' ideas.
With respect to students, in an ongoing study I used the NLP scoring to automatically branch students to targeted guidance. To explain to students why they might see different activities on their screen (this may not matter during online instruction but some teachers facilitate WISE during synchronous Zoom sessions), we added an explanation about the computer scoring their work and detecting ideas they might have missed. We are curious to learn what they think! A previous study from the group found that students trust computer feedback more when we transparently communicate what the computer does (and tell them that the computer tries to respond to them individually).
I'd love to hear your thoughts on what kinds of questions we could ask students to find out how they perceive NLP supported learning and what they would need to interact with automated guidance!
Inspiring work building on what I know to be decades of effort in science learning. So appreciate the attention to natural language processing as (I think?) a way to help teachers assess students' learning.
I'd love to know more about how you created the backend of the parts that allowed students to look for connections between climate change issues and systemic racism.
Allison Bradford
Co-Presenter Graduate Student Researcher
May 14, 2021 | 09:48 a.m.
Dear Deborah,
Thanks for your comments and questions. Yes, we are partnering with a number of teachers to better understand how to leverage NLP to aid them in assessing and responding to the rich ideas their students have.
Designing the curriculum to help students investigate connections between climate change and systemic racism was also a partner effort. In the case of the climate change, we built off of our existing WISE unit about climate change. We worked with our partner teachers to identify resources, such as datasets, maps, dynamic visualizations and news articles, that we could integrate into the existing unit. Our open source authoring tools (here's a bit more about our tools) make it easy to edit our units and add new activities that give students opportunities to make sense of data and gather evidence about these connections. In our RPP workshop meetings, our teachers also spent time customizing the units in ways they thought would best support their students. We look forward to thinking more about how to support students to look for connections between systemic racism and NGSS-aligned science content.
Really timely and necessary work! I am curious about possible student outcomes from this unit, especially in terms of action and engagement. One part that was very clear in the video is related to "being part" of the context of the unit and understanding how students are personally affected by climate change and systemic racism. What about in terms of being part of a community? or a global citizen?
Kelly Billings
Co-Presenter Graduate Student Researcher
May 13, 2021 | 08:38 p.m.
Hi Dante, thank you for your question! We have worked with teachers to create activities that promote activism and empowerment for students. For example, one school site chose to do an activity where students created an infographic based on what they learned about. pollution, health, and issues of equity. These infographics focused on solutions, both personal (like biking to school, using less electricity) and larger, more policy-oriented solutions (holding policymakers and local government officials accountable). As we continue to do this work, we are excited to learn more about how to support students and teachers in ways that encourage student agency and empowerment and see the variety of ideas students have about solutions to systemic issues.
1
Discussion is closed. Upvoting is no longer available
Hi Marcia and team - it's so fascinating to see how WISE continues to support teachers and students in important ways! Kelly, I can see why you asked the questions you did of our project about caring for students' and teachers' social and emotional wellbeing as they are invited to grapple with racism. I see the connections between our two projects, and I'm eager to hear more about how you and your team prepare and support teachers and students for this challenging work that might feel, to some at first peripheral to science learning and doing. Thanks for thinking with me!
1
Discussion is closed. Upvoting is no longer available
Found helpful:
Kimberly Elliott
Kelly Billings
Co-Presenter Graduate Student Researcher
May 14, 2021 | 05:01 p.m.
Thank you for your comment April, as well as the answers to my questions on your video. As we have worked with teachers to develop these units, we have thought deeply about what supports look like for our participants. This included activities in a professional development workshop where teachers discussed the issues that appear in our units with each other and co-created ways to launch units and support their students. Teachers know their students best, so we wanted to create space for them to think through what would be best in the context of their classrooms and how we can support them. We would love to speak with you further about what you have done with your work, as we are working currently with teachers to develop a unit focus on COVID, data science, and health equity as a part of this project as well.
1
Discussion is closed. Upvoting is no longer available
Hi Marcia and team! What an interesting approach! In our video, "Learning with Purpose as a Cultural Strength," we make the case that having a purpose that benefits a larger group is a powerful motivator for learning, especially for children from underserved backgrounds. It seems that your project is harnessing that in a really productive way!
Best regards, Barbara
Roberta Marquez
Undergraduate Student
May 18, 2021 | 01:33 p.m.
Hi Marcia and team!
This research is fascinating and imperative. I often conceptualize white supremacy as a virus, education being one of its victors. It's inspiring to see real solutions to real problems. How would you hope to measure students' improved practice of anti-racism?
This is fascinating material and so central to helping children learn. Cross-connecting deep issues in STEM with the climate change, anti-racism, and other urgent contemporary issues is such a wise approach for engaging students. Congratulations on developing such a thought provoking and helpful program! Clever, practical, and uplifting!
Rui Li
Co-Presenter Researcher/Research Manager
May 18, 2021 | 07:04 p.m.
Dear Andrew,
Thank you for your kind words! I think the more we learn from our current units introducing social justice into science, the more topics we can add in the future to address the cross-connecting issues for students. We will also host our summer professional development workshop to learn more from our teachers' experience and perspectives and discuss with them about what other topics that we can add in the future to integrate different lens of social justice in science.
Kevin McElhaney
Hello Berkeley team! Thank you for your video, this is such a great way to customize a WISE unit. I was wondering if as part of this work you have tried to measure students' shifts related to their knowledge about (or positions on) racial equity, and if you have found any relationships between this and their science learning. One might hypothesize these learning outcomes might be synergistic but I imagine it would not be straightforward to measure this.
Sarah Bichler
Post-Doctoral Researcher
Dear Kevin, thank you for viewing our project! We have just started looking at students' responses and are in the process of refining our assessment items. Reviewing students' responses, we learned that many students appreciated learning that not all people are impacted by climate change in the same way or how redlining has led to fewer green spaces in black and brown neighborhoods. We noticed that students explain the relationship between how close someone lives to a freeway and the likelihood they get asthma, so they are linking the science to impacts on peoples health. We also noticed that most students do not yet include ideas about racial equity in their explanations. We are curious to see if our revised items will elicit these ideas as well so we can learn how students link science and racial equity ideas. We also discussed that students may not be used to discussing such questions/issues in science class and that they may link their ideas now that these questions are brought up more.
Thanks for the question!
Sarah
Brian Riordan
Hi Kevin, I'm the ETS PI on this collaborative NSF grant; I lead the development of NLP methods for analysis of student responses. In this work we've also developed new NLP methods to detect student ideas at multiple levels of granularity to help support teachers to understand the breadth of student ideas and facilitate discussion. One of the foci of future work on revised items will be better capturing the diversity of student ideas at the intersection of science and social justice issues. We described an early version of the idea detection approach in a short technical paper, "Identifying NGSS-Aligned Ideas in Student Science Explanations".
Caitlin Martin
What incredibly important and thoughtful work! I really appreciate the explicit integration of systemic racism into an existing science learning platform in such an authentic way. And in a way that stems from what the educators were interested in bringing into their conversations in the classroom. Such a powerful example of making data human-centered and personal, understanding the historical implications of the world we see today -- emphasizing the need for change in the future! Did you get feedback from teachers about how things went? Including what conversations came up?
Sarah Bichler
Post-Doctoral Researcher
Dear Caitlin, thank you so much for the feedback! Teachers who are running the units participated in a workshop in which we discussed ways to support students during conversations about racial equity in science class. Looking at student data, we get the impression that students really appreciate learning about issues like urban heat islands. Many of our teachers are still teaching distance. We are currently interviewing teachers - my colleagues will share more details about what we learn from teachers!
Best,
Sarah
Marcia Linn
Professor
Thank you for your comments and interest in our efforts to welcome each student to science. We are investigating ways to support teaches to explore social justice issues, consistent with many students' concerns about the environment, BLM, and personal health. We seek to transform the goals of science courses from primarily preparing students for careers in science, to ensuring that each student, independent of race, culture, sexual orientation, or economic status is welcomed in science. This means developing the identity of each student as a science learner and respecting the science accomplishments of each person. Please tell us how your work can help us advance this goal.
Lei Liu
Hi Marcia, Thanks for sharing such a great piece of work! I'd love to have a follow up discussion on applying NLP-supported innovations in classrooms to support teacher practices. Also quite interested in integrating social justice issues into student science learning. - Lei
Marcia Linn
Professor
Great. We will coordinate with you and Brian to set up a meeting.
Judi Fusco
I'd love to talk with you all more about how you are using NLP. It looks really powerful and I'd love to see more. Thanks for this work as a model for how to bring in new, important topics in science to make them more relevant for students.
Marcia Linn
Professor
Hi Judi,
Thank you for your comment. We are partnering with ETS to use their innovations. We can send you copies of our research papers. Here is one reference.
Brian Riordan
Here are some links to recent technical papers on our work:
BEA_2020
AI4EDU_2020
Sarah Bichler
Post-Doctoral Researcher
Dear Lei and Judi, thanks so much for showing interest in our work! We'd love to share more about how we use NLP and implement NLP innovations in classrooms. Here is some more info:
1. We designed "milestone items" for several inquiry units. These items are complex and students have to link multiple ideas to explain a phenomenon. For example, in our Genetics of Extinction unit, students are asked "Why do siblings look similar but not the same?". Students can use ideas about genetic inheritance and genetic variation (gathered in the unit) to explain.
2. We develop a rubric to assess Knowledge Integration and also code for NGSS 3-dimensional understanding (DCI, CCC, or SEP dependent on the item). For example, another milestone in the Genetics of Extinction unit ask students to use evidence from a graph (SEP) to explain how species adapt to their environment.
3. Brian Riordan at ETS trains an automated scoring model and then we implement it in the units. Students' responses are scored in real time and teachers can view a report that is generated from the automated scores.
4. The report shows score distributions for KI and NGSS subscores and features example student responses. The report also includes recommended instructional actions that promote KI and/or are tailored to the ideas present in the respective classroom. For example, if students score really high on explaining natural selection but do not yet include evidence from the graph, recommended actions suggest ways to support students to use evidence.
Our reports now also show how students progress when they revised their responses after a teacher's intervention. In addition, teachers can view initial and revised responses (and KI scores) for each individual student. This way, teachers can more easily help those students who need the most help.
Looking forward to talking more and learning about your ideas how we can improve this work or what other work we can build on!
Best,
Sarah
Jan Mokros
Hi Marcia and team,
The use of data in studying social injustices is really important, and you clearly have some rigorous research going on! It may be beyond the scope of your project (or too controversial) but how you look at changes in students' attitudes about issues like anti-racism or climate change? It might be interesting to use some common instruments across projects, and our project is in the early stages of research about youth's understanding of social issues related to COVID.
Marcia Linn
Professor
Hi Jan,
That is an excellent issue. We have had good success with questions that get at the controversy and connect to the science. We will be reporting on this work shortly. A question that is interesting concerns whether a particular scientific phenomena (like pollution) has equal or unequal impacts on each family. We are seeing big changes in students thinking about this issue. Take care, Marcia
Sarah Bichler
Post-Doctoral Researcher
Hi Jan!
Thanks so much for your feedback. It would be great to share resources and learning from other groups how we can best approach measuring change in attitudes towards and knowledge of anti-racism and/or racial equity in relation to climate change!
We are curious to learn more about your project and what items to measure understanding of social issues you are developing. We're also curious about your COVID instructional unit - we have designed a WISE unit about COVID and social equity too!
Kirby Whittington
Dear Berkeley team! Wow, what a great presentation. It is so interesting to hear about two really important topics: helping teachers to see components of student responses related to the three dimensions and the integration of issues of social justice in the classroom. I am curious to know more about the teachers and their views on the place of social justice in science. While it seems like the teachers in the video were very keen to integrate the two, many science teachers shy away from these topics because they are unsure how to handle the discussion or questions that students may bring up. Did you all experience any of this resistance? Or how might you all help a teacher see the importance of bringing anti-racism, social justice into science? This is something I am very interested in and would love the thoughts of others working in the area.
Marcia Linn
Professor
Dear Kirby,
Hi Kirby, Thank you for the kind words. The issues you raise are important. We will be reporting on the elements of our professional development approach that facilitated discussions around introducing social justice into science. The formation of a Research Practice Partnership was definitely a key element for addressing concerns of teachers. Best, Marcia
Pendred Noyce
Marcia, what a rich set of tools you have created! I'll want to look at them more. You showed a child at home working; are these materials primarily for remote or online learning, or are they accompanied by hands-on investigations? What is the balance? And I see a lot of text; how dependent are the materials on children's reading levels?
Marcia Linn
Professor
Dear Penny,
How great to hear from you. The units were introduced in science classes using remote instruction. The students discussed the issues in class and, of course, this often meant that the discussion extended into the home. We took advantage of interactive visualizations, concept maps, and graph tools to make sure that the reading was not a primary channel for learning. We also supported text to speech so students could hear the material if they preferred--one slight advantage of working remotely!!
We would love to set up a meeting with you to get your insights into ways we can strengthen the work.
Best, Marcia
Jodi Asbell-Clarke
Nice work....I am wondering if there were issues that were studied that were not only "urban" issues but also dealt directly with race? The two seem to get conflated sometimes....any ideas? thanks for the great work!
Marcia Linn
Professor
Hi Jodi,
This is a good point. Our activities enable students to use the design of graphs to sort out race in exploring topics such as global climate change or the incidence of Asthma or COVID. We agree that these issues require a way to work out the separate contribution of race and class. We would love to meet with you to follow up on these issues. Thank you, Marcia
Heidi Carlone
Distinguished Professor
Great work, team! This environment is a rich resource for students' and teachers' learning. Building on Pendred's question above, I wondered about affordances and constraints of exploring issues that relate to anti-racism in a virtual environment. Are there opportunities built in to engage social learning among the youth participants?
On a different note, I wondered if you're finding certain topics in the science curriculum that lend themselves better to integration with justice-oriented issues than others.
Thanks for sharing! I'd love to explore this platform in more depth and plan to be delving into the website!
Marcia Linn
Professor
Hi Heidi, Thank you for your comments. Finding the most compelling topics is indeed important. We very much appreciate the thoughtful contributions of all the members of our RPP in figuring out promising topics. We would love to meet with you to expand our discussion of these topics. I hope you are doing well. Best, Marcia
Heidi Carlone
Sarah Bichler
Post-Doctoral Researcher
Dear Heidi, thanks so much for your comment and questions! I personally think that yes, there are certain topics in the science curriculum that lend themselves better to integration with justice-oriented issues than others. Climate change is one of them. I think this is because climate change impacts the world and us living in it "visibly" and students care about the environment. It is, in my opinion, easy to expand this and include social or political issues when teaching the science. I personally don't see how I can link social issues as easily with other science units I work on, for example Musical Instruments and the Physics of Sound Waves. However, now I am thinking about this and already have ideas about noise pollution... So I guess the more we think about how any science topic could be related, the more we learn about how it is related to social issues or how it can be taught and learned in relation to social issues. Thanks so much for sparking new ideas with this question!
Sarah
Heidi Carlone
Sue Allen
What a great project! I agree it's wonderful to have an authentic connection between racial justice and climate change. I wonder if the students might consciously avoiding using racism in their explanations - it's such a brave topic to raise for them as well as the teachers... and probably particularly challenging in a virtual space where communication can feel so tenuous. Virtual bravery is the highest form! Do you know how the parents are perceiving this focus on racial justice in science? They may be listening/participating more than if their children were in an in-person class.
Allison Bradford
Graduate Student Researcher
Dear Sue,
Thanks for your comments and questions! We are definitely seeing some variation in how directly students talk about race during the activities in our units. We also have seen a lot of bravery to take on difficult conversations about racism from our students and teachers. From our observations and interviews with our partner teachers, it seems like it is much more challenging in the remote classroom to gauge what is coming up for students and how they are responding to the activities that raise the connections to racial justice. Most teachers are really excited to teach the units in person.
As far as the parent perspective, one of our teachers shared that received positive feedback from a parent about the importance of discussing racial justice in the science classroom. None have reported receiving any pushback, but several have voiced being aware that parents might be listening while they teach. We would love to learn more about how parents perceive this focus, particularly as we think about how to ensure our students and their families' perspectives are represented in our partnership. We would be interested to learn from any ideas or suggestions you have around this!
Allison
Daniel Damelin
Senior Scientist
There seem to be three key elements:
Are there other components that you feel are critical in the success of the project?
It seems that helping teachers facilitate and navigate potentially difficult conversations must be challenging. It would be great to learn more about the professional learning program and how it supports teachers beyond the STEM aspects of the curricular units.
I'm also curious about the decision to pull out specific scores for each dimension of the NGSS, separately scoring DCI, SEP, and CCC. In some of the NGSS-aligned assessment projects I've been involved in we had some deep discussions about how to score for progress toward NGSS performance expectations. The standards are written to integrate these dimensions, so how do you think about teasing them apart for separate scores?
Sarah Bichler
Post-Doctoral Researcher
Dear Daniel,
thanks for raising such important questions! Separately scoring DCI, CCC, and SEP is something we also discuss a lot, within and beyond our research group. Here are some of my thoughts:
Best,
Sarah
Daniel Damelin
Senior Scientist
I think the argument that convinced me to try to keep the scoring integrated is the notion that, for example, proficiency in a scientific practice is going to be highly influenced by their disciplinary knowledge. I might be much better at investigating, modeling, arguing, or explaining in a discipline that I am more comfortable in. If that is the case then what does a score on a scientific practice mean divorced from the context in which it was measured?
One attempt to measure student outcomes in an integrated way can be found here where several projects have contributed to development of NGSS-aligned formative assessments. If one creates a free teacher account then you have access to the "teacher guide" for a task, available for many of the tasks. This guide has an integrated rubric for giving students feedback. For example, see this task and its associated teacher guide.
Sarah Bichler
Post-Doctoral Researcher
Daniel, this is so helpful! Thanks for sharing these wonderful resources. I agree with what you are saying.
Dr. Julia V. Clark
Retired Federal Employee
Addressing anti-racism is a thoughtful idea. It is not clear to me how the teachers used the web-based units to customize instruction for the students. Were the teachers and the students from diverse populations? How are the goals/objectives of the project evaluated? How is the Black Lives Matters concept incorporated into the project? It is stated that the project wants the students to connect their experiences in project activities to systemic reform. More clarity is needed to explain how this goal is accomplished. The concept of the project is commendable but not certain if the results intended will be accomplished
Sarah Bichler
Post-Doctoral Researcher
Dear Julia,
Thanks so much for you comment! You are raising really important issues. I can speak to how teachers customized instruction. During our workshop, teachers brainstorm and formulate their instructional goals. These differ between teachers. One of our teachers is eager to make units more accessible for their students who are all English Language Learners. Other teachers choose to design activities around local issues. Teachers are also using the NLP results and review their students work to see how their students express ideas. Ideally, instruction can then be build around specific ways in which students think about science concepts.
With respect to evaluating our goal to help students to connect their experiences in project activities to systemic reform, I think we are in a very early stage of exploring how to best support teachers, what teacher and students' needs are, and how we can design activities that empower students to take action. Currently, we are investigating how students bring up social justice and racial equity arguments and how they are connecting these arguments with the science they learn. I hope our units and activities raise awareness. I am eager to work with our team and to learn from others to find ways in which we can extend this beyond the classroom!
Please share your thoughts on how achievement of these aims could be evaluated. What outcomes would you like to see? I am very much looking forward to learning from your ideas!
Sarah
Marcia Linn
Professor
Dear Julia
We are excited to have your comments on our project. You raise important questions. As Sarah indicated we are considering all these issues and welcome input. Our customizations resulted from a Research Practice Partnership and the Anti-Racism units emerged from teacher desires to respond to the interests of their students in these issues. We were very gratified by the ways that participating teachers localized the units to their communities. For example, around the topic of Asthma, teachers guided students to analyze distributions of particulate matter, incidence of Asthma, and access to medical care in their communities. The teachers also engaged students in comparisons of their community to other local regions. We are using embedded assessments as initial evidence of student ideas and concerns. We are building on student responses to create new ways to measure overall progress as well as understanding of systemic issues. For example, we are building on our success with concept mapping activities that have automated guidance. These items have shown promise for valid assessment of each student, especially those who are English Learners. We are refining our measures while investigating their value for guiding customization of the units and improving student success. We will keep you informed as our papers report on this work. Take care, Marcia
Helen Zhang
Hi Marcia, Libby, and the Berkeley team! Thank you for the excellent video. I love the way of applying NLP to support responsive teaching. What an excellent idea! The Climate Change unit is a great example of infusing social justice into science learning. I’m wondering if you see any difference in terms of student learning (conceptual and affective) using the social justice based approach? Do students incorporate more social justice related issues in their explanations? I’m also curious how teachers and students respond to the NLP supported teaching? Do teachers feel more confident in responsive teaching?
Sarah Bichler
Post-Doctoral Researcher
Hi Helen!
Thanks so much for all these wonderful questions! The teachers I have worked with expressed that the report helped them understand a) the ideas students have or struggle with and b) the ways in which students think about taught concepts. One teacher I worked with taught the Musical Instruments unit. From the report generated from responses in their classroom, the teacher learned that students confused pitch and volume. The teacher expressed that they wouldn't have known if it is amplitude, pitch, etc. that trips kids up and because they saw the report, they designed a hands-on activity (this was right before shelter-in-place) supporting students to connect different properties of a sound wave and the wave's visual representation. In sum, this teacher felt the report helped to respond more to students' ideas.
With respect to students, in an ongoing study I used the NLP scoring to automatically branch students to targeted guidance. To explain to students why they might see different activities on their screen (this may not matter during online instruction but some teachers facilitate WISE during synchronous Zoom sessions), we added an explanation about the computer scoring their work and detecting ideas they might have missed. We are curious to learn what they think! A previous study from the group found that students trust computer feedback more when we transparently communicate what the computer does (and tell them that the computer tries to respond to them individually).
I'd love to hear your thoughts on what kinds of questions we could ask students to find out how they perceive NLP supported learning and what they would need to interact with automated guidance!
Best,
Sarah
Deborah Fields
Inspiring work building on what I know to be decades of effort in science learning. So appreciate the attention to natural language processing as (I think?) a way to help teachers assess students' learning.
I'd love to know more about how you created the backend of the parts that allowed students to look for connections between climate change issues and systemic racism.
Allison Bradford
Graduate Student Researcher
Dear Deborah,
Thanks for your comments and questions. Yes, we are partnering with a number of teachers to better understand how to leverage NLP to aid them in assessing and responding to the rich ideas their students have.
Designing the curriculum to help students investigate connections between climate change and systemic racism was also a partner effort. In the case of the climate change, we built off of our existing WISE unit about climate change. We worked with our partner teachers to identify resources, such as datasets, maps, dynamic visualizations and news articles, that we could integrate into the existing unit. Our open source authoring tools (here's a bit more about our tools) make it easy to edit our units and add new activities that give students opportunities to make sense of data and gather evidence about these connections. In our RPP workshop meetings, our teachers also spent time customizing the units in ways they thought would best support their students. We look forward to thinking more about how to support students to look for connections between systemic racism and NGSS-aligned science content.
Dante Cisterna
Really timely and necessary work! I am curious about possible student outcomes from this unit, especially in terms of action and engagement. One part that was very clear in the video is related to "being part" of the context of the unit and understanding how students are personally affected by climate change and systemic racism. What about in terms of being part of a community? or a global citizen?
Kelly Billings
Graduate Student Researcher
Hi Dante, thank you for your question! We have worked with teachers to create activities that promote activism and empowerment for students. For example, one school site chose to do an activity where students created an infographic based on what they learned about. pollution, health, and issues of equity. These infographics focused on solutions, both personal (like biking to school, using less electricity) and larger, more policy-oriented solutions (holding policymakers and local government officials accountable). As we continue to do this work, we are excited to learn more about how to support students and teachers in ways that encourage student agency and empowerment and see the variety of ideas students have about solutions to systemic issues.
Kimberly Elliott
April Luehmann
Hi Marcia and team - it's so fascinating to see how WISE continues to support teachers and students in important ways! Kelly, I can see why you asked the questions you did of our project about caring for students' and teachers' social and emotional wellbeing as they are invited to grapple with racism. I see the connections between our two projects, and I'm eager to hear more about how you and your team prepare and support teachers and students for this challenging work that might feel, to some at first peripheral to science learning and doing. Thanks for thinking with me!
Kimberly Elliott
Kelly Billings
Graduate Student Researcher
Thank you for your comment April, as well as the answers to my questions on your video. As we have worked with teachers to develop these units, we have thought deeply about what supports look like for our participants. This included activities in a professional development workshop where teachers discussed the issues that appear in our units with each other and co-created ways to launch units and support their students. Teachers know their students best, so we wanted to create space for them to think through what would be best in the context of their classrooms and how we can support them. We would love to speak with you further about what you have done with your work, as we are working currently with teachers to develop a unit focus on COVID, data science, and health equity as a part of this project as well.
Kimberly Elliott
April Luehmann
I would love to stay in touch Kelly. Shoot me an email when you get a chance; we can set up a time to have coffee: April.luehmann@rochester.edu
Barbara Rogoff
Hi Marcia and team! What an interesting approach! In our video, "Learning with Purpose as a Cultural Strength," we make the case that having a purpose that benefits a larger group is a powerful motivator for learning, especially for children from underserved backgrounds. It seems that your project is harnessing that in a really productive way!
Best regards, Barbara
Roberta Marquez
Hi Marcia and team!
This research is fascinating and imperative. I often conceptualize white supremacy as a virus, education being one of its victors. It's inspiring to see real solutions to real problems. How would you hope to measure students' improved practice of anti-racism?
Best,
Roberta (she/her)
Roberta Marquez
*vectors, not victors!
Andrew Meltzoff
This is fascinating material and so central to helping children learn. Cross-connecting deep issues in STEM with the climate change, anti-racism, and other urgent contemporary issues is such a wise approach for engaging students. Congratulations on developing such a thought provoking and helpful program! Clever, practical, and uplifting!
Rui Li
Researcher/Research Manager
Dear Andrew,
Thank you for your kind words! I think the more we learn from our current units introducing social justice into science, the more topics we can add in the future to address the cross-connecting issues for students. We will also host our summer professional development workshop to learn more from our teachers' experience and perspectives and discuss with them about what other topics that we can add in the future to integrate different lens of social justice in science.
Best,
Rui
Further posting is closed as the event has ended.