Thanks for visiting our video, Pati. For your question about feedback to students, we are also collaborating with teachers to design automated feedback based on characteristics in student reasoning patterns. We are planning a small scale cog lab study to try out the automated diagnosis and feedback with a small group of students. The feedback will highlight both achievement and gaps in specific reasoning patterns. With the support of automated diagnosis/feedback, teachers will play multiple essential roles in the classroom orchestration. To list a few, based on identified patterns, teachers can group students and assign differentiated tasks to address gaps in their reasoning; they can probe further how students' background and life experiences relate to their habits of reasoning; they can adjust instruction accordingly to support multi-dimensioning reasoning. I will let our NLP expert from our team to answer your question about ML method.

Thanks for watching our video, Marcia. You are spot on that it's important to communicate and collaborate with teachers to support their practice in classrooms, which is our next step. Your video is quite inspiring for us to consider next step design. Would love to have a follow up discussion. - Lei

Thanks for checking out our video, Dalila. As Aoife mentioned below, we used macro f-score to calculate the human-machine agreement.

The human-machine agreement and human-human agreements fall in similar range. Those 20% are not necessarily inaccurately detected in my view. They could be ones that may need additional judgments (e.g., need more contextual information, or other factors may leads to this type of reasoning), which teachers may be able to leverage into their judgment.  For all student responses, in addition to identifying the reasoning pattern, we also annotated texts as evidence for certain pattern (e.g., orange highlights for DCI; blue highlights for SEP). This way, teachers can also make their judgments by looking at machine-generated pattern label. We are also trying different models to improve accuracy and hopefully can report some improved accuracy later.

Hello Suzanne! Thanks for watching the video and for your comments. Our goal is developing this model in a way that is useful for teachers and students. At this stage, we are developing the automated model to identify students' reasoning patterns and planning to receive teacher feedback.

Certainly, we have been discussing how to use the automated tool in the classroom. One idea that we have considered is to develop prompts for teachers with the goal of guiding student reasoning to more complex responses (based on the tridimensional framework) and based on certain reasoning patterns. Potentially, teachers can identify groups of students whose responses fit a particular reasoning pattern and provide feedback based on those prompts.

Hi Kelly, Thanks for your question. The current project is focusing on a proof of concept study focusing on the design of the automated tools. The scope of a large project is to embed the automated tools into classroom assessments. The front-end will be the classroom assessments where students can input their responses. Once students' inputs are in, there will be automated prompts generated by the back-end linked automated tools. The assessments will be delivered through a platform that includes teacher dashboard, where teachers can track student progress and automated diagnosis results as well as what type of feedback was generated for individual students. The diagnosis results will include highlighted texts associated with NGSS dimensions as evidence for specific reasoning patterns. We anticipate teachers can use this information to adjust their instruction to address individual students' needs, group them for peer discussion, and further probe additional evidence if highlighted evidence of the reasoning pattern diagnosis is not convincing. We are planning to work with teachers to design multiple approaches to leverage the automated diagnosed information. Hope this helps answer your question and let us know if you have suggestions from your work as well. Thanks!