Thanks very much for the nice comment, Ekundayo! Totally agree with the implementation challenges, which is not addressed in this project. We are proposing a new proposal to tackle the classroom implementation issues. We would also welcome any brilliant ideas from teachers and/or researchers here.

To piggy back on Ekundayo's comment - given the noted biases often inherent in AI, are you tracking in this study how these could become manifest in broader use of this technology?  And/or are you expressly looking for evidence that the AI algorithms employed in this tool do not reflect such biases?  Your research on ways to optimize technology to support learning is so important.  Thank you for sharing (your video was great!).

Hi Janet,

Thank you for your great comments!

We are exploring other funding opportunities to pilot broader use of this technology. I am a hundred percent with you about the importance of potential biases employed in AI and we did document a few potential ones. E.g., we noticed that our current AI algorithms didn't take into account how students' misunderstanding of certain vocabularies might have impacted their reasoning patterns (particularly when the sample size of such misunderstanding was not big enough). We have a list of such potential issues and will explore potential solutions (either through AI or through AI + human/teacher). This could be a limitation of using existing data in our study.

Lei

Thanks for visiting our video, Barry! Yes, student data from the cognitive interviews helped us identify quite a few granted assumptions about student knowledge about NGSS. Regarding the implementation, explicit guidance and training will be necessary. Currently we are thinking of focusing on what can AI tell, what AI (at least in this AI tool) cannot tell, what evidence highlighted by AI could be trusted, what are some potential biases. More importantly, we are also thinking about what teachers may supplement what AI cannot do (or do well). 

Thanks for your comment, Cathryn. SPIN-NGSS is a small-scale, proof-of-concept study in which we developed the framework for reasoning patterns, the AI annotation model, and the feedback tool. Certainly, we'd like to pursue further research on classroom implementation of the tool, including the exploration of the potentials and challenges for implementation. Regarding the versatility of the tool, the item we selected for the AI-based feedback tool is aligned with a particular NGSS performance expectation for middle school and part of a performance assessment task. For example, teachers can embed the tool (and the task) in their instructional planning. Developing some type of support and teacher PD would be needed as well.

Hi, Andresse! We only conducted cognitive interviews with students about the tool for this project. We hope to move to the classroom implementation in a new study and collect feedback from teachers. 

Thanks, Xiaoming!

Great questions, Sarah! I cannot agree more that what student know (what they don't know) often are underestimated. For example, many students haven't had direct instruction on the three dimensions in the NGSS at all, which required us to revise some terms used in our feedback.

Also thanks for sharing some of the feedback examples. Definitely it's important to make the feedback related to student response and their thinking as much as possible. We may be able to use some of the automated annotation to personalize the feedback based on student's own response. For example, if a student only talked about certain concept but didn't cite any data, the feedback could be "What is your evidence for [AI highlighted concept]?"

1. What was your strategy for designing the feedback, what kind of feedback did you provide? 

Our feedback design was based on the student reasoning pattern framework we developed. The feedback was to focus on the integration of dimensions of science learning and the depth of their reasoning. For example, for students who only talked about disciplinary principles, we would provide feedback asking students to use data to support their statements. 

2. How was the feedback different for students whose response fell into pattern 1 vs 2?

The feedback is focusing on the missing dimension in pattern 1 (i.e., DCI/CCC) vs pattern 2 (i.e. SEP).

3. What feedback did students whose response fell into pattern 3 get? What revisions did you see in pattern 3?

Pattern 3 included multiple dimensions, however not necessarily integrated. The feedback would probe the connections between dimensions to enhance the integrated reasoning.

4. What types of revisions did you see? Mostly adding a new idea (data or science idea) or did students "rework" their entire response?

Our cog lab data showed that many students made revisions (96%), among which 69% of them revised their original responses (Patter 1 or 2) into a pattern 3 response. Most students revised based on the feedback to add a new idea to fix the missing dimension issue. 

Hope these responses answered your questions to some extent. Thanks again for these great questions!

Lei

Thanks, Clausell for your comment. The item we selected for this study is part of a performance task with several items that can be used for classroom assessment purposes. One potential example of classroom use is working with teachers to embed the performance task in instructional units, so they can align instructional activities and resources with the task. Regarding the automated feedback tool, teachers can potentially use the information provided by a dashboard to group students, based on their reasoning patterns, and organize a peer-feedback activity in which students revise their initial responses.