Classroom interpersonal dynamics between students and teachers, and students and peers, predict learners' downstream academic as well as socioemotional outcomes in a variety of academic disciplines. In STEM in particular, ensuring that all learners in the classroom feel welcomed and supported by their peers and teachers is critical to broadening participation from underrepresented groups. In this project we seek to harness multi-modal machine learning to develop an automatic classroom observation system that can perceive fine-grained interactions between classroom participants and help teachers to become more aware of the subtle interpersonal dynamics in their own classrooms. We will also develop novel methods of professional development that help teachers to foster better dynamics in their own classrooms: Instead of receiving feedback just 1-2x per year, or watching videos of other teachers' interactions, our technology will enable teachers to receive frequent feedback about their own teaching. Moreover, instead of receiving just a single score about a 15-minute video, the technology we are developing will give information about specific students and teachers at specific moments in time.In our video presentation, we will describe our progress on designing and training neural networks that can recognize classroom dynamics automatically. In particular, our neural networks can estimate the Positive Climate and Negative Climate dimensions of the Classroom Assessment Scoring System (CLASS) with Pearson correlations of 0.6. We will also discuss challenges and limitations, such as causal versus correlational inference and algorithmic fairness, that we are examining while pursuing our goal of improving teacher training and feedback.
Emerging Technologies, PD Models, Workforce Development
WPI, Univ of Virginia
Cyberlearning and Future Learning Technologies (Cyberlearning), EHR Core Research (ECR)
Related Content for Multimodal Machine Learning for Automated Classroom Feedback
Discussion from the 2020 STEM For All Video Showcase (21 posts)
Stacy Shaw
Researcher
May 4, 2020 | 01:00 p.m.
This technology is incredible. Would it be possible to use this technology in something like more advanced math classes and alert the teacher that many of the students are showing signs of confusion/ disengagement? I could see real value in teachers getting notified (especially if it was possible) in real-time that they've lost the class, and need to stop and answer/questions or clarify a concept before continuing a lesson.
I had the same impression and question. It is indeed an amazing application of Machine Learning to education. I am part of a study groups at the University of Rochester about AI-Augmented Learning, but I had not before thought about this dimension of using machine learning as a tool for teachers (rather than a direct support to students learning).
I am looking forward to following your project in the years to come - and see what other applications you may developed.
Jacob Whitehill
Lead Presenter Assistant professor
May 4, 2020 | 07:26 p.m.
Thanks for your question! The system we trained is specific to pre-school classrooms. However, the same machine learning methods could likely be applied for a wide range of classroom contexts and target variables (including confusion and disengagement). What would be needed is a dataset of hundreds of classrooms and associated target values, which is the same order of magnitude of training data we had for the pre-school classes.
James Brown
Facilitator 5/6 STEM Teacher
May 5, 2020 | 10:09 a.m.
I love the idea of using technology to help guide teachers in producing better student outcomes! How have you planned to recruit teachers to use this?
James Brown
Facilitator 5/6 STEM Teacher
May 5, 2020 | 10:10 a.m.
Do you have plans to integrate this technology to grades beyond pre-school classrooms?
Jacob Whitehill
Lead Presenter Assistant professor
May 5, 2020 | 10:12 a.m.
For the past several months we have been developing an experimental protocol to explore how automated classroom feedback, as provided by our machine learning-based analysis system, can help teachers to become more perceptive of classroom interpersonal dynamics. We will begin the experiment with classroom teachers this summer. After a pilot run of our study, we will recruit participants more broadly. Since the experiment will be held purely online, we hope to cast a wide net for recruitment. Thanks for your interest!
So far we have focused mostly on pre-school classrooms, but we have also conducted machine learning analyses on elementary and middle-school students. While each new population of students requires its own dataset for training, the general machine learning architecture that we use can likely generalize to a wide variety of students and learning contexts.
3
Discussion is closed. Upvoting is no longer available
Found helpful:
DeLene Hoffner
Rachel Navarro
James Brown
I think this methodology would yield fascinating results in college STEM classroom and could really heighten professors' awareness of their own biases and tendencies toward groups of students. This could be an excellent way of promoting inclusion and equity in STEM classrooms.
1
Discussion is closed. Upvoting is no longer available
This is a fascinating tool for teachers! I'd be interested in learning about the findings from the classroom intervention. The possibilities for expanding this beyond preschools is noted above. I think there is also potential for applying this tool with learning of students from underrepresented groups in STEM. Great work!
DeLene Hoffner
Facilitator Program Coordinator
May 7, 2020 | 04:40 p.m.
Good point. I see potential for a wide range of use at any grade level.
Co-Presenter Assistant Professor of Learning Sciences and Technology
May 5, 2020 | 02:35 p.m.
Thanks Lorna and Lisa! Its been a really fun collaboration and experience bringing together teams who think about the same problems with totally different methodological strengths and challenges!
1
Discussion is closed. Upvoting is no longer available
Love it! I can't wait to see how this plays out and where it leads you all with the research and implementation.
DeLene Hoffner
Facilitator Program Coordinator
May 6, 2020 | 02:25 a.m.
Wonderful video and project. It's very intriguing and inspiring. What have you found to be your biggest challenges?
Jacob Whitehill
Lead Presenter Assistant professor
May 6, 2020 | 08:13 a.m.
One of the fun and challenging aspects of the project has been bridging between our disciplines: education, learning science, and computer science. On the artificial intelligence and machine learning side, there is no dearth to computational challenges of the automatic perception of school classrooms! These include background noise, overlapping speech, uncontrolled lighting, people moving in and out of the scene...not to mention the semantically very high-level construct we are trying to estimate with our system.
1
Discussion is closed. Upvoting is no longer available
Found helpful:
DeLene Hoffner
DeLene Hoffner
Facilitator Program Coordinator
May 8, 2020 | 12:05 a.m.
Thank you , Jacob. That makes sense.
Stacey Forsyth
Facilitator Director
May 6, 2020 | 03:45 p.m.
This is a really interesting and exciting project! I'm curious to hear more about classroom teachers' reactions to this technology. Do they tend to be intrigued and interested by the idea of receiving feedback from an AI system? Or skeptical? I think it will be fascinating to see how the system's perception of classroom dynamics supports, or differs from, that of participating teachers. Can't wait to hear more!
Jacob Whitehill
Lead Presenter Assistant professor
May 7, 2020 | 07:08 a.m.
I think there is a mixture of both :-). We are currently planning an experiment for this summer to study how teachers interact with, and explore whether they benefit from, our technology.
DeLene Hoffner
Facilitator Program Coordinator
May 8, 2020 | 12:49 a.m.
I wondered if you feel you have achieved the goals you set out to with your project.
Jacob Whitehill
Lead Presenter Assistant professor
May 8, 2020 | 06:50 a.m.
I would say we have made significant progress towards our goals. Our current stage is to start looking at how to develop new teacher training experiences to help teachers to perceive classroom interpersonal dynamics more accurately.
DeLene Hoffner
Facilitator Program Coordinator
May 8, 2020 | 08:41 p.m.
Fantastic... your data can give strong examples and evidence for the range of interpersonal actions teachers demonstrate.. both consciously and subconsciously. I am glad to hear that new teacher training is in development.
What great work. These real-time feedback classroom detection systems have the potential to be great dashboard tools for teachers. In our work with the JSMF Teaches as Learners program, we have funded work that may be of interest to this project: Please check out Amy Ogan's lab at CMU and the work they are doing too: https://www.amyogan.com
Stacy Shaw
This technology is incredible. Would it be possible to use this technology in something like more advanced math classes and alert the teacher that many of the students are showing signs of confusion/ disengagement? I could see real value in teachers getting notified (especially if it was possible) in real-time that they've lost the class, and need to stop and answer/questions or clarify a concept before continuing a lesson.
Raffaella Borasi
I had the same impression and question. It is indeed an amazing application of Machine Learning to education. I am part of a study groups at the University of Rochester about AI-Augmented Learning, but I had not before thought about this dimension of using machine learning as a tool for teachers (rather than a direct support to students learning).
I am looking forward to following your project in the years to come - and see what other applications you may developed.
Jacob Whitehill
Assistant professor
Thanks for your question! The system we trained is specific to pre-school classrooms. However, the same machine learning methods could likely be applied for a wide range of classroom contexts and target variables (including confusion and disengagement). What would be needed is a dataset of hundreds of classrooms and associated target values, which is the same order of magnitude of training data we had for the pre-school classes.
James Brown
5/6 STEM Teacher
I love the idea of using technology to help guide teachers in producing better student outcomes! How have you planned to recruit teachers to use this?
James Brown
5/6 STEM Teacher
Do you have plans to integrate this technology to grades beyond pre-school classrooms?
Jacob Whitehill
Assistant professor
For the past several months we have been developing an experimental protocol to explore how automated classroom feedback, as provided by our machine learning-based analysis system, can help teachers to become more perceptive of classroom interpersonal dynamics. We will begin the experiment with classroom teachers this summer. After a pilot run of our study, we will recruit participants more broadly. Since the experiment will be held purely online, we hope to cast a wide net for recruitment. Thanks for your interest!
So far we have focused mostly on pre-school classrooms, but we have also conducted machine learning analyses on elementary and middle-school students. While each new population of students requires its own dataset for training, the general machine learning architecture that we use can likely generalize to a wide variety of students and learning contexts.
DeLene Hoffner
Rachel Navarro
James Brown
Rachel Navarro
I think this methodology would yield fascinating results in college STEM classroom and could really heighten professors' awareness of their own biases and tendencies toward groups of students. This could be an excellent way of promoting inclusion and equity in STEM classrooms.
DeLene Hoffner
Lisa Flores
This is a fascinating tool for teachers! I'd be interested in learning about the findings from the classroom intervention. The possibilities for expanding this beyond preschools is noted above. I think there is also potential for applying this tool with learning of students from underrepresented groups in STEM. Great work!
DeLene Hoffner
Program Coordinator
Good point. I see potential for a wide range of use at any grade level.
Lorna Quandt
Really a fascinating cyberlearning project!
Erin Ottmar
Assistant Professor of Learning Sciences and Technology
Thanks Lorna and Lisa! Its been a really fun collaboration and experience bringing together teams who think about the same problems with totally different methodological strengths and challenges!
Lorna Quandt
Brett Jones
Love it! I can't wait to see how this plays out and where it leads you all with the research and implementation.
DeLene Hoffner
Program Coordinator
Wonderful video and project. It's very intriguing and inspiring. What have you found to be your biggest challenges?
Jacob Whitehill
Assistant professor
One of the fun and challenging aspects of the project has been bridging between our disciplines: education, learning science, and computer science. On the artificial intelligence and machine learning side, there is no dearth to computational challenges of the automatic perception of school classrooms! These include background noise, overlapping speech, uncontrolled lighting, people moving in and out of the scene...not to mention the semantically very high-level construct we are trying to estimate with our system.
DeLene Hoffner
DeLene Hoffner
Program Coordinator
Thank you , Jacob. That makes sense.
Stacey Forsyth
Director
This is a really interesting and exciting project! I'm curious to hear more about classroom teachers' reactions to this technology. Do they tend to be intrigued and interested by the idea of receiving feedback from an AI system? Or skeptical? I think it will be fascinating to see how the system's perception of classroom dynamics supports, or differs from, that of participating teachers. Can't wait to hear more!
Jacob Whitehill
Assistant professor
I think there is a mixture of both :-). We are currently planning an experiment for this summer to study how teachers interact with, and explore whether they benefit from, our technology.
DeLene Hoffner
Program Coordinator
I wondered if you feel you have achieved the goals you set out to with your project.
Jacob Whitehill
Assistant professor
I would say we have made significant progress towards our goals. Our current stage is to start looking at how to develop new teacher training experiences to help teachers to perceive classroom interpersonal dynamics more accurately.
DeLene Hoffner
Program Coordinator
Fantastic... your data can give strong examples and evidence for the range of interpersonal actions teachers demonstrate.. both consciously and subconsciously. I am glad to hear that new teacher training is in development.
Michael I. Swart
What great work. These real-time feedback classroom detection systems have the potential to be great dashboard tools for teachers. In our work with the JSMF Teaches as Learners program, we have funded work that may be of interest to this project:
Please check out Amy Ogan's lab at CMU and the work they are doing too: https://www.amyogan.com
Further posting is closed as the event has ended.