Hi! We did collect data on problem performance. The study was set up so that students had to get the problem correct before moving on, so one thing we looked at was how many incorrect answers they had before getting it right. There wasn't a clear relationship with Q or I codes, but right now we are looking at whether some affect/arousal features in the speech are related to correct/incorrect problem solving.

I am thinking of a learning scenario where there is a group design task which is commonly found in engineering, maker-oriented projects, or robotics team projects where the both hands are engaged with tools and materials. Learning is gestural and embodied. One could imagine the role of the computer as a collaboration detector in the environment to help identify and assess levels of collaboration and engagement. In an assessment situation when you need to figure out individual performance and contribution to a group ask, one could identify who was really contributing to the problem solving. 

Can you say more about what you have planned next? Those Q codes are fascinating.

Thanks! Although this is an early-stage research project, we believe that it could lead to a wide range of applications for different learning contexts. PISA has recently added a collaboration task to its assessment, and perhaps in the future one of the data streams collected will be audio to help better understand how students are working together. 

This technology has the potential to reach all sorts of learners in many different contexts and support teachers in a) understanding how and when their students are learning and b) providing just-in-time support when it's needed. We might even be able to translate this into informal learning spaces as well (such as museums) or distance education. There are a lot of possibilities, which makes this work so exciting!

Yes, we are trying to assess/measure face-to-face collaboration. I think the eventual system could definitely be used in online learning systems, as long as there was high quality audio coming from the participants. This would be a really interesting extension of the work. 

Thanks!

The group collaboration codes were trying to capture how much the individuals in the group were intellectually contributing to the problem solving. So, in a high quality group, all three members would be actively intellectually engaged (which could involve mostly listening and asking the right questions at the right of the other members). The individual codes were specific behaviors or actions, like asking a question, agreeing with someone, or verbalizing their thinking out loud to the group. 

The issue about embodied action is really interesting and I'm glad you asked about it. One thing we were really concerned with early on is how much non-verbal communication would play a role in the collaboration. The video data helps in two ways: 1) it makes it a lot easier for the researchers to apply the codes to the data, especially the individual codes, and 2) it helps us see how much non-verbal communication is going on and whether or not that is having an impact on our coding. There was a good amount of non-verbal communication, but in general, that added nuance would not have changed our group-level codes. However, it did make it easier for the researchers to understand the group dynamics and apply the individual codes. 

Thanks!

Thanks for this detailed response! I look forward to seeing more of this work.

I'm sure the type of activity had an influence on this. We picked this type of activity specifically so that we could capture a good data set (one with a lot of communication). We needed to have lots of opportunities for students to talk to each other to solve the problem, and needed a range of outcomes (good collaboration to not collaborating) in order for our machine learning to work well. These activities were relatively short closed tasks, with a known answer. The nature of the collaboration would likely look different with longer open-ended tasks. 

I think there are a lot of implications for teaching. One thing in particular would be that typically teachers, when going around to monitor groups, only see a small snapshot of the collaboration of the group (and one that is usually colored by the students knowing that the teacher is watching them). One take-away would be to not rely too much on that little snapshot, since we know that there is a lot of variation within groups and within tasks. 

As of right now, we are not looking at the words and phrases. This may be something that we have time to investigate in more detail later this year, but it's not part of our current analysis. 

We are looking at the relative contribution from group members. A lot of the speech metrics are normalized by individual and/or by group so as not to conflate more talkative groups with less talkative groups. 

The Q codes do capture timely student contributions to the group effort, even if that student hadn't been talking a lot. If they are intellectually contributing, even in a small way, then they are helping that group.

In Phase 1, many of the children knew each other from class, but not all of them. In Phase 2, it was intact classes that we went into to collect data so they all knew each other.

Thanks!

Thank you, Cynthia, this information is quite helpful.

Thanks Kathy! I think these are really good questions for the next stage of our work. We want to look at longer, more open-ended tasks where things like assigned or not-assigned roles could make a difference, or how task milestones could affect how well they are able to self-manage their problem solving. There are so many possibilities and it's really exciting to see where this might go. I think incorporating interactive simulations (like PhET) could be really interesting down the road!