Dear Nanye,

Will it be possible to obtain a sample of the rubric for collaboration? 

 

Hi Cristo,

Thank you for asking. We are still refining, validating, and improving our rubric and will be prepared and happy to share it at a later date. We will make it available in the coming months.

Hi Patrick,

This is a great question and one that we get all of the time. It is a question that we try to take great care in! Yes, we are using body language and facial expressions to identify collaborative, and not so collaborative behaviors. It is important to note that how collaboration is defined and characterized differs from person to person- right? If you ask 10 people, you will most likely get 10 different answers. Because of that, we spent a significant amount of time surveying the literature and all kinds of research on how people determine what is constructive and detrimental to collaboration. We used research in computer mediated collaboration space, the military (which uses collaboration in high stress situations), and various other collaboration practices and theories, like T7 model for team effectiveness, Team Basics Model, Five Dynamics of Teamwork and Collaboration Model, Conditions for Team Effectiveness Model,  Five dysfunctions of a Team Model, Shared Plans theory and Joint Intentions Theory-- just to name a few. We spent a lot of time doing an analysis of the research to develop a framework and pull many of these ideas together into observable practices.

I cannot say that the there are universal behaviors associated with collaboration, but we have use research to inform our work, and continue to build on that work as we learn and read more. In addition, we perform validation tests on our rubric and framework using expert reviews and disconfirmation approaches for that very reason. The hope is that we can get different viewpoints (research based, experts, student surveys, etc) about what collaborative behaviors look like. Furthermore, we are working to diversify our participant population. Students are volunteers, which instantly biases the data, but we are reaching out to participants across the country to obtain a diverse population to help provide variety. With that said- we are aware that our participants are neurotypical. There is some other work being done in our research group, not related to this project (yet! :)) that works with this population. But, we are not yet incorporating that line of work into this project at the moment.

In terms of teachers using this- we really hope so! In our pilot of the work, teachers were excited about an instructional resource like this; something that could help make their already difficult jobs a little easier. The idea is that teachers would be given information on individual student behaviors that lead to an overall collaboration quality. The teacher would also get information on the group quality. Teachers would then use that information to help students continue to strengthen their contributions to the contributions to the group setting by either increasing or minimizing certain behaviors. We are working on a recommendation system that will provide the teachers with suggestions about what to highlight and what to minimize. However, we do not want to replace teachers- we want this to be something that helps them.

I hope this helps!

Thanks for the detailed response. It was indeed helpful! I assumed that the underlying theory was subtle and complex, and you've given me a very nice overview of the thinking that has gone into this.

It's still not entirely clear to me how I might use this information as a classroom teacher, but it's certainly an interesting direction for research.

 

Hi Patrick, Thank you for pushing us with this question. The goal is to make sure that teachers find this sort of thing useful. We are still developing utility cases for this so that we can provide a more realistic and implementation-based response, but in theory, this is an example of what we envision:

Students are working in groups during class- let's say they are working on an online activity. While working on the activity, each student has a camera that records their interactions for the duration of the activity. While the activity is being captured, the behaviors are being processed and analyzed using machine learning models that have been trained on diverse data. The machine learning models use this data as input to determine how each students is performing on different levels. For example-What role is Student 2 playing? What role was Student 3 playing? The machine learning models also analyze the input data to determine how well the group is doing. For example, is the group working effectively together; where they are sharing ideas equally and displaying respectful interactions while doing so?

The teacher gets that information and determines how much support to give each student and the group as a whole. For example, the teacher might see that Student 1 was offtask for most of the activity. The teacher can then talk directly to the student and provide appropriate supports for them. The recommendation system will try to help the teacher by providing suggestions of the kinds of interactions that Student 1 might want to do more of. For example, the recommendation system might say that Student 1 should perhaps spend more time following along or helping coordinate the group. The teacher may use that suggestion as part of their conversation with Student 1. 

This still down the line, but it is the direction we are heading.

I hope this helps! Please let me know if you have other thoughts.

Hi Marion,

Thanks for your question. Patrick (above) asked a similar question, so I will repeat some of my response here. It is important to note that how collaboration is defined and characterized differs from person to person. Because of that, we spent a significant amount of time surveying the literature and all kinds of research on how people determine what is constructive and detrimental to collaboration. Our work is informed by research in computer mediated collaboration space, the military, and various other collaboration practices and theories, like T7 model for team effectiveness, Team Basics Model, Five Dynamics of Teamwork and Collaboration Model, Conditions for Team Effectiveness Model, Five dysfunctions of a Team Model, Shared Plans theory and Joint Intentions Theory-- just to name a few. We spent a lot of time doing an analysis of the research to develop a framework and pull many of these ideas together into observable practices.

From this we created a rubric that has 3 concepts or dimensions of collaboration, broken down into 6 themes, 17 subthemes, and over 70 classes (behaviors) spread across the subthemes. An example, of this is at the concept called "individual awareness". That is broken down into  building collaboration and detracting from collaboration. Those are further delineated into subthemes like information gathering, sharing ideas, and disengaging from the group. The subthemes are further broken down into classes like comforting others, setting up roles, working off-task. We created so many categories to address the fact that each behavior has a function based on the context it is happening in. The categories help provide explainability for function (not intention) of the behaviors as they relate to the overall collaboration process. 

In terms of consistency, we perform validation and reliability tests on our rubric and framework using expert reviews, disconfirmation approaches, and Cohen's kappa calculations to achieve and maintain consistency. We are also working to diversify our participant population. Students are volunteers, which instantly biases the data, so we are reaching out to participants across the country to obtain a diverse population to help provide variety of body language in group settings.

Hope that helps!

Hi Kimberly,

I see that you are an informal educator. I want to mention that so far, this work is being conducted in a lab setting. So, there is very little noise, in terms of external distractions. Students are working small groups after school or after class. It would be quite interesting to implement this in an informal setting.

In terms of the setting that we are working in, we created a rubric that has 3 concepts or dimensions of collaboration, broken down into 6 themes, 17 subthemes, and over 70 classes (behaviors) spread across the subthemes. An example, of this is the concept called "individual awareness". That is broken down into  building collaboration and detracting from collaboration. Those are further delineated into subthemes like information gathering, sharing ideas, and disengaging from the group. The subthemes are further broken down into classes like comforting others, setting up roles, working off-task-- these are the behaviors we identify from the video capture. We created so many categories to address the the fact that each behavior has a function based on the context it is happening in. The categories help provide explainability for function (not intention) of the behaviors as they relate to the overall collaboration process.

Hope that helps!

Hi Kevin! Thanks. Congrats on the submission of your video as well. It looks great.

That would be an interesting study; to determine how well do our collaboration framework and machine learning models transfer across disciplines. Our hope is that the use of behaviors as the driving factor, as opposed to speech, will make our collaboration model and machine learning models transferable to other disciplines within science, hopefully STEM. Because our literature research and analysis of collaboration as a practice was not specific to science, it is possible that it could be transferred to other disciplines outside of STEM, but we don't have data for that. In addition, the tasks that the participants engage in are in life science and physical science, so many of the practices might be science specific, even though the codes used to identify the behaviors seem generic. One of the lines of study on the project is to take a look the relationships between the collaboration activities and the collaboration behaviors. We want to explore what kinds of behaviors might be associated with the different task characteristics. For example, do tasks that require the building of models elicit different kinds of behaviors and interactions? Does the quality of collaboration change? If so, how? Those kinds of questions would be interesting for other disciplines like math, or even social studies.

Hope that helps!

Hi Satabdi. Congrats on your video submission. Nice work to you, too!

This is such a good question. We have spent all of this year working on this transition. We know that working online generates similar behaviors, but there are also many changes that occur. We learned that, for example, exhibiting attentiveness in person will look different from when working online. In a face to face setting, an attentive participant will most likely look at their peers directly. In an online setting, everyone is looking at the screen, so it becomes harder to tell. We had to pay more attention to minute changes in behavior, such as head positioning and eye gaze. In an online setting, you can usually tell if someone is looking at their cell phone vs paying attention to the task being completed based on where their eyes are looking and the way their head is tilted. 

In terms of age, we didn't find much differences in age-based interactions, surprisingly. There is an article that said that after the age of 15, collaboration can be recorded successfully. However, because the middle school students that we started with were all volunteers, we had biased data and their behaviors tended to conform in a particular direction. We are still exploring this question as we collect more data.

Hope this helps!

Hi Brian!

My colleague Anirudh is better equipped to provide this information in the next day or two. But in short, we are looking at accuracy in 2 ways: from the collaboration conceptual model perspective and from the machine learning perspective. We are conducting validation studies on our model and rubric. So far, we have been able to reach reliability agreement above 90%. Our Kappa scores for reliability have been hovering around 60%- if I remember correctly. Our validity emerges from the results of the reliability scores (using a disconfirmation approach thus far), leading to refinements to the conceptual model and rubric. In terms of the machine learning models, our F1 score performance is between 80-85%. So, we believe that we are achieving good accuracy in both perspectives.

yes- we code at individual and group levels. We code them independently to avoid bias in the coding and to avoid propagating error. The levels are designed to be interdependent (one level will build to the next) but they are coded independently.

Yes, we have found that levels that require more nuance have lower accuracy than levels that look more specifically at gestures and movements.

Hope that helps!

Hi Brian!

Thank you for finding our work fascinating! As Nonye mentioned, currently we observe the F1 score performance to be around 80-85% for both individual roles and behaviors. Details about the data curation, data processing and machine learning modeling steps can be found in the following paper: https://arxiv.org/pdf/2007.06667.pdf. This paper was published last year and describes our initial attempt at using machine learning to automatically assess group collaboration quality from individual roles and behavior annotations. For the feature representation described in our paper, we did find that it was slightly easier to estimate collaboration quality from individual roles than from individual behaviors.

Since then we have developed more sophisticated machine learning models that offer more explainability and insight. Our most recent work, accepted in the EDM 2021 conference, will soon be made available in the open-access proceedings: https://educationaldatamining.org/edm2021/accep...

Very helpful, thanks for the detailed explanations Nonye and Anirudh, and linking the paper! Sounds like very positive results in terms of accuracy and reliability so far. Nice to see such a strong emphasis on a conceptual model to drive the rubrics and validation work!

Nonye - can you explain more about the disconfirmation approach you use? 

Hi Brian. In short, it looks for cases where the researchers might disagree on whether something fits with the larger idea of the study. We look for disconfirming cases. Here is a paper that describes it:

Booth, A., Carroll, C., Ilott, I., Low, L. L., & Cooper, K. (2013). Desperately seeking dissonance: identifying the disconfirming case in qualitative evidence synthesis. Qualitative health research, 23(1), 126-141.

Hi Kemi,

Nice to see you here!

As of now, we have been looking at science activities. When we started the work, students worked on model building projects face to face, so our coding rubric was designed to capture hands-on activities and behaviors. With the transition to online settings (due to COVID and other educational changes), we modified our rubric to capture online interactions, where the participants solve problems virtually. We tested various platforms and performed some case studies before completely transitioning.

Yes- the rubric uses both verbal cues and behavioral cues. In fact, in the first year of our study, we tested whether the use of behavior alone was at least as good as using both verbal and behavioral interactions. We found that, of course, using both verbal and nonverbal interactions was more accurate than nonverbal alone, but our reliability between the two modalities was in the moderate Kappa range- which is a much more strict reliability calculation. This showed us that the rubric was working. Moving forward, we are experimenting with the use of verbal interactions as part of the machine learning training.

Hi Khyati

This is a wonderful question. We keep this as part of our work constantly. We do a few things to diversify our groups. Students are volunteers, which instantly biases the data, but we are reaching out to participants across the country to obtain a diverse population to help provide variety. We reach out to groups that have diverse student populations and our network system that works with various groups of people. We have used research to inform our work, and continue to build on that work as we learn and read more. Our coders are also trained with the rubric to ensure that bias is recognized and addressed. In addition, we perform validation tests on our rubric and framework using expert reviews and disconfirmation approaches for that very reason. The hope is that we can get different viewpoints (research based, experts, student surveys, etc) about what collaborative behaviors look like.

With that said- we are aware that our participants are neurotypical. There is some other work being done in our research group, not related to this project (yet! :)) that works with this population. But, we are not yet incorporating that line of work into this project at the moment.

I hope that helps!

Hi Marcia. Thanks for your question. Can you explain what you mean by status? Are you referring to SES? Or group dynamics related to hierarchy? Or something else?

Thank you!

Hi Lindsay-

Can you say a little more about what you are interested in concerning the machine learning approach? For example, are you interested in how the machine learning models are trained? How the data is analyzed? Something else? I will also bring in Anirudh, one of the new co-PI's, to help answer this question.

In terms of the nonverbal behaviors we use, we created a rubric that has 3 concepts or dimensions of collaboration, broken down into 6 themes, 17 subthemes, and over 70 classes (behaviors) spread across the subthemes. An example, of this is the concept called "individual awareness". "Individual awareness" is broken down into building collaboration and detracting from collaboration. Those are further delineated into subthemes like information gathering, sharing ideas, and disengaging from the group. The subthemes are further broken down into classes like comforting others, setting up roles, working off-task. We created these categories to address the fact that each behavior has a function based on the context it is happening in. The categories help provide explainability for function (not intention) of the behaviors as they relate to the overall collaboration process. Int he upcoming months, we will be sharing our rubric, which will hopefully provide more details and useful information!

I hope that helps.

Thank you, Laura!

Hi Suzanne. Thank you for sharing! I got a chance to browse through your website and watch your video. Very inspiring- such great work! I am looking forward to sitting with all of the different rubrics that you have developed.

We are hoping that our work will lead to a feedback and recommendation system to help with that.

Thanks for engaging with our video!

Thanks Donna. We are looking forward to sharing what we learn.

Thanks for watching our video!

Thanks! We will have to check that out!

Thanks Judi! We hope that the rubric will contribute to the field.

Hi Sam,

1) Yes. our long term goal is to be able to provide the students with their own feedback individually and as a group. The goal is to give the teachers AND students feedback so that students can monitor themselves.

2) Are you asking if we are also analyzing the quality of their products that emerge from the collaboration? Yes- one line of our work is to look at the relationship between the artifact quality and the collaboration quality. We spent a lot of time designing the collaboration tasks so that we could analyze how different task features and resulting artifact may relate to different collaboration behaviors and/or interactions.

We also have codes aimed specifically to capture the use of tools as they collaborate. So, we incorporate how they interact with the collaboration tools- for example, are they contributing to the group effort; are they just fidgeting with the tools? What the video doesn't show is that the students are working on open ended questions that require them to build models. They are using paper manipulatives to create things like molecular models- these serve as the tools they are using. 

I hope that helps!