Hi Vivian. Thanks for the great questions. We've seen a few examples of pilot teachers differentiating based on gameplay data analysis. One example comes from the use of gameplay data about "supporting claims with evidence that is related to and in support of the claim." Students seem to struggle with this argumentation skill because the differences in the language of the evidence can be subtle (especially as players also need to relate evidence to one of four different types of "argument schemes"). Teachers have tended to use one of two sets of materials/questions depending on their diagnoses for why students are struggling: one set of materials that helps students focus on specific words to determine whether the evidence is irrelevant or contradictory; and one set of materials that gives students prompts they can use to think back to the point of their claim and then determine whether their evidence supports it. Teachers have then used these materials with small groups, or whole class. It's too early to tell whether these have an impact on student outcomes.

Interestingly, most of the pilot teachers (middle school science teachers) have tended not to differentiate using small groups—rather, they differentiate by classes (e.g., period 1 vs. period 2). That is, they make decisions about changes to instruction at the class level, rather than at the individual student level.

As for your second question about pilot teachers talking with each other: no, but it's an absolutely fantastic idea and I'm going to see if we try it in the next few weeks (we're just finishing up a pilot now and then heading into an impact study in the fall). Individually, pilot teachers have many opportunities to influence the design of the Report Helper, as we do three think-aloud sessions with them as they use the data dashboard and Report Helper, as well as follow up interviews.

Thanks again for the great questions, and the excellent suggestion about having the pilot teachers talk with each other.

Your observations about how middle school science teachers differentiate instruction/gameplay are VERY interesting! Do you have any ideas about how to shift their decision making from the classroom level to that of individual students.

In our work with K-5 teachers as part of Math for All, we are asking teachers to pick a focal student (a student whom they have questions about regarding math) and we are asking them to use a neurodevelopmental framework to observe this student throughout the school year so they get a better sense of what this student's strengths and needs are and so they can use this information to provide alternative means to help this student achieve high-quality learning outcomes. Teachers find that by focusing their planning in this way often helps many students in their classroom to succeed. We are also having general and special education teachers who serve the same student collaborate on planning instruction, which also helps to focus them on individual students. You can find more information about our work here: http://stemforall2017.videohall.com/presentations/1011

Hi Wendy,

Lots of questions here! Thanks. I won't do most of these justice, as you're asking some very good questions.

For starters, you can learn more about the game here: https://www.glasslabgames.org/games/AA-1

The game wasn't developed to help students "master" argumentation. Rather, it's designed to introduce students to the "parts" of an argument, and to evaluate the strength of arguments by asking critical questions and determining whether evidence is well-backed. Students do not generate original arguments specifically, but they can choose from multiple, "pre-packaged" arguments. (This certainly requires more discussion than is possible here.)

Our study is not actually about gameplay and student achievement. SRI conducted studies on the relationship between gameplay and argumentation skills (as measured by ETS's CBAL, though I can't remember the specifics now). What we are looking at is the relationship between teacher content knowledge, pedagogical content knowledge, and how well they use the gameplay data for formative assessment. We are ALSO looking at student impact measures to see if there are differences between teacher groupings: those teachers who receive the "educative materials," which are designed to help teachers make use of the gameplay data for assessment and tying it to other classroom-based activities, and those who do not have those materials.

The game does not include "science levels." As originally designed, the game was intended to be a stand alone exercise. In fact, it wasn't developed specifically for science classrooms. But we have developed an associated "mini-unit" on energy and argumentation that enables teachers to tie gameplay to argumentation in the classroom, using specific content objectives.

Your question about other games is an excellent one. Should we find meaningful differences in this work, we hope to scale it out to other games, where the questions would relate to whether the presence of a "data dashboard with educative materials" helps teachers integrate the use of gameplay data into their formative assessment practices.

As for "space in the curriculum," we have designed the intervention such that the game is tied (though not completely prescribed) to a three-week, supplemental mini-unit that helps teachers integrate argumentation practice into their regular materials for a middle school energy unit.

Lastly, the "educative materials" that I noted above are the "training." That is, through iterative design, we seek to create materials that should help teachers become better at using the specific data from this game, as well as help to help them understand how games CAN operationalize targeted learning objectives (which isn't to say that they always do—that's for a teacher to evaluate, we hope).

Thanks again for these good questions Wendy.

Jim

• Hello Miriam. Thank you for your comment and your questions. I am one of the researchers working specifically on the dashboard revisions and the educative materials. In response to your question about the process of making the revisions, we have used a design-based research approach for this project. We've conducted several cycles of data collection and iterations on the dashboard and accompanying materials, during which we have interviewed teachers, observed them in the classroom, and conducted think alouds where we asked them to use the dashboard and Report Helper to plan for upcoming instruction. This process has been very useful, if a little challenging to keep up with the timeline required by a research AND development project, while ensuring that our changes are driven by the data we’ve collected.
• The game we’re using for this study already had a data dashboard prior to our project, so we started by interviewing teachers, asking them to explain how they understood each aspect of the dashboard. From these interviews, we generated three major findings: 1) although all the teachers we interviewed had already used the game with students, many only had vague notions of what gameplay entailed, 2) teachers struggled to make connections between data on the dashboard—which was organized around gameplay events—and student performance on the targeted learning objectives, and 3) teachers wanted to know how students were progressing more than whether they had struggled along the way.  
• So we wanted to redesign the dashboard to make it more evident the ways in which gameplay related to argumentation skills. The game was built to operationalize four aspects of the Toulmin model of argumentation (using evidence of different argument schemes, supporting claims with evidence, querying others’ arguments with critical questions, and using backing to strengthen your argument), and so we decided to use these skills to organize student performance. This led us to articulate a learning sequence featuring these four skills, to create a new set of reporting rules that approximate student performance on these skills, and to disaggregate performance by game mission so teachers could better target students who were consistently struggling.
• As for the educative curriculum, we have several goals around helping teachers to use gameplay data to inform instruction. Data from gameplay have specific characteristics—one of them being that since it is built on gameplay moves teachers often don’t understand what student action the data represent. This makes it difficult for teachers to assess the extent to which they trust the data and to develop plans to support student success.
• So in the educative curriculum, one goal we have is to help teachers understand what it is that students are doing in the game, how the game operationalizes the four argumentation skills, and ways that students are likely to struggle. We’ve tried to organize this material in a way that makes the information actionable and helps teachers use the gameplay as a launching point for a more in-depth exploration of argumentation within the context of their content and curriculum. 

Miriam—

Thanks for these fantastic comments. With respect to your comments about "educative resources"—that's exactly right! (Only we've been calling them "educative materials.") As Noah said, the idea is to present those materials in a way that teachers will find them immediately actionable. As we are trying to help teachers become better at "using gameplay for formative assessment," we present the material in a three-step process that broadly mirrors one framing of formative assessment: 1. What are the learning targets? 2. Where are my students now? 3. How do get from here to there?

So, these materials are less linked to specific content (as is usually the case with educative curriculum, to your point) than they are to specific teaching practices. We'll see whether and in what ways they influence practice in the impact study...

Thanks again for these great comments.

Jim