As I watched the video, I was fascinated by the potential of students’ academic conversations. Can the project share some examples of these conversations, either in video or in transcripts? Also, I’m interested in what would sustain the level of conversation beyond the treatment phase, which is another way of asking why conversations diminished during the second baseline phase.

Great goal! Academic discussion is a big part of many of our projects also, and I am intrigued by your focus on this. Sustainability is a major challenge for many of us, and I’d like to hear your take on it. Aspects that we tend to look at include the length of the treatment as well as the cultural changes and technical systems put into place by the partnership to extend the gains beyond the project. I would love it if you could share some thoughts on the issue.

We trained teachers on specific routines that they articulated really buying into. They expressed ‘hating’ to have to go back to instruction without conversation routines. They all commented “do we really have to back to our old way of teaching?” “…this is powerful …my kids love it and I could see their thinking better now.”

Of course, buy is from all teachers is very difficult. From my past experiences, getting the principle to buy in and make teachers accountable helps motivate more at least try the approach. Once they try, they see almost immediately that their shyer and even more challenging students engage productively.

Very interesting; I am a Computer Science and bilingual teacher that works in an inner city school whose student population is 90% on free or reduced lunch programs and 70% have either been previously identified or are currently English Language Learners. Many of my students struggle with reading comprehension and understanding of technical and abstract terminology required in Computer Science courses. I would have loved to have seen video excerpts of students engaging in conversation and utilizing the engineering STEM materials. I am also interested to know if follow up studies are being considered to continue tracking these students as they progress through subsequent grades? And would STEM engineering materials be provided at appropriate grade levels as students continue to learn about STEM engineering design in 1st, 2nd or 3rd grades? Have you consider the evidence that you can provide that determines the impact of the STEM engineering materials as children get older? I also wonder if the ELL students would rapidly improve their language skills in both their primary and English languages? I really think it is great that you are doing research at such an early and critical childhood stage!

Very interesting project. And I think the questions asking for examples, about the teacher change model, and about making the culture stick (which is really what you need to happen and learn about for real impact) are right on. I look forward to more detailed answers on those things.

As well, I wonder what was the difference between the treatment and other group. Did the other group use the same materials but not engage in the same discussions? Or something else? When I know, i will ask some additional questions and make some additional comments.

I wonder, too, how you extended what the EIE folks learned while developing their curriculum. I know EIE a bit, though I don’t know much about the evaluations they did while designing it or after, and I don’t know what their teacher instructions say about promoting discussion.

Finally, your conclusions are nice — they tell us that it is worth continuing to do design activities with those who do not yet know how to have an idea-based kind of discussion and that EIE curriculum materials are good for promoting such discussion when used a certain way. But I wonder if you are finding out some other things: (1) Are your results just for EIE and the particular way your teachers are using it, or do they generalize beyond that? And, if so, how do they generalize? (2) How, exactly, are your teachers using EIE? That is, I want a lot more detail about the treatment. (3) Finally, there is literature out there (some I wrote) about using design activities well to foster learning. What might you add to that literature? And how did that literature help you? I see you used brain literature, but I don’t see how you used cognitive literature or design-based learning literature to guide your work, and I think bringing those literatures in will allow you to get to more specificity about how to foster the kinds of discussion you want and also help you generalize from your results beyond EIE.

One more thing, there is some literature on creating culture in the classroom — not so much with quantitative results, but ideas that come from what we know about cognition (some I wrote here also). That should help as you move forward from finding it is worth using design activities to foster ideas-based discussion towards figuring out how to sustain the culture being created.

Janet

Thank you for your great comments and great questions Janet. Our project was focused on development of the intervention, not instructional change. We needed to train teachers on the use of the units and in the conversation routines in order to test the degree of feasibility for young elementary students. I absolutely agree that real impact on teacher practice would require an examination of their practice after the professional development. I am also aware of the literature that demonstrates that teacher beliefs about learning and instruction are associated with the degree of change in practice and therefore effective PD would target these beliefs and examine their practice more systematically. We did examine instruction for degree of fidelity to the treatment, but the outcome measures were focused on student engagement. (In case you are curious our fidelity was high and reliability on fidelity coding was very strong-exact agreement and Kappa.)

In terms of design, for a single subject design, a separate control group is not necessary. The students act as their own control, which is why we included 2 baseline periods. This design is rigorous and conforms to the What Works Clearinghouse standards. But our next step is to conduct a group design where we would have an intervention group and a control. So your question is timely and is the next step. Since there was not enough past research to guide reasonable hypothesis on the direction of student outcomes with respect to our combination of activities with ELLs, a single subject design made the most sense to start. If all goes well we will be reporting on results from a group design this time next year.

With respect to our adaptations to EiE, yes we did modify! But all 4 lessons were implemented. We reduced the length of the story context (lesson 1), in this iteration, our text modification may have led to the reduction in student engagement during the reading of the story context. We used the hand pollinator unit which included a play, so it could also be an artifact of the genre for this particular unit. We took what we learned from this initial test and made guidelines for text modifications and have implemented it again. We are looking at the data and will know soon if our text modifications were better this time around in terms of keeping students engaged. Other modifications were to the worksheets students used to record what they learned about the materials in lesson 2 and 3. We lessoned the linguistic load to be appropriate for emergent readers. Lesson 4 was implemented as designed by EiE. We also added assessments to their existing assessments, a card sort that targeted, understanding of technology, the design cycle and the science concepts. We also implemented a think a loud protocol while they are doing the card sort so we can understand their thinking process while engaged in that task. We also embedded academic conversation routines in each of the 4 EiE lessons as well as the technology in a bag ‘pre lesson’.

In terms of your other questions: 1) Are your results just for EIE and the particular way your teachers are using it, or do they generalize beyond that? And, if so, how do they generalize?
Since we were able to find the same trend across types of students (low and high ELLs, and low, moderate, high ability students), and across classrooms (4; 2 mainstream contexts and 2 dual language immersion) we achieved a critical standard for a single subject design—replication across settings and individuals. Thus, we can generalize across to these two types of classrooms contexts. That is, we would expect that mainstream and dual language teachers who engage students in academic conversations while engaging in hands-on science and engineering design tasks (EiE includes both) would have similar results for their ELLs.

(2) How, exactly, are your teachers using EIE? That is, I want a lot more detail about the treatment.
See response above regarding EiE use. I hope that is enough detail.
(3) Finally, there is literature out there (some I wrote) about using design activities well to foster learning. What might you add to that literature? And how did that literature help you?
Janet, your work very much aligns with ours. We use the same research base in terms of what we know from the learning sciences. The brain stuff I like because of the great visual it provides and emphasizes the need to highlight the importance of scaffolding disciplinary discourse—which you also use. We also use systemic functional linguistics to frame the discourse work because it provides very specific language targets for teachers and students. Since the focus is on ELLs, we find that language framework useful when examining discourse patterns and in providing teachers with cues for what to listen for when they monitor student conversations. Most of the work in academic conversations is qualitative and corpus based, we hope to add to the field by confirming what qualitative (corpus-based) researchers (applied linguists) have been claiming for at least 15 years (much longer for learning sciences researchers)—in short, discourse matters. The nature and quality of students’ disciplinary discourse matters in both the development of understanding and in the ultimate attainment of targeted concepts. But we hope to provide implementable routines teachers can be trained to use and modify to fit the needs of their students.
I will definitely look for more of your work to help us shape the classroom culture from a cognitive perspective within an engineering education context. The work we site is more general work in cognition. Thank you for that tip!

So glad you did not have a control group. It seemed way too early for that, but the way you described the research, I thought you were doing that. Phew!!!

More later on the rest.

Congratulations for this important work! As a linguist engaged in using local languages for transforming the teaching of STEM in Haiti from rote memorization to active learning, I am fascinated by your research to improve education for underserved populations. I couldn’t agree more with you that, indeed, “discourse matters.” So language choice, too, matters: Students cannot effectively explain their thinking in a language that they do not speak fluently. And as Janet Kolodner points out, classroom culture too matters. In the Haiti case, we’ve seen first-hand the importance of choosing the native language of Kreyòl for teaching STEM instead of French which, for most students and teachers, are a (more or less distant) second language. Once students (and teachers!) are free to use their native language, and encouraged to do so by technology in that language, they become so much more effective in expressing and sharing their thought processes in academic conversations. I too would love to see samples of the conversations. I would also like to learn more about the native-language background of your emergent bilinguals and what role, if any, their native language played in their process of engaging in these academic conversations that you’re studying in your project. Thank you!

Thank you for this comment Michel. We will work on the cultural component more as we move forward. Your point about the role of the student native language in their process of engagement is well taken.

Good start. This is as very interesting way of engaging students and getting them to communicate. How do you account for their level of reading as it would seem this would hinder understanding of the academic language where the words are not your simple daily conversation words? Also I wonder how fifteen days can actually give you data that is truly indicative of a change achieved. Nevertheless I see the potential in this work when the underlying issues reading level, school readiness etc. have been handled.