NSF Awards: 1418149
2017 (see original presentation & discussion)
Grades 9-12
Computational thinking (CT) is an important set of 21st century knowledge and skills that has implications for the heavily technological world in which we live. Multiple industries indicate the under supply of those trained to be effective in the computer science (CS) workforce. In addition, there are increasing demands for broadening the participation in the CS workforce by women and members of minority populations.
In the Principled Assessment of Computational Thinking (PACT) suite of projects, SRI International is partnering with Exploring Computer Science (ECS) curriculum developers and teachers to address the need to fill the computer science pipeline with a more diverse population of students. Through this partnership, SRI International has designed and validated assessments of CT. High-quality assessment tools and resources will lower barriers for adopting and using the ECS curriculum, as well as other CS curricula, and will pave the way to reporting evidence of student progress and readiness to engage in further learning.
In the Computer Science in Secondary Schools (CS3) study, a part of the PACT suite of projects, we are using the assessments to examine the relationships among the factors that influence the implementation of ECS. This study is exploring how variation in curricular implementation influences student learning and determines not only what works, but also for whom and under what circumstances.
Other PACT projects are investigating formative assessment, teacher professional development, automated scoring of open-response items, and interactive online assessments. We hope the findings from these studies will help support the development of CT teaching and learning in secondary CS.
Eric Snow
Senior Education Researcher
The K12 Computer Science Framework was recently released (https://k12cs.org/), but contains limited support for teachers to assess the target practices. How can/should CS teachers, particularly at the secondary level, align their assessment practices with the Framework?
Daniel Heck
Thank you for sharing your work on assessing computational thinking. It is addressing an important gap in a rapidly developing area of K-12 education. I will definitely look into more information on your website and am eager to see the assessment tasks.
I'd be interested in knowing if any of your assessments have been used outside the ECS curriculum, or the extent to which you think that could be used as is, or in modified form, to assess computational thinking more generally.
Daisy Rutstein
Senior Education Researcher
Thanks for the great question. We haven't currently used our assessment items outside of the ECS curriculum, although we have used a similar process to develop CT practice items for other projects. In our design process we did try to not align too closely with ECS in order to make our assessment items usable outside of ECS. If other curriculum are teaching similar concepts then I do believe they would find our questions useable. For example, we have an item that's on comparing two algorithms. The algorithms we used are general and not code specific - so other curriculum that includes the concept of comparing algorithms could use that question.
Michelle Huynh
Eric Snow
Neil Plotnick
Teacher
Assessment is easily one of the more difficult tasks that a teacher will face. I viewed the Unit-2 ECS assessment and was pleased to note that they require the students to write how they got their answers. Having students justify their thinking with writing is so much more powerful than just filling in an scantron multiple choice test. Do you have a community of practice in which teachers create and share assessments that they write?
Michelle Huynh
Daisy Rutstein
Senior Education Researcher
Another great question - part of our current project is mapping out the development of an item bank. This would be a place that teachers and researchers could go to have access to CS assessment items. We are still working out details, such as how teachers or others might be able to create assessment tasks. So stay tuned! This summer our focus will be on the development of items beyond our ECS items that we can add to the item bank. For updates on this work you can go to pact.sri.com - once we have this work further along we will post something there.
Michelle Huynh
Eric Snow
Omar Ruvalcaba
I love that you're building this tool for teachers. Considering that you're doing this work with students of color who have been disenfranchised, I'd love to hear more about how you're considering race/culture and issues related to SES. I love the work you're doing but I would love to hear more about how you're involving community members to inform the research and how you're addressing needs specific to students of color. You have my vote! Good luck!
Michelle Huynh
Daisy Rutstein
Senior Education Researcher
Thanks for voting for us! When developing the assessments we have tried to make sure that the scenarios we picked are situations that would be familiar to a wide range of students. We have talked with some students and teachers about the situations and we are hoping to do more of that.
Nicole Reitz-Larsen
Educator
Thank you for sharing your thoughts on computational thinking and being aware that there are students who have not had access to computer science in their schools. Helping students be producers and be able to use computational thinking practices to solve a variety of problems is what will help them in school, outside of school and their career.
It is nice that there are both paper and online tests available for students to take. Have you noticed any differences in scores when students who haven't had a lot of access to computers take the online assessments vs paper?
Another question I have is what are the types of features built into the online testing that assess the students answers? If they are particular vocabulary words, or other items - are teachers scaffolding discussions or activity wrap ups to make sure they are using those vocabularies and concepts so that students are able to naturally use such in their explanations?
Michelle Huynh
Daisy Rutstein
Senior Education Researcher
We have done some initial analysis of the differences between the paper and pencil versions and the on-line versions. We found the differences to be item specific and often there was an interaction effect with the teacher - so the differences were not the same across all of the classrooms. Some questions in which the students performed better on the paper/pencil version had issues such as students had to scroll in the on-line environment. We are still working on addressing this so that there won't be an advantage of the paper versions.
Daisy Rutstein
Senior Education Researcher
We do not currently have any systems in place that automatically provide feedback to the students or teachers. However, when designing the items we did try to stay away from vocabulary terms. If we did need to use a term that we didn't think was familiar to students we tried to support it. In our scoring we concentrated on looking for evidence that students understood the concept -more than if they new particular terms.
Nicole Reitz-Larsen
Educator
Thank you for answering questions around the two types of testing. I appreciate the work you are doing.
Lien Diaz
Sr. Director
I've been following the work of this project for quite some time now and it's been wonderful to see such great progress with helping to define computational thinking more clearly through these assessments, both in paper-pencil and online formats. One thing that caught my attention was Mr. Redhill's testimonial on how the knowledge that he's gained has increased his self-confidence as a teacher and has observed growth in his students by leaps and bounds. For me, this is golden --- that teacher efficacy can be increased via these assessments. Perhaps, it's in learning more about the content that the assessments are targeting and pedagogy that best supports student achievement that are factored in this type of teacher development. Does your project have more data/information on the impact of teacher knowledge gained on student achievement? If so, how is teacher knowledge of the assessments connected to how students perform on the assessments?
Neil Plotnick
Teacher
Agreed! The best assessments should have two purposes. One, to allow teachers to gauge student mastery of a topic. Second, to inform teachers of the quality of their instruction.
Michelle Huynh
Daisy Rutstein
Senior Education Researcher
Thanks for your comments! In our initial study we hadn't really looked at gains for the teacher knowledge. We are currently in the beginning phase of a follow-up study though that is focused more on teacher knowledge of assessment practices in general and the ECS assessments. In this study we are going to be working with teachers to see what their needs are in terms of professional development related to assessments. We are going to provide support for teachers through the use of PD and then follow up with teachers to see how this has helped them. Hopefully this will give us more information on what information about the assessments can help teachers with their assessment and with their teaching. So hopefully in the next few years we'll be able to provide more information on your question.
Caitlin Martin
This is such valuable work. Thanks for sharing. I realize that this was designed around the ESC curriculum, but it sounds like it is broad enough that it could potentially be adapted for use in other cases where CT assessments were needed. One question is that while ESC is primarily a high school course, do you think this PACT assessment could be used with younger grades (middle school, or even earlier) as this type of course is increasingly being implemented with younger students?
Michelle Huynh
Further posting is closed as the event has ended.