Dear Angela, Thank you for watching our video. We appreciate your comments and value the opportunity to share our work here. We felt it was important to highlight how students' voices can impact our work, and are glad you found this interesting. Thanks again!

Hi Myriam, Great questions! 

For each subcomponent within our learning progression we designed two test items. Our blueprint focused on making those two test items maximally different for the student. Maximal difference in many of the subcomponents included the use of a pictorial representation vs a physical manipulative (or real life scene of a miniature playground). We analyzed the results qualitatively to compare the depth of student reasoning and any construct irrelevant variance that could have been caused by the different test item designs. 

In the item with the student video, we were assessing perspective taking on the construction of a 3D space when given different views (top, front, and/or side view). We were hoping that the student would use more than one of the views of the city to determine where one of the items was was on the map, but instead she used the size of the buildings to make her decision, which was not the intent. 

Iterating on assessment design based on what we intended vs the reality of what students do and say, is very insightful! 

Thank you for elaborating. Yes, very insightful!! 

In the first one, did one of them show deeper student reasoning than the other?

In that particular item the pictorial representation functioned better than the physical materials.

I think it is is also important to note that in addition to comparing the two models at the skill level, we also reviewed the core concept that the skills would be assessed within. Switching between the two models after each item could impact setup time and increase the cognitive load for the student and assessor to orient themselves to the item. When we didn't have consistency across the core concept, we revised and retested individual items. It was a very iterative process. 

Hi David,

The goal of the project is to develop formative assessments focused on numeric relational reasoning and spatial reasoning. We are looking beyond correctness and looking to support teachers with better understanding the level of student reasoning on a skill. 

Prior to working in research, I was a middle school math teacher and elementary math specialist. As a teacher, I remember thinking that some assessment items didn't function well for students but not really knowing why. I believe that conducting think aloud interviews with students adds an extremely valuable lens on assessment development. Through this process we have been able to see construct irrelevant variance through the students eyes. It was amazing to see how students reasoned through an item.

I think it would be extremely valuable for researchers to consider the value of think alouds with students for the purpose of identifying construct irrelevant variance in an assessment item. 

Hi Noelani,

Excellent question! The individual assessment items we used with students during the think aloud interviews were based on the learning progression associated with each construct. The items are differentiated by grade--so a kindergarten student would see a simpler version of a puzzle, for example, than the second grade student during the concurrent portion of the think aloud interview. But the main think aloud prompts and retrospective questions we posed were the same for students at all three grade levels. 

Thank you for your interest in our study!

Hi Eva,

Thank you for viewing our video and asking about bias. Before any of the assessment items are finalized we share the wording as well as the supplemental materials used with external experts in the field. One component of this review includes evaluating the items and materials for possible sources of bias that might unintentionally privilege one group of test takers over another. Specifically, do the items require background knowledge unrelated to the concept being tested that would differ for students with different backgrounds? Is the language sensitive to students from diverse backgrounds, students with limited English proficiency and students with disabilities? Ultimately, we want to ensure that all the items are free of bias in language and content. 

Thank you for viewing our video, Meagan. We appreciate your feedback!

Thank you for your interest in our work! I think recording students' responses can be valuable to understand their reasoning. Video recording is especially illuminating with younger students who often demonstrate their thoughts via actions (using pattern blocks to fill a puzzle, etc.) alongside verbal communication. And yes--it is a lot of fun! 

Thank you so much for watching our video and sharing your thoughts. We appreciate your comments. 

Thank you for your interest in our work and question, Paulina. The purpose of the think aloud interviews for this component of the project was to determine whether test items or tasks elicit the intended problem-solving processes. We worked with different cohorts of students across different time spans and constructs. (Early cohorts of students participated via zoom due to COVID and later cohort interviews were conducted in-person.) The early remote interviews enabled us to work with students in varied geographic locations. We also paid close attention to recruit a balance in grade levels K-2 and gender as well as a range of reading levels. Finally, we collected information about students' participation in preschool and after school programs, home language, and ethnicity and are still analyzing those data. 

Thanks for watching and your comment! The old saying "kids say the darnedest things" is true! We learned so much!