914 Views
  1. Leanne Ketterlin Geller
  2. https://www.smu.edu/simmons/About-Us/Directory/Education-Policy-Leadership/Ketterlin-Geller
  3. Professor
  4. Presenter’s NSFRESOURCECENTERS
  5. Southern Methodist University
  1. Cassandra Hatfield
  2. https://www.smu.edu/Simmons/About-Us/Directory/Research-in-Mathematics-Education/Hatfield
  3. Research Project Manager
  4. Presenter’s NSFRESOURCECENTERS
  5. Southern Methodist University
  1. Jennifer McMurrer
  2. https://www.smu.edu/Simmons/About-Us/Directory/Research-in-Mathematics-Education/McMurrer
  3. Senior Research Specialist
  4. Presenter’s NSFRESOURCECENTERS
  5. Southern Methodist University

Measures of Early Mathematical Reasoning Skills

NSF Awards: 1721100

2022 (see original presentation & discussion)

Grades K-6

The purpose of this video is to showcase the recent work conducted as part of the Measuring Early Mathematical Reasoning Skills: Developing Tests of Numerical Relational and Spatial Reasoning (MMaRS) project. The goal of the MMaRS project is to create formative assessments that measure numerical relational reasoning and spatial reasoning with the purpose of guiding teachers’ instructional decisions. As part of the development process, we conducted think aloud interviews with students in Kindergarten to Grade 2 to examine the functioning of various item features. We analyzed the results to determine whether students understood the assessment prompts, the items adequately elicited the intended content and reasoning strategies, and the task (e.g., materials, manipulatives and/or instructions) was free from bias or construct irrelevant variance. In this video, we describe more detail the process for conducting and analyzing think aloud interviews for the tests of Spatial Reasoning, and explain how we used this information to finalize the items prior to pilot testing.  

This video has had approximately 178 visits by 129 visitors from 80 unique locations. It has been played 77 times.
activity map thumbnail Click to See Activity Worldwide
Map reflects activity with this presentation from the 2022 STEM For All Video Showcase website, as well as the STEM For All Multiplex website.
Based on periodically updated Google Analytics data. This is intended to show usage trends but may not capture all activity from every visitor.
show more
Discussion from the 2022 STEM For All Video Showcase (23 posts)
  • Icon for: Cassandra Hatfield

    Cassandra Hatfield

    Co-Presenter
    Research Project Manager
    May 10, 2022 | 07:21 a.m.

    The Measuring Early Mathematical Reasoning Skills: Developing Tests of Numerical Relational and Spatial Reasoning (MMaRS) project is developing a suite of formative assessments to support teacher decision making. As a part of our design process we have been conducting think aloud interviews with students. The purpose is to compare the functionality of two items that were designed to assess the same content, while also looking for ways we could revise the items to reduce construct irrelevant variance. 

    What impact do you think this research methodology could have on the future of assessment development?

     
    2
    Discussion is closed. Upvoting is no longer available

    Ateng' Ogwel
    Alejandra Duarte
  • May 10, 2022 | 11:41 a.m.

    Thanks for sharing this work.  It is so interesting and needed, I love the approach of think aloud interviews, students teach us so much from our interview protocols, we find it to be a powerful tool.  Great work!

     

  • Icon for: Leanne Ketterlin Geller

    Leanne Ketterlin Geller

    Lead Presenter
    Professor
    May 10, 2022 | 02:08 p.m.

    Dear Angela, Thank you for watching our video. We appreciate your comments and value the opportunity to share our work here. We felt it was important to highlight how students' voices can impact our work, and are glad you found this interesting. Thanks again!

  • Icon for: Myriam Steinback

    Myriam Steinback

    Facilitator
    Consultant
    May 10, 2022 | 03:09 p.m.

    Your project sounds very interesting. Think aloud interviews are such a good way to assess! I'm curious about the two examples you shared: in the first one, the one with manipulatives as compared with a 'real life' scenario, what were you seeking to assess? And in the second one, where after the interview you changed the drawing so that the buildings were of the same size, what were you looking for?

  • Icon for: Cassandra Hatfield

    Cassandra Hatfield

    Co-Presenter
    Research Project Manager
    May 10, 2022 | 03:20 p.m.

    Hi Myriam, Great questions! 

    For each subcomponent within our learning progression we designed two test items. Our blueprint focused on making those two test items maximally different for the student. Maximal difference in many of the subcomponents included the use of a pictorial representation vs a physical manipulative (or real life scene of a miniature playground). We analyzed the results qualitatively to compare the depth of student reasoning and any construct irrelevant variance that could have been caused by the different test item designs. 

    In the item with the student video, we were assessing perspective taking on the construction of a 3D space when given different views (top, front, and/or side view). We were hoping that the student would use more than one of the views of the city to determine where one of the items was was on the map, but instead she used the size of the buildings to make her decision, which was not the intent. 

    Iterating on assessment design based on what we intended vs the reality of what students do and say, is very insightful! 

     
    1
    Discussion is closed. Upvoting is no longer available

    Myriam Steinback
  • Icon for: Myriam Steinback

    Myriam Steinback

    Facilitator
    Consultant
    May 10, 2022 | 05:41 p.m.

    Thank you for elaborating. Yes, very insightful!! 

    In the first one, did one of them show deeper student reasoning than the other?

  • Icon for: Cassandra Hatfield

    Cassandra Hatfield

    Co-Presenter
    Research Project Manager
    May 11, 2022 | 07:41 a.m.

    In that particular item the pictorial representation functioned better than the physical materials.

    I think it is is also important to note that in addition to comparing the two models at the skill level, we also reviewed the core concept that the skills would be assessed within. Switching between the two models after each item could impact setup time and increase the cognitive load for the student and assessor to orient themselves to the item. When we didn't have consistency across the core concept, we revised and retested individual items. It was a very iterative process. 

     
    1
    Discussion is closed. Upvoting is no longer available

    Myriam Steinback
  • Icon for: David Kung

    David Kung

    Facilitator
    Director of Policy
    May 10, 2022 | 10:50 p.m.

    Very intriguing work - I love watching videos of kids working through tasks. Can you say a little more about what the goal of the project is? How do you imagine this research as being used by others? Toward what end?

  • Icon for: Cassandra Hatfield

    Cassandra Hatfield

    Co-Presenter
    Research Project Manager
    May 11, 2022 | 07:52 a.m.

    Hi David,

    The goal of the project is to develop formative assessments focused on numeric relational reasoning and spatial reasoning. We are looking beyond correctness and looking to support teachers with better understanding the level of student reasoning on a skill. 

    Prior to working in research, I was a middle school math teacher and elementary math specialist. As a teacher, I remember thinking that some assessment items didn't function well for students but not really knowing why. I believe that conducting think aloud interviews with students adds an extremely valuable lens on assessment development. Through this process we have been able to see construct irrelevant variance through the students eyes. It was amazing to see how students reasoned through an item.

    I think it would be extremely valuable for researchers to consider the value of think alouds with students for the purpose of identifying construct irrelevant variance in an assessment item. 

     
    2
    Discussion is closed. Upvoting is no longer available

    Paulina Romero
    Myriam Steinback
  • Icon for: Noelani Ogasawara Morris

    Noelani Ogasawara Morris

    Facilitator
    Demonstration Teacher
    May 11, 2022 | 12:01 a.m.

    I'm curious to know more as well!  I'd love to know what other type of think aloud questions you might be using along the K-2 age levels.  For example, do you use more spatial reasoning and pattern type questions at the younger age and then grow to using think alouds with numbers or even more complex visuals with students in the second grade ?  

  • Icon for: Jennifer McMurrer

    Jennifer McMurrer

    Co-Presenter
    Senior Research Specialist
    May 11, 2022 | 09:45 a.m.

    Hi Noelani,

    Excellent question! The individual assessment items we used with students during the think aloud interviews were based on the learning progression associated with each construct. The items are differentiated by grade--so a kindergarten student would see a simpler version of a puzzle, for example, than the second grade student during the concurrent portion of the think aloud interview. But the main think aloud prompts and retrospective questions we posed were the same for students at all three grade levels. 

    Thank you for your interest in our study!

     
    1
    Discussion is closed. Upvoting is no longer available

    Noelani Ogasawara Morris
  • May 11, 2022 | 10:18 a.m.

    Hi I really enjoyed your video and was wondering about the construct of "bias free" ... how do you conceptualize that?

  • Icon for: Jennifer McMurrer

    Jennifer McMurrer

    Co-Presenter
    Senior Research Specialist
    May 11, 2022 | 04:37 p.m.

    Hi Eva,

    Thank you for viewing our video and asking about bias. Before any of the assessment items are finalized we share the wording as well as the supplemental materials used with external experts in the field. One component of this review includes evaluating the items and materials for possible sources of bias that might unintentionally privilege one group of test takers over another. Specifically, do the items require background knowledge unrelated to the concept being tested that would differ for students with different backgrounds? Is the language sensitive to students from diverse backgrounds, students with limited English proficiency and students with disabilities? Ultimately, we want to ensure that all the items are free of bias in language and content. 

  • May 11, 2022 | 10:24 a.m.

    Thanks for this video! I'm so glad to see such thoughtful and rigorous work to evaluate whether the assessments tap the intended cognitive processes. This step in assessment development is not easy. Thanks for explaining the process you've followed!

  • Icon for: Jennifer McMurrer

    Jennifer McMurrer

    Co-Presenter
    Senior Research Specialist
    May 11, 2022 | 04:27 p.m.

    Thank you for viewing our video, Meagan. We appreciate your feedback!

  • Icon for: Peter Tierney-Fife

    Peter Tierney-Fife

    Senior Curriculum/Instructional Design Associate
    May 12, 2022 | 02:56 p.m.

    Thanks for sharing this interesting and important work! It must be so interesting (and so much fun) working with students in this way. Although not the intent of your current project, I'm wondering if you have thoughts about using recorded (audio or video) student think aloud interviews as part of student mathematics activities (or assessments, although I'm most interested in their use in mathematics classrooms as learning activities).

  • Icon for: Jennifer McMurrer

    Jennifer McMurrer

    Co-Presenter
    Senior Research Specialist
    May 12, 2022 | 03:06 p.m.

    Thank you for your interest in our work! I think recording students' responses can be valuable to understand their reasoning. Video recording is especially illuminating with younger students who often demonstrate their thoughts via actions (using pattern blocks to fill a puzzle, etc.) alongside verbal communication. And yes--it is a lot of fun! 

  • Icon for: Sarah Bichler

    Sarah Bichler

    Researcher
    May 13, 2022 | 04:52 p.m.

    Thank you for including students in the design of assessment! It's such a big limitation of educational research, in my opinion, that we adults anticipate how a student (of any age) interprets a task we design and almost never validate whether we actually anticipate correctly.

    Thank you!

     
    1
    Discussion is closed. Upvoting is no longer available

    Paulina Romero
  • Icon for: Leanne Ketterlin Geller

    Leanne Ketterlin Geller

    Lead Presenter
    Professor
    May 14, 2022 | 09:31 p.m.

    Thank you so much for watching our video and sharing your thoughts. We appreciate your comments. 

  • Icon for: Paulina Romero

    Paulina Romero

    Graduate Student
    May 16, 2022 | 04:29 p.m.

    Very interesting topic. While I'm not in K-12, I am interested in teaching methods as a mom to a young one. 

    One question--can you share some background on how you selected your test group for the interviews? How much diversity did you have in primary/home language, ethnicity, reading levels, and geographic location? Did you find any nuances based on diversity that may impact your study?

  • Icon for: Jennifer McMurrer

    Jennifer McMurrer

    Co-Presenter
    Senior Research Specialist
    May 17, 2022 | 11:29 a.m.

    Thank you for your interest in our work and question, Paulina. The purpose of the think aloud interviews for this component of the project was to determine whether test items or tasks elicit the intended problem-solving processes. We worked with different cohorts of students across different time spans and constructs. (Early cohorts of students participated via zoom due to COVID and later cohort interviews were conducted in-person.) The early remote interviews enabled us to work with students in varied geographic locations. We also paid close attention to recruit a balance in grade levels K-2 and gender as well as a range of reading levels. Finally, we collected information about students' participation in preschool and after school programs, home language, and ethnicity and are still analyzing those data. 

  • May 17, 2022 | 08:36 a.m.

    Very interesting work!  I appreciate that you made children a central part of your design process - anticipation of how students think and reason is so important.

    -Jessica

  • Icon for: Cassandra Hatfield

    Cassandra Hatfield

    Co-Presenter
    Research Project Manager
    May 17, 2022 | 09:39 a.m.

    Thanks for watching and your comment! The old saying "kids say the darnedest things" is true! We learned so much! 

  • Further posting is closed as the event has ended.

Multiplex Discussion
  • Members may log in to post to this discussion.