7134 Views (as of 05/2023)
  1. Edys Quellmalz
  2. http://www.wested.org/personnel/edys-quellmalz/
  3. Director Technology Enhanced Assessment and Learning Systems
  5. WestEd
  1. Matt Silberglitt
  2. Senior Research Associate
  4. WestEd

SimScientist Assessments: Physical Science Links

NSF Awards: 1221614

2015 (see original presentation & discussion)

Grades 6-8

The SimScientists program is developing a multilevel assessment system for middle school science. This project will create formative and summative assessments and companion classroom reflection activities for four of the most widely taught topics in middle school physical science: Matter, Motion, Energy, and Waves. The project is in the design and development phase that includes feasibility studies of assessments for Energy and Waves, conducted in the classrooms of two teacher co-developers.
Validity of the assessments:
1. How well do the embedded, benchmark, and signature assessment tasks and items align with the Framework for K-12 Science Education, NGSS and state standards, and each other?
2. How well do assessment tasks and items meet standards for scientific accuracy, grade-level appropriateness, and task quality?
3. How well do the assessments meet technical standards for reliability and validity?
Classroom use of assessments:
4. Can teachers implement the assessments?
5. In what ways do teachers integrate the assessments into curricula to support learning?
6. Do teachers and students find the simulation-based assessments valuable for monitoring progress, adjusting instruction, and reporting?
7. Are students engaged in the assessments?
Policy implications:
8. Do the cross-validation studies of SimScientists assessments in multiple states support their appropriateness for testing the targeted science knowledge and practices across curricula and as components of a district or state science test?
9. Do state, county, and district representatives consider the simulation-based classroom and year-end assessments credible components of their state science assessment systems?

This video has had approximately 480 visits by 419 visitors from 125 unique locations. It has been played 160 times as of 05/2023.
Click to See Activity Worldwide
Map reflects activity with this presentation from the 2015 Teaching & Learning Video Showcase website, as well as the STEM For All Multiplex website.
Based on periodically updated Google Analytics data. This is intended to show usage trends but may not capture all activity from every visitor.
show more
Discussion from the 2015 Teaching & Learning Video Showcase (12 posts)
  • Icon for: Jackie DeLisi

    Jackie DeLisi

    Research Scientist
    May 11, 2015 | 11:41 a.m.

    This is a very interesting project. Can you say more about how you are establishing the validity of the assessment?

  • Icon for: Edys Quellmalz

    Edys Quellmalz

    Lead Presenter
    Director Technology Enhanced Assessment and Learning Systems
    May 11, 2015 | 05:16 p.m.

    We begin with external reviews of alignment, science content and item quality, conduct cognitive labs, then analyses of reliability and validity. Please go to simscientists.org to find the article published on this Quellmalz, Timms, et al 2012.

  • Icon for: Randy Kochevar

    Randy Kochevar

    Senior Research Scientist
    May 11, 2015 | 06:35 p.m.

    These look like really engaging simulations! What kind of teacher training goes into implementing them in a classroom?

  • Icon for: Matt Silberglitt

    Matt Silberglitt

    Senior Research Associate
    May 11, 2015 | 08:01 p.m.

    We provide teacher professional development to participants in our research projects. This has been either a primarily face-to-face workshop, with ongoing support, or, more recently, a web-based program. Each type of teacher professional development is about 8–12 hours total, although the web-based program is designed to allow for more and deeper exploration of model-based learning, formative assessment, and how these strategies apply to SimScientists.

  • Icon for: Kathy Perkins

    Kathy Perkins

    May 12, 2015 | 12:58 a.m.

    It’s great to hear about how you’ve been approaching new forms of assessment with the sims.
    What types of data are you collecting from the sim interaction, and how important is that to your assessment?
    Also, what are some of the effective teacher faciliation strategies you see teachers using around these tools, when engaging in formative assessment?

  • Icon for: Edys Quellmalz

    Edys Quellmalz

    Lead Presenter
    Director Technology Enhanced Assessment and Learning Systems
    May 12, 2015 | 01:23 p.m.

    The SimScientists assessments use Evidence-Centered Assessment Design (ECD) to specify the knowledge and skills to be assessed (Student Model), the features of the assessment tasks and questions that will elicit explicit evidence of learning on the targets (Task Models), and the rules for collecting, scoring, and aggregating, and reporting the evidence for the targets (Evidence Model). These three components for designing and collecting the student data are ESSENTIAL for using student response as evidence to support the claims of student learning of the specified science knowledge and practices.

    The curriculum-embedded simulation assessments collect data on the targeted NGSS core ideas and practices specified at the outset of the design of the assessments. Students responses are collected on tasks and questions requiring building models, designing, conducting, saving, and interpreting investigations, plus writing explanations, critiques, and arguments. The Learning Management System underlying the simulation interactions uses scoring and aggregation rules to produce Progress Reports on the targeted science knowledge and practices. Reports classify student responses to the questions and tasks for these content and practice targets into three categories: “On Track” “Progressing” and “Needs Help”.

    In the professional development provided before teachers use the SimScientists simulation-based assessments, then during the implementation of the assessments, strategies are shared for how to use the Progress Reports as a resource for formative assessment to adjust instruction. Ideas for grouping students based on the Progress Reports for classroom, off-line Reflection Activities are shared and developed during the professional development.

    An end-of unit simulation-based benchmark assessment for a unit provides a summative proficiency report for a student’s performance on the specified content and practices targets.

  • Icon for: Ben Sayler

    Ben Sayler

    Professor of Physical Science and Mathematics
    May 12, 2015 | 08:01 a.m.

    Fascinating work. Having students conduct an experiment by way of simulation seems like an excellent tool for regular instruction — not just as a way to assess. Might teachers be able to use your simulations steadily throughout their teaching, thereby making assessment essentially indistinguishable from instruction?

  • Icon for: Edys Quellmalz

    Edys Quellmalz

    Lead Presenter
    Director Technology Enhanced Assessment and Learning Systems
    May 12, 2015 | 01:30 p.m.

    We are developing strands of SimScientists assessments for middle school curricula in life, physical, and Earth science. There are three or four suites of SimScientists assessments in each of those three content areas. In the Reflection Activities following student use of one of the curriculum-embedded simulation assessments, teachers can use the Progress Reports on student performance to provide follow up, adjusted instruction. One of the strategies is for teachers to use the simulations to review and reinforce content and/or practices needing more attention.

    We are also developing simulation-based instructional modules to provide more review and reinforcement of core ideas and practices and cross-cutting concepts. Please go to the website, simscientists.org for more information on the projects and resources.

  • Icon for: Ben Sayler

    Ben Sayler

    Professor of Physical Science and Mathematics
    May 13, 2015 | 09:59 a.m.

    Will do. Thanks!

  • Icon for: Meredith Thompson

    Meredith Thompson

    Post doctoral fellow
    May 13, 2015 | 02:10 p.m.

    The simulations are wonderful. I especially like how the water molecules are vibrating even when water is in the solid state!

    I am curious about the details for the task and evidence model for the assessments. Are they on your website? I have read your J Ed. Psych article from 2013 but did not see the specific on those aspects of the assessment. For example, how is a “designing experiments” simulation scored in SimScientists Assessment? Are the criteria established within the team, or by reference to NGSS, or elsewhere?


  • Icon for: Edys Quellmalz

    Edys Quellmalz

    Lead Presenter
    Director Technology Enhanced Assessment and Learning Systems
    May 13, 2015 | 02:59 p.m.

    The specifications for the task and evidence models for the simulation tasks are internal design documents and not published. Criteria for scoring the investigations’ designs, trials, and interpretations are developed by our team and referenced to the NGSS.

  • Icon for: Meredith Thompson

    Meredith Thompson

    Post doctoral fellow
    May 13, 2015 | 03:02 p.m.

    OK! Thank you.

  • Further posting is closed as the event has ended.