1. Elizabeth Rowe
  2. Director of Research
  4. TERC, Landmark College, Massachusetts Institute of Technology
  1. Jodi Asbell-clarke
  2. https://edge.terc.edu/
  3. Director, EdGE at TERC
  5. TERC
  1. Ibrahim Dahlstrom-Hakki
  2. Director
  4. Landmark College
  1. Kelly Paulson
  3. TERC

Revealing the Invisible

NSF Awards: 1417967

2018 (see original presentation & discussion)

Grades 9-12, Undergraduate, Informal / multi-age

Researchers from EdGE at TERC, Landmark College, and MIT have come together to Reveal the Invisible. As part of a collaborative partnership resulting from an NSF Ideas Lab, our team is creating tools and methods that will transform how learning can be measured. Challenged with the question, “What is the most audacious question you could answer about education with big data?, we answered “We want to watch implicit learning happen in process.”

Implicit knowledge is knowledge that can be demonstrated through activity and behaviors but is not necessarily expressed in words. Rather than relying on tests with written and verbal expressions of knowledge, our study of implicit learning draws uses methods and instruments from multiple disciplines—science education, game-based learning, educational data mining, cognitive psychology, and neuroscience—to build multimodal models of learning.

We are collecting data from neurotypical learners and learners with cognitive differences who play the physics game Impulse while wearing an eye-tracking device so we can record their gameplay and eye movements simultaneously. We then create a visualization of the game from the players’ data logs, incorporating detected behaviors (from data mining models built in previous research) that are indicators of implicit knowledge of physics in Impulse gameplay. Finally, we layer the visualization with the eye-tracking data to show what parts of the screen (objects in the game) the player is attending to while demonstrating the physics learning behaviors (or not). This allows us to study the relationship between visual attention and learning.

The data architecture we have built for this work, DataArcade, is powerful and can be repurposed to collect a variety of types of multimodal data and synchronize the streams to a precision of 10 ms, which is required for multimodal analytics of learning in a fast action game. Our team is preparing to include additional data stream, such as EEG, physiological, and other sensor data, to build comprehensive multimodal models of implicit learning.

This video has had approximately 248 visits by 213 visitors from 128 unique locations. It has been played 150 times.
activity map thumbnail Click to See Activity Worldwide
Map reflects activity with this presentation from the 2018 STEM for All Video Showcase: Transforming the Educational Landscape website, as well as the STEM For All Multiplex website.
Based on periodically updated Google Analytics data. This is intended to show usage trends but may not capture all activity from every visitor.
show more
Original Discussion from the 2018 STEM for All Video Showcase
  • Icon for: Ibrahim Dahlstrom-Hakki

    Ibrahim Dahlstrom-Hakki

    May 13, 2018 | 10:56 p.m.

    Thank you for your interest in the Revealing the Invisible project. We hope this video gave you a basic idea of the work we are pursuing in this collaborative effort. We'd be happy to answer any questions that you may have and would love to hear your thoughts on how we can further extend our tools to support your work and the work of others in the field.

    Thank you,

    The Revealing the Invisible Team

  • Icon for: Louis Gross

    Louis Gross

    May 14, 2018 | 05:05 p.m.

    Ibrahim et al., 

    Thanks for an enticing look at the possibilities to enhance learning research using eye tracking in game settings. This provides a great opportunity as well to collect data that can analyze dynamic changes as the participants become more adept at the rules for a particular game. Can you say something more about what methods you are using for data driven discovery? Are you using data mining methods, neural nets? I've heard that commercial game companies are manipulating in real-time the rules of games - so can you "experiment" with the games you are using to help tease apart what I imagine are complex responses?



  • Icon for: Ibrahim Dahlstrom-Hakki

    Ibrahim Dahlstrom-Hakki

    May 15, 2018 | 09:53 a.m.

    Hi Lou,

    Thank you for watching our video and for your questions. We share your excitement at the prospect of using this type of data to better understand and eventually adapt learning games to improve student leaning. In terms of data driven discovery, we currently have a number of detectors built based on game behavior. This work was done by the team at EdGE and was based on the hand coding of specific game strategies to establish a ground truth that served as the basis for validating the detectors. We are currently extracting new features from the eye tracking data stream and are working on analyzing those using both top-down and bottom-up approaches. We will be looking for patterns that one can predict will be consistent with an understanding of newton's laws of motion (e.g. anticipatory eye movements) and we will be looking for emergent patterns associated with the currently available behavioral detectors.

    The eye tracking record will give us better insight in a player's cognitive processing but is a resource intensive endeavor. Our ultimate goal is to use this rich data to develop new detectors that can be implemented at scale using either pure game behavior or supplementing it with webcam eye data to allow us to adapt the game in realtime. We also have in place the data collection backend and are developing new visualization tools that can be used across different games to allow us to bring this level of multimodal data collection to other learning games.



  • Icon for: Louis Gross

    Louis Gross

    May 16, 2018 | 01:33 p.m.

    Ibrahim, thanks for the detailed response. It is fascinating that you are thinking ahead to development of new detectors, and I expect that there are many new opportunities to use these if they can be produced at scale.

  • Icon for: Courtney Arthur

    Courtney Arthur

    May 15, 2018 | 01:24 p.m.

    I really love the opportunity you are providing to students who often may struggle with conceptual concepts through gaming mechanisms. What successes have you had so far?

  • Icon for: Ibrahim Dahlstrom-Hakki

    Ibrahim Dahlstrom-Hakki

    May 15, 2018 | 01:31 p.m.

    We certainly find that many students who typically struggle in a traditional classroom can find learning through games a good way of developing basic conceptual understanding. This is not the case for all students and a bridging activity is needed to help students translate those concepts into explicit knowledge, but for a segment of students that are typically at risk including some students from low ses backgrounds and some students with disabilities, this can be a very effective approach. As we finish up analysis of our most recently collected data, we hope to find ways to improve our ability to support STEM learning in a broader range of learners.

  • Icon for: Joseph Reilly

    Joseph Reilly

    Graduate Student
    May 15, 2018 | 02:16 p.m.

    Very interesting work! How easy might it be to use those visualization tools and the data collection backend with other games or virtual environments for learning? I know many of the emergent features and behaviors you detect will be specific to Impulse but I'd love to try a similar suite of tools with other activities.

    Thanks, -Joe

  • Icon for: Ibrahim Dahlstrom-Hakki

    Ibrahim Dahlstrom-Hakki

    May 15, 2018 | 02:33 p.m.

    Our goal is to have something in place that others can use for their own work. Currently, we've only implemented this with TERC games but the games are quite varied and we've designed things with an eye to allow them to be used more broadly. Please feel free to get in touch if you have a specific application in mind.

  • Icon for: Dave Barnes

    Dave Barnes

    May 15, 2018 | 10:00 p.m.

    Elizabeth, Ibrahim, and team,

    Very interesting!  I'm intrigued.  I'm not sure what to ask, but it seems that there might be opportunities to look at the actions of novice players as compared with experts and that dichotomy might increase learning of participants.  

  • Icon for: Ibrahim Dahlstrom-Hakki

    Ibrahim Dahlstrom-Hakki

    May 16, 2018 | 11:13 a.m.

    Hi Dave,

    Absolutely, that is the direction we'd like to take. One of the challenges we need to address is domain expertise as opposed to game expertise as we tease apart players who understand the learning concepts but struggle with game mechanics and players who find strategies to succeed in the game without developing an understanding of the underlying concepts. The game is designed to minimize the distance between those two things (i.e. game mechanics and learning mechanics are aligned) but despite that we still see players exhibiting both of those behaviors.


  • Further posting is closed as the event has ended.

Multiplex Discussion
  • Post to the Discussion

    If you have an account, please login before contributing. New visitors may post with email verification.

    For visitors, we require email verification before we will post your comment. When you receive the email click on the verification link and your message will be made visible.



    NOTE: Your email will be kept private and will not be shared with any 3rd parties