4757 Views (as of 05/2023)
  1. Renee Cole
  2. https://chem.uiowa.edu/people/renee-s-cole
  3. Professor
  4. Presenter’s NSFRESOURCECENTERS
  5. University of Iowa
  1. Juliette Lantz
  2. Associate Dean of Curriclum, Professor
  3. Presenter’s NSFRESOURCECENTERS
  4. Drew University
  1. Kathryn Mauger-Sonnek
  2. Graduate Research Assistant
  3. Presenter’s NSFRESOURCECENTERS
  4. University of Iowa
  1. Gil Reynders
  2. https://orcid.org/0000-0003-1981-922X
  3. Postdoctoral Researcher
  4. Presenter’s NSFRESOURCECENTERS
  5. Virginia Commonwealth University
  1. Suzanne Ruder
  2. Professor
  3. Presenter’s NSFRESOURCECENTERS
  4. Virginia Commonwealth University

Collaborative Research: Eliciting and Assessing Process Skills in STEM

NSF Awards: 1524936, 1524399, 1524965

2020 (see original presentation & discussion)

Undergraduate

Skills such as communication, teamwork, critical thinking, and problem solving are frequently cited as important outcomes for STEM degree programs. However, the development of these skills is often taken for granted, and they are not often explicitly discussed or assessed in the classroom. Assessment serves two purposes: (1) providing a measure of achievement, and (2) facilitating learning by conveying what is valued in a course. This project has developed feedback-focused rubrics that serve as a resource for instructors to assess and support student skill development. Faculty and students from multiple institutions representing multiple STEM disciplines have verified the validity and utility of these rubrics in a variety of STEM courses. Implementation of the rubrics has been shown to positively impact faculty reflection and student outcomes.

This video has had approximately 452 visits by 332 visitors from 237 unique locations. It has been played 285 times as of 05/2023.
Click to See Activity Worldwide
Map reflects activity with this presentation from the 2020 STEM For All Video Showcase website, as well as the STEM For All Multiplex website.
Based on periodically updated Google Analytics data. This is intended to show usage trends but may not capture all activity from every visitor.
show more
Discussion from the 2020 STEM For All Video Showcase (35 posts)
  • Icon for: Renee Cole

    Renee Cole

    Lead Presenter
    Professor
    May 4, 2020 | 06:28 p.m.

    Thank you for viewing our video! We are in the fifth year of this project called ELIPSS (Enhancing Learning by Improving Process Skills in STEM). Initially, we developed rubrics to assess process skills, and now we’re enhancing these efforts by optimizing ways to provide feedback to students and instructors. We’re interested in hearing your thoughts about how you assess these transferable  skills, how you provide feedback to students, and what resources you would need to implement the ELIPSS rubrics. Particularly ...

    • In what ways does this video give you ideas about how to assess skills in your courses?
    • How could STEM instructors provide effective feedback to students about their process skill development?
    • In what ways could you change your courses (activities and/or facilitation) to encourage process skill development?
     
    1
    Discussion is closed. Upvoting is no longer available

    Kristin Flaming
  • Icon for: Shirley Lin

    Shirley Lin

    Higher Ed Faculty
    May 5, 2020 | 10:45 a.m.

    Hi, Renee,

    Thanks to you and the team for the great video. I had heard about ELIPSS several years ago at an ACS meeting. It would be interesting to ask students to record themselves solving problems aloud and then apply the process skills rubrics. In this era of online teaching, that could be a way to interact and give feedback to students.

    I am curious in your experience how students react to the feedback they get from these rubrics when it is provided? I have observed that sometimes students don't know how to use or value feedback on their work when it is provided, whether it is grades or comments or other types of information from instructors or peers.

    Thanks again for your work!

  • Icon for: Suzanne Ruder

    Suzanne Ruder

    Co-Presenter
    Professor
    May 6, 2020 | 01:57 p.m.

    Shirley,

    We have recorded some students as they talk through solving problems but have not given feedback after that. Switching to more online might make that an option for a greater number of students, but for large classes this is difficult. 

    In my experience the best way is to get students to reflect on the feedback they are given, and give those reflections some participation points. We had some positive comments on the skill development after doing it this way.

  • Icon for: Juliette Lantz

    Juliette Lantz

    Co-Presenter
    Associate Dean of Curriclum, Professor
    May 7, 2020 | 10:10 p.m.

    Hi Shirley - that's a really interesting idea to have students provide a recording of their problem solving, and be able to use the rubrics on those submissions.  We are thinking of strategies for rubric usage now that many classes are being taught online. 

    On your feedback insight- I've give my Learning Teams feedback on their interpersonal communication and teamwork, and asked them to talk about it and write back to me with one agreed upon improvement they'd try next class.  I gave them class time to do this, which might be key.

  • Icon for: Sara Yeo

    Sara Yeo

    Facilitator
    Assistant Professor
    May 5, 2020 | 04:23 p.m.

    I also have a question related to the transition to online learning given the situation in which we find ourselves. Do you have recommendations for how we might scale this to online courses? Looking forward to a fun discussion, thanks!

  • Icon for: Suzanne Ruder

    Suzanne Ruder

    Co-Presenter
    Professor
    May 6, 2020 | 02:02 p.m.

    Sara,

    We are thinking that skills in an online environment might be more focused on communication, information processing and metacognition. Separating students into zoom rooms could provide an opportunity to assess these skills; we tried this a little bit this semester. With a TA to monitor a discussion question for each room, the instructor could jump between rooms to assess skills. The biggest problem is getting everyone to join in the discussion, which if we have more time to prepare for online classes, this could be structured into a participation grade.

     
    1
    Discussion is closed. Upvoting is no longer available

    Sara Yeo
  • Icon for: Sara Yeo

    Sara Yeo

    Facilitator
    Assistant Professor
    May 7, 2020 | 02:23 p.m.

    Thanks for your response, Suzanne. I suppose TAs have to be trained to look for the specific skills online, too. Was this a challenge? Did changes have to be made for online assessment vs. face-to-face assessment?

  • Icon for: Juliette Lantz

    Juliette Lantz

    Co-Presenter
    Associate Dean of Curriclum, Professor
    May 7, 2020 | 10:13 p.m.

    HI Sara - we are piloting an online rubric system that allows an instructor to indicate ratings and feedback electronically, (perhaps by observing interpersonal skills in a Learning Team) and then quickly email this to all students in that team.  Stay tuned for us to roll this out!

     
    1
    Discussion is closed. Upvoting is no longer available

    Sara Yeo
  • Icon for: Renee Cole

    Renee Cole

    Lead Presenter
    Professor
    May 5, 2020 | 04:28 p.m.

    Hi Shirley,

    The notion of having students record themselves engaging in a skill and then applying the rubric to the video is a really interesting idea. That would be a good idea for an implementation to study, particularly this summer or for the fall. 

    The student reactions to feedback has been mixed (as you note) between generally ignoring feedback and using it to inform future practice. When faculty have a mechanism to encourage reflection we've seen better outcomes.

  • Icon for: Gil Reynders

    Gil Reynders

    Co-Presenter
    Postdoctoral Researcher
    May 5, 2020 | 04:45 p.m.

    Hi Sara,

    We're currently developing a web-based version of the rubrics. They were originally intended for live classes, but they should allow us to assess student groups in online settings as well. After an instructor or TA fills out the digital rubrics, all students in the group will receive an email with their scores and feedback. We're still in the early testing phases, but hopefully the web-based rubrics will be ready for wider testing in the fall.

  • Icon for: Nancy Shapiro

    Nancy Shapiro

    Facilitator
    Associate Vice Chancellor
    May 5, 2020 | 05:18 p.m.

    I think you have captured and tapped into an important element of STEM disciplines and majors that is valued, but which is sometimes hard to focus on in a STEM class.  Using rubrics has been a useful strategy in many other disciplines--I'm thinking of writing classes (both general education and technical writing), and other subjects.  Have you engaged any colleagues outside of the STEM disciplines to review your instruments, to offer insights or feedback to the process of the development of the rubric?  I'm always looking for ways to "cross over" between hard sciences and the social sciences and humanities--I think there is a lot to learn when we cross those bridges.

  • Icon for: Suzanne Ruder

    Suzanne Ruder

    Co-Presenter
    Professor
    May 6, 2020 | 02:06 p.m.

    Nancy,

    Since this project is NSF supported, our focus has been in STEM disciplines. However, we have a strong connection to the POGIL project, which does have people outside of STEM who have used our rubrics. The skill definitions were in fact developed within a team of instructors that included STEM and non-STEM disciplines to try to make the language useful across disciplines. 

  • Icon for: Renee Cole

    Renee Cole

    Lead Presenter
    Professor
    May 7, 2020 | 12:41 a.m.

    As Suzanne indicated, we're always interested in thinking about how the resources can be used by a variety of disciplines. We'd love to hear more about how well the categories and descriptors translate to your field.

  • Icon for: Juliette Lantz

    Juliette Lantz

    Co-Presenter
    Associate Dean of Curriclum, Professor
    May 7, 2020 | 10:18 p.m.

    Hi Nancy - as an assoc dean of curriculum at my institution, it's my intent to bring these forward across the college as we move to a more skills-based curriculum. It was a significant effort to land on language/word choices that were agreeable across the range of STEM disciplines represented in our team - I imagine that may be a stumbling block in trying to move these out to a broader community. 

  • Icon for: Joe Heimlich

    Joe Heimlich

    Researcher
    May 5, 2020 | 05:28 p.m.

    This is great! Working on a project on a field-wide learning framework, and soft skills are those that cut across all positions. Can't wait to look at these and see how we might be able to build on and use the amazing work you've done!  THANKS!

  • Icon for: Suzanne Ruder

    Suzanne Ruder

    Co-Presenter
    Professor
    May 6, 2020 | 02:10 p.m.

    Thanks Joe! We hope you find the rubrics useful. Some of us have combined portions of different rubrics, or a single row, to better assess the skill relevant to the particular task the student is undertaking.

  • Icon for: Hiller Spires

    Hiller Spires

    Higher Ed Administrator
    May 5, 2020 | 07:41 p.m.

    This is such an interesting project! These rubrics and feedback forms are so versatile and provide students with insight into their process skills that are too often glossed over. I believe that these documents would be beneficial when providing feedback to students for any kind of assignment. I think there may be a potential to build on the metacognitive aspect through having students submit a reflection on previous feedback with plans to address areas of weakness in future work. These reflections may enable students to iteratively work on these skills and be able to see explicit improvement as they progress through the course. I think this work is very beneficial to STEM teachers and am interested in potentially applying the documents to my projects as well. Thank you for sharing this video!

  • Icon for: Suzanne Ruder

    Suzanne Ruder

    Co-Presenter
    Professor
    May 6, 2020 | 02:13 p.m.

    Hiller,

    Thanks for your comment. We have had students reflect on the feedback received and had some really positive comments. They got some credit for their reflection which required them to look at the rubric, which we could track online. 

  • Icon for: Renee Cole

    Renee Cole

    Lead Presenter
    Professor
    May 7, 2020 | 12:45 a.m.

    Hi Hiller - we'd love to hear more about how you apply the rubrics to your project. We're very interested in the different ways people are implementing the rubrics.

  • Icon for: Joanne Stewart

    Joanne Stewart

    Higher Ed Faculty
    May 6, 2020 | 10:45 a.m.

    Such thoughtful and detailed work! Thank you for sharing it here. If you were working with a medium-sized chemistry program interested in improving the development of student skills, where would you suggest that they start?

  • Icon for: Suzanne Ruder

    Suzanne Ruder

    Co-Presenter
    Professor
    May 6, 2020 | 02:17 p.m.

    Joanne,

    Most faculty have found it easiest to start with interpersonal communication and/or teamwork if they are using some kind of active learning. The easiest cognitive skill to start with is information processing, which is needed to move on to higher order tasks like problem solving or critical thinking. This will depend on the learning environment and content being discussed, but these three are generally the most common with chemistry faculty we have worked with.

     

  • Icon for: Juliette Lantz

    Juliette Lantz

    Co-Presenter
    Associate Dean of Curriclum, Professor
    May 7, 2020 | 10:39 p.m.

    Joanne - I'd suggest starting somewhere in the curriculum it seems sensible to name skill development as a learning outcome - in many institutions, the laboratory setting might be a natural place to start.  It's key that activities are designed to give students strong cues to reveal these skills, so that you can assess and provide feedback.  In other words, if an instructor wants to facilitate the growth of teamwork, the assigned task needs to have teamwork as a crucial element in that activity, and it has to be made apparent to the instructor in some way.

  • Icon for: Parvaneh Mohammadian

    Parvaneh Mohammadian

    Higher Ed Faculty
    May 6, 2020 | 12:42 p.m.

    Thank you for sharing your project! I will be looking into using the rubric in my future classes.

     

    Would you be available for a short zoom presentation to the STEM faculty?

  • Icon for: Suzanne Ruder

    Suzanne Ruder

    Co-Presenter
    Professor
    May 6, 2020 | 02:18 p.m.

    Parvaneh,

    Thanks for your comment! We could definitely arrange for a short presentation.

  • Icon for: Parvaneh Mohammadian

    Parvaneh Mohammadian

    Higher Ed Faculty
    May 6, 2020 | 02:25 p.m.

    That's wonderful. Would you mind sharing your email with me? 

    mohammp@lamission.edu

  • Icon for: Suzanne Ruder

    Suzanne Ruder

    Co-Presenter
    Professor
    May 6, 2020 | 02:31 p.m.

    Just  sent you an email!

  • Icon for: Ivory Toldson

    Ivory Toldson

    Facilitator
    President
    May 6, 2020 | 02:31 p.m.

    Well done. I agree with many of the comments above which highlight the importance of these skills in this field. Feedback is a critical component that is too often overlooked or implemented at a time when there are not adequate time to address improve noted areas for growth. I appreciate the implementation of feedback throughout the process as it lends to a more tailored approach to learning.  I can also appreciate the accommodations you have made to transition to an online platform as it speaks to your adaptability to current needs.

    I'm interested in knowing what aspect of this five year project has been most meaningful to you. What evidences have you seen that make you to believe that you have made an impact on the field?

  • Icon for: Renee Cole

    Renee Cole

    Lead Presenter
    Professor
    May 7, 2020 | 12:56 a.m.

    The primary evidence for impact on the field has been the change in instructional practices of the faculty who have adopted and implemented the rubrics. Explicitly assessing process skills and paying attention to how learning materials and classroom facilitation support students in developing them has resulted in many faculty revising course materials and how they make skills visible in the classroom. In the case studies we've done with intentional implementation, we see improvements in student skills as well.

     

     
    1
    Discussion is closed. Upvoting is no longer available

    Ivory Toldson
  • Icon for: Barbara Reisner

    Barbara Reisner

    Higher Ed Faculty
    May 7, 2020 | 07:48 a.m.

    I attended an ELIPSS workshop at BCCE two years ago. Learning about this project helped me realize that process skills are an integral part of the classes I teach - I just hadn't been articulating that to my students. Since attending that workshop, I've been much more explicit about explaining this to students.

    I've used the Interpersonal Communication and Teamwork rubrics in a team-based class where we are now explicit that these are skills we want students to improve on. I don't have any data other than personal observation, but my co-instructor and I felt that students were working better by the end of the semester than any other term that we'd taught the course. Now that we're more familiar with the rubrics, we intend to more fully use them in our class this fall (which may be online) and will ask the students to reflect more on their skill development.

  • Icon for: Renee Cole

    Renee Cole

    Lead Presenter
    Professor
    May 11, 2020 | 11:04 a.m.

    Hi Barb,

    That's encouraging to hear that you've seen improvement. We'll be interested to hear more about your fall implementation.

  • Icon for: Juliette Lantz

    Juliette Lantz

    Co-Presenter
    Associate Dean of Curriclum, Professor
    May 7, 2020 | 10:27 p.m.

    Hi Hiller - as we are moving further into the use of reflection in our curriculum, I'm eager to give this kind of iterative reflection a try.  Great suggestion!

  • Icon for: Kristin Flaming

    Kristin Flaming

    Higher Ed Faculty
    May 9, 2020 | 08:02 p.m.

    This is a much needed program that I suspect will have a quick uptake in the coming years as universities and industry begin to focus on these areas more/

  • Icon for: Renee Cole

    Renee Cole

    Lead Presenter
    Professor
    May 11, 2020 | 11:05 a.m.

    We're hoping to be able to facilitate this. Propagation of resources is always a challenge. Do you have ideas about how to better reach these audiences?

  • Icon for: Hannah Sevian

    Hannah Sevian

    Higher Ed Faculty
    May 11, 2020 | 12:37 p.m.

    It's so great that your project is contributing rich resources based on well designed research. Regarding process skills, I wonder if you could give some insights into how you consider process skills to be related to the "science practices" that have been articulated in the K-12 domain (Framework and NGSS). I think the relationships between process skills at university level to practices that are being increasingly emphasized in K-12 are going to become increasingly more important as students move through the educational continuum into university.

  • Icon for: Renee Cole

    Renee Cole

    Lead Presenter
    Professor
    May 11, 2020 | 05:33 p.m.

    One of the challenges of this project has been figuring out what to call the skills so we connect with the different communities that often use different terms. We used "process skills" initially because the work grew out of the POGIL project, but they are analogous to the "practical and intellectual skills" used by AAC&U, as well as lots of other descriptors such as soft skills, 21st Century Skills, or employability skills. In terms of the science practices, there is certainly some overlap. Aspects of "Analyzing and interpreting data" are addressed in the Information Processing, Critical Thinking, and Problem Solving rubrics. Aspects of "Engaging in argument from evidence" are included in the Critical Thinking rubric. Aspects of "Obtaining, evaluating, and communicating information" are included in the Information Processing, Critical Thinking, Problem Solving, and Communication rubrics. It's common for instructors to create composite rubrics for grading purposes that pull categories from different rubrics to address the specific skills that should be addressed when completing a task. This allows there to be a clear connection to some of the formative assessments discussing skill development in general.

  • Further posting is closed as the event has ended.