Hi Jeff, great question! There were indeed some large differences, especially in forum participation, though it is difficult to say for sure whether they are due to a particular STEM topic, student population, or other reasons like those you mention. But, I think our best guesses for drivers of differences, based on interviews with teachers and reading syllabi, come down to course design. Most notably, there are various strategies for structuring discussion forums and encouraging them, such as making it a large, small, or non-existent component of students' grades, or putting students into small group forums versus a free-for-all approach. There is some literature on how to engender discussion (including lots of things that don't work!), but it's far from a solved problem. Definitely an exciting area for future research!

Good presentation.  Along those lines, what were your standards for detecting gendered language by professors or by students?

Separately did you notice differences (e.g. more or less) in "help-seeking" behaviors for online versus in-person classes.  In some sense students might feel more engaged to seek help through networks out of necessity, but some may also be discouraged to do so because of the distance. So, the recipe for success is the same, but whether or not the engagement is the same could be different.  Also, do you see racial or ethnic differences in who sought out more support resources in online learning.  This would also be particularly useful for considering design of supports for students with disabilities as well.

We’re glad that you enjoyed our video!

Now, to address your first question, to detect gendered language, we followed Newman et al.’s (2008), which used Linguistic Inquiry and Word Count (LIWC), which is a program for text analysis. The categories that Newman et al. used were based on Tannen’s classic (1990) work on gendered discourse, which classified text as report- (male) or rapport (female)-oriented language. The specific report categories that we examined were analytic, clout, number use, and certainty; the specific rapport categories that we examined were authenticity, pronouns, social processes, discrepancies, tentativeness, and affiliation. For more information on this analysis, here’s a link to the article recently accepted for publication: http://publish.illinois.edu/ilearngroup/files/2021/05/Hendricks-et-al.-2021.pdf

Unfortunately, addressing your next point, although we agree that there should be both benefits and barriers to seeking help online compared to in-person classes, we can’t answer your question about comparing help-seeking across these contexts because we didn’t directly compare online and in-person classes. We also didn’t separate out racially minoritized students from other groups of students underrepresented in STEM, given our adherence to privacy protections. We did consider the role of student engagement in predicting student improvement in the course and found that engagement (operationalized as posting to the discussion forum) was not related to improvement in the course. For more details about these analyses, I recommend reading our recently published paper at https://dl.acm.org/doi/10.1145/3448139.3448159. 

Moving forward, I think it will be important to understand the ways in which the online space provides both natural and instructor-created supports for students to get the help they need, when they need it. 

And thanks for these great questions!

I do think that is a strong possibility, and one which bodes well for the potential of future online courses. We're hoping to study the role of flexible anonymity that online spaces bring in promoting in future work, especially as it relates to help-seeking and help-giving among peers. With reduced stereotype threat, students might feel more inclined to ask (or answer) questions without fear of fitting into stereotypes they or their peers hold about their place in STEM.

By the way, do you remember the name of that study? I would love to read it!

It's a dissertation and it's available online (see the link below). The author doesn't address stereotype threat, but does talk about the other issues you mention. It may be an interesting reference for your work. 

Ratnabalasuriar, S. (2012). Forging Paths Through Hostile Territory: Intersections of Women's Identities Pursuing Post-Secondary Computing Education (Doctoral dissertation, Arizona State University). https://core.ac.uk/download/pdf/79563797.pdf

Thank you! Looking forward to diving into this!

Thank you for the reference!!! 

I was just rereading Nigel's initial comment and the concept of flexible anonymity caught my eye. Can you talk a little more about it? How do you conceptualize it? 

Certainly! In the online courses we've explored in this project, students have relative anonymity. Names are shown to other students when they interact in discussion forums, but little else is. In some courses, students can upload a picture of themselves; however, the data show that few students do so. On the other hand, students do tend to share personal details about themselves in the discussion forums, such as their place of birth, class experiences, hobbies, and other interests. So, students have some control over how much others' see of themselves, which may lower barriers to participation for some.

Not all courses provide such flexibility, however. All the courses we've examined show student names to each other in their online discussions, which may allow peers to infer some ethnicity or gender information, if imperfectly. So none of them allow full anonymity, but some more than others. A course we plan to examine in the near future shows peers their name, picture, and email address with every interaction, for example. It will be interesting in the future, I think, to see if there is a trade-off between building a sense of community by helping students get to know more about each other, versus increasing the barrier to entry for students who may not feel comfortable with this, and whether that affects different groups of students differently.

I try to encourage students to post a picture because it does seem to help me and them develop a bit deeper connection. I find that having the option to post anonymously in the discussion helps those that are not comfortable early on. Usually later in the semester there is very little posting anonymously. 

That seems like an excellent idea! I had not thought much about the temporal dynamics of it, but indeed perhaps the anonymity vs. comfort decision doesn't have to be much of a trade-off at all if it can be a gradual acclimation process. Thanks for that thought!

That is a really interesting question, which we have not considered much in this project -- partly because we don't have data on executive dysfunction (all data here are retrospective which makes it tough to ask students new questions) but also because we haven't considered that enough, and should! Some of the courses definitely engendered more participation and discussion than others based on how activities were structured, and I would suspect that students with disabilities related to executive dysfunction would benefit even more than average from an engaging course structure. The most engaging course sections tended to be those with a large amount of structure, including small-group forum discussions, frequent "reporting out" from groups to the whole class, and grades to go with each of those.

I also 100% agree with you from observations in my own courses about some students struggling more than most with online, and especially asynchronous, courses. In my courses over the last year I've tried to adopt some of the practices that seem to be most effective, like requiring forum participation (often), providing several different types of participation, and contacting students at very regular intervals. It isn't a perfect substitute for coming to class at a specific time, but anecdotally it seems to be helping!

We specifically focused on early, introductory STEM courses where we expected college students' experiences would be most formative. In some courses, students were required to post to forums in different ways, but not necessarily asking for help (e.g., posting reflections, answering other students' questions). The requirement varied quite a bit though, from around 5% to 25% of students' grades, whereas some courses had no grade attached. It certainly seems to make a difference! However, it is also hard to distinguish from the community building strategies you mention, which were roughly correlated with the grade percentage: courses with more grade emphasis on forums were also courses where there were more types/variety of participation encouraged (e.g., small group and large group discussion, specific discussion topics).

Regarding metacognition though, none of the courses instructed students to engage in metacognitive strategies or discourse, though there is quite a bit of evidence that some students do anyway. However, there seems to be a lot of opportunity for explicitly coaching this in the future, especially to promote learning in asynchronous courses and courses where students don't have as much support (i.e., where the need for self-regulation is even higher than usual).

I also am very curious about the double-edged sword of anonymity that you mention. We don't have a lot of variation in anonymity in the current courses, but are hoping to explore this a bit more in future courses where this can be more explicitly explored.

Thank you for these questions!

These courses don't have lab work in the typical sense (largely because they're asynchronous, I suspect). However, one course in particular does have somewhat related activities where the students brainstorm solutions to a thought experiment using the discussion forums to discuss and present their solutions, which appeared to be an effective approach for nearly all students based on what we can garner from graded outcomes and participation.

The question of mixed methods is something that would be really good to get to, and will likely be a big part of our next iteration of this work. In the current project, we are focusing on retrospective examination of courses that are completed. This enables us to take a slightly higher-level view because these data are anonymized and archival, which means we can get data from 100% of students. However, there is a lot missing from this high-level picture. We would love to do interviews and focus groups to obtain students' perspectives on some of these issues more directly (e.g., when and why they express metacognitive evaluations of their process), especially as they relate to things like gender, race, and first-generation college student status.

Excellent questions. We are indeed looking at patterns of regularity in time, specifically over observation periods through the duration of the course, operationalized as the time between each assignment's release date and its due date. The patterns themselves are derived from sequences of 'events,' where each event corresponds to the resource accessed (e.g., forum post, grade view). If you would like to know more about the details of our approach, we recommend reading our recent LAK paper: 

Zhou, J., & Bhat, S. (2021, April). Modeling Consistency Using Engagement Patterns in Online Courses. In LAK21: 11th International Learning Analytics and Knowledge Conference (pp. 226-236).

Overlap at the intersection of these methodologies is something we've just begun to explore (e.g., improvement and social network characteristics). We have not yet looked at consistency and social network size, but I think that is a really interesting question since it seems quite plausible that a well-connected network of peers would encourage regular interaction. Certainly something for us to examine going forward!

I think methods for engendering effective discussion in forums are still sorely needed. However, there are some previous projects in this area, showing mixed results; it is a tough problem! Here are a few of my favorites (many of those that cite these are also relevant):

Mazzolini, M., & Maddison, S. (2007) paper on the role of instructors in discussion forums

Kizilcec et al. (2020) paper on interventions intended to improve discussion forums (though in very large classes)

Schworm & Gruber (2012) paper on methods to encourage help-seeking in discussion forums, and interesting (if troublesome) results regarding the psychological barriers to help-seeking

Thank you! And yes, I am both encouraged and intrigued by that myself, because it may suggest that our online courses are simply modeled very closely after face-to-face courses. Hopefully, there is a great deal of future work to do discovering ways in which we can make them much better by leveraging their own unique affordances.