Two great questions.

One advantage we have is that lab reports (which we model on scientific articles) have several well-defined elements that do not require AI to identify. Right now, all of the feedback students are getting is pattern- and rules-based feedback from our automated system. We identified a lot of our search patterns by looking at our own students' writing; for example,we screen for the presence of a hypothesis statement located in the Introduction section by searching for an "if...then" clause. Papers that earned an A or B almost always had it; reports with lower grades usually did not. 

Since students had not read scientific literature before, we found they did not know where most of the pieces went, or how to format them. Getting the basics in the right place is where the automated system is helpful, because students can submit as often as they want. The instructors can focus on deep reading and spend their comments focused on global issues. All that said, what we're working on right now is a machine algorithm that can "read" the stepwise moves students make and provide even deeper early feedback.

Deciding what to make a writing task was something we committed to as a department. Every student needs to be able to communicate precisely regardless of their future career. Writing regularly with multiple rounds of feedback over the course of 1-2 years is the best way to build that skill. We're fortunate in all students in our 100-level lab courses design and complete several experiments of their own as part of inquiry-based labs. Writing is the natural end point for those experiments, because it's what we do as professionals.

Applying data science to student writing requires thinking about writing tasks a bit differently than what most instructors are used to. We tend to think in terms of individual grades; the DS approach requires thinking this way: "what data can I extract from this writing assignment? What can those data tell me about my class as a whole?"

We're very aware that using AI raises concerns about eliminating human teachers, and that's not our goal. What we are using the AI for is to identify the PATTERNS in student responses, so the teacher can target the specific problems their students have. For instance, with our annotation exercise this past semester, one TA found that 12% of their students mis-identified where data interpretation statements go (discussion, not results). This is only a couple students, but misplacing data interpretation is a major error in scientific writing. So the TA stepped back and addressed this misconception again in the class debrief of the annotation. 

The data we are collecting about student writing is telling us what the most frequent errors are, so we address those first. That brings us to the automated feedback question; some students do not pay attention to it. We do not pressure them about it, because they find out from peers very quickly that SAWHET would have flagged an error that cost them a good score. Their own desire to get a good grade is the motivator.

Students complained the first couple of semesters about the upload process, but we had surprisingly little negative response to the feedback itself. When we asked students, they said they got auto-feedback during their activities online already, so did not feel intrusive.

We tell students the first day of a class that uses this system that they are welcome to ask us how their feedback is being generated, and have a dedicated email address to do so. The fact we offer transparency seems to work well, because in 4 years we have had less than a dozen questions. It also helps that we tell students on their feedback that SAWHET is a computer system, and might be wrong. The final decision rests with their TA, and if they have questions, they should talk to their TA. This reiterates our belief that a human has final decision-making authority about grades.

A couple students have tried to game the system, but doing so takes MUCH more time and effort than they would spend following instructions and writing a good report. We count on this "time tax" to limit gaming. Having a live person at the end also limits it.

 We agree that AI will not replace the human at the end; it is a tool for getting more information so we can help students master basic scientific writing skills sooner. What you describe as tiers is exactly what we are doing. Students submit reports to SAWHET as many times as they like to get feedback on basic structural issues: I think of them as the "what goes where"  and "how do I format X" questions. After the deadline for submission passes, TAs get the most recently submitted version of the report, which is what they score using our bins-based model. Concurrently, students bring copies of the report for peers to review. The peer review and comments from TA on the submission go back to the student, who has a week to upload a revision version. The REVISED version is what gets graded for credit.

This combination has worked very well at improving student writing. We found that peer feedback did little to improve report scores, and that omissions and basic errors persisted even after peer reviews. Using SAWHET, students literally cannot submit work with missing elements; the intake form flags the errors even before submission. Using this approach, we went from an average of 64% of reports meeting basic criteria with the initial submission to >90% of reports meeting basic criteria in 1 year.

Affective outcomes have been positive too. We compared the number of negative comments about peer review and report grading processes on end-of-course lab evaluations, before vs. after implementing SAWHET. Total number of negative comments related to report grading dropped more than 50%, and negative comments about peer review dropped ~30% (this has been harder to put a firm number on.)

One of the unanticipated benefits of SAWHET and bins-based grading is that the remaining negative comments have shifted away from the instructors to the grading model. In the past, we routinely got "my TA is harder/more unfair than every other TA" comments for nearly every TA. This puts the TAs in a positive mentoring position, rather than the position of being the judge of students' abilities.

Tackling that is our project for the fall. What we want to try is flipping the order of experience so that students read skill-level appropriate primary literature first, discuss the findings, then re-analyze the article asking "what is in this section, and what does it accomplish? WHY does it belong there?" We also are testing having the undergraduates actually annotate their peers' papers using our grading rubric, so we can see what they are saying to one another. Big challenge for this latter approach is how best to collect the data so we can analyze it.

I really like the idea of having the students train to review literature and incorporating your grading rubric but I see the challenge you proposed. Good luck with the project!

Absolutely. The way we approach writing instruction feels counter-intuitive at first, especially for faculty who are comfortable with points-based grading scales. What we hope is that we can refine the TA training materials we are building right now for fall semester so that faculty can adopt them without direct supervision. That's a bit further down the road.

Definitely we are planning to publish. One of my hats is being part of the BioTAP Network and Association for Biology Laboratory Education. Both are interested in seeing this published. We are looking for a formal publication in the next year or so. In the mean time, anyone interest is welcome to contact me and find out more on an informal basis.

We definitely have been working with our Writing Center.  As more students whose first language is not English came onto campus, we had to start making more referrals. At the same time, we built a schematic that shows students "Have this problem? Talk to your TA. Have this problem? Talk to the Writing Center." We also make sure they have a print copy and direct digital access to our writing resource guide and requirements.

I am curious about audio narratives - thanks for pointing me that way!

 First semester we had enormous push-back, in part because we could not predict with certainty what the final averages would be. Turned out that average course grades went UP about 5%. We thought it was inflation in scaling, but we pulled reports and made random comparisons using our old rubric. The scores were higher because the reports had fewer errors and omissions.

Once we had numbers, we started telling students on Day 1 that we use this grading model, why we use it, and that we see higher scores as a result. That eliminated about 1/3 of the complaints between first and second semester. We are in the 6th semester of using bins grading now, and the number of complaints has dropped to about half of the number that we used to get on our old points-based grading rubric. The largest drop was in complaints that a particular TA was unfair or graded too harshly.

Anyone thinking about using bins-based grading needs to be ready for a couple of semesters of very loud student objections. Most of it is because it is different, and students cannot argue for points anymore. After a couple groups pass through, it becomes part of local dept. culture. It helps if you can use it for more than one course in a sequence too.

I would say 30 minutes

As I told Erin Kraal below, our TAs averaged 35 minutes per report pre-SAWHET/bins. Currently, TAs average 12-15 minutes per report. Some of my experienced TAs are down to 8-10 minutes.

We know this is not reporting bias because we compared their estimates to the metadata we extracted from commented Word documents. The timestamps on the comments corroborated their estimates.

We've not looked at that formally, but informally, I sat on several senior honors project committees in our department this past April, and about half said they appreciated how we had taught them writing, and that it made thinking about their research easier. Of those who did not say anything specifically, about 1/3 had elements in their writing and presentation style that I recognized as coming from our first year training program. We cannot claim sole credit, but I hope that we can start looking at longer-term impacts soon.

One random observation from this semester: we teach students to use a slightly unusual citation style that avoids favoring any particular field. I saw several senior students use it in their formal honors presentations. It's like a hidden Easter egg in software, or an inside joke. Not a big thing, but it warms your heart to know we made an impression.

This also is anecdotal, but faculty in our more advanced courses report that they don't have to explain the difference between private (meaning, incomplete) drafts and initial submissions anymore to their students. Juniors and seniors tend to turn in draft work that is more fully developed, so faculty can provide more meaningful feedback.

We do not have any immediate plans to develop materials ourselves, but I'd be happy to talk strategies and tools that your team could consider.

Big datasets like the one you have are ideal for training text and linguistic analyses. To give you some idea of what is possible, one of our analysis tasks is to read and classify every comment TAs make on student reports each semester, which is 10,000+ comments. By hand that takes about 2 months. We created a text classifier in R (which is free) that can classify the comments in about 15 minutes.

I'd love to talk further. Hit me up by email if you are interested.

Thanks for your questions Erin. Transfer to another format takes some work, but is definitely possible. Reports have a single standard format, which makes searching for structural issues easier, but if someone has a format they expect students to follow, they can implement form-based automated feedback. The limiting step is identifying what features to screen for, then writing rules to identify them. Fortunately, some features are pretty universal, like citing sources, or formats for summarizing statistics.

In free response writing the process is more complicated. What we did, and what I would suggest anyone thinking of this do, is collect a set of responses that cover the range of quality/scores. Identify the most important features on which the grade is based, ideally ones that can be scored as present/absent, or yes/partly/no. Those are the targets for automated feedback. We also looked at vocabulary range and readability scores (definitely easier with computational support.) They were not useful for our purposes, but the metrics are well-supported by research, and in a particular situation they may be useful.

What we have seen so far from bins grading is that grading time goes down drastically. Prior to using bins, our TAs spent an average of 35 minutes per paper, and had no criteria on which to say "this report does not even have enough content to grade." With bins, average grading time has dropped to 12-15 minutes, and there are clear criteria for handing a report back un-commented. The latter sounds harsh, but we do it very rarely; students know that they have to provide a complete report to get any feedback.

Based on the success with reports, we are now using bins grading for short answer questions on quizzes, notebooks, and oral presentations too.

One thing we found very useful was to remove %s and letter grades from our bins. We use "Acceptable", "Minor revisions needed", "Major revisions needed", and "Unacceptable" as the grading levels.