Tue Jul 19

We have developed a way to screen student feedback to ensure it's useful, not abusive (and academics don't have to burn it)

Written by: Abby Cathcart, Professor of Higher Education & Governance, Queensland University of Technology

This week, many Australian universities will be sending academics the results of the first semester student evaluation surveys.

For some this will be a worrying and unpleasant time. The comments university students make anonymously in their teaching evaluations can leave academics feeling fearful ^[1], distressed ^[2] and demoralised ^[3].

And with good reason. As a 2021 survey ^[4] of Australian academics and their experiences of student feedback found:

Personally destructive, defamatory, abusive and hurtful comments were commonly reported.

Hurtful or abusive comments can remain permanently on record as a measure of performance ^[5]. These records can affect applications for promotion or for secure continued employment.

The authors of the 2021 survey, led by Richard Lakeman at Southern Cross University have been among those calling for ^[6] anonymous online surveys to be scrapped. Some academics, burned by their experience of student feedback, say they no longer open or engage ^[7] with student evaluation reports. They said the risk of harm outweighed any benefits.

In the Netflix show, The Chair, a memorable scene sees the character Professor Joan Hambling burn ^[9] her student evaluations. Clearly, a different solution is needed.

Feedback from students can still be valuable for lifting teaching standards and it’s important students have their say.

We have developed a screening system ^[10] using machine learning ^[11] (where software changes its behaviour by “learning” from user input) that allows students to talk about their experiences while protecting academics from unacceptable comments.

Why a new approach is needed

University codes of conduct remind students of their general obligation to refrain from abusive or discriminatory behaviour, but not specifically in regard to student evaluations.

Instead, universities rely on self-regulation or on others to report incidents. Some institutions use profanity blockers to screen comments. Even then, these often fail to detect emerging terms of abuse in online speech.

So, in setting up our screening system, we wanted to:

promote staff and student well-being
enhance the reliability and validity of student feedback
improve confidence in the integrity of survey results.

We developed a method using machine learning and a dictionary of terms to screen for unacceptable student comments. The dictionary was created by QUT drawing on historically identified unacceptable comments and incorporating prior research into abusive and discriminatory terms.

Our ‘Screenomatic’ solution

There is not a lot of published work on the detection of unacceptable or abusive comments in student evaluation surveys. So our team adapted earlier research ^[12] on detecting misogynistic tweets. This worked because often the student comments we looked at were similar in length to a tweet’s 280-character limit.

Our approach, which we call “Screenomatic ^[13]”, automatically reviewed more than 100,000 student comments during 2021 and identified those that appeared to be abuse. Trained evaluation staff members manually reviewed about 7,000 flagged comments, updating the machine-learning model after each semester. Each update improves the accuracy of auto-detection.

Ultimately, 100 comments were removed before the results were released to educators and supervisors. University policy enables comments to be re-identified in cases of potential misconduct. The central evaluations team contacted these students and reminded them of their obligations under the code of conduct.

The Screenomatic ^[15] model can help protect both educators and students. Staff are safeguarded from abuse, and students at risk – who make comments that indicate they need mental health help, include allegations of bullying or harassment, or that threaten staff or other students – can be offered support. Universities can share data to train the model and maintain currency.

Importantly, the process enables universities to act morally to harness student voices while protecting people’s well-being.

Useful feedback, not abuse

The number of educators who receive abusive feedback may be relatively small ^[16]. However, it’s still unacceptable for universities to continue to expose their staff to offensive comments in the full knowledge of their potential impact.

With last year’s High Court ruling on liability for defamatory posts ^[18], and attempts to improve online safety ^[19], there is a growing acknowledgement that people should not be able to post anonymous, harmful messages.

After all, the cost of screening responses is nothing compared to the cost to individuals (including mental health or career consequences). And that’s ignoring the potential costs of litigation and legal damages.

At the end of the day, the anonymous comments are read by real people. As a tweet in response to the Lakeman findings ^[20] noted:

The Screenomatic model ^[21] goes a long way towards enabling the “tons of useful feedback” to serve its intended purpose while ensuring people aren’t harmed in the process.