Tue Aug 15

Can human moderators ever really rein in harmful online content? New research says yes

Written by: Marian-Andrei Rizoiu, Senior Lecturer in Behavioral Data Science, University of Technology Sydney

Social media platforms have become the “digital town squares” of our time, enabling communication and the exchange of ideas on a global scale. However, the unregulated nature of these platforms has allowed the proliferation of harmful content such as misinformation, disinformation and hate speech.

Regulating the online world has proven difficult, but one promising avenue is suggested by the European Union’s Digital Services Act, passed in November 2022. This legislation mandates “trusted flaggers” to identify certain kinds of problematic content to platforms, who must then remove it within 24 hours.

Will it work, given the fast pace and complex viral dynamics of social media environments? To find out, we modelled the effect of the new rule, in research ^[1] published in the Proceedings of the National Academy of Sciences.

Our results show this approach can indeed reduce the spread of harmful content. We also suggest some insights into how the rules can be implemented in the most effective way.

Understanding the spread of harmful content

We used a mathematical model of information spread to analyse how harmful content is disseminated through social networks.

In the model, each harmful post is treated as a “self-exciting point process ^[2]”. This means it draws more people into the discussion over time and generates further harmful posts, similar to a word-of-mouth process.

The intensity of a post’s self-propagation decreases over time. However, if left unchecked, its “offspring” can generate more offspring, leading to exponential growth.

A constellation of lights in a dark room, with a group of people silhouetted against the light.

Social media posts spread online through a process much like word of mouth. Robynne Hu / Unsplash ^[3]

The potential for harm reduction

In our study, we used two key measures to assess the effectiveness of the kind of moderation set out in the Digital Services Act: potential harm and content half-life.

A post’s potential harm represents the number of harmful offspring it generates. Content half-life denotes the amount of time required for half of all the post’s offspring to be generated.

We found moderation by the rules of the Digital Services Act can effectively reduce harm, even on platforms with short content half-lives, such as X (formerly known as Twitter). While faster moderation is always more effective, we found that moderating even after 24 hours could still reduce the number of harmful offspring by up to 50%.

The role of reaction time and harm reduction

The reaction time required for effective content moderation increases with both the content half-life and potential harm. To put it another way, for content that is longer-lived and generates large numbers of harmful offspring, intervening later can still prevent many harmful subsequent posts.

This suggests the approach of the Digital Services Act can effectively combat harmful content, even on fast-paced platforms like X.

We also found the amount of harm reduction increases for content with greater potential harm. While apparently counterintuitive, this indicates moderation is effective when it targets the offspring of offspring generation – that is, when it breaks the word-of-mouth cycle.

Making the most of moderation efforts

Prior research has shown tools based on artificial intelligence struggle to detect online harmful content. The authors of such content are aware of the detection tools, and adapt their language to avoid detection.

The Digital Services Act moderation approach relies on manual tagging of posts by “trusted flaggers”, who will have limited time and resources.

To make the most of their efforts, flaggers should focus their efforts on content with high potential harm for which our research shows that moderation is most effective. We estimate the potential harm of a post at its creation by extrapolating its expected number of offspring from previously observed discussions.

Implementing the Digital Services Act

Social media platforms already employ content moderation teams, and our research suggests the major platforms at least already have enough staff to enforce the Digital Services Act legislation. There are, however, questions about the cultural awareness of the existing staff as some of these teams are based in different countries to the majority of content posters they are moderating.

The success of the legislation will lie in appointing trusted flaggers with sufficient cultural and language knowledge, developing practical reporting tools for harmful content, and ensuring timely moderation.

Our study’s framework will provide policymakers with valuable guidance in drafting mechanisms for content moderation that prioritise efforts and reaction times effectively.

A healthier and safer digital public square

As social media platforms continue to shape public discourse, addressing the challenges posed by harmful content is crucial. Our research on the effectiveness of moderating harmful online content offers valuable insights for policymakers.

By understanding the dynamics of content spread, optimising moderation efforts, and implementing regulations like the Digital Services Act, we can strive for a healthier and safer digital public square where harmful content is mitigated, and constructive dialogue thrives.