Identifying inappropriate comments in German-language online newspapers

In online discussions, users are called "trolls" when they provoke others, try to dominate discussions or manipulate the opinions of other users. With the rise of social media, trolling has become a prominent term. The discussion culture suffers from the presence of trolls. But there are also more extreme effects, such as paid propaganda trolls who aim to influence elections. Some trolls are easy to recognize because they obviously spread hate, while other trolls behave more subtly. Therefore, removing troll comments is laborious work.

This paper focuses on German-language user comments in (mostly Swiss) online newspapers. The goal is to develop classification algorithms that can automatically and reliably detect unwanted behavior in comments. In a first step, existing literature and solutions were analyzed and evaluated. In addition, it was examined how various Swiss newspapers deal with trolls. Subsequently, training data was collected and processed. Among other data, a labeled dataset of over 2 million comments was compiled by 20 Minuten. Classifiers were developed with different training data to detect three categories of trolls: Hate trolls, Off-topic trolls and State-linked propaganda trolls.

Hate trolls are detected with a recall of 84%, precision of 83% and accuracy of 83% using a combination of BERT models. Off-topic trolls are detected with a recall of 78%, precision of 83%, and accuracy of 80% mainly by calculating the cosine similarity from a comment to other comments and the article content. State-linked propaganda trolls are detected with a recall of 92%, precision of 90% and accuracy of 91% for training data from Twitter. For comments from 20 Minuten, a classifier can predict with a recall of 62%, precision of 76% and accuracy of 71% whether comments will be accepted or rejected by the moderation. The results have shown that a metadata-only approach is not feasible for the analysis.

To make the results and the algorithms accessible to lay users, a web application was developed. This proof of concept allows trying the classifiers using custom or existing comments.

These classifiers cannot completely replace the manual moderation process. However, the classifier can be used to support the human moderators. With an adjusted threshold, about 20% of the unwanted comments can be automatically detected with almost no false positives. In future research, it could be investigated how well the developed classifiers perform with different data from other domains. Furthermore, the analysis could be further extended by not only analyzing individual comments but accounts of comment authors (user-based approach compared to post-based approach).