Toxicity is turning one in five people away from engaging online
The internet has enabled huge advances in society and, for many, the experience has been positive. More than 80%
of people think that the internet has made life better for ‘people like them’.
Yet for many, the story is not entirely positive. Recent analysis
of more than 100bn lines of chat suggests that while the vast majority of interactions (82%) were healthy, 3% were actively harmful, and the remaining 15% nuanced and falling somewhere in between.
This underlying level of abuse is causing around one in five people to disengage entirely from online communities:
19% of adults have stopped playing certain online games as a result of in-game harassment;
21% of people stopped using social media after facing online harassment.
Public opinion is also hardening against harmful interactions, and the companies that enable it. More than one in four
people say they would stop using a platform if it kept allowing harmful content. And many
people do not believe that tech companies are designing their products and services with users' best interests in mind.
Harmful content has real costs to business, as well as to society
These levels of toxicity are having an impact on companies’ bottom line, as well as on user health and wellbeing. Companies of all sizes are beginning to take action. A recent quote
from Chris Buzzo, Chief Marketing Officer at Electronic Arts, one of the largest gaming companies, says it all:
“In a study that we did, 58% of players say they experienced some form of toxic or disruptive behaviour in the last year. And it is one of the primary reasons that they choose not to play. So we don't really need much more evidence than that to say that toxicity is ruining play.”
Action is becoming an economic, as well as a moral, imperative. No company wants to lose engagement with an audience it has taken a lot of time, effort and money to acquire. In fact, in purely financial terms, engaged users can be worth up to 20 times more to a company than non-engaged users. For example, Canadian company Two Hat found
that the LTV (lifetime value) of players who did not participate in public chat was about $57, compared to the $563 LTV of players who did engage. The LTV of those who participated in private group chats was even higher, at more than $1,200.
Toxicity also takes its toll on tech workers. Not many people want to stay working on products they can see are causing actual harm to society. More than a quarter (28%
) of tech workers in the UK have seen decisions made about a technology that they felt could have negative consequences for people or society, the UK thinktank Doteveryone recently found.
Nearly one in five (18%) of these workers went on to leave their companies as a result. The number of employees leaving due to ethical concerns is even higher - 27% - for employees working specifically on AI. Given that every technology worker that leaves a company does so at a cost of £30,000
, this in itself imposes a significant cost.
Safety tech helps to prevent harm, and helps human moderators do what they do best
Given the above, it’s not hard to see the economic, as well as the social, arguments for investing in online safety.
Choosing, and using, the right technology is just one part of the mix. Other design elements are being considered within the emerging discipline of ‘safety by design’, which is currently gaining momentum in particular in the UK
The right choice of tech can help to prevent harms happening in the first place, helping to nudge users into rethinking potentially abusive content before they click ‘send’. A recent study by OpenWeb, for example, found an overall 12.5%
lift in civil and thoughtful comments after using the ‘nudge’ technique to challenge toxic behaviour, and an increase in community approval rate of up to 4.5%.
Safety tech can also significantly reduce moderator workload by blocking known illegal content, spotting known patterns of abuse, and by helping to prioritise the most urgent and important cases in moderators’ work queues. This frees human moderators up to focus on what they do best - understand context, empathise and make fair, nuanced decisions, and communicate the results. It also helps to reduce the physical and emotional burden on moderators themselves.
In one case
, adding a proactive chat filter to a popular mobile game delivered an 88% decrease in moderation volume, decreasing user-generated reports of abusive behaviour from 500 to 50 cases a day. And in another, emerging social media platform Popjam
found that using safety tech to help moderate images and content helped to reduce moderator workload by 65
How to invest in online safety - next steps
The evidence suggests that any company seeking to maintain safe, healthy online communities should check whether they have the right tech in place to support moderators.
There are products and services available to help you block known illegal content, deliver kid-safe experiences, detect known patterns of abuse, and fight disinformation. You can choose from free tools provided by some of the biggest international tech companies, to bespoke and supported solutions from specialist safety tech companies in the UK and overseas.
To find out more, check out our guide
to safety tech, and directory
of providers.[Many thanks to Qasim Shafi for his help researching this article]