Automated moderation arrives, and the old problems come with it

Slurs, invite links, a message with forty mentions, the same line pasted six times, a banned word in a display name and, in Call of Duty, speech on voice chat: software now judges each of these before a moderator sees it. It is faster and steadier than any staff team and no better at context, and when it gets something wrong, "the bot did it" answers nothing.

Where the automation comes from now

Word filters are old, and the earlier piece on handling toxic players covers why string matching is weak. What changed is that the automation now arrives from outside the community, written by a platform or a vendor.

Discord launched AutoMod on 16 June 2022. At launch a server could set up to three custom keyword filters, switch on Discord's own lists of commonly flagged words, time out anyone who tripped them and send the flagged message to a moderators' channel. Discord's developer documentation for AutoMod now allows six keyword rules of up to 1,000 terms each, alongside preset word lists, one of them for slurs, that Discord maintains and server owners do not write.

Machine learning followed. Jigsaw's Perspective API gave anyone with a key a toxicity score for a piece of text. On 9 March 2023 Discord began an experiment called AutoMod AI in a limited number of servers, using OpenAI technology to alert moderators when a server's own rules might have been broken. Voice came next: Activision began rolling out Modulate's ToxMod in Call of Duty on 30 August 2023 in North America, with a worldwide release outside Asia timed to Modern Warfare III on 10 November. That is a publisher's purchase. A community server gets what its chat platform ships, plus whichever free API a bot author happened to wire up.

What it does well

The case for automation is mostly about hours. A volunteer staff team is thin at four in the morning in its own timezone, and a bot is not. It treats the owner's friends exactly as it treats strangers. Against a raid it reacts in the time it takes a message to arrive, and it leaves a record of every action, which a person moderating at speed often does not.

For volume problems like spam and mention floods, a rule that fires instantly beats a human who arrives ten minutes later, and covering every hour of the day is one of the harder parts of moderating a game community in the first place.

What it cannot see

Context comes first. The same word is abuse in one message and a quotation in the report about it, and a server's running joke may use a term a preset list hates. Discord's rules accept an allow list for exactly this reason (up to 100 exempted terms for a keyword rule, 1,000 for the preset lists), which is an admission that context exists, kept up to date by hand. Substring matching has had a name for its failures since the 1990s: the Scunthorpe problem, after a town whose name contains a word no filter likes.

Language is harder. Call of Duty's voice moderation began with English only, with other languages promised later. A keyword list written by English speakers misses a Brazilian or Polish server's slang and trips over its ordinary words, and a classifier is only as good in a language as the text it learned from.

Then there is harassment that stays polite. Twenty accounts asking one player the same innocent question, or a pile-on of compliments whose meaning everyone in the community understands, breaks no rule in any single message, so no rule fires. Automated tools judge messages one at a time, and this kind of harassment lives in the pattern between them.

Rules nobody on staff wrote

A community that writes its own word list knows what is on it. One that switches on a platform's preset list, or scores messages with a vendor's model, is enforcing rules it did not write and can only carve exceptions out of, and the vendor's plans become part of its policy. Jigsaw is sunsetting Perspective: the service stays up until 31 December 2026, and Jigsaw's explanation is that AI capabilities have moved on and demand for a standalone tool has fallen. A bot that scores messages with it needs a replacement by the new year, and the replacement will score differently, so a message that passed in December may earn a timeout in January without anyone on staff changing a thing.

When the bot did it

The appeal is where automation hurts most. A player timed out by AutoMod asks why, and the moderator answering did not make the decision. The most they can usually offer is which rule fired, which explains what happened without saying whether it was fair. A classifier's score is worse, because a number is not a reason, and "the model rated you as toxic" is not an explanation anyone should be asked to accept. Appealing a ban as a player only works if there is a person who can look again at what happened, so automation has to be set up to leave one.

That means dull procedure. Every automated rule should have an owner on staff who switched it on and reads what it catches. Appeals against automated actions go to a human who can reverse them, under the same process that keeps appeals from becoming a popularity contest, and a small community can run the whole thing through ticket-based player support without a helpdesk team.

Which rules a machine should enforce alone

Discord's own experiment drew the line in a telling place: AutoMod AI was built to alert moderators when rules might have been broken, and it left the decision with them. That split is the useful one for a community server. Blocking a message is cheap to get wrong, since the sender can rephrase, so blocking invite links, mention floods, pasted spam or a known slur before it posts can happen automatically. A timeout or a ban for anything that depends on who said what to whom, and in what spirit, should raise a flag and wait for a person, even if the flag sits there until somebody on staff wakes up.