Outline of a human head and brain with brightly colored lights.
Photo by Silver Blu3, Flickr CC

Moderating online comments and communities is not for the faint of heart. Racism and sexism run rampant in some online forums, as emojis and crass language regularly target users of various demographics. Sociologist Katherine Cross suggests that development of artificial intelligence (AI) could be a way to lighten the load for human moderators currently attempting to manage the behavior of millions of users. In an interview with The Verge, Cross uses an AI character named Ami that she created as part of a fictional short story as an exemplar of what AI could be in the future: Ami exercises empathy and emotion in ways that current AI technology can’t, but eventually should. To this end, Cross brings up a few important considerations about AI and moderating digital communities:

  1. Development of AI technology that will replace humans is inevitable. Cross points out that most tech companies are looking to automate their moderation processes. “It’s the only solution they can see to the scalability problem that’s particularly acute on social media platforms or huge games where human mods can’t keep up with the actions of millions of users,” Cross says.
  2. There needs to be a bigger workforce of human community managers who can use AI tools to aid their work. While the future of AI is bright, Cross notes there are some community behaviors that AI can’t pick up as threatening — like the use of emojis. “Black Twitch streamers, for instance, have been harassed by people using monkey, fried chicken, or banana emoji, whose racist semiotics are easily understood by other humans, but that would be totally baffling to even the most sophisticated machine learning algorithm.”
  3. “Reason and emotion are not opposites; they inform one another, powerfully.” Cross notes that Ami, as a fictional AI character, is an exemplar of how technology and empathy could combine: “Ami is truly intelligent and, above all, empathetic. Her distinguishing feature as an AI is her capacity to feel the pain of others and feel a responsibility to do something about it, while also possessing the suprahuman powers of a computer. Currently, our real-world AI are only pseudo-AI, and they are woefully inadequate when pitted against the powerful, dynamic forces of human creativity at its worst.”