League of Legends Riot’s takeaway
Cases in the Tribunal were evaluated independently by both player juries and staff
from Riot Player Support. In over a year’s worth of cases, Riot found that the
community verdict agreed with the decision of the staff moderators 80 percent of the
time. The other 20 percent of the time, the players were more lenient than Riot’s
staff would have been (players were never harsher than the staffers).
Riot’s takeaway from the Tribunal experiment was that League players were not only
unwilling to put up with toxic behavior in their community, but they were willing to
be active participants in addressing the problem. This success inspired Riot to
assemble a team of staffers that would make up its formal player behavior initiative,
launched just over a year ago.
Jeffrey Lin holds a PhD in Cognitive Neuroscience from the University of Washington.
Before joining Riot’s player behavior team and eventually becoming lead designer of
Social Systems, Lin worked at Valve Software with experimental psychologist Mike
Ambinder, conducting research on games like Left 4 Dead 2 and DOTA 2. The other
founding members of the player behavior team were Renjie Li, who holds a PhD in Brain
and Cognitive Sciences from the University of Rochester, and Davin Pavlas, who holds
a PhD in Human Factors Psychology from the University of Central Florida.
All three doctors are hardcore gamers, a necessary prerequisite for the core team
members. “A big part of Riot Games in general is we want to be the most player-
focused game company in the world,” Riot producer T. Carl Kwoh told Ars Technica. “
Part of that player focus is really understanding that experience and living and
breathing that experience.”
Before their experiments could go forward, the team had to create some sort of
baseline for what constituted bad behavior in player chat rooms. This meant hand-
coding thousands of chat logs and designating each line as positive, neutral, or
negative. “Going through that exercise once has provided us with good data that we
can rely on as far as intuition goes,” Kwoh said. The player behavior team can now
categorize the nature of chat logs quickly.
With this hand-coding system in place, the Riot team conducted a little experiment
with its player base by messing with the default setting for cross-team chat, which
lets players broadcast a message to everyone on other teams. Players could still turn
the feature on, but they had to actively go into the settings to do so.
Riot then compared the quality of the chat log in the week before and the week after
the switch and noted a more than 30 percent swing from negative coded messages to
positive ones. And it wasn’t just because fewer people were chatting across teams,
either—overall usage of the feature stayed about the same, even after the switch.