League of Legends Riot’s takeaway

Cases in the Tribunal were evaluated independently by both player juries and staff

from Riot Player Support. In over a year’s worth of cases, Riot found that the

community verdict agreed with the decision of the staff moderators 80 percent of the

time. The other 20 percent of the time, the players were more lenient than Riot’s

staff would have been (players were never harsher than the staffers).

Riot’s takeaway from the Tribunal experiment was that League players were not only

unwilling to put up with toxic behavior in their community, but they were willing to

be active participants in addressing the problem. This success inspired Riot to

assemble a team of staffers that would make up its formal player behavior initiative,

launched just over a year ago.

Jeffrey Lin holds a PhD in Cognitive Neuroscience from the University of Washington.

Before joining Riot’s player behavior team and eventually becoming lead designer of

Social Systems, Lin worked at Valve Software with experimental psychologist Mike

Ambinder, conducting research on games like Left 4 Dead 2 and DOTA 2. The other

founding members of the player behavior team were Renjie Li, who holds a PhD in Brain

and Cognitive Sciences from the University of Rochester, and Davin Pavlas, who holds

a PhD in Human Factors Psychology from the University of Central Florida.

All three doctors are hardcore gamers, a necessary prerequisite for the core team

members. “A big part of Riot Games in general is we want to be the most player-

focused game company in the world,” Riot producer T. Carl Kwoh told Ars Technica. “

Part of that player focus is really understanding that experience and living and

breathing that experience.”

Before their experiments could go forward, the team had to create some sort of

baseline for what constituted bad behavior in player chat rooms. This meant hand-

coding thousands of chat logs and designating each line as positive, neutral, or

negative. “Going through that exercise once has provided us with good data that we

can rely on as far as intuition goes,” Kwoh said. The player behavior team can now

categorize the nature of chat logs quickly.

With this hand-coding system in place, the Riot team conducted a little experiment

with its player base by messing with the default setting for cross-team chat, which

lets players broadcast a message to everyone on other teams. Players could still turn

the feature on, but they had to actively go into the settings to do so.

Riot then compared the quality of the chat log in the week before and the week after

the switch and noted a more than 30 percent swing from negative coded messages to

positive ones. And it wasn’t just because fewer people were chatting across teams,

either—overall usage of the feature stayed about the same, even after the switch.