As the organizational role of trust and safety continues to evolve, the field has shifted from a primarily reactive approach—responding to legal requests, removing content after it has been flagged by users, and adding friction to prevent abuse after such abuse has been discovered—to a more proactive model. Today, trust and safety teams are often consulted during the early design of a product to help anticipate abuse and build in safety features at the onset.
In this chapter, we trace this evolution, starting with a nascent field focused on catching and removing spammers and trolls to a complex global infrastructure involving moderating trillions of bits of data, managing millions of users, and hiring thousands of trust and safety practitioners. We explore key elements that make up trust and safety as well as different approaches and their tradeoffs. We also discuss the importance of where a trust and safety team fits within an organization.
The aim of this chapter is to help the reader understand the key elements of trust and safety so that they can help shape its complex future.
