The internet has significantly shifted how people around the world interact with each other. Online service providers, internet companies, and various technologies have become the vehicles that allow people to engage and interact at a scale not previously possible. While these companies and technologies drive innovation and growth and connect people in unprecedented ways, they also serve as vehicles for harmful and unwanted behaviors. To address such problems, companies have invested in efforts to prevent, mitigate, minimize, and—where possible and warranted—remove unwanted content or conduct.
While individuals and teams have diligently worked to protect user safety and ensure trust in digital products and services since the advent of the World Wide Web, the term “trust & safety” (often shortened to T&S) emerged from 1999 to 2010. It’s now commonly used as an umbrella term to describe the teams at internet companies and service providers that work to ensure users are protected from harmful and unwanted experiences. Long before the phrase became popularized, eBay was an early adopter of the term “trust and safety,” where “trust” referred to trust among eBay users and of the company itself, and “safety” referred to keeping platform users safe (Boyd, 2022).
Since that time, the growth of trust and safety reflects the confluence of four factors:
- The proliferation of online services and user-generated content;
- The recognition among many large internet companies that their products can be exploited in ways that bring about unintended consequences;
- The realization that if efforts are not undertaken to mitigate such misuse and abuse, both users’ experiences and company reputations and profits are likely to erode;
- The recognition that trust and safety issues as well as techniques used by bad actors are constantly developing, requiring ongoing and evolutionary responses.
Early trust and safety teams responded to common forms of online abuse (e.g., fraud, scams, phishing, spam, trolling, malware) which surfaced during the commercialization of the internet and the rapid growth of the web in the 1990s. Since then, and especially with the advent of social media in the mid-2000s and the proliferation of user-generated content (UGC) that came with it, trust and safety has evolved from identifying, minimizing, and merely removing bad content to having a much larger and more central role in creating the content and product policies governing a company’s products and services, as well as developing the tools, systems, and techniques for enforcing those policies. Developing this trust and safety framework often starts with a company defining its core values and trust principles, which in turn leads to tailored trust and safety models, policy development, and enforcement operations. This chapter provides a broad overview of the trust and safety functions within the majority of today’s internet companies.