Each company’s trust and safety team is structured slightly differently, depending on its services or purpose, and the company’s platform architecture, resourcing, and development. However, there are common trust and safety elements or functions shared across the industry that help map out what constitutes a developed trust and safety team. In early-stage companies or companies with a relatively small user base, the business owner or single employee may be responsible for all trust and safety functions, frequently on a “best effort” or “as needed” basis. In these instances, policy and enforcement decisions are often ad hoc and may go untracked.
The descriptions below are general functions or roles found in many mature trust and safety teams at companies with established operations, although we often reference how smaller companies handle similar issues by comparison. Companies may include all, combine, or provide only some of these various roles and functions. Other companies may not consider some of these functions as part of trust and safety at all and organize them in another part of the business.
The basic component of many trust and safety teams consists of policy, individuals and sub-teams that set the “rules of the road,” and enforcement, individuals and teams who ensure those rules are followed by identifying policy-violating content or behavior and taking action (which can include warning, removal, blocking, etc.). Policy is generally composed of individuals who develop and revise policies on particular products, services or issue areas. Depending on the role, they may work with various internal stakeholders to drive the strategy, vision, and execution for preventing and reducing policy violating content and behavior on the platform. Enforcement is generally a function of the operations team, who is responsible for day-to-day services, including moderation activities who review content or behavior and determine based on policy whether it is allowable. Their work may include incident management and responding to high-profile or widespread trust and safety issues affecting a particular product or service.
Most teams in mature platform settings have a much larger constellation of elements and functions that support the policy and enforcement effort. This includes engineers who develop the tools used by content moderators as part of conducting a review to enforce policy and developing and implementing new product features to mitigate harmful experiences, educate users about policies, and improve users’ trust. Smaller companies often aim to outsource some of these needs to external providers. Over the years an entire ecosystem of trust and safety service providers that aim to provide some of these capabilities has emerged. Another element that supports policy and enforcement, especially in scaled settings, are those who create knowledge management systems and the training resources content moderators use to learn how to apply policy as well as those who work on content design and strategy to educate users on the platform’s policies.
In scaled settings, researchers, data analysts, or engineers support policy and enforcement by providing crucial insight into users’ experience, the prevalence of abusive content, user reaction to that content, and how users perceive policy and enforcement, among other areas. For example, research insights often guide the development of new policies, enforcement processes, or new product features aimed at creating a safer user experience. An intelligence and discovery capability, which can be a part of research but also be an independent function, can help identify new abuse trends or abusive actors that may inform policy development and/or be used by enforcement teams.
Other teams that support enforcement efforts are law enforcement response and compliance teams, who are responsible for reviewing and accurately assessing legal requests from law enforcement officials, while ensuring compliance with applicable law and platform’s terms of service. Members from these teams may work closely with internal public policy and legal teams. In smaller companies these tasks are often handled either directly by an attorney or by the enforcement team in consultation with the company’s legal function.
These are just a few of the elements that form the cornerstone of a mature trust and safety apparatus. Established trust and safety teams also work with a number of other stakeholders, from communications to customer support to sales and advertising (in for-profit contexts). A more comprehensive list and description of trust and safety-related functions and roles can be found in the Key Functions and Roles overview.
