Approaches to Trust & Safety

Where the trust and safety function sits within a company can have implications on the perception, influence, and impact of its work. How a trust and safety team is structured organizationally and the approaches it takes in its policy and enforcement models varies from company to company. The policy and enforcement model a company takes—and how much, if any of it, to potentially outsource—depends on the company’s platform, values, and community, but each model has its tradeoffs. This section explores the different ways organizations structure their trust and safety teams and various approaches to content moderation.

Organizational Approaches

Organizing the elements of trust and safety into a coherent structure is critical for a well-functioning trust and safety practice. The way trust and safety is structured impacts an organization’s ability to prevent and detect policy-violating content and behavior, how key decisions about direction are decided, how edge cases (unprecedented, nuanced, and/or complex incidents) are resolved, and even whether and to what extent outside experts or advisors are included in the decision-making process. The trust and safety field is relatively young but developing quickly. As a result, there is no single standard for how it is organized within a company.

The first decision an organization typically makes is whether their trust and safety function should be centralized or dispersed.

A centralized trust and safety model consists of a single, discrete team that is responsible for all trust and safety components from end-to-end. This single team works hand-in-hand with product teams but is ultimately tasked with owning policy creation and enforcement, as well as advising at the product ideation and development stage to ensure safety is incorporated into the product, a concept known as Safety by Design.

A dispersed trust and safety approach is a model in which trust and safety team members are distributed throughout a company, and there is no single, central team. Rather, trust and safety professionals are embedded throughout the company and report to the head of each business unit, rather than a singular head of trust and safety.

These two models each have tradeoffs. A centralized model can provide consistency across a company’s products and services. It can also provide a greater level of accountability because there is one individual directly responsible for trust and safety, from product advisory efforts to abuse enforcement. A centralized model may also provide trust and safety with a “seat at the table” at high-level meetings and a single, executive-level, independent voice focused on user and customer protection. However, a centralized model may be viewed as a cost center for the company and an obstacle to developing new and innovative services because its perspective may run contrary to business goals or it may recommend caution in place of rapid growth. This may foster an adversarial relationship with other business units, leading to trust and safety being “sidelined” or left out of conversations on new service offerings that may have trust and safety implications.

A dispersed model—embedding trust and safety professionals within each business unit—provides a more seamless method for trust and safety knowledge and expertise to play a role in product design decisions, minimizing abuse or misuse of the product or service. Although a dispersed model may foster a deeper integration between various business units and trust and safety, it may be at the expense of potential consistencies. For example, if multiple services at the same company each designed a separate method for users to report potentially policy-violating content and each report is evaluated by independent teams of reviewers that do not communicate with each other, it is likely the user experience will be confusing. Furthermore, outcomes, as well as policy-making, may be contradictory. In addition, reporting directly to business unit owners can create an incentive to follow business leader decisions that focus solely on growth and profit, minimizing trust and safety concerns.

Substructure of a Centralized Model

Most large tech companies have adopted a centralized approach; however, hybrid and alternative structures are also used, especially in companies with multiple services that each have hundreds of millions or billions of users. Centralized models must also contend with how a single trust and safety team should be organized. A trust and safety team can broadly be broken down by function, service, or abuse type.

In a function-based structure, sub-teams are formed based on each core job role within trust and safety. There is a product management sub-team, an engineering sub-team, an operations sub-team, a policy sub-team, an incident management sub-team, and so on.

In a service-based structure, sub-teams are centered around each product or service the company offers, but ultimately report to the head of trust and safety. For example, a company offering an online shopping service, a cloud infrastructure service, and a ratings and review service would have a trust and safety sub-team dedicated to developing policy and preventing, detecting, and mitigating abuse on each service, but all sub-teams would report to one head of trust and safety.

In an abuse-type structure, sub-teams are based on each major form of abuse, with the sub-team responsible solely for the abuse area they are focused on, while a single policy team drafts policy and enforcement guidance for all sub-teams to follow. A trust and safety team structured in this way may have a sub-team focused on spam, a sub-team focused on child safety, a sub-team focused on copyright and trademark infringement, and a sub-team focused on dangerous organizations and violent extremism, for example.

Perception and Influence

No matter the approach, selecting where trust and safety fits within the broader company’s organizational structure can have important implications for how trust and safety is perceived and the level of influence a trust and safety team has within an organization. Additionally, to whom trust and safety reports internally can also directly or indirectly affect its ability to have a voice independent of competing incentives, such as revenue, public policy, or public relations considerations. In a centralized model, the leader of trust and safety most often reports to either the Chief Executive Officer (CEO), the Chief Operations Officer (COO), or the Chief Legal Officer/General Counsel (CLO). Reporting directly to the CEO conventionally signals the team’s importance and that trust and safety is a major priority for the company. A trust and safety leader who reports to the COO can indicate that trust and safety is tightly aligned with the company’s business needs and objectives, but may also suggest that if trust and safety needs conflict with revenue generating interests, trust and safety needs may be deprioritized. Reporting up to the CLO can mean the trust and safety team is more compliance-focused and may not directly influence new and emerging products and services.

While organizational structures differ across companies, trust and safety’s primary role is protecting users from abuse and misuse through the creation of policies, guidelines, detection systems, and product interventions. However, it should be recognized that where the trust and safety function sits within a company has important implications for the perception, influence, and impact of trust and safety work.