In trust and safety, the structure, roles, and responsibilities of investigations and intelligence gathering teams may differ depending on the organization’s resources and size.
In smaller companies, the individual contributors (ICs, an industry-wide term to describe an employee who is not in a manager position) may handle both investigations and intelligence gathering responsibilities across several types of abuse. This is because smaller companies may not have additional staff (such as data analysts and scientists) to handle the data analysis for intelligence gathering requests. The ICs’ domain expertise across the several abuse types may vary among each employee. The organization can determine whether investigation requests are assigned to the employee based on their domain expertise, or have the investigations be handled by anyone on the team. Larger companies, however, may hire investigators with specific or various domain expertises and expect them to handle investigations primarily of those domains.
Investigations and intelligence gathering teams may be separated into their own teams by categories such as abuse type, product, or geographical region. The teams could also include not only investigators, but also data scientists, and Artificial Intelligence/Machine Learning engineers. However, employees in those roles may report to their manager who oversees that respective function (for example, a Data Scientist may report to the Data Science Team) as opposed to an Investigations Manager. This means an investigator may not have a data scientist or engineer as their supervisor, but the investigator could have the data scientist or engineer as their teammate.
Team Structures and Responsibilities
The table below further explains how team structures and responsibilities may differ depending on an organization’s size and scope.
| Small to Medium (< 10k employees worldwide) | Large (>=10k employees worldwide) | |
|---|---|---|
| Team Structure | One manager for a small set of ICs (1-4) Investigations Manager would report directly to the Head of Trust & Safety, Product, Legal/Compliance, or a similar Business Operations-related department Investigations Team would be located globally and expected to handle issues across all geographical regions and time zones | Head of Investigations with multiple Investigations Managers as their direct report Head of Investigations may report to a broader Operations, Product, Security, Legal, or Risk & Response. department with additional chains of command above the Head in each department Each Investigation Manager manages a set of ICs and may be its own sub-team within the Investigations Team Investigations sub-team or functions may be separated by geographical region, time zone, or abuse type May have additional ICs such as Program Managers, Data Analysts, Data Scientists, or Security Engineers, to oversee certain intelligence gathering and compliance initiatives more broadly |
| Investigations | Team handles all investigations for the Trust & Safety organization across various abuse types, such as Child Safety, Fraud, or Spam issues Each IC may be expected to cover all abuse types, or be assigned to investigations for a certain abuse type Investigation reports may be used in company legal proceedings | ICs may be more specialized, handle investigations based on the domain/abuse type they were hired for Investigations requests may be more technical in nature, thus requiring more skills with data analysis and programming languages Investigation reports may act more as an audit/compliance mechanism and only be used in company-wide legal proceedings on a case-by-case basis. This is because the company may have a separate Legal Department handling all of the reporting for a legal proceeding |
| Intelligence | Done on an ad-hoc basis per requests from organizational leadership or other departments Handled by the Investigations Team, with potential partnership with Product Analysts, Data Scientists, and Engineers from other teams and departments within the organization Insights may be shared in org-wide meetings but not to the broader company May be deprioritized if Investigations Team has several investigation requests that require quicker deadlines | Intelligence gathering may be its own separate team from the Investigations Team to analyze ongoing/known abuse types and threats at a larger scale, instead of taking enforcement actions on each reported or detected abuse May have Investigators, Data Scientists, and Engineers, all in one team but report to the same manager Have a routine cadence to generate insights internally to the broader company, but require additional Legal review and approval to publish insights externally Occur regardless of Investigation Team’s request count because it acts separately |
| Deliverables | Investigation Reports describing the nature of abuse, its prevalence, potential impact on company, and attribution Proposals to create or update existing policies and processes External documentation on an identified threat or abuse type (such as company blog posts, Support Articles, court documents) Referrals to law enforcement agencies Suggestions for defining safeguards and other mitigative processes | The Investigations, Intelligence, and Risk Mitigation functions may each have their own versions of a deliverable, such as: A machine learning model to detect or action on certain user behaviors Research documents on user behaviors or product development Assessments on the effectiveness of existing risk mitigation controls Taking scaled enforcement action to disrupt abusive behavior Strategic intelligence reports on emerging harm and threats |
Roles
In trust and safety, job roles related to investigations, intelligence gathering, and risk mitigation may fall under the titles Analyst, Investigator, Investigations Analyst, Abuse Analyst, or something similar. Below are some examples of what an investigations role might look like under an assigned abuse type. Consequently, the list is not exhaustive, and these roles may overlap with one another.
Child Safety Analyst/Investigator. This investigator would review external and/or internally submitted reports of alleged harm specifically towards underage users that go beyond individual user-submitted reports that a Content Moderation Team handles. This role can be further specified in its title or job description to indicate a certain type of abuse towards underage users or a certain product or service that the organization offers, such as cyberbullying and harassment investigator, live video streams analyst, suicide and self-harm analyst.
Fraud Analyst/Investigator. This role is one of the broadest types in trust and safety, so the scope of fraud may differ across organizations and companies, even if the job skills and day to day job responsibilities may look the same on paper. For some organizations, this role may be named more specifically to focus on the financially motivated crimes aspect of fraud, such as money laundering, or fraudulent activity at a platform-wide scope instead of certain incidents.
Therefore, it is up to the organization to define what types of fraudulent activities that an investigator would cover. Both prospective and current Fraud Investigators would benefit from asking recruiters and hiring what types of specific fraud the investigator would be expected to handle in the job application process because an organization may not be able to specify types of fraud in a public job posting.
Fraud investigations in the trust and safety do not always have to involve financial or e-commerce related issues. Technical abuses within an organization’s product or platform, such as injection attacks into databases and codes or breaking weak algorithms to steal customer data to commit fraud, may also not fall directly under a Trust and Safety Investigation team’s scope. Instead, fraud in trust and safety may include but not be limited to:
- Account takeovers (ATO), or account activity resulting from hacked accounts like account ransom and SIM Swapping;
- Ads, user reviews, or product listings that may be misleading or inauthentic in nature;
- Scams designed to take a victim’s identity, data, money, or other assets;
- Content or activity designed to redirect users off of the platform to another third party service;
- Phishing;
- Scraping;
- API misuse;
- Social engineering campaigns and techniques.
Counterterrorism (CT) or Harms Analyst/Investigator. This investigator role’s availability in an organization highly depends on what the organization considers a threat or a harm. Smaller companies may not have a dedicated investigations team or members focus specifically on counterterrorism, so this role could be housed under a different branch of Trust and Safety, like the Law Enforcement Response Team (LERT). In general, a CT investigation investigates user behavior, account creation, and engagement related to a known terrorist or extremist organization to prevent potential harm from a terrorist attack, or prevent the organization from recruiting and growing on a platform.
Site Integrity Analyst/Investigator. This type of investigator focuses on the abuses that target the authenticity of the platform. Common integrity abuses, particularly among social media, manifest in the forms of disinformation, misinformation, impersonation, and other methods of deceptive or inauthentic activities. At a glance, these types of threats may not be inherently or presently violent like terroristic threats, or threats to child safety. However, they can negatively disrupt social communities from engaging in healthy exchange of ideas and communicating accurate information, which may consequently shape the discourse of civic or crisis events in the real world. Sometimes, these abuses are undertaken in a coordinated manner and investigators may discover a network of actors working together to carry out these abuses, such as a disinformation campaign. Thus, complex investigations on integrity abuses often require investigators to study not just the content, but also the behavioral aspect of the violating actor.
