Investigation
Investigation is the process of examining and establishing facts about online behavior using an array of techniques via tools and resources that are either open-source (available for free to anyone with internet access) or internal to a company. An investigation’s end goal is to develop a deeper, more comprehensive understanding of online behavior, including but not limited to unique users, platform-wide activities, or wider user patterns and engagement. Trust and safety teams undertake investigations for many purposes like helping organizations define and/or improve their policies, improving community guidelines, understanding product usage, recommending product developments, or measuring reputation.
Investigation, Content Moderation, and Operations. Investigations and content moderation are often mentioned alongside one another in trust and safety; however, in practice they involve separate processes.
- Content moderation refers to the broader process of detecting and reviewing content that violates a platform’s Terms of Service, then taking steps to block, filter, label or remove it. Content moderation typically falls under the scope of Operations. Content moderation may involve investigative “elements” such as reviewing content to detect and identify violating user behavior, but it does not require thorough investigations into finding new or emerging types of abusive content or behavior. (For more information about this topic, see the Content Moderation and Operations chapter.)
- Investigations, however, focus on reviewing potentially violating user behaviors that may not be tied to a known abuse or process that a content moderation team can currently handle. So, an investigations team takes additional time to review the user(s), the user-generated content (UGC) or the user’s behavior, and historical context as to how their organization or similar organizations have handled the abusive behavior. Because of this, an investigation’s findings may help content moderation teams shape their existing content moderation policies and processes to handle reported and detected abuse more effectively and at a broader scale.
These differences are why content moderation and investigation require a different operational workflow and skill set for each team and process.
Intelligence
Intelligence is the process of collecting actionable, relevant, and timely information to influence enforcement, policy, and product, with the chief goal of mitigating or preventing threats. While intelligence and investigation share similarities in outcomes, the processes and structures are different. Investigation studies detail user behavior in the past or present, whereas intelligence gears towards assessing the trends, threats, and patterns on the horizon that can then be applied to various functions. Both play an important role in shaping policy and product development, as well as operational enforcement.
Risk Mitigation
Risk can be defined as the potential, possibility, or likelihood of harm, costs, and damage to a company, its user community, or other stakeholders. Risk mitigation is the process of identifying, assessing, controlling and/or reducing, and reviewing risks to a company.
Common Types of Risks. The following is a non-exhaustive list of common trust and safety risks a company may face:
- Reputational/PR: A company may suffer public relations or reputational damage from negative media coverage, resulting in a loss of confidence in and demand for its product or service. This may require huge communications resources and investments to mitigate negative coverage and explain how the company plans to rebuild users’ trust and improve itself.
- Regulatory/Legal: A company may be targeted by governments or state agencies through strengthened legislation or regulation. This often comes with the belief that the company’s product or service has harmful effects on society and needs to be regulated to mitigate negative externalities.
- Brand Safety: For companies and user-generated content platforms that rely on advertising for revenue, advertisers and businesses may avoid displaying ads on these platforms as being associated with these platforms may damage their own reputations.
- Employee Safety: A company may also suffer threats to the safety and well-being of its employees if bad actor behavior targets the company’s staff. Bad actor behavior may come from individuals wanting to abuse a company’s services for personal or financial gain, or retaliating against the company for decisions with which they do not agree.
- Monetary: A company may face monetary risk tied to trust and safety issues occurring on its platforms. Although monetary risk is baked into Regulatory, Brand Safety and Reputational risks, it is worth mentioning explicitly as companies often perform a cost/benefit analysis related to this risk alone.
The Role of Investigations & Intelligence in Risk Mitigation
Investigations and intelligence teams help with identifying, assessing, and reviewing risks by:
- Identifying: Uncovering patterns and manifestations of user behavior that present risks to a company and its community.
- Assessing: Gauging the extent of negative impact (harm, costs) to a company and its community.
- Reviewing: Building a situational picture of user behavior before and after risk mitigation interventions and controls have been made to evaluate the effectiveness of such controls.
- Recommending: Drafting security recommendations as part of the investigative and intelligence process. Some common recommendations are enforcement actions, automated system improvements or policy changes.
Close working relationships with data analysts and security engineers are important for progressing in these areas. Investigations and intelligence can focus on more qualitative and hard-to-measure aspects of user behavior, while data analysts and engineers can focus on quantitative indicators.
Broader Impacts of Risk Mitigation
As executive teams focus on the growth and expansion of a company, risk mitigation helps to assess and reduce costs that might befall the business and its users. It limits the potential and likelihood of downside when pursuing growth, expansion, and improvement of the product or service. Risk mitigation processes identify potential, present, and future weaknesses and threats. That way, a company/product will not be blindsided by pitfalls as it pursues additional product and business developments.
Supporting Safety by Design. Investigation and intelligence teams help companies understand user behavior patterns, which can shape improvements in both product and policy to mitigate risks.
