In final section of this chapter, we outline the current opportunities and challenges for investigations and intelligence teams.
Opportunities
Cross-functional and external collaboration. Investigations and intelligence professionals have a diverse skill set and scope that allows them to navigate between legal and technical teams. They can increase their value to an organization by partnering with other internal stakeholders in their Trust and Safety units. They can partner with Operations teams to increase bad actor detection and enforcement accuracy, with Risk Management teams to identify and gauge risks to the platform and to the company, and with Policy and Legal teams to review and improve existing Community Guidelines or Terms of Service. They can translate policy or legal issues to a technical audience such as Data Science, Engineering, Product Management, and Security, and vice versa. They can also partner externally with intelligence vendors who can provide a suite of capabilities and tools at scale, within a short onboarding time. This may be beneficial to companies with limited internal headcount and resources to obtain internal or external labor and resources quickly.
Advancements in tooling and technology. Investigations and intelligence professionals can amplify their impact by helping internal teams within an organization develop and improve existing tools, or help find and create resources for better internal tools. Tools that improve search functionality across different types of content (e.g., user information, text, video, or images), or tools that provide tagging or labeling capabilities can greatly speed up and sharpen the process of obtaining user behavior insights. For this, it would be vital to partner with Product or Engineering teams continuously to develop tooling, and to keep these tooling development projects prioritized vis-a-vis other product launches and initiatives. Tooling should aim for improving detection both at scale and with speed.
Trust and safety professionals in investigative and risk management fields also have industry knowledge of what types of internal tools, data resources, or tooling designs are effective vs. ineffective. This knowledge helps product and engineering teams have strategic approaches to software development more quickly than researching best practices from scratch. At times, trust and safety professionals may choose to purchase licenses from third parties that offer abilities to conduct investigations, such as link analysis.
Increased public awareness and demand for online safety. With the rise of high-profile incidents involving misinformation, hate speech, child harms, financial scams, and terrorist or extremist content, there has been increased public awareness and demand for technology companies to prioritize Trust and Safety resourcing. This presents an opportunity for Investigations and Intelligence professionals to obtain executive support and resources to strengthen their capabilities to obtain insights into user behavior.
Driving policy creation and refinement. Since T&S professionals in investigations, intelligence, and risk mitigation-like roles often see emerging threats to an organization or platform, they are instrumental in proposing changes to Terms of Service or company-wide processes. Their investigative findings may challenge existing policies or add more room for nuance in interpretation and enforcement.
Career growth opportunities. Organizations creating a new T&S organization or are expanding their current T&S team can use the skills and roles listed in this curriculum chapter to help plan their hiring and retention plans and create clearer roles and responsibilities across employees. Investigations and intelligence are typically mid-level or senior Trust and Safety professionals with multiple domains of expertise. Therefore, individuals currently working as content moderators, operations analysts, or customer support specialists can take their foundational skills and experience to transition into investigators or threat intelligence analysts. An organization’s management can up-skill these individuals by providing them projects and training in certain skill and workflows, such as:
- Data analysis or programming languages such as SQL and Python, which will make them able to handle platform abuse at a larger scale;
- Specific abuse types, such as child safety and financial fraud involving graphic or sexual content such as sextortion and romance scams, instead of content moderation from a nudity or sexual lens. Or, taking a fraud analyst and specializing them into spam and phishing, account takeover, social engineering, or misinformation;
- Training sessions on effective report writing, business proposal, and threat modeling.
This way, an organization can retain existing trust and safety professionals and offer them a clearer path to career advancement, and give their professionals transferable skills that can enable an analyst in trust and safety to move into other industries such as corporate security, compliance, or healthcare. If the organization does not have any current staff with investigative abilities or experience with risk mitigation, then they can hire for these roles and skills more strategically.
Investigations and intelligence experience, coupled with opportunities for cross-functional collaboration with Product, Policy, and Risk teams, can also be a good foundation for further leadership in the T&S space. A current investigations analyst who may have a technical background but lack policy writing or workflow design skills can shadow someone from Product Management or Policy. Or, rotation programs between these teams can be established to position current investigators into future team lead or manager roles.
Challenges
In-house data and engineering capabilities. Investigations and Intelligence teams can be greatly empowered—or limited—by the wider data science or engineering capabilities of a company. It is, therefore, important to develop or directly embed data science and engineering resources to Investigations/Intelligence teams and wider T&S teams. To dive deep into user behavior patterns and trends on the platform, analysts would require the following:
- A comprehensive and convenient data platform where user/account/content data can be queried, compiled, sorted, and (as a bonus) visualized;
- Internal tools that can amplify analysts’ abilities to combine and compare information from multiple data sets, view and handle user reports, and batch-enforce against bad actor accounts;
- Clear data access policies and chains of command for reviewing and approving data access and collection requests.
Keeping up with evolving threats. As technology continues to advance and new platforms continue to emerge, so too do the tactics used by bad actors to harm users. Bad actors may exploit the simplest of product changes (e.g., expanded methods to search for users or channels, resulting in easier pathways for grooming), or may seek to gain an advantage into new forms of technology (e.g., video-enabled e-commerce, or the metaverse). It is important for Investigations and Intelligence professionals to stay ahead of the curve and adapt their tactics accordingly.
Lack of resources and/or privacy concerns. Investigations and Intelligence professionals are limited by internal tools and information sources provided to them by the company. While developing effective internal tools can present a great opportunity, the lack of political and financial support for internal tool development can severely limit the speed and depth to which investigations and intelligence can yield insights into user patterns and threat vectors. Access to user data can also raise privacy concerns—an issue that is now under the spotlight for a number of technology companies. It is important to ensure the protection and appropriate use of user data while also fulfilling responsibilities to investigate and mitigate harm.
Potential limits on subsequent impact. The translation of investigative and intelligence findings into real impact on a company’s users and its business is highly dependent on whether these insights are acted upon by a company’s product, policy, or business teams. To mitigate this outcome, Investigations and Intelligence teams need to continually demonstrate their potential value and impact, produce accessible and actionable insights, as well as build an effective escalations and review process between them and other cross-functional teams.
Balancing data collection with protecting privacy and public trust. Although this challenge exists in any investigative role regardless of the industry, a trust and safety professional has to ensure that an emerging or evolving threat is identifiable with as minimum information as possible because a business or organization is not a law enforcement agency. The business or organization sets its own policies, and an investigations or intelligence staff member ultimately needs to follow their employer’s policies and goals. A trust and safety investigation could also become publicly available if used in a legal proceeding or government regulation.
There are situations where a stakeholder or victim of an abuse type may ask investigation teams to search deeper into an issue. However an investigator and their team may have to push back on requests if they may be considered too reliant on using data about one’s personal information, or usage of that person’s platform or service, to make the investigation doable. Even if the abuse’s severity or prevalence is not known enough, it may not be worth looking into user data or behavior to try and find evidence of said abuse if it risks or harms an organization’s reputation and commitment to user privacy. Investigators from the cybersecurity industry may have authorization from their organization’s leadership to look into employees’ user data. Law enforcement officers and government employees may have authorization from their local or national government to look into potential abuses. However, a T&S professional may not have similar levels of business or legal protection to conduct such investigations.
Therefore, this curriculum encourages aspiring or current professionals in investigations, intelligence gathering, and risk mitigation roles to consider:
- Approach investigation requests and processes based on its feasibility, benefits to the organization based on the potential findings, and availability of existing data and resources to start and complete the investigation;
- Create investigative processes that are repeatable, regardless of the investigator’s skill level. While it is possible for an investigator to obtain user data more easily than others depending on their technical skillset, it may not be worth using for a Trust and Safety investigation if those data collection techniques collect more user data than what someone else employed for the same role could not also collect;
- Set clear policies on when a T&S professional can access user data, and when additional management or leadership approval is needed to collect user data that is currently not available or obtainable by others within the organization.
Investigator burnout. Trust and safety investigators often get placed on several tasks outside of their investigation case workload. These tasks may include incident response, on-call shifts, quality assurance for ops workflows, training for junior analysts, policy/product/workflow design and documentation, data analysis, or internal tooling design. Although these professionals are given these responsibilities due to their experience and skills being valuable for other teams, these additional tasks burn out staff members at an accelerated pace.
Some investigators may initially find these increased responsibilities as a sign that their organization values their experience. Their impact from one or more of these tasks may carry more weight than their investigative findings because investigations often take weeks to complete and may not always result in an impactful finding. Whereas, one on-call shift may result in a new, critical business need.
The additional tasks may make it harder for investigations teams to complete their cases if they become expected to take on other operational, policy, or data querying responsibilities. Operations analysts that were promoted internally into investigators become stretched thin more quickly than external hires due to their institutional knowledge. If these investigators become unable to work due to burnout or leave the organization due to feeling unable to complete all of their responsibilities, then the organization may lose years of knowledge that will be hard to replace with external hiring.
Investigator burnout can be mitigated in two ways:
- An organization’s senior departmental leadership and manager clearly define scope across its Trust and Safety functions;
- The additional tasks such as projects or data analysis, do not take precedence over investigative report findings and report quality. Some organizations or managers may treat the additional tasks as equal to an investigator’s core work because they may be deliverables that are provided more quickly to an organization. However, just because the investigator’s investigation is incomplete and there are no deliverables resulting from the investigation does not make the investigation an impactful one.
Conclusion
Investigations and intelligence capabilities are vital to Trust & Safety teams across the technology industry. They provide companies and user communities the capacity to dive deeply into user patterns and behavior, and to identify threats or bad actor networks that may not be detected by surface-level content enforcements. This work then allows companies to proactively build safety into the foundation of their platforms, and to stay ahead of threats on the horizon.
Acknowledgements
Authors│Jonathan Lim, Marie McCauley, David Lim
Contributors│Assaf Kipnis, Max Aliapoulos
Special Thanks│Harsha Bhatlapenumarthy
