Opportunities

The primary opportunity is to proactively prevent harm to people, property, and democracies. These goals overlap with those of law enforcement. To accomplish these shared goals, T&S teams must start by assessing strengths and limitations. For example, T&S professionals may have access to user information and data, proprietary algorithms, and tools. However, T&S professionals do not have the legal authority or capability to intervene in the physical world. Put another way, T&S professionals are not law enforcement and cannot make an arrest, question potential witnesses or suspected criminals, or conduct judicial investigations. Conversely, it must be recognized that law enforcement does not have the mission, authority, or capabilities to provide the many services platforms deliver to users. They also do not have the same contractual obligations to provide users a certain degree of anonymity and freedom of expression, or to determine if harmful content violates the terms of service and should be removed from a platform. Through a carefully crafted team approach, T&S professionals can fulfill this common objective and uphold the terms of service expected by users and their platform’s ethos. 

There are two reasons to establish these points of connectivity. The first is time. When we consider time, we recognize that some threats identified by either law enforcement or a platform may have an imminent component. To prevent imminent harm, knowing who to contact in advance may make the difference. Establishing and documenting these relationships in advance can be enormously important when a crisis occurs.  Setting up effective communication channels is essential to ensure that incoming law enforcement requests can be triaged, prioritized, and actioned, and that proactive notification of harm from a platform is passed to the appropriate law enforcement agency.

The second reason is resource constraints. T&S professionals must consider resource limitations, both those of a platform or law enforcement, as in both cases, resources  are finite. This recognition helps to put into perspective the reason to establish relationships well in advance of an instance of imminent harm. Experience has shown that each organization can only do so much so quickly during periods of peak threat volume. Therefore, deploying limited resources for maximum effect is tied to establishing internal policies and external relationships. 

Sharing intelligence between platforms and law enforcement can provide substantial benefits to both. Platforms can often identify unique behavior involving organized networks or scalable actions that would not normally be visible to law enforcement. At the same time, law enforcement can provide high-quality insights into threats that can help trust and safety teams more effectively identify threat actors and policy violations.

Sharing tactical intelligence (the names of accounts and users deemed a threat) and strategic intelligence (information on the broader threat landscape) can be important avenues to advance public safety.  

A word on the critical importance of strategic intelligence sharing: Understanding emerging threats and trends are at the core of preparedness. No one faction, law enforcement, or T&S professionals alone is in a position to find and prevent future threats.  

Finally, the ability to make meaningful change with law enforcement and prevent harm is intrinsically satisfying for T&S professionals. Knowing that T&S professionals have agency and can “do something” to disrupt acts of violence, fraud, or other forms adds some psychological safety. Without law enforcement partnerships, T&S professionals may feel powerless.

Challenges

Good relationships with law enforcement must be constructed with platform users in mind. Trust and safety is strongly aligned with law enforcement in the desire to prevent serious harm, but the professions operate differently and their goals will not always be in full alignment.  For example, T&S are not government agents and also may act more restrivetly than law enforcement.  A detailed understanding of law enforcement authorities can further equip T&S professionals to discern important distinctions between their role and that of law enforcement.

Implementing terms of service is a strength for obtaining mutual goals with law enforcement and can also be a point of divergence.  For example, law enforcement may legally use a range of investigative techniques, such as the use of sock puppets and undercover accounts. These techniques can be used in both open and closed social media environments. Law enforcement does not need a warrant to utilize such techniques and instead can rely upon governing policies.  However, such techniques are not a legal right and don’t carry the weight of a judicial branch court order; therefore, they may violate a platform’s terms of service. This point of departure between certain law enforcement techniques and the terms of service or ethics of a social media platform is not necessarily good or bad. Still, it is a point of delineation.  To reduce suspicion and maintain professional relationships, proactively discussing this distinction is important. 

Another mission delineation from law enforcement can occur over hateful language and hate groups. As disturbing as some hate groups can be, in almost all cases, they have a legal right to express their ideas,provided the speech does not fall into one of the excepted First Amendment categories such as advocating for violence or the sexual abuse of minors. T&S professionals may make referrals to law enforcement expecting action to be taken. Such requests are unlikely to be actioned if they are based on speech alone. Therefore, it may be more prudent and expeditious for T&S professionals to seek other avenues to mitigate such threats. Examples are listed below. By establishing a dialogue with law enforcement, T&S professionals can be better informed about when and where law enforcement may be able to exercise authority.      

Opportunities Beyond Law Enforcement

There are at least four different relationships and programs T&S professionals can consider which are not directly related to law enforcement but can assist in fulfilling the mission of trust and safety. First, having relationships with non-law enforcement partners is encouraged and can provide additional opportunities. Partnerships may include formalized bodies, which lack legal authorities but serve critical roles in helping understand and mitigate the nature of the evolving threats.  

Some NGO’s, such as the Global Internet Forum to Counter Terrorism (GIFCT), assist industry with proactive detection designed to prevent harms such as terroristic threats and violent extremists. They equip digital platforms with tools designed to disrupt terrorist and violent extremist activity. Other prominent partners include the International Centre for Missing & Exploited Children (icmec.org), and the National Center for Missing and Exploited Children.

Second, a particular source of strength internal to the T&S community is cross-platform signal sharing. Threat actors are not confined to one platform and often will seek to utilize a variety of them. This is particularly the case after an individual or group is de-platformed by one social media company. Sharing tactical and strategic intelligence among platforms serves as an additional layer of awareness. As with the case of law enforcement, making contact with T&S peers well in advance of a crisis is prudent.  Furthermore, establishing protocols on what can be shared with peers and when it can be shared has long-term benefits.

Third, there are communities of interest that may be able to play a role in mitigating threats. T&S professionals should have procedures and criteria to determine if such an organization is a preferable channel over law enforcement when dealing with threats.  For example, Life After Hate offers a number of programs to support individuals trying to leave hate groups. They also provide training programs. 

Fourth, establishing a landing page on the platform which provides information for users to contact law enforcement directly should a user become concerned about a piece of content is important. Tailoring the landing page to explain to users the difference in a specific law enforcement agency’s mission or geographic jurisdiction helps to facilitate such activity.

The following are good points of contact in the US for the relevant topics:

• Terrorism and threats of violence: FBI National Threat Operations Center (make a report)
• Human/child exploitation: National Human Trafficking Hotline (submit a tip)
• CSAM: NCMEC Cyber Tipline (report an incident)
• Cybersecurity: FBI National Threat Operations Center or the local FBI field office (list of field office contact information)
• Suicide and self injury: no centralized reporting system; recommend reaching out to any pre-established law enforcement point of contact, known local service numbers, 911 or a local suicide prevention organization.

Challenges Not Directly Related to Law Enforcement

Resources

T&S professionals will encounter a series of ongoing challenges in securing public safety that, at first look, are not clearly connected to their relationship with law enforcement. For example, T&S professionals have to determine how much they can look into a threat issue. As stated before, T&S professionals do not have infinite resources. Established platforms are likely in a position to devote more resources when compared to their smaller peers. Emerging and smaller platforms may not be able to devote many resources at all. Their business growth and revenue may not afford them the opportunity to fully scale up a T&S group within their respective organization. No matter the size of the team, to determine the level of available resources applied to a particular threat, T&S professionals should establish prioritization criteria. Smaller companies can benefit from contacting their larger peers or the organizations listed in this section.     

Litigation

As T&S resources are balanced with other company interests, T&S professionals should be cognizant of the risk of legal liability. For example, a platform may be subject to civil action if the platform does not sufficiently address the threat. It is recommended for T&S professionals to review the Legal section of this manual for more information. Additionally, if a platform makes a statement, perhaps in their terms of service, about data protection, that platform can be considered in breach of their terms of service if they overshare with law enforcement or expose users to such things as a violation of privacy or infringe upon a user’s civil rights. 

Public Attention

Next, there is increased regulatory attention from the public, and governments (e.g., U.S. state, local, and federal,non-U.S. governments), concerning how T&S is included in a platform’s work. Having defined policies and articulating the resources assigned to trust and safety relative to the company’s budget can be helpful when working with lawmakers. Similar to other issues T&S professionals face, scrutiny does not just include government involvement. Equal levels of attention can come from privacy and civil liberty advocacy groups. Striking a balance is bolstered by policy and procedures.

Best Practices to Consider

T&S professionals will encounter several issues when interacting with law enforcement requests. One of these is data retention. Having company policies about what data is stored and for how long is the first step. Such a framework will enable T&S professionals to inform law enforcement what is available and what records have been purged.

The next is establishing policies over encryption and anonymization.  Both afford users enhanced security.  However, one of the challenges for a company that provides either or both is balancing the company’s trust and safety responsibilities with their users’ privacy.  There are several best practices to consider that maintain user privacy and apply the appropriate trust and safety scrutiny.  For instance, platforms can actively track and monitor trends other companies may be experiencing.  This puts the company offering the services in a position to be better aware should they see similar trends.  In response to a change in the threat environment adding in system features and making alterations to the terms of service could be outcomes.  Additionally, establishing robust and accessible user reporting options can assist in identifying abusive users or behavior.  Lastly, companies can take a “safety by design” approach which puts user safety and rights at the center of the design and development of products and services. The goal is to anticipate and reduce harm which might occur while using products, rather than trying to implement remedies after the harm has occurred. An example of this approach is the use of age verification processes to ensure only those who are old enough can gain access to certain websites.

Conclusion

The intersection between trust and safety and law enforcement is one of the most critical and challenging parts of effective trust and safety operations. This intersection covers both a variety of legal obligations for platforms and service providers, and some of the most harmful content and behavior that trust and safety professionals may face during their careers.

Whether responding to legal and emergency requests or proactively reaching out to law enforcement about potential threats, establishing clear and efficient processes is essential to ensure legal obligations are fulfilled and the best possible actions are taken as quickly as possible. Building professional working relationships with law enforcement is valuable both to improve these processes and to allow both parties to provide insights and feedback to each other so that they can work together to more effectively prevent harm.

---

Acknowledgements

Authors│Karen Brown, James Gresham, Christine Kalaveshi, Holly Lawrence, Brian Murphy, Evelyn Tzeng
Contributors│Wesley Rich
Special Thanks│Michelle Solis, Rolando Vega