During our latest webinar, Jen Weedon from Niantic's Trust and Safety team, shared insights into her team's Safety by Design approach, as well as other preventative safety measures the company employs to ensure a safer and more engaging experience for users on their platforms.


As online gaming experiences become increasingly interconnected and attract diverse audiences, including young users, the importance of robust safety measures increases. This is why at Niantic, Jen's team is informed by a Safety by Design approach, that prioritises prevention, education and proactive thinking.

“We are trying to empower users to control their experiences. This isn’t about designing a utopia; this is about empowering people to create the spaces they want.”

By combining expertise from across departments and drawing from the experiences of different teams like Policy, Detection and Moderation Operations, Investigations and Enforcement, and Child Safety, their team aims to minimise risk from every angle. To ensure that their platforms are being developed responsibly, Jen's team employs strategies like risk analysis, and adversarial planning (thinking like a bad actor), and redteaming that help them spot potential safety risks in the development process.

What is Redteaming?

Redteaming involves assembling stakeholders to rigorously test products or features, to uncover potential safety risks that might otherwise be overlooked. This proactive approach helps mitigate risks before they materialise. Jen illustrated the redteaming process with anonymised case studies and emphasised its iterative nature, advocating for regular reevaluation as products evolve. Following this approach not only benefits online players, but can also positively impact business and organisational success.

Business Benefits

• Launch readiness: Businesses are informed by where and how to prepare for product or functionality launches.
• Informing decisions: Informs businesses on implementation of certain trade-offs or mitigations to implement.
• Harm prevention: Through redteaming, businesses work with their product to prevent harm.
• Resource allocation: Based on the information from red teaming, resources can be allocated differently, which demonstrates cost-efficiency.
  
  

Organisational Benefits

• Opens a safe space: Teams are encouraged to discuss difficult topics and share their perspectives in a safe space, promoting trust and accountability.
• Critical thinking: Promotes critical thinking early on in the development of the product or functionality. By thinking outside of the box, it also breeds a culture that challenges the status quo.
• Robust plans: Teams benefit from redteaming by developing more resilient and well-thought-out plans, motivated by collaboration.
  
  

Safety Considerations outside Traditional Content Moderation

Once a product has been 'redteamed' or 'pressure tested' Jen and her team work together to think outside the box on how to limit abuse threats for various different demographics. This includes features like restricting certain levels or game features for young players, gathering community feedback before wide releases, and implementing comprehensive in-game safety features like blocking and reporting mechanisms. For location-based games like Pokémon Go, Niantic also has to consider the location of their games to ensure that all gameplay takes place in safe locations worldwide.

Overall, creating safe products is about creative thinking and adaptability, stressed Jen. The gaming industry frequently innovates and and adopts new technology, requiring Trust and Safety teams to think outside the box to stay ahead of the curve. Amidst the dynamic nature of online gaming, fundamental toolsets and best practices remain crucial for ensuring safe-by-design experiences. As a member of the Trust and Safety Professional Association , Jen collaborates with other experts to publish and reshare best practice guidelines for designing safe experiences.

Safety by Design Guidelines & Resources

• Safety by Design - eSafety Commissioner
• Toolkit for Digital Safety Design Interventions and Innovations: Typology of Online Harms
• A Framework of Severity for Harmful Content Online
• Prosocial Design Network
• Basic Online Safety Expectations - eSafety Commissioner
• Global Principles on Digital Safety: Translating International Human Rights for the Digital Context
• The Safe Framework
• The Procedural Justice Framework for Tech Professionals
• Shouting into the Void: Why Reporting Abuse to Social Media Platforms Is So Hard and How to Fix It
• A Menu of Recommender Transparency Options
• Harms modeling - Azure Application Architecture Guide
• Disruption and Harms in Online Gaming Framework
• Safety by Design for generative AI
  
  

Find out more about Niantic and their Safety by Design approach here.