Develop platform safety defaults certification program

[anivar]

AILuminate Issue Draft: Platform Safety Defaults Certification

Title

Develop platform safety defaults certification program

Issue Description

Problem Statement

AILuminate provides excellent safety evaluation capabilities, but real-world platforms often deploy with safety features disabled, undermining the value of safety testing. We need standards for platform safety defaults, not just model safety capabilities.

Evidence: The Platform Deployment Gap

Discovery: Major platforms like Google AI Studio ship with ALL safety features disabled by default:

• Harassment → OFF
• Hate → OFF
• Sexually Explicit Content → OFF
• Dangerous Content → OFF

This demonstrates a fundamental disconnect: we rigorously test safety through AILuminate, but platforms deploy unsafe by default.

Supporting Technical Evidence

Field work on MLCommons inference today revealed additional patterns:

• Configuration errors silently compromise safety guarantees
• Version mismatches create untested safety scenarios
• Metric misunderstandings lead to incorrect safety assumptions
• Documentation gaps result in unsafe deployments

Proposed Solution: "Safe by Default" Certification

Phase 1: Standards Definition

• Define "Safe by Default" requirements for AI platforms
• Create platform certification criteria based on AILuminate benchmarks
• Establish transparency standards for safety threshold disclosure

Phase 2: Certification Framework

• Develop assessment methodology for platform safety defaults
• Create scoring system for platform safety configuration
• Build certification badge/recognition program

Phase 3: Industry Adoption

• Launch public reporting of platform safety defaults
• Create competitive incentives for safety leadership
• Advocate for regulatory alignment with certification standards

Certification Criteria (Draft)

1. Default Safety Settings: Safety features enabled by default
2. Threshold Transparency: Clear disclosure of safety thresholds
3. Override Friction: Appropriate barriers for disabling safety
4. Regular Validation: Ongoing safety benchmark compliance
5. User Education: Clear communication about safety features

Benefits

1. Market Differentiation: Platforms compete on safety, not just performance
2. User Protection: Ensures safety defaults protect users by design
3. Benchmark Value: Makes AILuminate testing meaningful in deployment
4. Industry Standards: Establishes MLCommons as platform accountability leader

Global Considerations

This certification should address:

• Multi-language safety considerations
• Cultural context in safety thresholds
• Regional regulatory requirements
• Diverse deployment contexts

Implementation Strategy

• Start with voluntary certification
• Build industry coalition for adoption
• Create public transparency dashboard
• Develop integration with existing AILuminate infrastructure

Background

This proposal comes from a Safety WG member with 20+ years platform technology experience and direct evidence of deployment-safety gaps from today's field work.

Vision: Transform AILuminate from "test safety" to "ensure safety" - making platforms accountable for their safety defaults.

"Safety isn't a toggle. It's a baseline."

[anivar]

Closing this issue as it was submitted without proper implementation or verification. The platform certification framework referenced does not exist in the current codebase. I'll focus on more concrete contributions to existing AILuminate capabilities.