Skip to content

🎯 AudioNews.uk v2.3.0 - AI Deduplication & Story Quality Enhancement

Choose a tag to compare

View all tags
@ajlennon ajlennon released this 15 Oct 15:25
v2.3.0
8a9c992

🔧 Major AI Analysis Improvement

This release addresses the duplicate story issue reported by Molloy, where multiple instances of the same news topic (e.g., China-related stories) were appearing in the daily digest. The AI analysis system has been comprehensively enhanced with intelligent deduplication.

🎯 Issue Reported by Molloy

  • Multiple duplicate stories about the same events (China economy, China trade, China GDP)
  • Redundant content reducing digest quality
  • Need for intelligent story consolidation

✅ Deduplication Solution Implemented

🤖 Enhanced AI Analysis

  • Explicit deduplication instructions: AI now told to select only the BEST/MOST COMPREHENSIVE version of each story
  • Clear examples in prompts: 'If you see China economy and China trade and China GDP - pick the best one'
  • Strict uniqueness rules: 'Each theme should have UNIQUE, DISTINCT stories - no duplicates allowed'

🔍 Programmatic Safety Net

  • Keyword overlap detection: 40% similarity threshold for identifying duplicate stories
  • Automatic filtering: Stories with significant keyword overlap are automatically skipped
  • Debug logging: Shows when duplicates are detected and removed for transparency

📝 Improved Content Synthesis

  • Anti-repetition prompts: 'AVOID repetitive content - synthesize into ONE coherent narrative'
  • Story combination: 'If multiple stories are about the same event, combine them into one summary'
  • Enhanced system messages: Both OpenAI and Anthropic prompts updated for consistency

🛡️ Fallback Protection

  • Enhanced fallback mode: Even if AI fails, keyword-based categorization includes deduplication
  • Stricter thresholds: 50% overlap threshold for fallback mode ensures quality

📊 Before vs After Example

Before (Reported Issue):

  • 'China's economy shows growth'
  • 'China trade figures released'
  • 'China GDP data published'

After (Fixed):

  • 'China's economy shows growth' (most comprehensive version selected)

🔧 Technical Implementation

  • Two-layer deduplication: AI-level + programmatic validation
  • Keyword extraction: Smart filtering based on significant terms (>3 characters)
  • Overlap calculation: Jaccard similarity for accurate duplicate detection
  • Significance scoring: Stories ranked by comprehensiveness and detail

🎧 User Impact

  • ✅ No more duplicate stories in daily digests
  • ✅ Higher quality content with unique, distinct news items
  • ✅ More concise digests without redundant information
  • ✅ Better synthesis combining related events into coherent narratives
  • ✅ Professional news service matching broadcast standards

🚀 Deployment

  • ✅ CI/CD pipeline tested and validated
  • ✅ Both AI providers (OpenAI GPT-4, Anthropic Claude 4.5) updated
  • ✅ Fallback mode enhanced for reliability
  • ✅ Next digest (6 AM UK time) will use improved deduplication

🔗 Live Site: https://audionews.uk
🎧 Service: Daily AI-enhanced news digest at 6 AM UK time
♿ Purpose: Accessible news service for blind and visually impaired users
🤖 AI: Claude 4.5 Sonnet analysis with Microsoft Edge TTS neural voices
🙏 Credit: Issue reported by Molloy - thank you for the quality feedback!

Assets 2
Loading