Skip to content

copilot-pr-nlp-analysis.md

Latest commit

 

History

History
506 lines (388 loc) · 17.5 KB

copilot-pr-nlp-analysis.md

File metadata and controls

506 lines (388 loc) · 17.5 KB
Raw
name Copilot PR Conversation NLP Analysis
description Performs natural language processing analysis on Copilot PR conversations to extract insights and patterns from user interactions
true
schedule workflow_dispatch
cron
daily around 10:00 on weekdays
permissions
contents pull-requests actions issues
read
read
read
read
engine copilot
network
allowed
defaults
python
node
sandbox
agent
awf
imports
uses with
shared/daily-audit-base.md
title-prefix expires
[nlp-analysis]
1d
uses with
shared/repo-memory-standard.md
branch-name description
memory/nlp-analysis
Historical NLP analysis results
shared/copilot-pr-analysis-base.md
shared/python-dataviz.md
shared/python-nlp.md
steps
name env run
Fetch PR comments for detailed analysis
GITHUB_TOKEN GH_TOKEN
${{ secrets.GITHUB_TOKEN }}
${{ secrets.GITHUB_TOKEN }}
# Create comments directory mkdir -p /tmp/gh-aw/pr-comments # Fetch detailed comments for each PR from the pre-fetched data PR_COUNT=$(jq 'length' /tmp/gh-aw/pr-data/copilot-prs.json) echo "Fetching comments for $PR_COUNT PRs..." jq -r '.[].number' /tmp/gh-aw/pr-data/copilot-prs.json | while read -r PR_NUM; do echo "Fetching comments for PR #${PR_NUM}" gh pr view "${PR_NUM}" \ --json comments,reviews,reviewComments \ > "/tmp/gh-aw/pr-comments/pr-${PR_NUM}.json" 2>/dev/null || echo "{}" > "/tmp/gh-aw/pr-comments/pr-${PR_NUM}.json" sleep 0.5 # Rate limiting done echo "Comment data saved to /tmp/gh-aw/pr-comments/"
timeout-minutes 20
features
copilot-requests
true
tools
cli-proxy
true

Copilot PR Conversation NLP Analysis

You are an AI analytics agent specialized in Natural Language Processing (NLP) and conversation analysis. Your mission is to analyze GitHub Copilot pull request conversations to identify trends, sentiment patterns, and recurring topics.

Mission

Generate a daily NLP-based analysis report of Copilot-created PRs merged within the last 24 hours, focusing on conversation patterns, sentiment trends, and topic clustering. Post the findings with visualizations as a GitHub Discussion in the audit category.

Current Context

  • Repository: ${{ github.repository }}
  • Analysis Period: Last 24 hours (merged PRs only)
  • Data Location:
    • PR metadata: /tmp/gh-aw/pr-data/copilot-prs.json
    • PR comments: /tmp/gh-aw/pr-comments/pr-*.json
  • Python Environment: NumPy, Pandas, Matplotlib, Seaborn, SciPy, NLTK, scikit-learn, TextBlob, WordCloud
  • Output Directory: /tmp/gh-aw/python/charts/

Task Overview

Phase 1: Load and Parse PR Conversation Data

Pre-fetched Data Available: The shared component has downloaded all Copilot PRs from the last 30 days. The data is available at:

  • /tmp/gh-aw/pr-data/copilot-prs.json - Full PR data in JSON format
  • /tmp/gh-aw/pr-data/copilot-prs-schema.json - Schema showing the structure

Note: This workflow focuses on merged PRs from the last 24 hours. Use jq to filter:

# Get PRs merged in the last 24 hours
DATE_24H_AGO=$(date -d '1 day ago' '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || date -v-1d '+%Y-%m-%dT%H:%M:%SZ')
jq --arg date "$DATE_24H_AGO" '[.[] | select(.mergedAt != null and .mergedAt >= $date)]' /tmp/gh-aw/pr-data/copilot-prs.json

  1. Load PR metadata:

    cat /tmp/gh-aw/pr-data/copilot-prs.json
echo "Total PRs: $(jq 'length' /tmp/gh-aw/pr-data/copilot-prs.json)"

  2. Parse conversation threads using jq:

    • For each PR in /tmp/gh-aw/pr-comments/pr-*.json, extract:
      • Comments (from comments array)
      • Review comments (from reviewComments array)
      • Reviews (from reviews array)
    • Identify conversation participants (human vs Copilot)
    • Structure as message exchanges

  3. Create structured conversation dataset:

    • Save to /tmp/gh-aw/python/data/conversations.csv with columns:
      • pr_number: PR number
      • pr_title: PR title
      • message_type: "comment", "review", "review_comment"
      • author: Username
      • is_copilot: Boolean
      • text: Message content
      • created_at: Timestamp
      • sentiment_polarity: (to be filled in Phase 2)

Phase 2: Preprocess with jq and Python

  1. Use jq to extract conversation threads:

    # Example: Extract all comment bodies from a PR
jq '.comments[].body' /tmp/gh-aw/pr-comments/pr-123.json

  2. Create Python script (/tmp/gh-aw/python/parse_conversations.py) to:

    • Read all PR comment JSON files
    • Extract and clean text (remove markdown, code blocks, URLs)
    • Combine PR body with conversation threads
    • Identify user ↔ Copilot exchange patterns
    • Save cleaned data to CSV

  3. Text preprocessing:

    • Lowercase conversion
    • Remove special characters and code snippets
    • Tokenization
    • Remove stopwords
    • Lemmatization

Phase 3: NLP Analysis

Create Python analysis script (/tmp/gh-aw/python/nlp_analysis.py) to perform:

3.1 Sentiment Analysis

  • Use TextBlob or NLTK VADER for sentiment scoring
  • Calculate sentiment polarity (-1 to +1) for each message
  • Aggregate sentiment by:
    • PR (overall PR sentiment)
    • Message type (comments vs reviews)
    • Conversation stage (early vs late messages)

3.2 Topic Extraction and Clustering

  • Use TF-IDF vectorization to identify important terms
  • Apply K-means clustering or LDA topic modeling
  • Identify common discussion themes:
    • Code quality feedback
    • Bug reports
    • Feature requests
    • Documentation discussions
    • Testing concerns

3.3 Keyword and Phrase Analysis

  • Extract most frequent n-grams (1-3 words)
  • Identify recurring technical terms
  • Find common feedback patterns
  • Detect sentiment-laden phrases

3.4 Temporal Patterns

  • Analyze sentiment changes over conversation timeline
  • Identify if discussions become more positive/negative over time
  • Detect rapid sentiment shifts (controversy indicators)

Phase 4: Generate Visualizations

Create the following charts in /tmp/gh-aw/python/charts/:

  1. sentiment_distribution.png: Histogram of sentiment polarity scores
  2. topics_wordcloud.png: Word cloud of most common terms (colored by topic cluster)
  3. sentiment_timeline.png: Line chart showing sentiment progression across conversation stages
  4. topic_frequencies.png: Bar chart of identified topic clusters and their frequencies
  5. keyword_trends.png: Top 15 keywords/phrases with occurrence counts

Chart Quality Requirements:

  • DPI: 300 minimum
  • Size: 10x6 inches (or appropriate for data)
  • Style: Use seaborn styling for professional appearance
  • Labels: Clear titles, axis labels, and legends
  • Colors: Colorblind-friendly palette

Phase 5: Upload Visualizations as Assets

For each generated chart:

  1. Verify chart was created:

    find /tmp/gh-aw/python/charts/ -maxdepth 1 -ls

  2. Upload each chart using the upload asset MCP tool (call it directly — do NOT wrap in a shell command or use $() to capture the URL)

  3. Record the returned URL from each upload by writing it to a plain text file in /tmp/gh-aw/agent/ immediately after the MCP tool returns:

    • sentiment_distribution.png → write URL to /tmp/gh-aw/agent/url-sentiment-distribution.txt
    • sentiment_timeline.png → write URL to /tmp/gh-aw/agent/url-sentiment-timeline.txt
    • topic_frequencies.png → write URL to /tmp/gh-aw/agent/url-topic-frequencies.txt
    • topics_wordcloud.png → write URL to /tmp/gh-aw/agent/url-topics-wordcloud.txt
    • keyword_trends.png → write URL to /tmp/gh-aw/agent/url-keyword-trends.txt

    For example, after the upload asset tool returns https://github.com/.../chart.png, write it with:

    echo -n "https://github.com/.../chart.png" > /tmp/gh-aw/agent/url-sentiment-distribution.txt

    Do NOT store URLs in shell variables or use command substitution ($(...)) — this triggers the security harness.

Phase 6: Create Analysis Discussion

Build the discussion body by reading the URL files saved in Phase 5, then post a comprehensive discussion.

Before constructing the body, read the uploaded chart URLs:

SENTIMENT_DIST_URL=$(cat /tmp/gh-aw/agent/url-sentiment-distribution.txt 2>/dev/null || echo "")
SENTIMENT_TIME_URL=$(cat /tmp/gh-aw/agent/url-sentiment-timeline.txt 2>/dev/null || echo "")
TOPIC_FREQ_URL=$(cat /tmp/gh-aw/agent/url-topic-frequencies.txt 2>/dev/null || echo "")
TOPICS_CLOUD_URL=$(cat /tmp/gh-aw/agent/url-topics-wordcloud.txt 2>/dev/null || echo "")
KEYWORD_TRENDS_URL=$(cat /tmp/gh-aw/agent/url-keyword-trends.txt 2>/dev/null || echo "")

Use a Python script to write the fully-substituted discussion body to /tmp/gh-aw/agent/discussion_body.md, inserting the literal URL strings directly (no shell variable expansion in the final body). Then pass the body to the create_discussion safe-output tool.

Post a comprehensive discussion with the following structure:

Title: Copilot PR Conversation NLP Analysis - [DATE]

Content Template (substitute [SENTIMENT_DIST_URL], [SENTIMENT_TIME_URL], [TOPIC_FREQ_URL], [TOPICS_CLOUD_URL], and [KEYWORD_TRENDS_URL] with the literal URL strings read from the files above):

# 🤖 Copilot PR Conversation NLP Analysis - [DATE]

## Executive Summary

**Analysis Period**: Last 24 hours (merged PRs only)  
**Repository**: ${{ github.repository }}  
**Total PRs Analyzed**: [count]  
**Total Messages**: [count] comments, [count] reviews, [count] review comments  
**Average Sentiment**: [polarity score] ([positive/neutral/negative])

## Sentiment Analysis

### Overall Sentiment Distribution
![Sentiment Distribution]([SENTIMENT_DIST_URL])

**Key Findings**:
- **Positive messages**: [count] ([percentage]%)
- **Neutral messages**: [count] ([percentage]%)
- **Negative messages**: [count] ([percentage]%)
- **Average polarity**: [score] on scale of -1 (very negative) to +1 (very positive)

### Sentiment Over Conversation Timeline
![Sentiment Timeline]([SENTIMENT_TIME_URL])

**Observations**:
- [e.g., "Conversations typically start neutral and become more positive as issues are resolved"]
- [e.g., "PR #123 showed unusual negative sentiment spike mid-conversation"]

## Topic Analysis

### Identified Discussion Topics
![Topic Frequencies]([TOPIC_FREQ_URL])

**Major Topics Detected**:
1. **[Topic 1 Name]** ([count] messages, [percentage]%): [brief description]
2. **[Topic 2 Name]** ([count] messages, [percentage]%): [brief description]
3. **[Topic 3 Name]** ([count] messages, [percentage]%): [brief description]
4. **[Topic 4 Name]** ([count] messages, [percentage]%): [brief description]

### Topic Word Cloud
![Topics Word Cloud]([TOPICS_CLOUD_URL])

## Keyword Trends

### Most Common Keywords and Phrases
![Keyword Trends]([KEYWORD_TRENDS_URL])

**Top Recurring Terms**:
- **Technical**: [list top 5 technical terms]
- **Action-oriented**: [list top 5 action verbs/phrases]
- **Feedback**: [list top 5 feedback-related terms]

## Conversation Patterns

### User ↔ Copilot Exchange Analysis

**Typical Exchange Pattern**:
- Average messages per PR: [number]
- Average Copilot responses: [number]
- Average user follow-ups: [number]

**Engagement Metrics**:
- PRs with active discussion (>3 messages): [count]
- PRs merged without discussion: [count]
- Average response time: [if timestamps available]

## Insights and Trends

### 🔍 Key Observations

1. **[Insight 1]**: [e.g., "Code quality feedback is the most common topic, appearing in 78% of conversations"]

2. **[Insight 2]**: [e.g., "Sentiment improves by an average of 0.3 points between initial comment and final approval"]

3. **[Insight 3]**: [e.g., "Testing concerns are mentioned in 45% of PRs but sentiment remains neutral"]

### 📊 Trend Highlights

- **Positive Pattern**: [e.g., "Quick acknowledgment of suggestions correlates with faster merge"]
- **Concerning Pattern**: [e.g., "PRs with >5 review cycles show declining sentiment"]
- **Emerging Theme**: [e.g., "Increased focus on documentation quality this period"]

## Sentiment by Message Type

| Message Type | Avg Sentiment | Count | Percentage |
|--------------|---------------|-------|------------|
| Comments | [score] | [count] | [%] |
| Reviews | [score] | [count] | [%] |
| Review Comments | [score] | [count] | [%] |

## PR Highlights

### Most Positive PR 😊
**PR #[number]**: [title]  
**Sentiment**: [score]  
**Summary**: [brief summary of why positive]

### Most Discussed PR 💬
**PR #[number]**: [title]  
**Messages**: [count]  
**Summary**: [brief summary of discussion]

### Notable Topics PR 🔖
**PR #[number]**: [title]  
**Topics**: [list of topics]  
**Summary**: [brief summary]

## Historical Context

[If cache memory has historical data, compare to previous periods]

| Date | PRs | Avg Sentiment | Top Topic |
|------|-----|---------------|-----------|
| [today] | [count] | [score] | [topic] |
| [previous] | [count] | [score] | [topic] |
| [delta] | [change] | [change] | - |

**7-Day Trend**: [e.g., "Sentiment trending upward, +0.15 increase"]

## Recommendations

Based on NLP analysis:

1. **🎯 Focus Areas**: [e.g., "Continue emphasis on clear documentation - correlates with positive sentiment"]

2. **⚠️ Watch For**: [e.g., "Monitor PRs that generate >7 review comments - may need earlier intervention"]

3. **✨ Best Practices**: [e.g., "Quick initial acknowledgment (within 1 hour) associated with smoother conversations"]

## Methodology

**NLP Techniques Applied**:
- Sentiment Analysis: TextBlob/VADER
- Topic Modeling: TF-IDF + K-means clustering
- Keyword Extraction: N-gram frequency analysis
- Text Preprocessing: Tokenization, stopword removal, lemmatization

**Data Sources**:
- GitHub PR metadata (title, body, labels)
- PR comments and review threads
- Review comments on code lines
- Pull request reviews

**Libraries Used**:
- NLTK: Natural language processing
- scikit-learn: Machine learning and clustering
- TextBlob: Sentiment analysis
- WordCloud: Visualization
- Pandas/NumPy: Data processing
- Matplotlib/Seaborn: Charting

## Workflow Details

- **Repository**: ${{ github.repository }}
- **Run ID**: ${{ github.run_id }}
- **Run URL**: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
- **Analysis Date**: [current date]

---

*This report was automatically generated by the Copilot PR Conversation NLP Analysis workflow.*

Edge Cases and Error Handling

No PRs in Last 24 Hours

If no Copilot PRs were merged in the last 24 hours:

  • Create a minimal discussion noting no activity
  • Include message: "No Copilot-authored PRs were merged in the last 24 hours"
  • Still maintain cache memory with zero counts
  • Optionally show historical trends

PRs with No Comments

If PRs have no conversation data:

  • Analyze only PR title and body
  • Note in report: "X PRs had no discussion comments"
  • Perform sentiment on PR body text
  • Include in "merged without discussion" metric

Insufficient Data for Clustering

If fewer than 5 messages total:

  • Skip topic clustering
  • Perform only basic sentiment analysis
  • Note: "Sample size too small for topic modeling"
  • Focus on keyword extraction instead

Empty or Invalid JSON

Handle parsing errors gracefully:

try:
    data = json.load(file)
except json.JSONDecodeError:
    print(f"Warning: Invalid JSON in {filename}, skipping")
    continue

Success Criteria

A successful analysis workflow:

  • ✅ Fetches only Copilot-authored PRs merged in last 24 hours
  • ✅ Pre-downloads all PR and comment data as JSON
  • ✅ Uses jq for efficient data filtering and preprocessing
  • ✅ Applies multiple NLP techniques (sentiment, topics, keywords)
  • ✅ Generates 5 high-quality visualization charts
  • ✅ Uploads charts as assets with URL-addressable locations
  • ✅ Posts comprehensive discussion in audit category
  • ✅ Handles edge cases (no data, parsing errors) gracefully
  • ✅ Completes within 20-minute timeout
  • ✅ Stores analysis metadata in cache memory for trends

Important Security and Data Guidelines

Data Privacy

  • No sensitive data: Redact usernames if discussing specific examples
  • Aggregate focus: Report trends, not individual message content
  • Public data only: All analyzed data is from public PR conversations

Rate Limiting

  • Sleep 0.5 seconds between PR API calls to avoid rate limits
  • Batch requests where possible
  • Handle API errors with retries

Resource Management

  • Clean up temporary files after analysis
  • Use efficient data structures (pandas DataFrames)
  • Stream large files rather than loading all into memory

Cache Memory Usage

Store reusable components and historical data:

Historical Analysis Data (/tmp/gh-aw/cache-memory/nlp-history.json):

{
  "daily_analysis": [
    {
      "date": "2024-11-04",
      "pr_count": 8,
      "message_count": 45,
      "avg_sentiment": 0.32,
      "top_topic": "code_quality",
      "top_keywords": ["fix", "test", "update", "documentation"]
    }
  ]
}

Reusable NLP Helper Functions (save to cache):

  • Text preprocessing utilities
  • Sentiment analysis wrapper
  • Topic extraction helpers
  • Chart generation templates

Before Analysis: Check cache for helper scripts After Analysis: Save updated history and helpers to cache

Remember: Focus on identifying actionable patterns in Copilot PR conversations that can inform prompt improvements, development practices, and collaboration quality.

{{#import shared/noop-reminder.md}}