Prompt benchmark (#34)

* Add prompt evaluation system for keyword extraction benchmarking

- Implement automated prompt evaluation framework with Jaccard similarity scoring
- Add dataset generator: creates ground truth test cases from keywords using LLM
- Add prompt evaluator: tests prompts against dataset and calculates metrics
- Add batch evaluator: compares multiple prompts and generates ranking report
- Create 6 alternative prompt variations for testing (v1-v6)
- Add metrics module with Jaccard similarity calculation and keyword normalization
- Include rate limiting support (configurable delay) for API requests
- Organize outputs: default paths use evaluation_data/ and evaluation_results/ directories
- Update .gitignore to exclude evaluation data and results directories

The system enables systematic benchmarking of keyword extraction prompts, with
prompt_v1_concise showing 4.6x improvement over baseline (0.3465 vs 0.0747 Jaccard score).

* FIXES MADE BY CURSOR - fixed pylint

* Add comprehensive prompt engineering evaluation system with best practices testing

- Add 10 new prompt variants (v7-v16) systematically testing prompt engineering best practices:
  * Individual practices: persona, delimiters, explicit steps, few-shot examples, detailed instructions, format specification
  * Combined practices: persona+examples, steps+delimiters, persona+steps+examples, all best practices
- Expand evaluation dataset from 10 to 28 test cases covering diverse ML/NLP/CV domains
- Enhance metrics module with Precision, Recall, and F1 score calculations
- Update evaluation scripts to calculate and report all metrics (Jaccard, Precision, Recall, F1)
- Update prompts README with documentation of new variants and best practices tested
- Results show 3.8x improvement over baseline with best-performing prompt (prompt_v2_single_words)

* fixed pylint #1

* fixed pylint #2

* fixed pylint #3

* fixed error in OpenAI provider

* fixed the one trailing white space, which caused the pylint to reject this minor change (had 9.99/10 code quality)

* revert changes in openAI provider.py

---------

Co-authored-by: Tobias Geilen <tgeilen@mail.uni-mannheim.de>
Co-authored-by: Moritz <moritzschmidt1@gmail.com>

Author: 3 people

.gitignore

@@ -19,4 +19,10 @@ backend/ingestion/data
 backend/inquiro-env/
 
 # Claude Code
-.claude
+.claude
+
+# Evaluation data and results
+backend/evaluation_data/
+backend/evaluation_results/
+backend/dataset.json
+backend/results.json

backend/app/llm/evaluation/README.md

@@ -0,0 +1,140 @@
+# Prompt Evaluation System
+
+This module provides tools for benchmarking and comparing different keyword extraction prompts.
+
+## Overview
+
+The evaluation system works in two stages:
+
+1. **Dataset Generation**: Create ground truth test cases (keywords → user inputs)
+2. **Prompt Evaluation**: Test prompts against the dataset and calculate Jaccard similarity scores
+
+## Quick Start
+
+### 1. Generate Evaluation Dataset
+
+First, create a keywords file with one keyword set per line (comma-separated):
+
+```bash
+# keywords.txt
+transformer, attention mechanism, BERT
+reinforcement learning, Q-learning, DQN
+convolutional neural network, CNN, image classification
+```
+
+Then generate the dataset:
+
+```bash
+cd backend
+python -m app.llm.evaluation.dataset_generator \
+    --keywords-file keywords.txt \
+    --output evaluation_data/dataset.json
+```
+
+Note: The default output is `evaluation_data/dataset.json` (directories are created automatically).
+
+This will create the dataset file with test cases containing:
+- `ground_truth_keywords`: The original keywords
+- `user_input`: Generated natural language query
+- `metadata`: Generation info
+
+### 2. Evaluate a Prompt
+
+Create a prompt file (e.g., `my_prompt.txt`):
+
+```
+You are an expert in academic information retrieval. Extract 5 short search queries...
+```
+
+Then evaluate it:
+
+```bash
+python -m app.llm.evaluation.prompt_evaluator \
+    --prompt-file my_prompt.txt \
+    --dataset evaluation_data/dataset.json \
+    --output evaluation_results/results.json
+```
+
+Note: The default output is `evaluation_results/results.json` (directories are created automatically).
+
+The results will include:
+- `mean_jaccard`: Average Jaccard similarity score
+- `std_jaccard`: Standard deviation
+- `min_jaccard` / `max_jaccard`: Score range
+- `per_case_scores`: Detailed results for each test case
+
+## Metrics
+
+**Jaccard Similarity**: Measures overlap between extracted and ground truth keywords
+- Formula: `|A ∩ B| / |A ∪ B|`
+- Range: 0.0 (no overlap) to 1.0 (perfect match)
+- Keywords are normalized (lowercase, stripped) before comparison
+
+## File Structure
+
+```
+app/llm/evaluation/
+├── __init__.py
+├── metrics.py              # Jaccard similarity calculation
+├── dataset_generator.py     # Generate test dataset
+├── prompt_evaluator.py      # Evaluate prompts
+└── README.md               # This file
+```
+
+## Example Workflow
+
+```bash
+# 1. Create keywords file
+cat > keywords.txt << EOF
+transformer, attention, BERT
+reinforcement learning, DQN, policy gradient
+CNN, image classification, ResNet
+EOF
+
+# 2. Generate dataset
+python -m app.llm.evaluation.dataset_generator \
+    --keywords-file keywords.txt \
+    --output evaluation/dataset.json \
+    --delay 20.0
+
+# 3. Evaluate a single prompt
+python -m app.llm.evaluation.prompt_evaluator \
+    --prompt-file app/llm/openai/prompts.py \
+    --dataset dataset.json \
+    --output results_baseline.json \
+    --delay 20.0
+
+# 4. Evaluate all prompts and compare (RECOMMENDED)
+python -m app.llm.evaluation.evaluate_all_prompts \
+    --prompts-dir app/llm/evaluation/prompts \
+    --dataset evaluation_data/dataset.json \
+    --output-dir evaluation_results \
+    --delay 20.0 \
+    --include-baseline
+```
+
+## Evaluating Multiple Prompts
+
+The `evaluate_all_prompts.py` script evaluates all prompts in a directory and generates a comparison report:
+
+```bash
+python -m app.llm.evaluation.evaluate_all_prompts \
+    --prompts-dir app/llm/evaluation/prompts \
+    --dataset evaluation_data/dataset.json \
+    --output-dir evaluation_results \
+    --delay 20.0 \
+    --include-baseline
+```
+
+This will:
+- Evaluate all prompts matching the pattern (default: `prompt_*.txt`)
+- Save individual results for each prompt
+- Generate a comparison report ranking prompts by mean Jaccard score
+- Optionally include the baseline prompt from `app/llm/openai/prompts.py`
+
+## Notes
+
+- Requires `OPENAI_API_KEY` to be set in your environment
+- Dataset generation uses the same OpenAI model as production
+- Evaluation runs sequentially (one test case at a time)
+- Results are saved as JSON for programmatic analysis

backend/app/llm/evaluation/__init__.py

@@ -0,0 +1 @@
+"""Evaluation module for prompt benchmarking."""

backend/app/llm/evaluation/common.py

@@ -0,0 +1,99 @@
+"""Common utilities for evaluation scripts."""
+
+import argparse
+import logging
+from typing import Dict, List, Optional
+
+from app.core.config import settings
+from app.llm.openai.provider import OpenAIProvider
+
+logger = logging.getLogger(__name__)
+
+
+def add_delay_argument(parser: argparse.ArgumentParser) -> None:
+    """
+    Add the standard --delay argument to an argument parser.
+
+    Args:
+        parser: ArgumentParser instance to add the argument to
+    """
+    parser.add_argument(
+        "--delay",
+        type=float,
+        default=20.0,
+        help="Delay in seconds between API requests (default: 20.0 for 3 RPM limit)",
+    )
+
+
+def add_dataset_argument(parser: argparse.ArgumentParser) -> None:
+    """
+    Add the standard --dataset argument to an argument parser.
+
+    Args:
+        parser: ArgumentParser instance to add the argument to
+    """
+    parser.add_argument(
+        "--dataset",
+        type=str,
+        required=True,
+        help="Path to dataset JSON file",
+    )
+
+
+def validate_openai_api_key() -> bool:
+    """
+    Validate that OPENAI_API_KEY is set in the environment.
+
+    Returns:
+        True if API key is set, False otherwise
+    """
+    if settings.OPENAI_API_KEY is None:
+        logger.error("OPENAI_API_KEY is not set. Please configure it in your environment.")
+        return False
+    return True
+
+
+async def call_openai_api(
+    provider: OpenAIProvider,
+    developer_prompt: str,
+    user_content: Optional[str] = None,
+    reasoning_effort: str = "low",
+) -> str:
+    """
+    Make a standardized OpenAI API call for evaluation scripts.
+
+    Args:
+        provider: OpenAIProvider instance
+        developer_prompt: Prompt for the developer role
+        user_content: Optional user content (if None, only developer prompt is sent)
+        reasoning_effort: Reasoning effort level (default: "low")
+
+    Returns:
+        Response text from the API
+
+    Raises:
+        Exception: If the API call fails
+    """
+    input_messages: List[Dict[str, str]] = [
+        {
+            "role": "developer",
+            "content": developer_prompt,
+        },
+    ]
+
+    if user_content is not None:
+        input_messages.append(
+            {
+                "role": "user",
+                "content": user_content,
+            }
+        )
+
+    # Accessing _model is necessary for evaluation scripts
+    # pylint: disable=protected-access
+    response = await provider.client.responses.create(
+        model=provider._model,
+        reasoning={"effort": reasoning_effort},
+        input=input_messages,
+    )
+    return response.output_text.strip()