MagpieMcp

A web search and web fetch MCP (Model Context Protocol) server built with .NET 10, using SearXNG for web search functionality.

Overview

MagpieMcp is a Model Context Protocol (MCP) server that provides web search and content fetching capabilities through a secure, isolated environment. It uses SearXNG as the search backend, offering privacy-focused web search with configurable settings.

Features

• MCP Server: Implements MCP protocol for AI assistants
• SearXNG Integration: Privacy-focused web search backend
• Web Fetch Tool: Secure content fetching with URL validation and sanitization
• Modular Architecture: Configurable tool pipeline with service composition
• Secure Containerization: Docker-based deployment with security hardening
• Resource Isolation: Network-level isolation between services
• .NET 10: Built with the latest .NET framework

Architecture

┌─────────────┐
│   AI Client │
└──────┬──────┘
       │ MCP Request
       ↓
┌─────────────┐
│  MagpieMcp  │
│   (API)     │
└──────┬──────┘
       │ HTTP
       ↓
┌─────────────┐
│  SearXNG    │
│ (Search)    │
└─────────────┘

Prerequisites

• Docker\Podman (for containerized deployment)
• .NET 10 SDK (for local development)

Installation

Setup

• Change secret_key in searxng/settings.yml.
• Ports and IP addresses can be changes based on your preferences.

Using Docker Compose

1. Clone the repository:

git clone <repository-url>
cd MagpieMcp

2. Start the services:

docker-compose -f docker-compose.yml up -d --build

or

podman-compose -f podman-compose.yml up -d --build

This will start:

• mcp-api: The MCP server on port 5066
• searxng: The search backend (internal only)

Local Development

1. Restore dependencies:

dotnet restore

2. Build the project:

dotnet build

3. Run the API:

dotnet run --project MagpieMcp.Api

4. Run searxng from compose (use docker-compose.override.yml to expose on localhost):

./run_searxng_compose.sh

Configuration

MCP Client Configuration

Configure your MCP client (e.g., Claude Desktop) to connect to MagpieMcp:

{
  "mcpServers": {
    "magpiemcp": {
      "type": "http",
      "url": "http://localhost:5066"
    }
  }
}

Tool Configuration

The server provides two main tools configured in appsettings.json with the following service types:

Supported Service Types:

• OnlyExternalUrlService: Validates external URLs and blocks private/internal hosts
• HttpService: Makes HTTP requests to URLs (supports UrlTemplate and Headers configuration)
• HtmlSanitizationService: Sanitizes HTML content
• HtmlReadabilityExtractionService: Extracts readable content from HTML
• HtmlToMarkdownService: Converts HTML to Markdown format
• SearXngJsonToMarkdownService: Processes SearXNG JSON search results into Markdown
• InjectionSanitizationService: Sanitizes against prompt injection attacks

HttpService Configuration Parameters:

• UrlTemplate: Template for constructing HTTP requests (supports {0} placeholder for input)
• Headers: Dictionary of HTTP headers to include in requests

Note: Other services in the pipeline do not require configuration parameters - they operate with default behavior based on their function.

web_search: Performs web searches using the configured backend

{
  "Name": "web_search",
  "PreProcessSteps": [
    {
      "ServiceType": "HttpService",
      "UrlTemplate": "http://localhost:8080/search?q={0}&format=json"
    }
  ],
  "PostProcessSteps": [
    {
      "ServiceType": "SearXngJsonToMarkdownService"
    },
    {
      "ServiceType": "InjectionSanitizationService"
    }
  ]
}

web_fetch: Fetches web content with security validation and processing

{
  "Name": "web_fetch",
  "PreProcessSteps": [
    {
      "ServiceType": "OnlyExternalUrlService"
    },
    {
      "ServiceType": "HttpService",
      "UrlTemplate": "{0}"
    }
  ],
  "PostProcessSteps": [
    {
      "ServiceType": "HtmlSanitizationService"
    },
    {
      "ServiceType": "HtmlReadabilityExtractionService"
    },
    {
      "ServiceType": "HtmlToMarkdownService"
    },
    {
      "ServiceType": "InjectionSanitizationService"
    }
  ]
}

SearXNG Settings

Edit searxng/settings.yml to customize search behavior:

use_default_settings: true
server:
    secret_key: "your-secret-key"  # Change this for production
search:
  safe_search: 1
  default_lang: "en"
  formats:
    - html
    - json

Security Features

The Docker configuration includes several security measures:

• Read-only filesystem: /app and /etc mounted as read-only
• Non-root user: Runs as non-root user
• Capability dropping: cap_drop: ALL
• No new privileges: security_opt: no-new-privileges:true
• Resource limits: CPU and memory constraints
• Network isolation: Internal network for API, separate network for SearXNG egress

Usage

Once running, you can use the MCP server through your AI assistant:

1. Start the MCP server
2. Query your AI assistant with search or fetch requests
3. The assistant will use MagpieMcp to perform web searches or content fetching

Development

Project Structure

MagpieMcp/
├── MagpieMcp.Api/           # Main API project
├── MagpieMcp.Api.Tests/     # Test project
├── searxng/                 # SearXNG configuration
├── docker-compose.yml       # Docker services
├── Dockerfile               # Build configuration
└── .mcp.json               # MCP client configuration

The modular architecture uses a pipeline of tool services:

• Tool Services: Configurable processing steps (URL validation, HTTP fetching, HTML sanitization, etc.)
• Configuration: Tools defined with pre- and post-processing steps
• Service Composition: Each tool can have multiple processing stages using different service types

Running Tests

dotnet test --project MagpieMcp.Api.Tests

The test suite includes:

• Integration tests for the modular tool services pipeline
• Search integration tests for web search functionality

Running Scripts

Several helper scripts are provided:

• run_config_compose.sh: Check configuration
• run_full_compose.sh: Start full stack
• run_searxng_compose.sh: Start SearXNG only for debugging