|
| 1 | +# Agent Discovery |
| 2 | + |
| 3 | +MeshGuard's discovery module finds and tracks AI service usage across your enterprise, including unauthorized "shadow AI" that bypasses governance controls. It integrates with network proxies, SIEM systems, API gateways, and MeshGuard agents to provide a comprehensive view of which AI services are in use, who is using them, and whether they comply with your policies. |
| 4 | + |
| 5 | +## Architecture |
| 6 | + |
| 7 | +The discovery system has four layers: |
| 8 | + |
| 9 | +1. **Connectors** -- Data source integrations that ingest traffic events (proxy logs, SIEM, API gateways, agent reports). |
| 10 | +2. **Registry** -- A database of known AI services with domain patterns, risk profiles, and compliance flags. |
| 11 | +3. **Scanner** -- The orchestrator that runs discovery scans, matches traffic to known services, and triggers enforcement. |
| 12 | +4. **Enforcement** -- Integration with the MeshGuard policy engine to block, warn, or log discovered AI usage. |
| 13 | + |
| 14 | +``` |
| 15 | +Proxy Logs ─┐ |
| 16 | +SIEM ───────┤ |
| 17 | +API Gateway ┤──→ Connectors ──→ Scanner ──→ Registry Match ──→ Enforcement |
| 18 | +Agents ─────┘ ↓ |
| 19 | + Discovery DB |
| 20 | +``` |
| 21 | + |
| 22 | +## Connectors |
| 23 | + |
| 24 | +MeshGuard ships with four built-in connector types: |
| 25 | + |
| 26 | +### Proxy Log Connector |
| 27 | + |
| 28 | +Parses logs from corporate proxies (Squid, Zscaler, BlueCoat). |
| 29 | + |
| 30 | +```typescript |
| 31 | +{ |
| 32 | + id: 'corp-proxy', |
| 33 | + name: 'Corporate Proxy', |
| 34 | + type: 'proxy-log', |
| 35 | + enabled: true, |
| 36 | + config: { |
| 37 | + logPath: '/var/log/squid/access.log' |
| 38 | + }, |
| 39 | + syncIntervalMinutes: 15 |
| 40 | +} |
| 41 | +``` |
| 42 | + |
| 43 | +### API Gateway Connector |
| 44 | + |
| 45 | +Ingests logs from API gateways (Kong, Apigee, AWS API Gateway). |
| 46 | + |
| 47 | +```typescript |
| 48 | +{ |
| 49 | + id: 'api-gw', |
| 50 | + name: 'API Gateway', |
| 51 | + type: 'api', |
| 52 | + enabled: true, |
| 53 | + config: { |
| 54 | + endpoint: 'https://kong-admin.internal:8001', |
| 55 | + apiKey: 'your-api-key' |
| 56 | + }, |
| 57 | + syncIntervalMinutes: 10 |
| 58 | +} |
| 59 | +``` |
| 60 | + |
| 61 | +### SIEM Connector |
| 62 | + |
| 63 | +Pulls events from SIEM systems. Supports Splunk and Elasticsearch natively. |
| 64 | + |
| 65 | +```typescript |
| 66 | +{ |
| 67 | + id: 'siem-splunk', |
| 68 | + name: 'Splunk SIEM', |
| 69 | + type: 'siem', |
| 70 | + enabled: true, |
| 71 | + config: { |
| 72 | + endpoint: 'https://splunk.internal:8089', |
| 73 | + apiKey: 'your-splunk-token', |
| 74 | + siemType: 'splunk' |
| 75 | + }, |
| 76 | + syncIntervalMinutes: 30 |
| 77 | +} |
| 78 | +``` |
| 79 | + |
| 80 | +**Splunk** -- Searches the `proxy` index for traffic to known AI service domains and extracts source IP, destination host, path, and byte counts. |
| 81 | + |
| 82 | +**Elasticsearch** -- Queries `proxy-*` indices with wildcard domain matches and returns structured traffic events. |
| 83 | + |
| 84 | +### Agent Report Connector |
| 85 | + |
| 86 | +Receives real-time reports from MeshGuard agents about AI service access. This connector requires no external configuration -- agents report directly. |
| 87 | + |
| 88 | +```typescript |
| 89 | +{ |
| 90 | + id: 'agent-reports', |
| 91 | + name: 'Agent Reports', |
| 92 | + type: 'agent', |
| 93 | + enabled: true, |
| 94 | + config: {}, |
| 95 | + syncIntervalMinutes: 5 |
| 96 | +} |
| 97 | +``` |
| 98 | + |
| 99 | +## Known AI Service Registry |
| 100 | + |
| 101 | +MeshGuard includes a built-in registry of 25+ known AI services across nine categories: |
| 102 | + |
| 103 | +| Category | Services | |
| 104 | +|----------|----------| |
| 105 | +| `llm` | ChatGPT, OpenAI API, Claude, Anthropic API, Google Gemini, Mistral AI, Cohere | |
| 106 | +| `code-assistant` | GitHub Copilot, Cursor, Tabnine, Codeium, Amazon CodeWhisperer, Sourcegraph Cody | |
| 107 | +| `image-gen` | DALL-E, Midjourney, Stability AI, Leonardo.AI | |
| 108 | +| `voice` | ElevenLabs, Murf AI, AssemblyAI | |
| 109 | +| `automation` | Zapier AI, Make AI, n8n AI | |
| 110 | +| `analytics` | Databricks AI, Snowflake Cortex | |
| 111 | +| `search` | Perplexity | |
| 112 | + |
| 113 | +Each service entry includes: |
| 114 | + |
| 115 | +- **Domain patterns** -- Exact and wildcard matches (e.g., `api.openai.com`, `*.cloud.databricks.com`) |
| 116 | +- **Path patterns** -- Endpoint-specific matching (e.g., `/v1/chat/completions`) |
| 117 | +- **Header signatures** -- API key format detection (e.g., `sk-*`, `sk-ant-*`) |
| 118 | +- **Risk assessment** -- Default risk level, data exfiltration risk, code execution risk |
| 119 | +- **Compliance flags** -- Relevant standards (SOC2, HIPAA, GDPR, FedRAMP, ISO27001) |
| 120 | + |
| 121 | +### Adding Custom Services |
| 122 | + |
| 123 | +Register additional AI services specific to your environment: |
| 124 | + |
| 125 | +```typescript |
| 126 | +import { addCustomService } from './discovery'; |
| 127 | + |
| 128 | +addCustomService({ |
| 129 | + id: 'internal-llm', |
| 130 | + name: 'Internal LLM', |
| 131 | + vendor: 'YourCompany', |
| 132 | + category: 'llm', |
| 133 | + domains: ['llm.internal.company.com'], |
| 134 | + pathPatterns: ['/v1/generate', '/v1/chat'], |
| 135 | + defaultRisk: 'low', |
| 136 | + dataExfiltrationRisk: false, |
| 137 | + codeExecutionRisk: false, |
| 138 | + complianceFlags: ['SOC2'], |
| 139 | + description: 'Self-hosted internal LLM service', |
| 140 | +}); |
| 141 | +``` |
| 142 | + |
| 143 | +## Shadow AI Detection |
| 144 | + |
| 145 | +Shadow AI refers to unauthorized use of AI services that bypasses your governance controls. MeshGuard detects shadow AI by: |
| 146 | + |
| 147 | +1. **Matching network traffic** -- Connectors capture outbound traffic and the scanner matches destination hosts against the registry. |
| 148 | +2. **Classifying risk** -- Each discovery is assigned a risk level based on the matched service's profile. |
| 149 | +3. **Tracking status** -- Discoveries move through a lifecycle: `discovered` > `monitoring` / `sanctioned` / `blocked` / `retired`. |
| 150 | + |
| 151 | +### Discovery Statuses |
| 152 | + |
| 153 | +| Status | Description | |
| 154 | +|--------|-------------| |
| 155 | +| `discovered` | Newly found, not yet reviewed | |
| 156 | +| `sanctioned` | Reviewed and approved for use | |
| 157 | +| `blocked` | Explicitly blocked by policy | |
| 158 | +| `monitoring` | Under observation, not yet decided | |
| 159 | +| `retired` | Previously active, now inactive | |
| 160 | + |
| 161 | +### Risk Levels |
| 162 | + |
| 163 | +| Risk | Description | |
| 164 | +|------|-------------| |
| 165 | +| `critical` | Immediate action required. High data exfiltration and compliance risk. | |
| 166 | +| `high` | Significant risk. Code execution possible, sensitive data exposure likely. | |
| 167 | +| `medium` | Moderate risk. Data exfiltration possible but service has compliance certifications. | |
| 168 | +| `low` | Minimal risk. Limited data exposure, well-known vendor. | |
| 169 | +| `unknown` | Unclassified. Needs manual review. | |
| 170 | + |
| 171 | +## Running Scans |
| 172 | + |
| 173 | +### On-Demand Scan |
| 174 | + |
| 175 | +```typescript |
| 176 | +import { runScan } from './discovery'; |
| 177 | + |
| 178 | +const result = await runScan({ |
| 179 | + connectorIds: ['siem-splunk', 'corp-proxy'], |
| 180 | + timeRange: { |
| 181 | + from: new Date('2026-04-21'), |
| 182 | + to: new Date('2026-04-22'), |
| 183 | + }, |
| 184 | + orgId: 'org_abc', |
| 185 | + fullScan: false, // Incremental scan |
| 186 | +}); |
| 187 | +``` |
| 188 | + |
| 189 | +**Scan result:** |
| 190 | + |
| 191 | +```json |
| 192 | +{ |
| 193 | + "id": "scan_abc123", |
| 194 | + "startedAt": "2026-04-22T10:00:00.000Z", |
| 195 | + "completedAt": "2026-04-22T10:00:45.000Z", |
| 196 | + "status": "success", |
| 197 | + "newDiscoveries": 3, |
| 198 | + "updatedDiscoveries": 12, |
| 199 | + "totalServicesFound": 8, |
| 200 | + "criticalFindings": 0, |
| 201 | + "highRiskFindings": 2, |
| 202 | + "connectorResults": [ |
| 203 | + { "connectorId": "siem-splunk", "status": "success", "recordsProcessed": 4521 }, |
| 204 | + { "connectorId": "corp-proxy", "status": "success", "recordsProcessed": 1203 } |
| 205 | + ] |
| 206 | +} |
| 207 | +``` |
| 208 | + |
| 209 | +### Periodic Scans |
| 210 | + |
| 211 | +Enable automatic periodic scanning: |
| 212 | + |
| 213 | +```typescript |
| 214 | +import { startPeriodicScans, stopPeriodicScans } from './discovery'; |
| 215 | + |
| 216 | +// Scan every 60 minutes |
| 217 | +startPeriodicScans(60); |
| 218 | + |
| 219 | +// Stop periodic scanning |
| 220 | +stopPeriodicScans(); |
| 221 | +``` |
| 222 | + |
| 223 | +### Real-Time Agent Events |
| 224 | + |
| 225 | +Agents can report AI service access in real time without waiting for the next scan cycle: |
| 226 | + |
| 227 | +```typescript |
| 228 | +import { ingestAgentEvent } from './discovery'; |
| 229 | + |
| 230 | +const discovery = ingestAgentEvent({ |
| 231 | + agentId: 'agent_123', |
| 232 | + destinationHost: 'api.openai.com', |
| 233 | + destinationPath: '/v1/chat/completions', |
| 234 | + bytesSent: 4096, |
| 235 | + orgId: 'org_abc', |
| 236 | +}); |
| 237 | + |
| 238 | +if (discovery) { |
| 239 | + console.log(`Detected: ${discovery.serviceName} (${discovery.risk} risk)`); |
| 240 | +} |
| 241 | +``` |
| 242 | + |
| 243 | +## Policy Enforcement |
| 244 | + |
| 245 | +When a new AI service is discovered through an agent, MeshGuard automatically evaluates it against the agent's policy: |
| 246 | + |
| 247 | +- Actions are formatted as `ai:{category}:{service-id}` (e.g., `ai:llm:openai-api`) |
| 248 | +- The policy engine returns `allow`, `deny`, or a default |
| 249 | +- The enforcement result is recorded as an `EnforcementEvent` and logged in the audit trail |
| 250 | + |
| 251 | +Enforcement actions: |
| 252 | + |
| 253 | +| Action | When Applied | |
| 254 | +|--------|-------------| |
| 255 | +| `block` | Policy explicitly denies the AI service | |
| 256 | +| `allow` | Policy allows and the service is sanctioned | |
| 257 | +| `warn` | Policy allows but the service is not yet sanctioned | |
| 258 | +| `log` | Default action for unmatched policies | |
| 259 | + |
| 260 | +::: tip |
| 261 | +Create policy rules using the `ai:*` action prefix to control AI service access. For example, `ai:code-assistant:*` matches all code assistant services. |
| 262 | +::: |
| 263 | + |
| 264 | +## Sanctioning and Blocking |
| 265 | + |
| 266 | +### Sanction a Service |
| 267 | + |
| 268 | +Mark a discovered service as approved: |
| 269 | + |
| 270 | +```typescript |
| 271 | +import { sanctionService } from './discovery'; |
| 272 | + |
| 273 | +sanctionService('disc_abc123', 'admin@company.com', 'Approved for engineering team'); |
| 274 | +``` |
| 275 | + |
| 276 | +### Block a Service |
| 277 | + |
| 278 | +Explicitly block a discovered service: |
| 279 | + |
| 280 | +```typescript |
| 281 | +import { blockService } from './discovery'; |
| 282 | + |
| 283 | +blockService('disc_xyz789', 'admin@company.com', 'Not compliant with HIPAA requirements'); |
| 284 | +``` |
| 285 | + |
| 286 | +## Summary Dashboard |
| 287 | + |
| 288 | +Get a quick overview of your organization's shadow AI posture: |
| 289 | + |
| 290 | +```typescript |
| 291 | +import { getShadowAISummary } from './discovery'; |
| 292 | + |
| 293 | +const summary = getShadowAISummary('org_abc'); |
| 294 | +``` |
| 295 | + |
| 296 | +The summary includes: |
| 297 | + |
| 298 | +- **Discovery statistics** -- Total discoveries, active/sanctioned/blocked counts, by category and risk |
| 299 | +- **Recent discoveries** -- Last 10 discovered services |
| 300 | +- **Unreviewed count** -- How many discoveries are still in `discovered` status |
| 301 | +- **High-risk count** -- Count of critical and high-risk discoveries that are not yet blocked |
| 302 | + |
| 303 | +## Initialization |
| 304 | + |
| 305 | +Initialize the discovery module at application startup: |
| 306 | + |
| 307 | +```typescript |
| 308 | +import { initDiscovery } from './discovery'; |
| 309 | + |
| 310 | +initDiscovery({ |
| 311 | + dbPath: '/var/data/meshguard/discovery.db', |
| 312 | + enablePeriodicScans: true, |
| 313 | + scanIntervalMinutes: 60, |
| 314 | +}); |
| 315 | +``` |
| 316 | + |
| 317 | +## Best Practices |
| 318 | + |
| 319 | +1. **Start with SIEM and proxy connectors** -- These provide the broadest visibility into AI service usage across the enterprise. |
| 320 | +2. **Enable agent reporting** -- Real-time agent events catch AI usage that batch scans might miss. |
| 321 | +3. **Review discoveries promptly** -- Unreviewed discoveries are a blind spot. Assign someone to triage new findings weekly. |
| 322 | +4. **Sanction approved services explicitly** -- A sanctioned service will not trigger warnings in enforcement. This reduces noise. |
| 323 | +5. **Create AI-specific policies** -- Use `ai:*` action patterns in your MeshGuard policies to control which agents can access which AI services. |
| 324 | +6. **Monitor high-risk services** -- Set up alerts for discoveries with `critical` or `high` risk levels. |
| 325 | +7. **Add custom services** -- Register internal AI services and industry-specific tools that are not in the built-in registry. |
0 commit comments