This guide provides comprehensive instructions for deploying Mantissa Log to AWS.
For experienced users who have completed all prerequisites:
# Clone repository
git clone <repository-url>
cd mantissa-log-dev
# Run deployment
bash scripts/deploy.shFollow the interactive prompts to configure and deploy.
Before starting deployment, verify all prerequisites are met:
# Check Terraform
terraform --version
# Expected: Terraform v1.5.0 or higher
# Check AWS CLI
aws --version
# Expected: aws-cli/2.x.x or higher
# Check Python
python3 --version
# Expected: Python 3.11.x or higher
# Verify AWS credentials
aws sts get-caller-identity
# Should return your AWS account detailsIf any checks fail, see Prerequisites.
git clone <repository-url>
cd mantissa-log-devBefore running the deployment, understand the configuration options:
Environment:
dev- Development environment (lower redundancy, cost-optimized)staging- Pre-production environment (medium redundancy)prod- Production environment (high redundancy, multi-AZ)
AWS Region:
- Select based on data residency, cost, and service availability
- Verify Bedrock availability if using AWS Bedrock
- Recommended: us-east-1, us-west-2, eu-west-1
LLM Provider:
bedrock- AWS Bedrock (recommended, no API keys needed)anthropic- Anthropic API (requires API key)openai- OpenAI API (requires API key)
Log Sources:
- VPC Flow Logs - Network traffic analysis
- GuardDuty - Threat detection findings
- CloudTrail - AWS API activity (recommended)
Start the deployment:
bash scripts/deploy.shThe script will guide you through the deployment process.
The deployment script will prompt for configuration:
Configuration:
Environment name (dev/staging/prod) [dev]: prod
AWS Region [us-east-1]: us-east-1
Project prefix [mantissa-log]: mantissa-log
S3 bucket for Terraform state [mantissa-log-terraform-state]:
Enable VPC Flow Logs ingestion? (y/n) [y]: y
Enable GuardDuty integration? (y/n) [y]: y
LLM Provider (bedrock/anthropic/openai) [bedrock]: bedrock
Configuration Summary will be displayed. Review carefully:
Configuration Summary:
Environment: prod
Region: us-east-1
Project Prefix: mantissa-log
State Bucket: mantissa-log-terraform-state
VPC Flow Logs: y
GuardDuty: y
LLM Provider: bedrock
Proceed with deployment? (y/n):
Type y to continue.
The script will now:
- Check prerequisites - Verify tools and AWS access
- Create state bucket - S3 bucket for Terraform state
- Configure backend - Set up Terraform remote state
- Package Lambda functions - Build deployment packages
- Deploy infrastructure - Create AWS resources with Terraform
Terraform Plan Review:
The script will show a Terraform plan. Review the resources to be created:
Terraform will perform the following actions:
# module.storage.aws_s3_bucket.logs_bucket will be created
# module.compute.aws_lambda_function.detection_engine will be created
# module.data.aws_dynamodb_table.state_table will be created
...
Plan: 45 to add, 0 to change, 0 to destroy.
Type y to apply the plan.
Deployment time: 5-10 minutes depending on region and resources.
After infrastructure deployment completes:
CloudTrail Setup:
The script will optionally create a CloudTrail trail:
Checking CloudTrail configuration...
Creating CloudTrail trail: mantissa-log-trail
CloudTrail configured and started
Admin User Creation:
Create an admin user for the web interface:
Setting up Cognito admin user...
Create admin user? (y/n) [y]: y
Admin email address: admin@example.com
Admin password (min 8 chars, upper, lower, number, special):
Admin user created: admin@example.com
Password requirements:
- Minimum 8 characters
- At least one uppercase letter
- At least one lowercase letter
- At least one number
- At least one special character
The deployment script automatically runs smoke tests:
Running smoke tests...
S3 Buckets:
Testing Logs bucket exists... PASS
Testing Logs bucket encryption enabled... PASS
Testing Logs bucket public access blocked... PASS
Testing Athena results bucket exists... PASS
Glue Data Catalog:
Testing Glue database exists... PASS
Found tables: cloudtrail vpc_flow_logs guardduty
Lambda Functions:
Testing Detection engine function exists... PASS
Testing Detection engine has execution role... PASS
Testing LLM query function exists... PASS
Testing Alert router function exists... PASS
DynamoDB Tables:
Testing State table exists... PASS
Testing State table has TTL enabled... PASS
EventBridge Rules:
Testing Detection schedule rule exists... PASS
Testing Detection schedule rule is enabled... PASS
Cognito:
Testing User pool exists... PASS
Testing User pool client exists... PASS
API Gateway:
API Endpoint: https://abc123.execute-api.us-east-1.amazonaws.com/prod
Testing API endpoint is accessible... PASS
Athena:
Testing Athena workgroup exists... PASS
Query Execution Test:
Testing Athena query execution... PASS
==========================================
Smoke Test Summary
==========================================
Tests Passed: 18
Tests Failed: 0
All smoke tests passed!
If any tests fail, see Troubleshooting.
After infrastructure deployment, the script will prompt to deploy the web application:
Web application deployment...
Deploy web application to CloudFront? (y/n) [y]: y
Type y to deploy the web interface.
Web Deployment Process:
The deployment script will:
- Load Terraform outputs - Retrieve S3 bucket and CloudFront details
- Create environment configuration - Generate
.env.productionwith API endpoints - Install dependencies - Run npm install for React application
- Build application - Compile with Vite build tool
- Deploy to S3 - Upload static assets with appropriate cache headers
- Invalidate CloudFront - Clear CDN cache to serve new version
Build output:
Building web application...
Running Vite build...
vite v5.0.8 building for production...
✓ 1250 modules transformed.
dist/index.html 0.45 kB │ gzip: 0.30 kB
dist/assets/index-a1b2c3d4.css 45.21 kB │ gzip: 12.34 kB
dist/assets/index-e5f6g7h8.js 234.56 kB │ gzip: 78.90 kB
Build completed successfully
Deploying to S3...
Syncing files to s3://mantissa-log-prod-web/
Uploading index.html with no-cache policy...
Files deployed to S3
Invalidating CloudFront cache...
Invalidation created: I2ABCDEFGHIJK
Waiting for invalidation to complete (this may take 1-2 minutes)...
CloudFront cache invalidated
==========================================
Web Deployment Complete!
==========================================
Application URL: https://d123abc456xyz.cloudfront.net
Configuration:
API Endpoint: https://abc123.execute-api.us-east-1.amazonaws.com/prod
User Pool ID: us-east-1_ABC123XYZ
Region: us-east-1
Next steps:
1. Open the application URL in your browser
2. Log in with your Cognito credentials
3. Configure LLM settings in Settings > LLM Configuration
4. Set up alert integrations in Settings > Integrations
Note: It may take a few minutes for CloudFront to serve the updated content globally
Deployment time: 2-5 minutes including build and CloudFront invalidation.
Skip web deployment:
To skip web deployment and deploy it later:
Deploy web application to CloudFront? (y/n) [y]: n
Skipping web deployment
You can deploy the web app later by running: ./scripts/deploy-web.sh
The script will display a deployment summary:
==========================================
Deployment Complete!
==========================================
Environment: prod
Region: us-east-1
API Endpoint: https://abc123.execute-api.us-east-1.amazonaws.com/prod
Web Application: https://d123abc456xyz.cloudfront.net
Cognito User Pool ID: us-east-1_ABC123XYZ
Cognito Client ID: 1a2b3c4d5e6f7g8h9i0j
Next steps:
1. Configure alert destinations in AWS Secrets Manager
2. Review and enable detection rules
3. Configure log sources (CloudTrail, VPC Flow Logs, etc.)
4. Access the web interface at: https://d123abc456xyz.cloudfront.net
Save these outputs - you'll need them for configuration.
Open the CloudFront URL in your browser:
https://d123abc456xyz.cloudfront.net
First-time login:
- Enter the admin email address created earlier
- Enter the admin password
- You may be prompted to change your password
- Accept terms and complete profile setup
Web Interface Features:
- Query Interface - Natural language query with SQL preview
- Alerts Dashboard - View recent alerts and their status
- Detection Rules - Browse, edit, and enable/disable rules
- Settings - Configure LLM providers, integrations, and preferences
- Conversations - Multi-turn query sessions with context
Troubleshooting web access:
If the web interface doesn't load:
-
Check CloudFront distribution status:
DIST_ID=$(cat terraform-outputs.json | jq -r '.cloudfront_distribution_id.value') aws cloudfront get-distribution --id $DIST_ID --query 'Distribution.Status'
Expected:
"Deployed" -
Verify S3 bucket has files:
WEB_BUCKET=$(cat terraform-outputs.json | jq -r '.web_bucket_name.value') aws s3 ls s3://$WEB_BUCKET/
Expected:
index.htmlandassets/directory -
Check browser console for errors (F12 Developer Tools)
-
Redeploy web application:
bash scripts/deploy-web.sh
Configure where alerts should be sent:
Slack Integration:
-
Create a Slack webhook URL:
- Go to https://api.slack.com/apps
- Create new app
- Enable Incoming Webhooks
- Copy webhook URL
-
Store in Secrets Manager:
aws secretsmanager create-secret \ --name mantissa-log/alerts/slack \ --secret-string '{ "webhook_url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL", "channel": "#security-alerts", "username": "Mantissa Log" }' \ --region us-east-1
PagerDuty Integration:
-
Get PagerDuty Integration Key:
- Go to PagerDuty Service
- Add Integration
- Select "Events API v2"
- Copy Integration Key
-
Store in Secrets Manager:
aws secretsmanager create-secret \ --name mantissa-log/alerts/pagerduty \ --secret-string '{ "integration_key": "YOUR_INTEGRATION_KEY" }' \ --region us-east-1
Email Integration:
Option 1: SMTP
aws secretsmanager create-secret \
--name mantissa-log/alerts/email \
--secret-string '{
"smtp_host": "smtp.gmail.com",
"smtp_port": 587,
"smtp_username": "your-email@gmail.com",
"smtp_password": "your-app-password",
"from_address": "alerts@example.com",
"to_addresses": ["security-team@example.com"]
}' \
--region us-east-1Option 2: AWS SES
# First verify email addresses in SES
aws ses verify-email-identity --email-address alerts@example.com
aws secretsmanager create-secret \
--name mantissa-log/alerts/email \
--secret-string '{
"use_ses": true,
"from_address": "alerts@example.com",
"to_addresses": ["security-team@example.com"]
}' \
--region us-east-1Detection rules are stored in the rules/ directory and automatically uploaded during deployment.
Review default rules:
ls -la rules/Enable/disable specific rules:
Edit rule files to set enabled: true or enabled: false:
# rules/aws/cloudtrail-root-activity.yaml
name: "Root Account Activity"
enabled: true # Set to false to disable
severity: "critical"
...Upload updated rules:
# Get rules bucket name
RULES_BUCKET=$(cat terraform-outputs.json | jq -r '.rules_bucket.value')
# Upload rules
aws s3 sync rules/ s3://$RULES_BUCKET/rules/ \
--exclude "*.md" \
--exclude "README*"Create custom rules:
- Create new YAML file in
rules/directory - Follow the schema in
rules/schema.json - Upload to S3
- Rules are loaded on next detection cycle (5 minutes)
CloudTrail (Already configured by deployment script):
Verify CloudTrail is logging:
aws cloudtrail get-trail-status --name mantissa-log-trailExpected: "IsLogging": true
VPC Flow Logs:
Enable for VPCs you want to monitor:
# Get logs bucket
LOGS_BUCKET=$(cat terraform-outputs.json | jq -r '.logs_bucket.value')
# Enable VPC Flow Logs
aws ec2 create-flow-logs \
--resource-type VPC \
--resource-ids vpc-12345678 \
--traffic-type ALL \
--log-destination-type s3 \
--log-destination "arn:aws:s3:::$LOGS_BUCKET/vpc-flow-logs/" \
--log-format '${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}'GuardDuty:
Enable GuardDuty findings export:
# Get logs bucket
LOGS_BUCKET=$(cat terraform-outputs.json | jq -r '.logs_bucket.value')
# Get GuardDuty detector ID
DETECTOR_ID=$(aws guardduty list-detectors --query 'DetectorIds[0]' --output text)
# Create publishing destination
aws guardduty create-publishing-destination \
--detector-id $DETECTOR_ID \
--destination-type S3 \
--destination-properties DestinationArn=arn:aws:s3:::$LOGS_BUCKET/guardduty/,KmsKeyArn=arn:aws:kms:us-east-1:123456789012:key/abc-defCustom Log Sources:
For custom applications or services:
- Configure to write logs to S3 bucket in JSON format
- Use prefix:
s3://$LOGS_BUCKET/custom-logs/ - Create Glue table for log schema
- Write detection rules targeting the new table
Test Alert Routing:
Invoke the alert router Lambda directly:
# Get function name
ALERT_ROUTER=$(cat terraform-outputs.json | jq -r '.alert_router_function_name.value')
# Create test alert
cat > test-alert.json <<EOF
{
"alert_id": "test-001",
"title": "Test Alert",
"description": "This is a test alert from deployment verification",
"severity": "low",
"rule_name": "test_rule",
"source": "manual_test",
"timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"metadata": {
"test": true
}
}
EOF
# Invoke Lambda
aws lambda invoke \
--function-name $ALERT_ROUTER \
--payload file://test-alert.json \
--region us-east-1 \
response.json
# Check response
cat response.jsonYou should receive the test alert in configured destinations (Slack, email, etc.).
Test LLM Query:
Query your logs using natural language:
# Get API endpoint
API_ENDPOINT=$(cat terraform-outputs.json | jq -r '.api_endpoint.value')
# Get Cognito user pool details for authentication
USER_POOL_ID=$(cat terraform-outputs.json | jq -r '.user_pool_id.value')
CLIENT_ID=$(cat terraform-outputs.json | jq -r '.user_pool_client_id.value')
# Authenticate (get token)
# This requires setting up proper authentication flow
# See Cognito documentation for details
# Query logs
curl -X POST "$API_ENDPOINT/query" \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"question": "Show me failed login attempts in the last hour",
"execute": true
}'Test Detection Engine:
The detection engine runs automatically every 5 minutes. To trigger manually:
# Get function name
DETECTION_ENGINE=$(cat terraform-outputs.json | jq -r '.detection_engine_function_name.value')
# Invoke manually
aws lambda invoke \
--function-name $DETECTION_ENGINE \
--region us-east-1 \
response.json
# Check logs
aws logs tail /aws/lambda/$DETECTION_ENGINE --followCloudWatch Dashboards:
Access CloudWatch to view metrics:
- Lambda execution counts and errors
- DynamoDB read/write capacity
- S3 bucket sizes
- Athena query execution times
CloudWatch Logs:
View Lambda function logs:
# Detection Engine logs
aws logs tail /aws/lambda/mantissa-log-detection-engine --follow
# LLM Query Handler logs
aws logs tail /aws/lambda/mantissa-log-llm-query --follow
# Alert Router logs
aws logs tail /aws/lambda/mantissa-log-alert-router --followAthena Query History:
Monitor queries in Athena console:
- Query execution times
- Data scanned
- Failed queries
Cost Monitoring:
Set up billing alerts:
# Create SNS topic for billing alerts
aws sns create-topic --name mantissa-log-billing-alerts
# Subscribe email
aws sns subscribe \
--topic-arn arn:aws:sns:us-east-1:123456789012:mantissa-log-billing-alerts \
--protocol email \
--notification-endpoint your-email@example.com
# Create billing alarm (requires us-east-1)
aws cloudwatch put-metric-alarm \
--alarm-name mantissa-log-monthly-cost \
--alarm-description "Alert when estimated monthly cost exceeds threshold" \
--metric-name EstimatedCharges \
--namespace AWS/Billing \
--statistic Maximum \
--period 21600 \
--evaluation-periods 1 \
--threshold 100 \
--comparison-operator GreaterThanThreshold \
--alarm-actions arn:aws:sns:us-east-1:123456789012:mantissa-log-billing-alerts \
--region us-east-1Lambda functions use these environment variables:
Detection Engine:
GLUE_DATABASE- Glue catalog database nameATHENA_WORKGROUP- Athena workgroup for queriesRULES_BUCKET- S3 bucket containing detection rulesSTATE_TABLE- DynamoDB table for detection stateALERT_TOPIC_ARN- SNS topic for alerts
LLM Query Handler:
GLUE_DATABASE- Glue catalog database nameATHENA_WORKGROUP- Athena workgroupLLM_PROVIDER- bedrock, anthropic, or openaiLLM_MODEL- Model name (e.g., claude-3-haiku)SESSION_TABLE- DynamoDB table for sessions
Alert Router:
SECRETS_PREFIX- Prefix for Secrets Manager secretsENRICHMENT_ENABLED- Enable alert enrichment (true/false)
Key variables in terraform.tfvars:
environment = "prod"
aws_region = "us-east-1"
project_prefix = "mantissa-log"
# LLM Configuration
llm_provider = "bedrock"
llm_model = "anthropic.claude-3-haiku-20240307-v1:0"
# Detection Engine
detection_schedule = "rate(5 minutes)"
detection_timeout = 900
# Alert Configuration
alert_enrichment_enabled = true
# Log Retention
cloudwatch_log_retention_days = 30
# Data Retention
s3_lifecycle_days = 90
s3_glacier_days = 365
# Resource Sizing
lambda_memory_size = 512
dynamodb_billing_mode = "PAY_PER_REQUEST"Detection rules use this YAML schema:
name: "Rule Name"
description: "Detailed description"
enabled: true
severity: "critical|high|medium|low|info"
category: "access|network|data|compliance|threat"
query: |
SELECT
field1,
field2
FROM table
WHERE condition
threshold:
count: 1
window: "5m|1h|24h"
metadata:
mitre_attack:
- "TA0001"
references:
- "https://example.com/doc"
tags:
- "aws"
- "cloudtrail"To deploy to multiple regions:
- Run deployment script for each region
- Use different state buckets per region
- Configure cross-region replication for logs (optional)
- Use Route53 for failover between regions
For production deployments:
- Enable DynamoDB global tables (multi-region)
- Configure Lambda reserved concurrency
- Set up CloudWatch alarms for failures
- Enable S3 cross-region replication
- Use multiple alert destinations
To deploy Lambda functions in VPC:
- Edit
infrastructure/aws/terraform/modules/compute/main.tf - Uncomment VPC configuration sections
- Provide subnet IDs and security group IDs
- Ensure NAT Gateway or VPC endpoints exist
- Redeploy with
terraform apply
To use custom domain for API:
- Register domain in Route53
- Create ACM certificate
- Add custom domain to API Gateway
- Create Route53 alias record
- Update Cognito callback URLs
To update an existing deployment:
bash scripts/update.shThis will:
- Check Git status
- Pull latest code
- Check for breaking changes
- Package updated Lambda functions
- Update infrastructure
- Update Lambda function code
- Update detection rules
- Run smoke tests
To redeploy just the web application after making changes:
bash scripts/deploy-web.shThis is useful when:
- Updating UI components or styling
- Fixing bugs in the React application
- Adding new features to the web interface
- Changing environment configuration
The script will:
- Build the latest React application
- Deploy to S3 with optimized caching
- Invalidate CloudFront cache
- Verify deployment
Development workflow:
For active web development:
# Make changes to web/src/
cd web
npm run dev # Start development server
# Test locally at http://localhost:5173
# When ready to deploy:
cd ..
bash scripts/deploy-web.shCache busting:
The deployment script automatically handles cache busting:
- Static assets (JS, CSS, images) get 1-year cache with content hashes
index.htmlgets no-cache policy- CloudFront cache is invalidated after each deployment
The Terraform deployment includes health monitoring infrastructure for all log sources. The following resources are created by the log_source_health module:
| Resource | Purpose |
|---|---|
DynamoDB Table (mantissa-log-source-health) |
Stores health state for all sources |
| Health Check Lambda | Runs every 5 minutes via EventBridge |
| Baseline Lambda | Runs daily to compute volume baselines |
| Health API Lambda | Serves the health monitoring REST API |
| EventBridge Rules (2) | Schedules for health checks and baselines |
| CloudWatch Alarms (4) | Monitors health check Lambda errors and duration |
| API Gateway Routes (7) | REST endpoints under /health/ |
The health monitoring Lambdas use these environment variables (configured automatically by Terraform):
| Variable | Description | Default |
|---|---|---|
LOG_SOURCE_HEALTH_TABLE |
DynamoDB table name for health state | mantissa-log-source-health |
ATHENA_DATABASE |
Athena database for data lake queries | mantissa_logs |
ATHENA_OUTPUT_LOCATION |
S3 path for Athena query results | From Terraform output |
TENANT_ID |
Tenant identifier | default |
HEALTH_CONFIG_S3_BUCKET |
S3 bucket for custom health configs (optional) | None |
HEALTH_CONFIG_S3_KEY |
S3 key for health config JSON | config/health_configs.json |
DEFAULT_DESTINATIONS |
Comma-separated default alert destinations | slack |
SEVERITY_ROUTING |
JSON severity-to-destinations mapping | Built-in defaults |
# Check DynamoDB health table exists
aws dynamodb describe-table \
--table-name mantissa-log-source-health \
--query 'Table.TableStatus'
# Check health check Lambda exists
aws lambda get-function \
--function-name mantissa-log-prod-health-check \
--query 'Configuration.FunctionName'
# Check EventBridge rules
aws events list-rules \
--name-prefix mantissa-log-source-health
# Manually invoke health check
aws lambda invoke \
--function-name mantissa-log-prod-health-check \
--log-type Tail \
/dev/stdout
# Check health via API
API_ENDPOINT=$(cat terraform-outputs.json | jq -r '.api_endpoint.value')
TOKEN=$(cat terraform-outputs.json | jq -r '.admin_token.value')
curl -H "Authorization: Bearer $TOKEN" "$API_ENDPOINT/health/summary"To customize health thresholds per source, create a JSON file and upload to S3:
# Create custom config
cat > health_configs.json << 'EOF'
{
"okta": {
"source_type": "okta",
"expected_max_latency_seconds": 600,
"silence_threshold_seconds": 7200,
"alert_destinations": ["pagerduty", "slack"]
}
}
EOF
# Upload to S3
BUCKET=$(cat terraform-outputs.json | jq -r '.health_config_bucket.value')
aws s3 cp health_configs.json s3://$BUCKET/config/health_configs.jsonSee Log Sources Configuration for all available options.
To completely remove the deployment:
bash scripts/destroy.shWARNING: This will destroy all infrastructure and optionally delete all log data.
The script will:
- Confirm destruction multiple times
- Empty S3 buckets
- Destroy infrastructure with Terraform
- Delete S3 buckets (optional)
- Clean up local files
After successful deployment:
- Review Troubleshooting Guide
- Configure additional log sources
- Customize detection rules
- Set up dashboards and reports
- Train team on using the system
- Establish runbooks for common scenarios
- Schedule regular reviews of alerts and rules