Add format auto-detection for zero-config S3 ingestion

[Davidding4718]

Description

Add a format auto-detection library for Intelligent Ingestion that automatically identifies data format and compression from raw S3 object bytes, eliminating the need for customers to specify a codec in their pipeline configuration.

Key features:

• Standalone format detection module (data-prepper-plugins/format-detection/) with minimal dependencies (snappy-java for decompression)
• Detects compression: gzip, snappy, zstd (via magic bytes)
• Detects binary formats: Parquet, Avro, ORC, PDF, images (via magic bytes)
• Detects text formats: JSON arrays, ND-JSON, CSV, TSV, XML, plain text (via heuristics)
• Handles edge cases: UTF-8 BOM, quoted CSV fields, binary garbage detection, truncated JSON arrays
• Gzip and Snappy decompression for format detection on compressed content

S3 source integration:

• When no codec is specified in pipeline config, the S3 source enters auto-detect mode
• AutoDetectS3ObjectWorker reads first 64KB of each object, detects format, and selects the appropriate codec dynamically
• AutoDetectCodecFactory maps detected format to codec plugin (ndjson, json, csv, parquet, avro, newline)
• DetectionMetrics emits CloudWatch-compatible metrics via Micrometer with combined format+compression names and per-type duration timers

Performance: Average detection time is 1.1ms per file (tested with 5,000 files). Detection overhead is < 0.1% of total pipeline processing time.

---

Testing

Unit Tests (57 tests):

• Compression detection: gzip, zstd, snappy magic bytes
• Binary format detection: Parquet, Avro, ORC, PDF, images
• Text format heuristics: JSON arrays, NDJSON, CSV, TSV, XML, plain text
• Edge cases: empty files, binary garbage, UTF-8 BOM, quoted CSV fields, truncated JSON, large first lines
• Gzip and Snappy decompression integration

End-to-End Pipeline Testing (S3 → auto-detect → OpenSearch):

Tested with real S3 bucket using both S3 scan and SQS event-driven modes, with results indexed into an OpenSearch domain.

File Type	Detected Format	Compression	Confidence	Result
.ndjson	NDJSON	NONE	HIGH	✅ Fields parsed individually (timestamp, level, service, etc.)
.ndjson.gz	NDJSON	GZIP	HIGH	✅ Decompressed + parsed
.ndjson.snappy	NDJSON	SNAPPY	HIGH	✅ Decompressed + parsed
.json (array)	JSON	NONE	HIGH	✅ Each array element indexed as document
.csv	CSV	NONE	MEDIUM	✅ Header row used as field names
.csv.gz	CSV	GZIP	MEDIUM	✅ Decompressed + parsed
.csv.snappy	CSV	SNAPPY	MEDIUM	✅ Decompressed + parsed
.tsv	TSV	NONE	MEDIUM	✅ Tab-delimited parsed
.psv (pipe)	CSV	NONE	MEDIUM	✅ Pipe delimiter detected
.parquet	PARQUET	NONE	HIGH	✅ Detected and parsed
.pdf	PDF	NONE	HIGH	✅ Detected, skipped (unsupported codec)
.log (plain text)	TEXT	NONE	LOW	✅ Each line as one event
Binary garbage	UNKNOWN	NONE	LOW	✅ Skipped, not indexed
Empty file	UNKNOWN	NONE	LOW	✅ Skipped

Load Test (5,000 files, ~500MB):

• Detection: 0 failures, average 1.1ms per file, max 3.6ms
• Total detection overhead: 5.6 seconds (< 0.1% of 13 min total pipeline time)
• All 5,000 NDJSON files correctly detected and indexed

Mixed Format Test (3,600 files — NDJSON + CSV + JSON + compressed variants):

• All formats correctly detected and routed to appropriate codecs
• Per-type duration metrics confirm compression adds ~1-18ms overhead depending on algorithm
• CloudWatch metrics published successfully under dataprepper namespace

---

CloudWatch Metrics & Observability

Format detection emits metrics via Micrometer (Prometheus + CloudWatch compatible):

Per format+compression type:

• formatDetection.ndjson / formatDetection.ndjson.gzip / formatDetection.csv.snappy — count + duration per type
• formatDetection.total — aggregate count
• formatDetection.failed — unsupported/unknown files
• formatDetection.confidence.high/medium/low — confidence breakdown
• formatDetection.duration — overall detection timer

Prometheus endpoint sample (after 5,000 file load test):

s3_auto_detect_pipeline_s3_formatDetection_total 5000.0
s3_auto_detect_pipeline_s3_formatDetection_failed_total 0.0
s3_auto_detect_pipeline_s3_formatDetection_ndjson_total 5000.0
s3_auto_detect_pipeline_s3_formatDetection_duration_seconds_sum 5.631845329
s3_auto_detect_pipeline_s3_formatDetection_confidence_high_total 5000.0

Periodic log summary (every 60 seconds during activity, with final summary when ingestion stops):

╔══════════════════════════════════════════════════╗
║  Format Detection Summary                        ║
╠══════════════════════════════════════════════════╣
║  Total objects processed: 3600                   ║
║  Failed detections:       0                      ║
╠══════════════════════════════════════════════════╣
║  Type Distribution:                              ║
║    ndjson               1000  ( 27.8%)            ║
║    csv                  1000  ( 27.8%)            ║
║    json                 1000  ( 27.8%)            ║
║    ndjson.gzip           200  (  5.6%)            ║
║    csv.gzip              200  (  5.6%)            ║
║    ndjson.snappy         100  (  2.8%)            ║
║    csv.snappy            100  (  2.8%)            ║
╠══════════════════════════════════════════════════╣
║  Confidence:                                     ║
║    HIGH         2300  ( 63.9%)                  ║
║    MEDIUM       1300  ( 36.1%)                  ║
╚══════════════════════════════════════════════════╝

---

Issues Resolved

N/A

Check List

• New functionality includes testing.
• New functionality has a documentation issue. Please link to it in this PR.
  • New functionality has javadoc added
• Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ License Header Check Passed

All newly added files have proper license headers. Great work! 🎉