feat: Add macOS ESF pipeline (ecs_macos_esf)

[eric-nebulock]

Pull Request: Add macOS Endpoint Security Framework (ESF) Pipeline

Summary

This PR adds ecs_macos_esf, a processing pipeline for macOS Endpoint Security Framework (ESF) events that maps Sigma taxonomy fields to ECS (Elastic Common Schema) fields.

Type: New Pipeline
Platform: macOS
Data Source: Endpoint Security Framework (ESF) via eslogger

---

Description

The macOS Endpoint Security Framework (ESF) is Apple's modern security telemetry API that provides real-time notifications for security-relevant events including process execution, file operations, authentication, network connections, and more.

This pipeline enables Sigma rules to query ESF data that has been normalized to ECS format, following the same pattern as existing pipelines like ecs_windows and ecs_zeek_beats.

Data Flow

eslogger -> ESF Collector -> Elasticsearch (ECS fields)
                                   |
Sigma Rule -> ecs_macos_esf -> Lucene Query (ECS fields)

---

Files Changed

File	Change Type	Description
sigma/pipelines/elasticsearch/macos.py	New	Pipeline implementation
sigma/pipelines/elasticsearch/__init__.py	Modified	Added ecs_macos_esf export
tests/test_pipelines_macos.py	New	Unit tests (8 tests)
README.md	Modified	Added pipeline to documentation

---

Field Mappings

Legend:

• Standard = Exists in Windows/Sysmon Sigma taxonomy
• NEW = macOS/Unix-specific, not in standard Sigma taxonomy

---

Process Fields (Sigma to ECS)

Sigma Field	ECS Field	Status	Description
Image	process.executable	Standard	Process executable path
ProcessId	process.pid	Standard	Process ID
ProcessName	process.name	Standard	Process name
CommandLine	process.command_line	Standard	Full command line
CurrentDirectory	process.working_directory	Standard	Working directory
ParentImage	process.parent.executable	Standard	Parent executable
ParentProcessId	process.parent.pid	Standard	Parent PID
ParentProcessName	process.parent.name	Standard	Parent name
ParentCommandLine	process.parent.command_line	Standard	Parent command line

User/Group Fields (Sigma to ECS)

Sigma Field	ECS Field	Status	Description
User	process.user.name	Standard	Username
UserId	process.user.id	NEW	Unix user ID (no Windows equivalent)
EffectiveUserId	process.user.id	NEW	Unix effective UID (euid)
RealUserId	process.real_user.id	NEW	Unix real UID (ruid)
RealUser	process.real_user.name	NEW	Unix real username
GroupId	process.group.id	NEW	Unix group ID
EffectiveGroupId	process.group.id	NEW	Unix effective GID (egid)
RealGroupId	process.real_group.id	NEW	Unix real GID (rgid)
TargetUser	user.target.name	NEW	Target username (setuid)
TargetUserId	user.target.id	NEW	Target user ID (setuid)
TargetGroup	group.target.name	NEW	Target group (setgid)
TargetGroupId	group.target.id	NEW	Target group ID (setgid)

File Fields (Sigma to ECS)

Sigma Field	ECS Field	Status
TargetFilename	file.path	Standard
FileName	file.name	Standard
FileDirectory	file.directory	NEW
SourceFilename	file.source.path	NEW
DestinationFilename	file.target.path	NEW

Network Fields (Sigma to ECS)

Sigma Field	ECS Field	Status
DestinationIp	destination.ip	Standard
DestinationPort	destination.port	Standard
SourceIp	source.ip	Standard
SourcePort	source.port	Standard

Code Signature Fields (Sigma to ECS)

Sigma Field	ECS Field	Status	Description
Signed	process.code_signature.exists	Standard	Is signed (boolean)
SignatureStatus	process.code_signature.status	Standard	Signature status
SigningID	process.code_signature.signing_id	NEW	macOS code signing ID
TeamID	process.code_signature.team_id	NEW	Apple Developer Team ID

Process Injection Fields (Sigma to ECS)

Sigma Field	ECS Field	Status	Description
SourceImage	process.executable	Standard	Injecting process
SourceProcessId	process.pid	Standard	Injecting process PID
TargetImage	target.process.executable	Standard	Target process
TargetProcessId	target.process.pid	Standard	Target process PID
TargetProcessName	target.process.name	NEW	Target process name
TargetProcessGUID	target.process.entity_id	NEW	Target process entity ID

Unix/macOS-Specific Fields (All NEW)

Note: All fields in this section are NEW and have no Windows/Sysmon equivalent.

Sigma Field	ECS Field	Description
SignalNumber	signal.number	Unix signal number (e.g., 9=SIGKILL)
PtraceRequest	ptrace.request	Ptrace request type (e.g., PT_ATTACH)
XpcServiceName	xpc.service_name	macOS XPC service name
KextIdentifier	driver.name	Kext bundle identifier
MemoryProtection	memory.protection	Memory protection flags (mprotect)

---

Summary of New Fields

Total New Fields: 22

Category	New Fields
User/Group	UserId, EffectiveUserId, RealUserId, RealUser, GroupId, EffectiveGroupId, RealGroupId, TargetUser, TargetUserId, TargetGroup, TargetGroupId (11)
File	FileDirectory, SourceFilename, DestinationFilename (3)
Code Signature	SigningID, TeamID (2)
Process Injection	TargetProcessName, TargetProcessGUID (2)
Unix/macOS-Specific	SignalNumber, PtraceRequest, XpcServiceName, KextIdentifier, MemoryProtection (5)

Why These New Fields?

1. Unix User/Group Model: Unix has separate real/effective user/group IDs for privilege management (setuid/setgid), which Windows handles differently with tokens.

2. macOS Code Signing: Apple's code signing uses SigningID and TeamID for developer identification, unique to macOS.

3. Unix Process Control: SignalNumber and PtraceRequest are Unix-specific mechanisms for process control/debugging.

4. macOS IPC: XPC is Apple's inter-process communication framework, unique to macOS/iOS.

5. Kernel Extensions: macOS kernel extensions (kexts) have unique identifiers not found on Windows.

---

Supported Logsource Categories

Category	ESF Event Type	Condition Added
process_creation	9 (exec)	event.action:exec AND esf.event_type:9
file_event	Multiple	event.category:file
file_create	13 (create)	event.action:create AND esf.event_type:13
file_delete	19 (unlink)	event.action:unlink AND esf.event_type:19
file_rename	21 (rename)	event.action:rename AND esf.event_type:21
authentication	111	event.category:authentication AND esf.event_type:111
privilege_escalation	24, 25	event.category:iam
process_injection	64 (ptrace)	event.action:ptrace AND esf.event_type:64
process_access	64 (ptrace)	event.action:ptrace AND esf.event_type:64
process_signal	27 (signal)	event.action:signal AND esf.event_type:27
kernel_extension	17, 18	event.category:driver
driver_load	17, 18	event.category:driver
codesigning	62, 94	event.action:cs_invalidated
security_policy	Various	event.category:malware,configuration
mount	22	event.action:mount AND esf.event_type:22
memory_protection	20	event.action:mprotect AND esf.event_type:20

---

Example Usage

Sigma Rule

title: Suspicious Process Execution
status: stable
logsource:
    product: macos
    service: endpointsecurity
    category: process_creation
detection:
    selection:
        Image|endswith: '/curl'
        CommandLine|contains: 'malicious'
    condition: selection

Generated Lucene Query

(event.action:exec AND esf.event_type:9) AND (process.executable:*\/curl AND process.command_line:*malicious*)

CLI Usage

# Convert a Sigma rule to Lucene query
sigma convert -t lucene -p ecs_macos_esf rule.yml

# Convert with EQL backend
sigma convert -t eql -p ecs_macos_esf rule.yml

# Convert to Kibana NDJSON
sigma convert -t lucene -p ecs_macos_esf -f kibana_ndjson rule.yml

---

Tests

All 8 tests pass:

tests/test_pipelines_macos.py::test_ecs_macos_esf_process_creation PASSED
tests/test_pipelines_macos.py::test_ecs_macos_esf_file_event PASSED
tests/test_pipelines_macos.py::test_ecs_macos_esf_field_mapping PASSED
tests/test_pipelines_macos.py::test_ecs_macos_esf_authentication PASSED
tests/test_pipelines_macos.py::test_ecs_macos_esf_process_injection PASSED
tests/test_pipelines_macos.py::test_ecs_macos_esf_process_signal PASSED
tests/test_pipelines_macos.py::test_ecs_macos_esf_code_signature PASSED
tests/test_pipelines_macos.py::test_ecs_macos_esf_mount PASSED

Run tests with:

poetry run pytest tests/test_pipelines_macos.py -v

---

Checklist

• Pipeline follows existing ecs_windows/ecs_zeek_beats patterns
• Field mappings verified against official ECS specification
• All ECS fields use lowercase dot notation
• ProcessingItems have unique identifiers
• LogsourceCondition used for rule matching
• Unit tests cover all major field mappings and categories
• All tests pass
• README.md updated with new pipeline
• __init__.py exports the pipeline
• Allowed backends specified: elasticsearch, eql, lucene, opensearch
• Priority set to 30 (consistent with other ECS pipelines)

---

Related Resources

• Apple Endpoint Security Framework Documentation
• macOS eslogger man page
• Elastic Common Schema (ECS) Reference
• Elastic Defend for macOS

---

ESF Event Types Reference

Event Type	Name	Description
9	exec	Process execution
11	fork	Process fork
13	create	File creation
17	kextload	Kernel extension load
18	kextunload	Kernel extension unload
19	unlink	File deletion
20	mprotect	Memory protection change
21	rename	File rename
22	mount	Filesystem mount
24	setuid	Set user ID
25	setgid	Set group ID
27	signal	Process signal
64	ptrace	Process trace (debugging/injection)
65	xpc_connect	XPC connection
111	authentication	Authentication event

---

Notes for Reviewers

1. ECS Compliance: All field mappings have been verified against the official ECS specification. Some macOS-specific fields (like signal.number, ptrace.request) are custom extensions that follow ECS naming conventions.

2. Elastic Defend Compatibility: The ECS field names are aligned with Elastic Defend's field naming conventions for seamless integration with Elastic Security.

3. Cross-Platform Compatibility: The Sigma taxonomy fields used (Image, CommandLine, TargetFilename, etc.) are the same as Windows Sysmon, enabling cross-platform rule development.

4. Data Collection: This pipeline expects ESF data to be collected and normalized to ECS format. The ESF collector outputs data with ECS field names that this pipeline targets.

[eric-nebulock]

Closing duplicate - PR #171 already covers this feature. This PR was created in error.