Fix pipeline conditions (index/source) placement with OR-ed regex by Copilot · Pull Request #66 · SigmaHQ/pySigma-backend-splunk

Copilot · 2026-03-15T01:09:26Z

When a processing pipeline adds conditions via add_condition (e.g. index, source) and the rule contains OR-ed regex expressions, those conditions get placed inside the trailing | search clause instead of at the beginning of the query. This prevents Splunk from using them for efficient initial data retrieval.

Before:

| rex field=CommandLine "(?<CommandLineMatch>suspicious_command)"
| eval CommandLineCondition=if(isnotnull(CommandLineMatch), "true", "false")
| search index="test" source="test" (EventID=4688 CommandLineCondition="true") OR ImageCondition="true"

After:

index="test" source="test"
| rex field=CommandLine "(?<CommandLineMatch>suspicious_command)"
| eval CommandLineCondition=if(isnotnull(CommandLineMatch), "true", "false")
| search (EventID=4688 CommandLineCondition="true") OR ImageCondition="true"

Changes

sigma/backends/splunk/splunk.py: Added get_all_condition_fields() class method to SplunkDeferredORRegularExpression which returns the set of all condition field names created by deferred OR regex expressions. In finish_query(), this set is stored in state.processing_state["deferred_or_condition_fields"] for use by finalize methods. The prefix extraction logic (moving non-deferred leading field=value terms before the rex/eval pipeline commands) is performed in finalize_query_default() rather than finish_query(), so that finalize_query_data_model() continues to receive the expected query format and is not affected. Added _field_eq_val_re class-level pattern for matching leading field=value terms.
tests/test_backend_splunk.py: Added test_splunk_regex_query_explicit_or_with_add_condition covering the exact scenario from the issue.

Original prompt

This section details on the original issue you should resolve

<issue_title>Conditions added by processing pipelines are deferred with OR-ed regex</issue_title>
<issue_description>## Problem

Regular expressions logically linked with OR are deferred

Reproduction

Processing pipeline:

name: Example Sigma Pipeline Config
priority: 100
transformations:
  - id: prefix_source_and_index
    type: add_condition
    conditions:
      index: test
      source: test

Rule:

title: Example Sigma Rule
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    EventID: 4688
    CommandLine|re:
      - "suspicious_command"
  selection2:
    Image|re:
      - "suspicious_command"
  condition: selection or selection2

Result:

| rex field=CommandLine "(?<CommandLineMatch>suspicious_command)"
| eval CommandLineCondition=if(isnotnull(CommandLineMatch), "true", "false")
| rex field=Image "(?<ImageMatch>suspicious_command)"
| eval ImageCondition=if(isnotnull(ImageMatch), "true", "false")
| search index="test" source="test" (EventID=4688 CommandLineCondition="true") OR ImageCondition="true"

Expected result:

index="test" source="test"
| rex field=CommandLine "(?<CommandLineMatch>suspicious_command)"
| eval CommandLineCondition=if(isnotnull(CommandLineMatch), "true", "false")
| rex field=Image "(?<ImageMatch>suspicious_command)"
| eval ImageCondition=if(isnotnull(ImageMatch), "true", "false")
| search (EventID=4688 CommandLineCondition="true") OR ImageCondition="true"
```</issue_description>

## Comments on the Issue (you are @copilot in this section)

<comments>
<comment_new><author>@thomaspatzke</author><body>
Looks good to me, would be great to have it as PR!</body></comment_new>
</comments>

Fixes Conditions added by processing pipelines are deferred with OR-ed regex #51

📍 Connect Copilot coding agent with Jira, Azure Boards or Linear to delegate work to Copilot in one click without leaving your project management tool.

…ions When processing pipelines add conditions (index, source, etc.) and the rule contains OR-ed regex expressions, the conditions were being placed inside the `| search` clause instead of at the beginning of the query. This fix extracts Splunk metadata field conditions (index, source, sourcetype, host) from the query and places them before the deferred rex/eval pipeline commands for efficient data retrieval. Co-authored-by: thomaspatzke <1845601+thomaspatzke@users.noreply.github.com>

thomaspatzke · 2026-03-16T23:30:24Z

@copilot Instead of statically matching on certain condition parts in the #sym:finish_query method, the distinction can already be made while the conversion in #sym:convert_condition_field_eq_val_re and #sym:SplunkDeferredORRegularExpression by distinguishing between query expression parts that refer to fields created by the eval in #sym:SplunkDeferredORRegularExpression and fields that don't depend on this and can be moved to the beginning of the generated query.

…namically Instead of hardcoded index/source/sourcetype/host regex matching, use SplunkDeferredORRegularExpression.get_all_condition_fields() to identify which leading query parts don't depend on deferred eval fields and can be moved before the rex/eval pipeline commands. Co-authored-by: thomaspatzke <1845601+thomaspatzke@users.noreply.github.com>

Co-authored-by: thomaspatzke <1845601+thomaspatzke@users.noreply.github.com>

Copilot · 2026-03-16T23:35:48Z

@copilot Instead of statically matching on certain condition parts in the #sym:finish_query method, the distinction can already be made while the conversion in #sym:convert_condition_field_eq_val_re a...

Refactored in fdd5cd4. Removed the hardcoded _meta_field_re pattern for index|source|sourcetype|host. Now SplunkDeferredORRegularExpression.get_all_condition_fields() returns the set of condition field names created by deferred OR regex expressions. In finish_query, this set is used to identify which leading field=value terms in the query don't depend on any deferred eval field and can be moved before the rex/eval pipeline commands.

thomaspatzke · 2026-03-22T21:33:45Z

@copilot a test fails, please fix.

…conditions-regex

…mpat The prefix extraction in finish_query was breaking finalize_query_data_model because it altered the query format that data model handling expected. Now finish_query keeps the simple format (\n| rex...\n| eval...\n| search <query>) and stores the deferred condition field names on state. finalize_query_default uses those field names to extract non-deferred leading conditions and place them before the deferred pipeline commands. Co-authored-by: thomaspatzke <1845601+thomaspatzke@users.noreply.github.com> Agent-Logs-Url: https://github.com/SigmaHQ/pySigma-backend-splunk/sessions/a4e1c291-7dee-4028-96db-41812064da2a

Copilot · 2026-03-22T21:39:52Z

@copilot a test fails, please fix.

Fixed in 4f3ca56. The prefix extraction in finish_query was breaking finalize_query_data_model by altering the query format it expects. Moved the prefix extraction to finalize_query_default instead, so finish_query preserves the simple \n| rex...\n| eval...\n| search <query> format that both finalize_query_default and finalize_query_data_model can handle. Also merged main to include the data model deferred handling tests. All 124 tests pass.

Initial plan

69ff724

Copilot AI assigned Copilot and thomaspatzke Mar 15, 2026

Copilot started work on behalf of thomaspatzke March 15, 2026 01:09 View session

Copilot AI changed the title ~~[WIP] Fix deferred handling of OR-ed regex in conditions~~ Fix pipeline conditions (index/source) placement with OR-ed regex Mar 15, 2026

Copilot AI requested a review from thomaspatzke March 15, 2026 01:23

Copilot finished work on behalf of thomaspatzke March 15, 2026 01:23

Copilot started work on behalf of thomaspatzke March 16, 2026 23:30 View session

Copilot AI and others added 2 commits March 16, 2026 23:34

Move field_eq_val regex to class-level constant to avoid recompilation

fdd5cd4

Co-authored-by: thomaspatzke <1845601+thomaspatzke@users.noreply.github.com>

Copilot finished work on behalf of thomaspatzke March 16, 2026 23:36

thomaspatzke approved these changes Mar 22, 2026

View reviewed changes

Copilot started work on behalf of thomaspatzke March 22, 2026 21:33 View session

Copilot AI and others added 2 commits March 22, 2026 21:37

Merge remote-tracking branch 'origin/main' into copilot/fix-deferred-…

46fea1f

…conditions-regex

Copilot AI requested a review from thomaspatzke March 22, 2026 21:40

Copilot finished work on behalf of thomaspatzke March 22, 2026 21:40

thomaspatzke marked this pull request as ready for review March 22, 2026 21:53

thomaspatzke merged commit 1e5d1e5 into main Mar 22, 2026
4 checks passed

thomaspatzke mentioned this pull request Mar 22, 2026

Prepend necessary conditions to query #53

Closed

Navigation Menu

Uh oh!

Uh oh!

Fix pipeline conditions (index/source) placement with OR-ed regex - #66

Fix pipeline conditions (index/source) placement with OR-ed regex#66
thomaspatzke merged 6 commits into
mainfrom
copilot/fix-deferred-conditions-regex

Copilot AI commented Mar 15, 2026 •

edited

Loading

Uh oh!

thomaspatzke commented Mar 16, 2026

Uh oh!

Copilot AI commented Mar 16, 2026

Uh oh!

thomaspatzke commented Mar 22, 2026

Uh oh!

Copilot AI commented Mar 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

Copilot AI commented Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Reproduction

Uh oh!

thomaspatzke commented Mar 16, 2026

Uh oh!

Copilot AI commented Mar 16, 2026

Uh oh!

thomaspatzke commented Mar 22, 2026

Uh oh!

Copilot AI commented Mar 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Mar 15, 2026 •

edited

Loading