Fix deferred expressions in correlation rules

[Copilot]

Deferred expressions (| regex, | rex) in correlation rule sub-queries were dropped from output because finalization and deferred handling were conflated. pySigma 1.2.0 split these into finish_query() (deferred, called for all rules) and finalize_query() (output formatting, skipped for sub-rules). The Splunk backend needs to align with this split.

Changes

• Guard | table for correlation rules — finalize_query_default and finalize_query_savedsearches now only append | table for SigmaRule instances, preventing invalid output in correlation queries (Splunk rejects non-streaming ops in multisearch subsearches)
• Fix type signatures — finish_query, finalize_query_default, finalize_query_savedsearches now accept Union[SigmaRule, SigmaCorrelationRule] matching the pySigma base class
• Add correlation + deferred tests — regex deferred, regex-OR deferred (rex), single/multi-rule correlations, and sub-rule fields exclusion

Before

| multisearch
[ search fieldA="value1" fieldB="value2" | eval event_type="base_rule" ]
[ search fieldA="value1" | eval event_type="base_rule2" ]
                        ^ fieldB|re dropped

After

| multisearch
[ search fieldA="value1" fieldB="value2" | eval event_type="base_rule" ]
[ search fieldA="value1"
| regex fieldB="value2" | eval event_type="base_rule2" ]

Original prompt

---

This section details on the original issue you should resolve

<issue_title>deferred expressions not applied in correlation rules</issue_title>
<issue_description>## Description

When a correlation rule has, in one of its "sub rule" or "referenced rule" a deferred expression, like a regex (converted to | regex) or an regex oring, these are removed from the query.

Example :

title: Base rule
name: base_rule
status: test
logsource:
    category: test
detection:
    selection:
        fieldA: value1
        fieldB: value2
    condition: selection
---
title: Base rule 2
name: base_rule2
status: test
logsource:
    category: test
detection:
    selection:
        fieldA: value1
        fieldB|re: value2
    condition: selection
---
title: Multiple occurrences of base event
status: test
correlation:
    type: event_count
    rules:
        - base_rule
        - base_rule2
    group-by:
        - fieldC
        - fieldD
    timespan: 15m
    condition:
        gte: 10

will currently be converted into:

| multisearch
[ search fieldA="value1" fieldB="value2" | eval event_type="base_rule" ]
[ search fieldA="value1" | eval event_type="base_rule2" ]

| bin _time span=15m
| stats count as event_count by _time fieldC fieldD

| search event_count >= 10

We note that fieldB disappeared from the second sub search.

Explanation

This is because of this line commited following this issue. The thing is, we need to get deferred expressions in the search part.
And as deferred are handled in the finalize_query step, in my opinion, we need to reactivate the finalization for rules that are part of a correlation rule, and let the responsibility of the finalization to the backends (ie, let them select which part need to be in the search part of the query, for instance, here, we want deferred expressions to be in the search part, but we dont want this for fields which are converted to | table field1, field2 ...).

When I tried to include fields (so the | table) in the subsearch, splunk fired the error multisearch subsearches might only contain purely streaming operations)

So, in the splunk backend this could be done by handling this in the finalize_query_default:

def finalize_query_default(
        self, rule: SigmaRule, query: str, index: int, state: ConversionState
    ) -> str:
        if not rule._backreferences: # checking if rule is not part of a correlation rule
            table_fields = " | table " + ",".join(rule.fields) if rule.fields else ""
            return query + table_fields
        else :
            return query

I opened this issue as I was working on the #51, as I need to create new deferred expressions which are missing in the correlation rules.

Summary

So, in a nutshell, I propose:

• making a PR on pysigma and change this line by removing the if condition, allowing all rules to be finalized, even those which are part of a correlation rule.
• making a PR on the pysigma splunk backend to update the finalize_query_default, as mentioned previously.</issue_description>

<agent_instructions>Implement a fix for this issue by utilizing the query finalization feature from pySigma.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@thomaspatzke After thinking about this issue I believe that this is a structural issue in pySigma. Currently, generation of deferred query parts and output formats is handled in one finalization step that can be disabled as a whole. This was irrelevant before correlation rules were introduced, but isn't anymore after their introduction, as issues similar to this shown.

My proposal is to split the finalization step into two:

1. Query finalization: for addition of deferred and other query party like Splunk | fields ....
2. Output format finalization: for generating an output format, e.g. embedding the query into another data structure like JSON, savedsearches.conf etc. (which should be done with processing pipelines post-processing nowadays)</comment_new>
   <comment_new>@thomaspatzke
   Just implemented it in pySigma branch finalization-split and a test for correlation rules with deferred query expressions passes. The handling of deferred query parts is now located in finish_query() that is also called for correlation rules while the finalization isn't. Implementing a similar split in the Splun...

• Fixes deferred expressions not applied in correlation rules #52

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.