detect/absent: Add error_or, must_error, and must_succeed options

doc/userguide/rules/payload-keywords.rst

@@ -337,6 +337,8 @@ These differences are also discussed in :doc:`differences-from-snort`.
 A discussion of this difference can be found at
 https://redmine.openinfosecfoundation.org/issues/8031
 
+.. _rules-keyword-absent:
+
 absent
 ------
 
@@ -354,7 +356,67 @@ It can take an argument "or_else" to match on absent buffer or on what comes nex
 
 .. container:: example-rule
 
-   alert http any any -> any any (msg:"HTTP request without referer";  :example-rule-emphasis:`http.referer; absent: or_else;` content: !"abc"; sid:1; rev:1;)
+   alert http any any -> any any (msg:"HTTP request without referer";  :example-rule-emphasis:`http.referer; absent: or_else;` \
+       content: !"abc"; sid:1; rev:1;)
+
+It can also take an argument "error_or" to match on transform errors or on subsequent content matches.
+This is useful for detecting when data transformations fail (e.g., invalid base64 encoding) or when the
+decoded data matches a pattern:
+
+.. container:: example-rule
+
+   alert http any any -> any any (msg:"Detect base64 decode error or malicious content"; file.data; \
+       from_base64; :example-rule-emphasis:`absent: error_or;` content:"malicious"; sid:1; rev:1;)
+
+It can also take an argument "must_error" to match only when a transform fails.
+Unlike ``error_or``, there is no fallthrough to subsequent keywords — if the transform succeeds,
+the match fails:
+
+.. container:: example-rule
+
+   alert http any any -> any any (msg:"Detect base64 decode failure"; file.data; \
+       from_base64; :example-rule-emphasis:`absent: must_error;` sid:1; rev:1;)
+
+It can also take an argument "must_succeed" to ensure that subsequent keywords only match
+against successfully-transformed data. If a transform failure is detected, the match is
+rejected — preventing false positives from matching against the original (pre-transform) buffer:
+
+.. container:: example-rule
+
+   alert http any any -> any any (msg:"Detect malware only in decoded base64"; file.data; \
+       from_base64; :example-rule-emphasis:`absent: must_succeed;` content:"malware"; sid:1; rev:1;)
+
+The options differ as follows:
+
+* ``or_else`` matches if the buffer is absent (NULL) OR if subsequent keywords match
+* ``error_or`` matches if a transform operation fails (sets error flag) OR if subsequent keywords match
+* ``must_error`` matches only if a transform operation fails; no other keywords are allowed on the
+  same buffer, but other sticky buffers can follow
+* ``must_succeed`` rejects the match if a transform fails; subsequent keywords only run on
+  successfully-transformed data
+
+When a transform like ``from_base64`` encounters invalid data, it sets an error flag on the inspection
+buffer but leaves the buffer intact to preserve prefilter capability. Without ``must_succeed``, a rule
+inspecting content on the transformed buffer could inadvertently match against the original data.
+The ``absent: must_succeed`` keyword prevents this by failing the match whenever the error flag is set.
+
+.. note:: The ``error_or``, ``must_error``, and ``must_succeed`` options require the buffer to have a
+   transform that can fail. Currently, the transforms that can signal errors are ``from_base64`` and
+   ``pcrexform``. Using these options on a buffer without such a transform will cause a rule loading
+   error.
+
+   Conversely, ``or_else`` cannot be used on a buffer with a transform that can fail.
+   Use ``error_or`` instead to properly handle potential transform errors.
+
+   ``must_error`` cannot be combined with other keywords such as content on the same buffer.
+   However, it can be followed by other sticky buffers to combine error detection with
+   additional matching conditions.
+
+This example alerts when base64 decoding fails on the URI and the host matches::
+
+   alert http any any -> any any (msg:"base64 error and bad host"; \
+       http.uri; from_base64: mode strict; absent: must_error; \
+       http.host; content:"suspicious.example.com"; sid:1;)
 
 For files (i.e ``file.data``), absent means there are no files in the transaction.
 

doc/userguide/rules/transforms.rst

@@ -167,13 +167,43 @@ on.
     alert http any any -> any any (http_request_line; to_sha256; \
         content:"\|54A9 7A8A B09C 1B81 3725 2214 51D3 F997 F015 9DD7 049E E5AD CED3 945A FC79 7401\|"; sid:1;)
 
+.. _transform-error-signaling:
+
+Error-Signaling Transforms
+--------------------------
+
+Some transforms can signal an error when they cannot complete their operation. When an error
+is signaled, the transform sets an error flag on the inspection buffer and the original buffer
+content is preserved unchanged. Content keywords following a failed transform therefore match
+against the original (pre-transform) data, which provides pass-through semantics by default.
+
+The following transforms can signal errors:
+
+- :ref:`pcrexform <pcrexform>` — signals an error when the regular expression does not match
+- :ref:`from_base64 <from_base64>` — signals an error when the data cannot be decoded
+
+The ``absent`` keyword can be used to detect or guard against these failures:
+
+- ``absent: error_or`` — matches if the transform signals an error, OR if subsequent keywords match
+- ``absent: must_error`` — matches only when the transform signals an error; no other content keywords
+  are allowed on the same buffer
+- ``absent: must_succeed`` — rejects the match if the transform signals an error, ensuring content
+  keywords only run against successfully-transformed data
+
+See :ref:`rules-keyword-absent` for full documentation and examples of each mode.
+
+.. _pcrexform:
+
 pcrexform
 ---------
 
 Takes the buffer, applies the required regular expression, and outputs the *first captured expression*.
 
 .. note:: this transform requires a mandatory option string containing a regular expression.
 
+If the regular expression does not match the buffer content, the transform signals an error
+and the original buffer content is preserved. See :ref:`transform-error-signaling` for how to
+detect or guard against transform failures using ``absent``.
 
 This example alerts if ``http.request_line`` contains ``/dropper.php``:
 
@@ -182,6 +212,18 @@ This example alerts if ``http.request_line`` contains ``/dropper.php``:
     alert http any any -> any any (msg:"HTTP with pcrexform"; http.request_line; \
         pcrexform:"[a-zA-Z]+\s+(.*)\s+HTTP"; content:"/dropper.php"; sid:1;)
 
+This example uses pass-through semantics: ``pcrexform`` extracts the top-level domain from a
+subdomain (e.g., ``www.example.com`` → ``example.com``), or the original query is preserved
+when there is no subdomain. Either way, ``content:"example.com"`` matches::
+
+    alert dns any any -> any any (msg:"TLD match"; \
+        dns.query; pcrexform:"\.([^\.]+\.[^\.]+)$"; content:"example.com"; sid:1;)
+
+This example alerts if ``pcrexform`` fails to match (e.g., unexpected request line format)::
+
+    alert http any any -> any any (msg:"pcrexform match failure"; http.request_line; \
+        pcrexform:"[a-zA-Z]+\s+(.*)\s+HTTP"; absent: must_error; sid:2;)
+
 url_decode
 ----------
 
@@ -303,6 +345,11 @@ line breaks, and any non base64 alphabet.
 
 Mode ``strict`` will fail if an invalid character is found in the encoded bytes.
 
+.. note:: When ``from_base64`` encounters data it cannot decode (e.g., in ``strict`` mode
+   or invalid base64 data in ``rfc4648`` mode), it signals an error and the original buffer
+   content is preserved. See :ref:`transform-error-signaling` for how to detect or guard
+   against decode failures using ``absent``.
+
 The following examples will alert when the buffer contents match (see the
 last ``content`` value for the expected strings).
 

rust/Makefile.am

@@ -157,6 +157,7 @@ if HAVE_BINDGEN
 		--allowlist-function 'SC.*' \
 		--allowlist-var 'SC.*' \
 		--allowlist-var 'SIGMATCH.*' \
+		--allowlist-var 'DETECT_CI_FLAGS.*' \
 		--opaque-type 'SCConfNode' \
 		--allowlist-type 'ThreadVars.?' \
 		--opaque-type 'ThreadVars.?' \

rust/src/dcerpc/detect.rs

@@ -426,15 +426,15 @@ unsafe extern "C" fn dcerpc_tx_get_stub_data(
                 tx.stub_data_buffer_tc.len() as u32,
             )
         };
+        SCInspectionBufferSetupAndApplyTransforms(
+            det_ctx, list_id, buffer, data, data_len, transforms,
+        );
+
         if tx.endianness > 0 {
             (*buffer).flags |= DETECT_CI_FLAGS_DCE_LE;
         } else {
             (*buffer).flags |= DETECT_CI_FLAGS_DCE_BE;
         }
-
-        SCInspectionBufferSetupAndApplyTransforms(
-            det_ctx, list_id, buffer, data, data_len, transforms,
-        );
     }
     return buffer;
 }

rust/src/detect/mod.rs

@@ -89,6 +89,7 @@ pub use suricata_sys::sys::{
     SIGMATCH_INFO_MULTI_UINT, SIGMATCH_INFO_STICKY_BUFFER, SIGMATCH_INFO_UINT16,
     SIGMATCH_INFO_UINT32, SIGMATCH_INFO_UINT64, SIGMATCH_INFO_UINT8, SIGMATCH_NOOPT,
     SIGMATCH_OPTIONAL_OPT, SIGMATCH_QUOTES_MANDATORY, SIGMATCH_SUPPORT_FIREWALL,
+    SIGMATCH_TRANSFORM_CAN_FAIL,
 };
 
 #[repr(u8)]

rust/src/detect/transforms/base64.rs

@@ -19,20 +19,22 @@
 
 use crate::detect::error::RuleParseError;
 use crate::detect::parser::{parse_var, take_until_whitespace, ResultValue};
-use crate::detect::SIGMATCH_OPTIONAL_OPT;
+use crate::detect::{SIGMATCH_OPTIONAL_OPT, SIGMATCH_TRANSFORM_CAN_FAIL};
 use crate::ffi::base64::{SCBase64Decode, SCBase64Mode};
 use crate::utils::base64::get_decoded_buffer_size;
 
 #[cfg(test)]
 use crate::detect::transforms::base64::tests::{
-    SCInspectionBufferCheckAndExpand, SCInspectionBufferTruncate,
+    SCInspectionBufferCheckAndExpand, SCInspectionBufferSetError, SCInspectionBufferTruncate,
 };
 use suricata_sys::sys::{
     DetectEngineCtx, DetectEngineThreadCtx, InspectionBuffer, SCDetectHelperTransformRegister,
     SCDetectSignatureAddTransform, SCTransformTableElmt, Signature,
 };
 #[cfg(not(test))]
-use suricata_sys::sys::{SCInspectionBufferCheckAndExpand, SCInspectionBufferTruncate};
+use suricata_sys::sys::{
+    SCInspectionBufferCheckAndExpand, SCInspectionBufferSetError, SCInspectionBufferTruncate,
+};
 
 use nom8::bytes::complete::tag;
 use nom8::character::complete::multispace0;
@@ -273,6 +275,8 @@ unsafe extern "C" fn base64_transform(
     let num_decoded = SCBase64Decode(input.as_ptr(), input.len(), ctx.mode, output);
     if num_decoded > 0 {
         SCInspectionBufferTruncate(buffer, num_decoded);
+    } else {
+        SCInspectionBufferSetError(buffer);
     }
 }
 
@@ -283,7 +287,7 @@ pub unsafe extern "C" fn DetectTransformFromBase64DecodeRegister() {
         desc: b"convert the base64 decode of the buffer\0".as_ptr() as *const libc::c_char,
         url: b"/rules/transforms.html#from-base64\0".as_ptr() as *const libc::c_char,
         Setup: Some(base64_setup),
-        flags: SIGMATCH_OPTIONAL_OPT,
+        flags: SIGMATCH_OPTIONAL_OPT | SIGMATCH_TRANSFORM_CAN_FAIL,
         Transform: Some(base64_transform),
         Free: Some(base64_free),
         TransformValidate: None,
@@ -300,6 +304,7 @@ pub unsafe extern "C" fn DetectTransformFromBase64DecodeRegister() {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use suricata_sys::sys::DETECT_CI_FLAGS_ERROR;
 
     #[test]
     fn test_parser_invalid() {
@@ -412,6 +417,12 @@ mod tests {
         (*buffer).inspect_len = buf_len;
     }
 
+    #[allow(non_snake_case)]
+    pub(crate) unsafe fn SCInspectionBufferSetError(buffer: *mut InspectionBuffer) {
+        (*buffer).flags |= DETECT_CI_FLAGS_ERROR;
+        (*buffer).initialized = true;
+    }
+
     fn test_base64_sample(sig: &str, buf: &[u8], out: &[u8]) {
         let mut ibuf: InspectionBuffer = unsafe { std::mem::zeroed() };
         let mut input = Vec::new();
@@ -427,19 +438,65 @@ mod tests {
                 &mut ctx as *mut DetectTransformFromBase64Data as *mut c_void,
             );
         }
+        assert!(!ibuf.inspect.is_null());
         let ibufi = ibuf.inspect;
         let output = unsafe { build_slice!(ibufi, ibuf.inspect_len as usize) };
         assert_eq!(output, out);
     }
 
+    fn test_base64_error_flag(sig: &str, buf: &[u8]) {
+        let mut ibuf: InspectionBuffer = unsafe { std::mem::zeroed() };
+        let mut input = Vec::new();
+        input.extend_from_slice(buf);
+        ibuf.inspect = input.as_ptr();
+        ibuf.inspect_len = input.len() as u32;
+        ibuf.flags = 0;
+        let (_, mut ctx) = parse_transform_base64(sig).unwrap();
+        unsafe {
+            base64_transform(
+                std::ptr::null_mut(),
+                &mut ibuf as *mut InspectionBuffer,
+                &mut ctx as *mut DetectTransformFromBase64Data as *mut c_void,
+            );
+        }
+        assert_eq!(ibuf.flags & DETECT_CI_FLAGS_ERROR, DETECT_CI_FLAGS_ERROR);
+        assert!(ibuf.initialized);
+        assert!(!ibuf.inspect.is_null());
+        assert!(ibuf.inspect_len > 0);
+        /* original buffer must be preserved on error */
+        let ptr = ibuf.inspect;
+        let out = unsafe { build_slice!(ptr, ibuf.inspect_len as usize) };
+        assert_eq!(out, buf);
+    }
+
+    fn test_base64_no_error_flag(sig: &str, buf: &[u8]) {
+        let mut ibuf: InspectionBuffer = unsafe { std::mem::zeroed() };
+        let mut input = Vec::new();
+        input.extend_from_slice(buf);
+        ibuf.inspect = input.as_ptr();
+        ibuf.inspect_len = input.len() as u32;
+        ibuf.flags = 0;
+        let (_, mut ctx) = parse_transform_base64(sig).unwrap();
+        unsafe {
+            base64_transform(
+                std::ptr::null_mut(),
+                &mut ibuf as *mut InspectionBuffer,
+                &mut ctx as *mut DetectTransformFromBase64Data as *mut c_void,
+            );
+        }
+        assert_eq!(ibuf.flags & DETECT_CI_FLAGS_ERROR, 0);
+        assert!(!ibuf.inspect.is_null());
+        assert!(ibuf.inspect_len > 0);
+    }
+
     #[test]
     fn test_base64_transform() {
         /* Simple success case -- check buffer */
         test_base64_sample("", b"VGhpcyBpcyBTdXJpY2F0YQ==", b"This is Suricata");
         /* Simple success case with RFC2045 -- check buffer */
         test_base64_sample("mode rfc2045", b"Zm 9v Ym Fy", b"foobar");
-        /* Decode failure case -- ensure no change to buffer */
-        test_base64_sample("mode strict", b"This is Suricata\n", b"This is Suricata\n");
+        /* Decode failure case -- should set error flag */
+        test_base64_error_flag("mode strict", b"This is Suricata\n");
         /* bytes > len so --> no transform */
         test_base64_sample(
             "bytes 25",
@@ -462,13 +519,19 @@ mod tests {
             b"SGVs bG8 gV29y bGQ=",
             b"Hello Wor",
         );
-        /* input is not base64 encoded */
-        test_base64_sample(
-            "mode rfc2045",
-            b"This is not base64-encoded",
-            &[
-                78, 24, 172, 138, 201, 232, 181, 182, 172, 123, 174, 30, 157, 202, 29,
-            ],
-        );
+        /* rfc2045 partial decode: valid prefix decoded, invalid suffix skipped, no error */
+        test_base64_sample("mode rfc2045", b"ABCD!!!!", b"\x00\x10\x83");
+        test_base64_no_error_flag("mode rfc2045", b"ABCD!!!!");
+        /* input is not base64 encoded: decode should error */
+        test_base64_error_flag("mode rfc2045", b"!!!!");
+    }
+
+    #[test]
+    fn test_base64_transform_decode_error_sets_flag() {
+        /* Invalid at start: decode fails -> error flag */
+        test_base64_error_flag("mode strict", b"!!!!");
+
+        /* Invalid at start: decode fails -> error flag */
+        test_base64_error_flag("mode rfc4648", b"!!!!");
     }
 }

rust/sys/src/sys.rs

@@ -24,6 +24,7 @@ pub const SIGMATCH_INFO_ENUM_UINT: u32 = 524288;
 pub const SIGMATCH_INFO_BITFLAGS_UINT: u32 = 1048576;
 pub const SIGMATCH_BAN_FIREWALL_RULE: u32 = 2097152;
 pub const SIGMATCH_BAN_FIREWALL_MODE: u32 = 4194304;
+pub const SIGMATCH_TRANSFORM_CAN_FAIL: u32 = 8388608;
 pub type __intmax_t = ::std::os::raw::c_long;
 pub type intmax_t = __intmax_t;
 #[repr(u32)]
@@ -668,6 +669,12 @@ extern "C" {
         s: *mut Signature, transform: ::std::os::raw::c_int, options: *mut ::std::os::raw::c_void,
     ) -> ::std::os::raw::c_int;
 }
+pub const DETECT_CI_FLAGS_START: u8 = 1;
+pub const DETECT_CI_FLAGS_END: u8 = 2;
+#[doc = "< transformation/decode error occurred"]
+pub const DETECT_CI_FLAGS_ERROR: u8 = 16;
+#[doc = " buffer is a single, non-streaming, buffer. Data sent to the content\n  inspection function contains both start and end of the data."]
+pub const DETECT_CI_FLAGS_SINGLE: u8 = 3;
 #[repr(C)]
 #[derive(Debug, Copy, Clone, PartialEq, Eq)]
 pub struct InspectionBuffer {
@@ -723,6 +730,9 @@ extern "C" {
 extern "C" {
     pub fn SCInspectionBufferTruncate(buffer: *mut InspectionBuffer, buf_len: u32);
 }
+extern "C" {
+    pub fn SCInspectionBufferSetError(buffer: *mut InspectionBuffer);
+}
 extern "C" {
     pub fn SCInspectionBufferGet(
         det_ctx: *mut DetectEngineThreadCtx, list_id: ::std::os::raw::c_int,

src/bindgen.h

@@ -42,6 +42,7 @@
 #include "output-eve.h"
 #include "detect-engine-register.h"
 #include "detect-engine-buffer.h"
+#include "detect-engine-inspect-buffer.h"
 #include "detect-engine-helper.h"
 #include "detect-engine-state.h"
 #include "detect-parse.h"

src/detect-engine-analyzer.c

@@ -870,7 +870,7 @@ static void DumpMatches(RuleAnalyzer *ctx, SCJsonBuilder *js, const SigMatchData
             case DETECT_ABSENT: {
                 const DetectAbsentData *dad = (const DetectAbsentData *)smd->ctx;
                 SCJbOpenObject(js, "absent");
-                SCJbSetBool(js, "or_else", dad->or_else);
+                SCJbSetString(js, "mode", DetectAbsentModeStr(dad->mode));
                 SCJbClose(js);
                 break;
             }