IMAP protocol parser, logger and sticky buffers v8 by glongo · Pull Request #15648 · OISF/suricata

glongo · 2026-06-15T15:21:28Z

Changes:

Vec::with_capacity in literal is bounded with IMAP_MAX_BODY_SIZE
Unbounded email header allocation is bounded with IMAP_MAX_HEADERS
Line length bounded with IMAP_MAX_LINE_SIZE
Plain credentials are no longer logged
Added more SV tests (imap-body-too-large, imap-line-too-long, imap-too-many-headers)

Link to ticket: https://redmine.openinfosecfoundation.org/issues/8276

Previous PR: #15617

This introduces a parser for IMAP protocol. An IMAP transaction has two states driven by the 'complete' field: - Open (complete = false): waiting for a matching tagged response. - Complete (complete = true): tagged response received, or special conditions met (BYE, server greeting). Completion logic (is_complete): - If a tagged request exists: requires a matching tagged response. - No tagged request + BYE: complete (server closing). - No tagged request + any response: complete (server greeting). Transactions are created in three places: 1. Request parser: every parsed command creates a new tx. 2. Response parser (tagged, no matching request): midstream/async; orphan tagged response gets its own tx. 3. Response parser (untagged, no incomplete tx): server greeting or unsolicited response. Once open, messages accumulate: - Request side: continuation and literal data attach to the most recent incomplete tx. - Response side: untagged responses attach to the most recent incomplete tx; tagged responses attach via find_request(tag). Email extraction happens at most once per tx: - From literal data in APPEND commands (request side). - From FETCH response data (response side). Seven hard limits prevent unbounded growth: - IMAP_MAX_TX (256, configurable): total transactions. Exceeded: all incomplete txs force-completed with TooManyTransactions event. - IMAP_MAX_MSGS_PER_TX (512): requests/responses per tx. Exceeded: message silently dropped. - IMAP_MAX_LINES (512): request/response lines per tx. Exceeded: line silently dropped. - IMAP_MAX_BODY_SIZE (10MB): email body in ImapParsedEmail. Exceeded: body truncated, BodyTooLarge event. - IMAX_MAX_HEADERS (512): Numbers of headers parsed and stored. Exceeded: TooManyHeaders event. - IMAP_MAX_LINE_SIZE (8KB): request/response line max length. Exceeded: LineTooLong event. - Literal size (u32): bounded by declared size in {N} specifier. IMAP_MAX_TX only is configurable limit via app-layer.protocols.imap.max-tx. Ticket OISF#8276

This introduces a logger for IMAP protocol. Ticket OISF#8276

This implements the following sticky buffers for IMAP protocol: - imap.request - imap.response The following frames have been added: - imap.body - imap.headers - imap.pdu The following email sticky buffers have been updated to work with IMAP: - email.from - email.subject - email.to - email.cc - email.date - email.message_id - email.x_mailer The following email sticky buffers have been added and are supported only for IMAP: - email.command - email.body - email.header - email.header.name - email.header.value Ticket OISF#8276

codecov · 2026-06-15T15:58:16Z

Codecov Report

❌ Patch coverage is 84.86669% with 403 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.94%. Comparing base (28b10fb) to head (a61cf4c).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #15648      +/-   ##
==========================================
+ Coverage   82.90%   82.94%   +0.03%     
==========================================
  Files        1006     1009       +3     
  Lines      273648   276262    +2614     
==========================================
+ Hits       226869   229134    +2265     
- Misses      46779    47128     +349

Flag	Coverage Δ
fuzzcorpus	`60.84% <13.85%> (-0.52%)`	⬇️
livemode	`18.35% <11.96%> (-0.08%)`	⬇️
netns	`22.71% <11.96%> (-0.12%)`	⬇️
pcap	`45.44% <66.08%> (+0.21%)`	⬆️
suricata-verify	`66.81% <79.29%> (+0.09%)`	⬆️
unittests	`58.48% <53.43%> (-0.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

suricata-qa · 2026-06-15T16:55:02Z

Information: QA ran without warnings.

Pipeline = 32039

catenacyber

Thanks for the work

CI/QA : ✅
Git ID set : looks fine for me
CLA : you already contributed
Doc update : nice :-)
Redmine ticket : ok, should https://redmine.openinfosecfoundation.org/issues/3244 be closed as duplicate ?
Rustfmt : 🟡 please add the check in CI see scripts/rustfmt.sh
Tests : 🟡 some unanswered questions on SV PR like OISF/suricata-verify#2908 (comment)
Dependencies added: none
Code : looking
Commits segmentation : ok
Commit messages : cool, some questions below

bounded by declared size in {N} specifier

What is {N} specifier ?

silently dropped

Could we have events for it to be not silent ?

catenacyber · 2026-06-16T14:38:59Z

        header_names: Vec::new(),
        header_values: Vec::new(),
        direction,
+        body_md5,


Is this md5 with truncated data or not ?

With truncated data.

(maybe needs a mention in the docs)

catenacyber · 2026-06-16T14:40:08Z

+}
+
+#[no_mangle]
+pub unsafe extern "C" fn SCImapMimeConfigBodyMd5(val: bool) {


That should be only if IMAP_MIME_BODY_MD5_DISABLED is false (if we did not explicitily disable md5)

This should be for SMTP as well

Those are checked above, line 295-298.

Which file are you talking about ?

For

line 295-298.

I have

} } impl ImapState {

catenacyber · 2026-06-16T14:43:33Z

+This keyword works with both SMTP and IMAP protocols.
+For SMTP, the config option ``app-layer.protocols.smtp.mime.body-md5``
+must be enabled or auto. For IMAP, the config option
+``app-layer.protocols.imap.mime.body-md5`` must be enabled or auto.


Also, could we use email.body; to_md5; ?

catenacyber · 2026-06-16T14:47:50Z

Do we have file extraction ? (or a ticket for it) ?

Should we not log the email field ?

glongo · 2026-06-16T16:59:25Z

Do we have file extraction ? (or a ticket for it) ?

Should we not log the email field ?

Victor and I agreed on keeping email out for now, and work on it after this is merged since we want to refactor the current email code (which is strictly bounded to SMTP).

glongo · 2026-06-16T17:02:11Z

Thanks for the work

CI/QA : ✅ Git ID set : looks fine for me CLA : you already contributed Doc update : nice :-) Redmine ticket : ok, should https://redmine.openinfosecfoundation.org/issues/3244 be closed as duplicate ? Rustfmt : 🟡 please add the check in CI see scripts/rustfmt.sh Tests : 🟡 some unanswered questions on SV PR like OISF/suricata-verify#2908 (comment) Dependencies added: none Code : looking Commits segmentation : ok Commit messages : cool, some questions below

bounded by declared size in {N} specifier

What is {N} specifier ?

silently dropped

Could we have events for it to be not silent ?

{N} is the literal size (email body) and come from IMAP command, see below:

A001 APPEND INBOX {10485813+}

body-too-large event is set if literal size is greater than IMAP_MAX_BODY_SIZE. I missed to update update this part of commit.

catenacyber · 2026-06-16T18:28:15Z

{N} is the literal size (email body) and come from IMAP command, see below:

Ok, this was not clear for someone who does not know IMAP ;-)

catenacyber · 2026-06-16T18:29:02Z

Victor and I agreed on keeping email out for now, and work on it after this is merged since we want to refactor the current email code (which is strictly bounded to SMTP).

Please add a ticket about it and make it blocked by the current IMAP ticket :-) So I will not ask again next time

glongo · 2026-06-17T10:46:46Z

Done: https://redmine.openinfosecfoundation.org/issues/8666

catenacyber · 2026-06-17T13:48:30Z

Done: https://redmine.openinfosecfoundation.org/issues/8666

Not completely :-p

make it blocked by the current IMAP ticket

catenacyber

Needs some changes for the explicitly disabled md5 : see #15648 (comment)

and likely other changes from inline discussions

glongo · 2026-06-25T13:30:41Z

Replaced with #15730

glongo added 4 commits June 15, 2026 15:57

imap: implement logger

2c92bbb

This introduces a logger for IMAP protocol. Ticket OISF#8276

doc: add imap parser and sticky buffers

a61cf4c

glongo requested review from a team, jasonish, jufajardini and victorjulien as code owners June 15, 2026 15:21

glongo changed the title ~~Dev imap proto v8~~ IMAP protocol parser, logger and sticky buffers v8 Jun 15, 2026

glongo mentioned this pull request Jun 15, 2026

IMAP protocol parser, logger and sticky buffers v7 #15617

Closed

catenacyber reviewed Jun 16, 2026

View reviewed changes

Comment thread rust/src/imap/logger.rs

catenacyber reviewed Jun 16, 2026

View reviewed changes

Comment thread src/detect-email.c

catenacyber reviewed Jun 16, 2026

View reviewed changes

Comment thread doc/userguide/rules/email-keywords.rst

catenacyber reviewed Jun 16, 2026

View reviewed changes

catenacyber requested changes Jun 18, 2026

View reviewed changes

glongo mentioned this pull request Jun 25, 2026

IMAP protocol parser, logger and sticky buffers v9 #15730

Open

glongo closed this Jun 25, 2026

Uh oh!

Conversation

glongo commented Jun 15, 2026

Uh oh!

codecov Bot commented Jun 15, 2026

Codecov Report

Uh oh!

suricata-qa commented Jun 15, 2026

Uh oh!

catenacyber left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

catenacyber commented Jun 16, 2026

Uh oh!

glongo commented Jun 16, 2026

Uh oh!

glongo commented Jun 16, 2026

Uh oh!

catenacyber commented Jun 16, 2026

Uh oh!

catenacyber commented Jun 16, 2026

Uh oh!

glongo commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

catenacyber commented Jun 17, 2026

Uh oh!

catenacyber left a comment

Choose a reason for hiding this comment

Uh oh!

glongo commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

glongo commented Jun 17, 2026 •

edited

Loading