All about Email

[bschmalhofer]

Email-Handling in OTOBO is importants and complex. Improvements in that area migh have large benefits. So this discussion has three goals.

1. Collect the way Email are currently handled in OTOBO
2. Collect ideas about potential improvements

30_000 ft view

There is a plethora of email related modules on CPAN. Many modules are clearly obsolete or no longer maintained. The maintained modules can be classified into different groups:

1. The classic Mail Tools, https://metacpan.org/dist/MailTools
2. The Overmail universe, https://metacpan.org/search?q=author%3AMARKOV&q=Mail
3. The Email::* universe https://metacpan.org/search?q=author%3ARJBS&q=Email
4. Others, which may simply be alternatives or actually the standard module for their task

Use cases in OTOBO

Receiving mails

OTOBO fetches mails either with POP3 or IMAP. Both protocols have variants which encrypt the data transfer.

• IMAP: Mail::IMAPClient is consistently used, Eliminate usage Net::IMAP::Simple, only use Mail::IMAPClient for IMAP MailAccount #4174
• POP3: Net::POP3 is consistently used.

These choices look sensible. I looked at Mail::Transport::IMAP4 and it turns out that it is based on Mail::IMAPClient.

Parsing mails

This is done in Kernel::System::EmailParser. Mail::Internet and M̀IME::Tools are used internally. Using both modules is kind of redundant, as there is no good reason for parsing the Email twice. Even though Mail::Internet only extracts the headers and does not parse MIME.

Alternatives are Mail::Message and Email::MIME

This is addressed in #4469 . The approach is to stick with Main::Internet but avoid the double parsing.

Parsing Lists of Email Addresses

This is a common use case in OTOBO as MIME article data keeps the complete header line in the database. Currently Mail::Address is used for that. Grepping grep -r 'Mail::Address->parse(' | wc -l reports 72 lines.

However this module is not very performant. Email::Address or Email::Address::XS might be alternatives. See the benchmark
benchmark_address_parsing.pl.txt. In this benchmark caching is annulled.

                             Rate Mail::Address Mail::Message::Field::Fast Sisimai::Address Email::Address Email::Address::XS
Mail::Address              2.89/s            --                        -3%             -86%           -89%               -99%
Mail::Message::Field::Fast 2.99/s            3%                         --             -86%           -89%               -99%
Sisimai::Address           21.3/s          635%                       611%               --           -19%               -93%
Email::Address             26.2/s          807%                       778%              23%             --               -92%
Email::Address::XS          326/s        11188%                     10824%            1435%          1144%                 --

The fastest module is Email::Address::XS. The C library used in Email::Address::XS is based on Dovecot which I interpret as that it is battle tested.

The POD of Email::Address recommends using Email::Address::XS. If Email::Address::XS is used, then at least version 1.05 from 2022 should be required. See pali/Email-Address-XS#5 .

Here is the availability of the Email::Address::XS as reported by https://pkgs.org/search/?q=email-address-xs&on=name :

See also #5483 .

Distro	Email::Address::XS
Enterprise Linux 10 (RHEL 10, Rocky Linux 10, AlmaLinux 10, CentOS 10 Stream)	1.05
Enterprise Linux 9 (RHEL 9, Rocky Linux 9, AlmaLinux 9, CentOS 9 Stream)]	1.04
Enterprise Linux 8 (RHEL 8, Rocky Linux 8, AlmaLinux 8)	1.04
Debian 13 (Trixie)	1.05
Debian 12 (Bookworm)	1.05
Debian 11 (Bullseye)	1.04
Ubuntu 26.04 LTS (Resolute Raccoon)	1.05
Ubuntu 25.10 (Questing Quokka)	1.05
Ubuntu 24.04 LTS (Noble Numbat)	1.05
Ubuntu 22.04 LTS (Jammy Jellyfish)	1.04

Checking Email Addresses

This is done in Kernel::System::CheckItem::CheckEmail(). Internally some regexes and the CPAN modules Email::Valid, Net::DNS, and Net::DNS::Resolver are used.
The list of used modules looks sensible. Emai::Valid belongs to the Email::* universe. It could also do the mx check, that is the DNS check, using Net::DNS.

Email::Valid has some quirks, see Perl-Email-Project/Email-Valid#16 and Perl-Email-Project/Email-Valid#36 . But that seems to be considered in OTOBO, where CheckEmail() is usually called only on plain email addresses.

See also #637 and #5498 .

Constructing the From-line of emails to be sent

This depends on the queue, settings in the SysConfig, and the name of the agent.
See Kernel::System::TemplateGenerator::Sender. Mail::Address is used there.

Constructing mail

This is done in Kernel::System::Email::Send(). M̀IME::Entity`is used internally.

Sending mails

Besides the test backends there are two ways of actually sending Emails from OTOBO.

1. Piping the mail to a sendmail executable
2. SMTP using Net::SMTP

These look like sensible choices.

Actions

• Parsing mails Do not parse Emails twice in Kernel::System::EmailParser::new() #4469