Suboptimal (slow) "ugrep" behavior when used behind a pipe?

[jschleus]

On an Apache web server, I sometimes search for specific patterns at the end of a large log file (roughly 3,000,000 lines). To speed up the search, I use the tac command (which prints files in reverse order) and the option -m 1 to stop reading the file after the first matching line.

Here's an example searching explicitly for the last entry of the log file (previously determined for this example):

1. Using "grep":

time grep “04/Apr/2026:23:59:57” access_log > /dev/null
real    0m0.204s
user    0m0.164s
sys     0m0.040s

time tac access_log | grep -m 1 “04/Apr/2026:23:59:57” > /dev/null
real    0m0.002s
user    0m0.001s
sys     0m0.001s

2. Using “ugrep”:

time ugrep “04/Apr/2026:23:59:57” access_log > /dev/null
real    0m0.092s
user    0m0.036s
sys     0m0.056s

time tac access_log | ugrep -m 1 “04/Apr/2026:23:59:57” > /dev/null
real    0m0.641s
user    0m0.387s
sys     0m0.527s

In this example, ugrep is roughly twice as fast as grep in the standard case, but when used behind the pipe, the execution time of the entire command increases significantly and the hoped-for speedup effect from tac does not materialize (quite the opposite).
Also using the ugrep option --max-count=1 shows the same behavior.

For the sake of completeness, here is the raw time consumed by the tac command alone:

time tac fresh-access_log.260404 > /dev/null
real    0m0.526s
user    0m0.361s
sys     0m0.164s

As a layman, I'd venture a somewhat bold guess from the above data that grep immediately reads the data from the pipe and then, after the first match, stops the process before the pipe (here tac), while ugrep first reads all (!) the data and only then starts the search.

A similar behavior occurs when using cat, except that the times taken aren't quite as dramatically different, since cat is faster than tac.

But maybe I've overlooked something fundamental.

[genivia-inc]

Ugrep detects non-file inputs such as pipes to read them non-blocking, so not all at once. It does not read line-by-line as GNU grep though, so a whole block of data is read that is available on the pipe (i.e. data does not block), then it is searched.

Have you tried option --line-buffered?

[jschleus]

Thanks for the explanation.

I had overlooked the --line-buffered option. However, at least within my test scenario, its use doesn't seem to make any difference in terms of measurement accuracy, neither for ugrep nor for grep.

The situation is somewhat unfortunate for me, as I can't simply replace grep with ugrep in general from a performance perspective. So from a layperson's point of view, an additional option like --line-by-line or similar might be useful?

[genivia-inc]

Another part of the reason is actually the piping of the output to dev/null. This is similar to option -q which overrides non-blocking input and therefore forces reading. It's not useful to use in combination with -m 1 for that reason. If you don't pipe to dev/null then the results should be fast as expected.

We could change that behavior in an update perhaps.

[jschleus]

Piping of the output to /dev/null I have only done for clarity to suppress the essentially insignificant (u)grep output.
But even without that piping, the measured times are unfortunately practically unchanged.

[genivia-inc]

Sure, because ugrep still waits for the pipe to be closed by the sender. Otherwise you'd risk a "broken pipe" signal reported by senders that don't handle EPIPE see e.g. die with a broken pipe. But the difference is that a ugrep output (a match) won't be delayed since the first match found will be reported, i.e. without waiting for a larger block of input to arrive.

This is statically controlled, configured in ugrep.hpp:

// drain stdin until eof to keep reading from stdin such as a pipe (and prevent broken pipe signal)
#define WITH_STDIN_DRAIN

GNU grep has the same behavior to drain pipes with drain_input().

We could add a new option to let ugrep terminate prematurely before the sender is finished. This risks a broken pipe depending on the sending program, so it is not advisable. But it can be useful.

[genivia-inc]

We could make option -q (or piping output to /dev/null) faster by not draining the input pipe by also checking for flag_quiet to be false, like so in ugrep.cpp:3900:

    // drain stdin when non-seekable file such as a pipe until eof, unless option -q is used
    if (file_in == stdin && !flag_quiet && !feof(stdin) && fseek(stdin, 0, SEEK_END) < 0 && errno != EINVAL)
    {
      char buf[16384];
      while (input.get(buf, sizeof(buf)) > 0)
        continue;
    }

That is a better approach and aligns with GNU grep.

[jschleus]

OK.

To clarify: In my example, the first match (at the end of the large file) is found by using tac at the beginning of the pipe and is immediately output also by ugrep. However, it then takes a long time for the entire command to complete. This isn't the case with grep, and at least in my tests, I haven't seen a broken pipe error (though this might be due to the grep line by line reading). Sorry, I don't know much about pipe processing.

As already said: The output to /dev/null was used only to make the example lines simpler and clearer. In reality
the output is required.

[genivia-inc]

OK, I'm trying to figure out what's different to GNU grep. From this, it appears that GNU grep terminates at the first match with -m 1, but I'm not sure if that is always the case for all (historic) GNU grep versions. The GNU grep manual says:

       -m NUM, --max-count=NUM
              Stop reading a file after NUM matching lines.

So this suggests GNU grep will exit when the required number of matches has been found. We can do the same in ugrep.