Add engine profiler

[ChiahongHong]

Since all of our current performance benchmarks are based on unit tests, it has been difficult to identify bottlenecks across the end-to-end input flow. This PR adds an engine profiling workload that simulates real-world input and uses xctrace to generate .trace files that can be opened directly in Instruments.

Source/Tools/run-engine-profiler.sh

The report will be generated in Source/Engine/Report/.

Without NEON:

short_iterations=771129
medium_iterations=116411
long_iterations=30573

With NEON:

short_iterations=898449
medium_iterations=148747
long_iterations=43103

PS: short, medium, and long refer to the length of the bopomofo sequence in each ProfilingScenario.

The traces confirm that binary search is indeed the largest performance cost. After the recent PRs introduced SIMD optimizations, I wanted to approach the problem from another angle: can we reduce the number of binary searches performed in the first place?

For example, when entering ㄒㄧㄠˇ ㄇㄞˋ ㄓㄨˋ ㄧㄣ ㄕㄨ ㄖㄨˋ ㄈㄚˇ, the lookup results show a significant number of misses. As each new reading is entered, many of the same missing keys are searched again:

insert[1] keys=vul3 reading=ㄒㄧㄠˇ
  query[1] reading=ㄒㄧㄠˇ unigram_count=9
    value=小 (...remaining values omitted)
insert[2] keys=a94 reading=ㄇㄞˋ
  query[2] reading=ㄒㄧㄠˇ-ㄇㄞˋ unigram_count=1
    value=小麥
  query[3] reading=ㄇㄞˋ unigram_count=14
    value=賣 (...remaining values omitted)
insert[3] keys=5j4 reading=ㄓㄨˋ
  query[4] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ unigram_count=0
  query[5] reading=ㄇㄞˋ-ㄓㄨˋ unigram_count=0
  query[6] reading=ㄓㄨˋ unigram_count=60
    value=助 (...remaining values omitted)
insert[4] keys=up reading=ㄧㄣ
  query[7] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ unigram_count=0
  query[8] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ unigram_count=0
  query[9] reading=ㄇㄞˋ-ㄓㄨˋ unigram_count=0
  query[10] reading=ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ unigram_count=0
  query[11] reading=ㄓㄨˋ-ㄧㄣ unigram_count=1
    value=注音
  query[12] reading=ㄧㄣ unigram_count=51
    value=因 (...remaining values omitted)
insert[5] keys=gj reading=ㄕㄨ
  query[13] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ unigram_count=0
  query[14] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ unigram_count=0
  query[15] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ-ㄕㄨ unigram_count=0
  query[16] reading=ㄇㄞˋ-ㄓㄨˋ unigram_count=0
  query[17] reading=ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ unigram_count=0
  query[18] reading=ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ-ㄕㄨ unigram_count=0
  query[19] reading=ㄓㄨˋ-ㄧㄣ-ㄕㄨ unigram_count=0
  query[20] reading=ㄧㄣ-ㄕㄨ unigram_count=0
  query[21] reading=ㄕㄨ unigram_count=50
    value=書 (...remaining values omitted)
insert[6] keys=bj4 reading=ㄖㄨˋ
  query[22] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ unigram_count=0
  query[23] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ unigram_count=0
  query[24] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ-ㄕㄨ unigram_count=0
  query[25] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ-ㄕㄨ-ㄖㄨˋ unigram_count=0
  query[26] reading=ㄇㄞˋ-ㄓㄨˋ unigram_count=0
  query[27] reading=ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ unigram_count=0
  query[28] reading=ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ-ㄕㄨ unigram_count=0
  query[29] reading=ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ-ㄕㄨ-ㄖㄨˋ unigram_count=0
  query[30] reading=ㄓㄨˋ-ㄧㄣ-ㄕㄨ unigram_count=0
  query[31] reading=ㄓㄨˋ-ㄧㄣ-ㄕㄨ-ㄖㄨˋ unigram_count=0
  query[32] reading=ㄧㄣ-ㄕㄨ unigram_count=0
  query[33] reading=ㄧㄣ-ㄕㄨ-ㄖㄨˋ unigram_count=0
  query[34] reading=ㄕㄨ-ㄖㄨˋ unigram_count=1
    value=輸入
  query[35] reading=ㄖㄨˋ unigram_count=22
    value=入 (...remaining values omitted)
insert[7] keys=z83 reading=ㄈㄚˇ
  query[36] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ unigram_count=0
  query[37] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ unigram_count=0
  query[38] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ-ㄕㄨ unigram_count=0
  query[39] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ-ㄕㄨ-ㄖㄨˋ unigram_count=0
  query[40] reading=ㄒㄧㄠˇ-ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ-ㄕㄨ-ㄖㄨˋ-ㄈㄚˇ unigram_count=0
  query[41] reading=ㄇㄞˋ-ㄓㄨˋ unigram_count=0
  query[42] reading=ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ unigram_count=0
  query[43] reading=ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ-ㄕㄨ unigram_count=0
  query[44] reading=ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ-ㄕㄨ-ㄖㄨˋ unigram_count=0
  query[45] reading=ㄇㄞˋ-ㄓㄨˋ-ㄧㄣ-ㄕㄨ-ㄖㄨˋ-ㄈㄚˇ unigram_count=0
  query[46] reading=ㄓㄨˋ-ㄧㄣ-ㄕㄨ unigram_count=0
  query[47] reading=ㄓㄨˋ-ㄧㄣ-ㄕㄨ-ㄖㄨˋ unigram_count=0
  query[48] reading=ㄓㄨˋ-ㄧㄣ-ㄕㄨ-ㄖㄨˋ-ㄈㄚˇ unigram_count=0
  query[49] reading=ㄧㄣ-ㄕㄨ unigram_count=0
  query[50] reading=ㄧㄣ-ㄕㄨ-ㄖㄨˋ unigram_count=0
  query[51] reading=ㄧㄣ-ㄕㄨ-ㄖㄨˋ-ㄈㄚˇ unigram_count=0
  query[52] reading=ㄕㄨ-ㄖㄨˋ-ㄈㄚˇ unigram_count=1
    value=輸入法
  query[53] reading=ㄖㄨˋ-ㄈㄚˇ unigram_count=0
  query[54] reading=ㄈㄚˇ unigram_count=11
    value=法 (...remaining values omitted)
output=小麥注音輸入法

I am working on a follow-up PR to mitigate this issue. Since it will inevitably require changes to Gramambular, I am currently considering how to keep those changes as small and contained as possible.

---

ReadingGridUpdateTrace.cpp

#include <filesystem>
#include <iomanip>
#include <iostream>
#include <memory>
#include <string>
#include <utility>
#include <vector>

#include "Mandarin/Mandarin.h"
#include "McBopomofoLM.h"
#include "gramambular2/language_model.h"
#include "gramambular2/reading_grid.h"

namespace {

using Formosa::Gramambular2::LanguageModel;
using Formosa::Gramambular2::ReadingGrid;
using Formosa::Mandarin::BopomofoKeyboardLayout;
using Formosa::Mandarin::BopomofoReadingBuffer;
using McBopomofo::McBopomofoLM;

class TracingLanguageModel : public LanguageModel {
 public:
  explicit TracingLanguageModel(std::shared_ptr<LanguageModel> languageModel)
      : languageModel_(std::move(languageModel)) {}

  std::vector<Unigram> getUnigrams(const std::string& reading) override {
    std::vector<Unigram> unigrams = languageModel_->getUnigrams(reading);
    std::cout << "  query[" << ++queryCount_ << "] reading=" << reading
              << " unigram_count=" << unigrams.size() << '\n';
    for (const Unigram& unigram : unigrams) {
      std::cout << "    value=" << unigram.value()
                << '\n';
    }
    return unigrams;
  }

  bool hasUnigrams(const std::string& reading) override {
    return languageModel_->hasUnigrams(reading);
  }

 private:
  std::shared_ptr<LanguageModel> languageModel_;
  size_t queryCount_ = 0;
};

}  // namespace

int main(int argc, char* argv[]) {
  if (argc != 2) {
    std::cerr << "Usage: " << argv[0] << " <LANGUAGE_MODEL_PATH>\n";
    return 1;
  }

  const std::filesystem::path languageModelPath = argv[1];
  auto languageModel = std::make_shared<McBopomofoLM>();
  languageModel->loadLanguageModel(languageModelPath.c_str());
  if (!languageModel->isDataModelLoaded()) {
    std::cerr << "Failed to load language model: " << languageModelPath << '\n';
    return 1;
  }

  auto tracingLanguageModel =
      std::make_shared<TracingLanguageModel>(languageModel);
  ReadingGrid grid(tracingLanguageModel);
  BopomofoReadingBuffer readingBuffer(BopomofoKeyboardLayout::StandardLayout());

  const std::vector<std::string> keySequences = {
      "vul3", "a94", "5j4", "up", "gj", "bj4", "z83",
  };

  ReadingGrid::WalkResult walk;
  for (size_t index = 0; index < keySequences.size(); ++index) {
    const std::string& keySequence = keySequences[index];
    for (char key : keySequence) {
      readingBuffer.combineKey(key);
    }

    const std::string reading = readingBuffer.composedString();
    std::cout << "insert[" << index + 1 << "] keys=" << keySequence
              << " reading=" << reading << '\n';
    if (!grid.insertReading(reading)) {
      std::cerr << "Failed to insert reading: " << reading << '\n';
      return 1;
    }
    readingBuffer.clear();
    walk = grid.walk();
  }

  std::string output;
  for (const std::string& value : walk.valuesAsStrings()) {
    output += value;
  }
  std::cout << "output=" << output << '\n';
  return output == "小麥注音輸入法" ? 0 : 1;
}

./ReadingGridUpdateTrace Source/Data/data.txt

[gemini-code-assist]

Code Review

This pull request introduces an engine profiling workload (EngineProfile.cpp) along with a macOS-specific runner script (run-engine-profiler.sh) to measure performance with and without NEON SIMD support. Feedback on the changes includes resolving a cross-platform compilation issue on Windows by converting the filesystem path to a narrow string, preventing potential race conditions in the shell script by using mktemp instead of a fixed /tmp directory, and avoiding undefined behavior by validating argc and argv[0] before use.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

On Windows, std::filesystem::path::c_str() returns const wchar_t* rather than const char*. Since McBopomofoLM::loadLanguageModel expects a const char*, this will cause a compilation error on Windows.

To ensure cross-platform compatibility (as Windows is a supported platform in this repository), convert the path to a narrow string first using .string().

    languageModel_->loadLanguageModel(languageModelPath.string().c_str());

[gemini-code-assist]

Using a fixed path in /tmp (like /tmp/McBopomofoEngineProfilerBuild) can lead to permission conflicts and race conditions if multiple users or parallel processes run this script on the same machine.

According to the repository's general rules, atomic and unique creation functions like mktemp -d should be used for temporary directories to ensure isolation and avoid race conditions.

Suggested change

	BUILD_DIR="${BUILD_DIR:-/tmp/McBopomofoEngineProfilerBuild}"
	NEON_BUILD_DIR="${NEON_BUILD_DIR:-$BUILD_DIR-NEON}"
	BUILD_DIR="${BUILD_DIR:-$(mktemp -d -t McBopomofoEngineProfilerBuild)}"
	NEON_BUILD_DIR="${NEON_BUILD_DIR:-$(mktemp -d -t McBopomofoEngineProfilerBuild-NEON)}"

References

1. To ensure test isolation and avoid race conditions during parallel test execution, use atomic and unique creation functions like mkstemp() or mkdtemp() for temporary files and directories instead of fixed filenames or non-atomic randomized path checks.

[gemini-code-assist]

If the program is executed with an empty argument list (where argc is 0), argv[0] will be nullptr. Accessing argv[0] on lines 230 and 241 without checking if it is null leads to undefined behavior and potential crashes.

To ensure robust and defensive programming, verify that argc > 0 and argv[0] != nullptr before using it, falling back to a default program name if necessary.

int main(int argc, char* argv[]) {
  const char* programName = (argc > 0 && argv[0] != nullptr) ? argv[0] : "EngineProfile";

  if (argc != 2) {
    std::cerr << "Usage: " << programName << " <PROFILE_DURATION>\n";
    return 1;
  }

  const auto profileDuration = ParseProfileDuration(argv[1]);
  if (!profileDuration.has_value()) {
    std::cerr << "Profile duration must be an integer between 1 and 3600.\n";
    return 1;
  }

  const std::filesystem::path languageModelPath =
      ResolveLanguageModelPath(programName);

[lukhnos]

I wanted to approach the problem from another angle: can we reduce the number of binary searches performed in the first place?

Thanks for the insight and I think profiling is a good area to pursue. As for improvements, perhaps some form of caching, or remembering the last binary search pointer, could be useful? Especially since those misses are caused by repeatedly searhcing on the same prefix. Short of refactoring the whole DB to a trie, pehraps this would give us the most bang for the buck.

But let's also keep our perspective on how much this matters in the grand scheme. Our SIMD experiments are good because it taps underutilized CPU power for a relatively low engineering+maintenance cost (that is, if we eventually solve the deployment challenges) and they are very much confined in a small set of string search functions. Other optimization strategies may increase complexity—we'll have to see about that.

And don't lose sights on what we use the gains for. If walking, basically "re-rendering", a 100-character sentence is a sub-millisecond task, it seems that binary-search is a "done" problem here. I'll quote something I learned from a CG class long time ago: if you can already render a million polygons every frame without lag, it frees you to pursue whatever is it to make your application (your game, your CAD, and so on) better. :)

[ChiahongHong]

That makes sense. If I make any further changes, I’ll post them here in the comments for everyone’s reference rather than open a separate PR. Please don’t feel any obligation to review or respond to them. Just treat them as experiments I’m doing on my own~

[ChiahongHong]

For caching, I tried keeping an array in each span to record missing keys. I also tried using an unordered_map to remember which keys had already been looked up. I know 新酷音 uses a trie, but that feels too heavy for this. None of these approaches felt elegant enough to me.

In the end, this is the version I am happy with and wanted to share. It does not add any new data structure. The core change is just the loops in ReadingGrid::update().

void ReadingGrid::update(size_t loc, EditType editType) {
  // Spans that do not cross the edit retain their previous lookup result. A
  // node means that the lookup succeeded, while a null slot means that the
  // same reading was already looked up and did not exist. Only spans that
  // include an insertion or cross a deletion boundary need to be queried.
  size_t affectedLength = kMaximumSpanLength - 1;
  size_t begin = loc <= affectedLength ? 0 : loc - affectedLength;
  size_t end = editType == EditType::kInsertion ? loc + 1 : loc;
  end = std::min(end, readings_.size());

  for (size_t pos = begin; pos < end; pos++) {
    size_t minimumLength = loc - pos + 1;
    size_t maximumLength = std::min(kMaximumSpanLength, readings_.size() - pos);
    for (size_t len = minimumLength; len <= maximumLength; len++) {
      std::string combinedReading =
          combineReading(readings_.begin() + static_cast<ptrdiff_t>(pos),
                         readings_.begin() + static_cast<ptrdiff_t>(pos + len));

      if (!hasNodeAt(pos, len, combinedReading)) {
        auto unigrams = lm_.getUnigrams(combinedReading);
        if (unigrams.empty()) {
          continue;
        }

        insert(pos, std::make_shared<Node>(std::move(combinedReading), len,
                                           std::move(unigrams)));
      }
    }
  }
}

Let's do some simple math XDD, we only want update() to visit combinations that can reach the edit location:

size_t affectedLength = kMaximumSpanLength - 1;
size_t begin = loc <= affectedLength ? 0 : loc - affectedLength;

kMaximumSpanLength is 8, so if there are fewer than 7 readings on the left, we start at position 0. Otherwise, we start at loc - 7.

The end position depends on whether this is an insertion or deletion:

size_t end = editType == EditType::kInsertion ? loc + 1 : loc;

For an insertion, end is loc + 1. The outer loop is:

for (size_t pos = begin; pos < end; pos++)

Since the condition is pos < end, end must be loc + 1 if we want to include pos == loc.

Suppose we insert n between a and b:

a b c

After the insertion:

a n b c
  ↑
 loc

The affected combinations starting before n are:

a n
a n b
a n b c

The affected combinations starting at n are:

n
n b
n b c

All of them contain the inserted reading. Existing combinations such as b c do not contain the edit, so they can keep their previous lookup result.

For a deletion, end is just loc.

After deletion, loc is the first surviving reading on the right. For example:

a b c

After deleting b:

a | c
    loc = 1

Only spans starting to the left of loc can cross this new boundary. So deletion uses pos < loc and does not include pos == loc.

Then we calculate the minimum length:

size_t minimumLength = loc - pos + 1;

This is the minimum length needed for a span starting at pos to reach loc.

The maximum length is:

size_t maximumLength =
    std::min(kMaximumSpanLength, readings_.size() - pos);

This is the number of readings left between pos and the end of the vector, capped by kMaximumSpanLength.

That's all!! We've covered all the cases without adding a trie, a cache, or any other long-lived data structure.
My changes are available in https://github.com/ChiahongHong/McBopomofo/tree/reduce-binary-search

Without NEON

Stage	short_iterations	medium_iterations	long_iterations
Before	771,129	116,411	30,573
After	892,077	148,430	43,351

With NEON

Stage	short_iterations	medium_iterations	long_iterations
Before	898,449	148,747	43,103
After	952,787	174,237	66,322

[lukhnos]

That's all!! We've covered all the cases without adding a trie, a cache, or any other long-lived data structure.
My changes are available in https://github.com/ChiahongHong/McBopomofo/tree/reduce-binary-search

This is amazing work by identifying duplicate work that needs not doing! We'd be more than happy to review the change if you'd like to make a PR (or integrate into this one). Thank you so much for the efforts!

[ChiahongHong]

I’ve opened #886 separately to make the changes easier to review. Thank you so much for taking the time to look at this!

[lukhnos]

LGTM. Thanks!