Merge pull request #5 from zonble/copilot/fix-4

Add comprehensive Copilot instruction file with detailed framework and testing documentation

Author: zonble

.copilot-instructions.md

@@ -0,0 +1,370 @@
+# McBopomofo Copilot Instructions
+
+## What's the Project For
+
+McBopomofo (小麥注音輸入法) is a Traditional Chinese input method engine (IME) for macOS. It allows users to input Traditional Chinese characters using the Bopomofo phonetic system (注音符號), which is the standard phonetic notation system used in Taiwan. 
+
+The project is part of the OpenVanilla framework and provides:
+- Smart phonetic-to-character conversion
+- User-customizable phrase dictionaries
+- Associated phrase suggestions
+- Multi-candidate selection
+- Support for custom user phrases and exclusions
+
+## System Requirements
+
+### Runtime Requirements
+- macOS 10.15 (Catalina) or later
+
+### Development Requirements
+- macOS 14.7 or later
+- Xcode 15.3 or later
+- Python 3.9 (available through Xcode or homebrew)
+
+## Build Process
+
+The project uses Xcode as its primary build system:
+
+1. **Open Project**: Open `McBopomofo.xcodeproj` in Xcode
+2. **Select Target**: Choose "McBopomofoInstaller" target
+3. **Build**: Build the project (⌘+B)
+4. **Install**: Run the installer directly to install McBopomofo
+5. **Reinstall**: For subsequent updates, repeat the process
+
+### Important Notes
+- macOS may limit the number of times an input method process can be killed in a single login session
+- If installation issues occur after multiple installs, log out and log back in
+- The installer automatically kills and restarts the input method process
+
+## Project Components
+
+### Architecture Overview
+```
+McBopomofo/
+├── Source/                     # Main application source
+│   ├── Engine/                # C++ core engine
+│   │   ├── Mandarin/         # Bopomofo processing
+│   │   ├── gramambular2/     # Text segmentation library
+│   │   └── McBopomofoLM.*    # Language model
+│   ├── InputState.swift       # State machine implementation
+│   ├── InputMethodController.swift  # Main controller
+│   └── Data/                 # Language model data files
+├── McBopomofoTests/           # Test suite
+└── Packages/                  # Swift Package dependencies
+```
+
+### Technology Stack
+- **Swift**: UI layer, input method controller, state management
+- **Objective-C++**: Bridge between Swift and C++ components
+- **C++**: Core engine, language processing, data structures
+- **Xcode**: Build system and development environment
+
+### Framework Foundation
+McBopomofo is built on Apple's native input method frameworks:
+
+#### Cocoa Framework
+- **UI Components**: Native macOS user interface elements
+- **Event Handling**: Keyboard and mouse event processing
+- **Window Management**: Candidate window and preference panels
+- **Integration**: Seamless integration with macOS applications
+
+#### Input Method Kit (IMK)
+- **IMKInputController**: Base class for input method controllers
+  - `McBopomofoInputMethodController` extends `IMKInputController`
+  - Handles input events and text processing
+  - Manages communication with client applications
+- **IMKServer**: Input method server infrastructure
+  - Manages input method instances
+  - Handles system-level input method operations
+- **IMKCandidateController**: Candidate selection interface
+- **Protocol Compliance**: Implements IMK protocols for proper system integration
+
+### Key Files
+- `InputMethodController.swift`: Main input method logic
+- `InputState.swift`: State machine implementation
+- `Source/Engine/McBopomofoLM.h/cpp`: Language model management
+- `Source/Engine/Mandarin/Mandarin.h/cpp`: Bopomofo processing
+- `Source/Engine/gramambular2/`: Text segmentation algorithms
+
+## Design of Input States
+
+McBopomofo implements a finite state machine with multiple distinct states to handle various input scenarios:
+
+### Core Input States
+
+1. **Deactivated**: User hasn't activated McBopomofo
+2. **Empty**: McBopomofo is active but no input yet, or user just committed text
+3. **Inputting**: User is typing Bopomofo keys; input buffer is visible
+4. **Committing**: Sending text to client applications
+5. **Marking**: User is selecting text in buffer to create custom phrase
+6. **ChoosingCandidate**: Candidate window is open for user selection
+
+### Extended States
+
+The system includes additional specialized states:
+
+- **SelectingFeature**: User accessing special features menu
+- **SelectingDateMacro**: Date/time macro selection
+- **ChineseNumber**: Chinese numeral conversion
+- **Big5**: Big5 encoding conversion
+- **EnclosedNumber**: Circled/parenthesized number conversion
+- **AssociatedPhrases**: Associated phrase suggestions
+- **SelectingDictionary**: Dictionary lookup mode
+- **ShowingCharInfo**: Character information display
+- **CustomMenu**: Custom menu operations
+
+### State Properties
+- **Immutable**: States are immutable objects; transitions create new state instances
+- **One-way data flow**: UI and text output follow single data source
+- **Context-specific data**: Each state contains only relevant data (e.g., candidate list only exists in Choosing Candidate state)
+
+### Implementation Details
+- Base class: `InputState`
+- State-specific subclasses: `Deactivated`, `Empty`, `Inputting`, `Committing`, `Marking`, `ChoosingCandidate`
+- Controller creates new state objects instead of modifying existing ones
+
+## Design of Mandarin Package
+
+The Mandarin package handles Bopomofo phonetic input processing and conversion.
+
+### Core Classes
+
+#### BopomofoSyllable (BPMF)
+- **Purpose**: Represents a complete Bopomofo syllable
+- **Storage**: 16-bit integer with bit masks for components
+- **Components**: Consonant, Middle Vowel, Vowel, Tone Marker
+- **Features**:
+  - Conversion to/from Hanyu Pinyin
+  - Composed string representation
+  - Component validation and extraction
+  - Overlap detection between syllables
+
+#### BopomofoKeyboardLayout
+- **Purpose**: Maps keyboard keys to Bopomofo components
+- **Key Functions**:
+  - `syllableFromKeySequence()`: Converts key sequence to syllable
+  - `keySequenceFromSyllable()`: Converts syllable back to keys
+- **Layout Support**: Different keyboard layouts for Bopomofo input
+- **Validation**: Ensures valid key combinations (e.g., J/Q/X require I or UE vowels)
+
+#### BopomofoReadingBuffer
+- **Purpose**: Manages user input during syllable composition
+- **Features**:
+  - Key combination and validation
+  - Pinyin mode support
+  - Buffer clearing and state management
+  - Integration with keyboard layouts
+
+### Key Algorithms
+
+#### Syllable Construction
+1. Process each key in input sequence
+2. Check for valid key combinations
+3. Handle special cases (J/Q/X with I/UE vowels)
+4. Apply tone markers correctly
+5. Validate final syllable composition
+
+#### Component Mapping
+- Bit-masked representation for efficient storage
+- Separate masks for consonants, vowels, and tones
+- Support for multiple components per key
+
+## Keyboard Layouts
+
+McBopomofo supports multiple Bopomofo keyboard layouts to accommodate different user preferences and typing habits:
+
+### Standard Layout
+- **Description**: Traditional Bopomofo layout used in Taiwan
+- **Characteristics**: Direct mapping of Bopomofo symbols to QWERTY keys
+- **Usage**: Most common layout for Bopomofo input
+- **Implementation**: `BopomofoKeyboardLayout::StandardLayout()`
+
+### ETen Layout
+- **Description**: ETen Traditional layout
+- **Characteristics**: Alternative key mapping optimized for certain typing patterns
+- **Usage**: Popular among users familiar with ETen input systems
+- **Implementation**: `BopomofoKeyboardLayout::ETenLayout()`
+
+### Hsu Layout (許氏鍵盤)
+- **Description**: Hsu keyboard layout designed by Hsu Lian-chin
+- **Characteristics**: 
+  - Optimized for faster typing with fewer keystrokes
+  - Special heuristics for vowel combinations
+  - Automatic correction rules (e.g., GI/GUE → JI/JUE)
+- **Usage**: Preferred by advanced users for speed typing
+- **Implementation**: `BopomofoKeyboardLayout::HsuLayout()`
+
+### ETen26 Layout
+- **Description**: ETen 26-key layout variant
+- **Characteristics**: Extended ETen layout with additional key combinations
+- **Usage**: Enhanced version of ETen layout
+- **Implementation**: `BopomofoKeyboardLayout::ETen26Layout()`
+
+### Layout Architecture
+- **Key-to-Component Mapping**: Each layout defines mappings from keyboard keys to Bopomofo components
+- **Syllable Construction**: Layouts handle complex rules for syllable formation
+- **Special Rules**: Each layout can implement specific typing optimizations and corrections
+- **Runtime Switching**: Users can switch between layouts in preferences
+
+## Design of Language Model (McBopomofoLM)
+
+The language model manages text conversion, user customization, and phrase suggestions.
+
+### Architecture
+
+#### McBopomofoLM Class
+- **Inheritance**: Extends `Formosa::Gramambular2::LanguageModel`
+- **Purpose**: Central hub for all language processing
+- **Integration**: Combines multiple data sources and processing layers
+
+### Data Processing Pipeline
+
+When processing unigrams (single-character/phrase entries):
+
+1. **Original Unigrams**: Retrieve from primary language model
+2. **Exclusion Filtering**: Remove user-excluded phrases
+3. **Phrase Replacement**: Apply user-defined replacements
+4. **External Conversion**: Transform via external converter (if enabled)
+5. **Deduplication**: Remove duplicate entries
+6. **Return Results**: Provide final candidate list
+
+### Key Components
+
+#### Primary Language Model
+- **ParselessLM**: Main language model for character/phrase data
+- **Unigram-based**: Uses single-token probability model
+- **File-based**: Loads from bundled data files
+
+#### User Customization
+- **UserPhrasesLM**: User-defined custom phrases
+- **Exclusion List**: User-blocked phrases
+- **Replacement Map**: User-defined phrase substitutions
+- **Associated Phrases**: Context-based phrase suggestions
+
+#### External Processing
+- **Macro Converter**: Handles text macros and shortcuts
+- **External Converter**: Traditional/Simplified Chinese character conversion (OpenCC-based)
+- **Runtime Configuration**: Enable/disable features dynamically
+
+### File Management
+- **User Data Folder**: Configurable location for user files
+- **Template System**: Automatic creation of empty user files
+- **Atomic Updates**: Safe file writing and reloading
+
+## Algorithm of Gramambular2
+
+Gramambular2 is the core segmentation and input method library using statistical models.
+
+### Theoretical Foundation
+
+#### Hidden Markov Model (HMM)
+- **Observations**: Input characters or Bopomofo syllables
+- **Hidden States**: Possible character/phrase groupings
+- **Goal**: Find most likely segmentation given input sequence
+
+#### Naive Bayes Classification
+- **Approach**: Simplified probabilistic classification
+- **Assumptions**: Independence between features
+- **Efficiency**: Fast computation suitable for real-time input
+
+### Core Algorithm
+
+#### Segmentation Process
+1. **Input Sequence**: Receive series of observations (syllables/characters)
+2. **State Generation**: Generate possible hidden states (character combinations)
+3. **Probability Calculation**: Compute likelihood using unigram model
+4. **Path Selection**: Choose most probable segmentation path
+5. **Output Generation**: Return most likely character sequence
+
+#### Language Model Integration
+- **Unigram Model**: Simple single-token probability model
+- **Frequency-based**: Probabilities derived from corpus frequency
+- **Extensible**: Support for custom language models
+
+### Implementation Details
+
+#### Reading Grid
+- **Purpose**: Manages candidate generation and selection
+- **Structure**: Grid of possible readings and conversions
+- **Optimization**: Efficient storage and retrieval of candidates
+
+#### Language Model Interface
+- **Abstract Base**: `LanguageModel` interface for pluggable models
+- **Standard Methods**: `hasUnigrams()`, `getUnigrams()` for data access
+- **Extensibility**: Support for custom language model implementations
+
+### Performance Characteristics
+- **Real-time**: Optimized for interactive input method use
+- **Memory Efficient**: Minimal memory footprint
+- **Scalable**: Handles large dictionaries efficiently
+- **Fast Lookup**: Quick candidate generation and ranking
+
+### Use Cases
+1. **Input Method**: Convert Bopomofo sequences to Chinese characters
+2. **Text Segmentation**: Break Chinese text into meaningful units
+3. **Candidate Ranking**: Order possible conversions by probability
+4. **Context Awareness**: Consider surrounding text for better suggestions
+
+## Development Guidelines
+
+### Code Organization
+- **Swift**: Use for UI, state management, and application logic
+- **C++**: Use for performance-critical algorithms and data processing
+- **Objective-C++**: Use for bridging between Swift and C++
+
+### Testing
+McBopomofo employs a comprehensive testing strategy with both C++ and Swift test suites:
+
+#### C++ Engine Tests
+Located in `Source/Engine/`, these tests validate core algorithms and data structures:
+
+- **Unit Tests**: Individual component testing using Google Test framework
+- **Test Files**:
+  - `MandarinTest.cpp`: Bopomofo syllable and keyboard layout tests
+  - `McBopomofoLMTest.cpp`: Language model functionality tests
+  - `ParselessLMTest.cpp`: Language model parsing and data structure tests
+  - `UTF8HelperTest.cpp`: UTF-8 string processing tests
+  - `UserPhrasesLMTest.cpp`: User-defined phrase management tests
+  - `PhraseReplacementMapTest.cpp`: Phrase replacement logic tests
+  - `KeyValueBlobReaderTest.cpp`: Data file reading tests
+  - `MemoryMappedFileTest.cpp`: Memory-mapped file operations tests
+  - `AssociatedPhrasesV2Test.cpp`: Associated phrase suggestion tests
+  - `UserOverrideModelTest.cpp`: User override functionality tests
+- **Coverage**: Core engine components, data structures, algorithms
+- **Build**: Integrated with CMake build system
+
+#### Swift Application Tests
+Located in `McBopomofoTests/`, these tests validate application logic and UI integration:
+
+- **Test Framework**: XCTest framework for Swift testing
+- **Test Files**:
+  - `KeyHandlerBopomofoTests.swift`: Input processing and key handling tests
+  - `KeyHandlerPlainBopomofoTests.swift`: Plain Bopomofo input tests
+  - `PreferencesTests.swift`: User preferences and configuration tests
+  - `DictionaryServiceTests.swift`: Dictionary lookup service tests
+  - `ServiceProviderTests.swift`: Input method service provider tests
+  - `AssociatedPhrasesTests.swift`: Associated phrase functionality tests
+  - `InputMacroTests.swift`: Text macro processing tests
+  - `VersionUpdateTests.swift`: Version management and update tests
+- **Integration**: Tests complete input workflows and state management
+- **UI Testing**: Validates user interface components and interactions
+
+#### Mixed Language Tests
+- **UTF8HelperTest.mm**: Objective-C++ test bridging Swift and C++ components
+- **Bridging Header**: `McBopomofoTests-Bridging-Header.h` enables Swift-C++ interop in tests
+
+#### Testing Approach
+- **Isolated Testing**: C++ tests focus on algorithmic correctness
+- **Integration Testing**: Swift tests validate complete user workflows
+- **Cross-Language Testing**: Objective-C++ tests ensure proper bridging
+- **Continuous Validation**: Both test suites run during development cycles
+
+### Debugging
+- **State Inspection**: Monitor InputState transitions
+- **Language Model**: Verify unigram processing pipeline
+- **Syllable Processing**: Check Bopomofo key handling
+
+### Performance Considerations
+- **Memory Management**: Careful handling of large language model data
+- **Real-time Constraints**: Input method must be responsive
+- **Battery Usage**: Optimize for minimal system impact