zonble
diff --git a/‎Source/Data/AGENTS.md‎
Lines changed: 242 additions & 10 deletions b/‎Source/Data/AGENTS.md‎
Lines changed: 242 additions & 10 deletions
diff --git a/‎Source/Data/Makefile‎
Lines changed: 10 additions & 10 deletions b/‎Source/Data/Makefile‎
Lines changed: 10 additions & 10 deletions
diff --git a/‎Source/Data/README‎
Lines changed: 51 additions & 19 deletions b/‎Source/Data/README‎
Lines changed: 51 additions & 19 deletions
diff --git a/‎Source/Data/bin/C_Version/Makefile‎ ‎Source/Data/bin_legacy/C_Version/Makefile‎Source/Data/bin/C_Version/Makefile renamed to Source/Data/bin_legacy/C_Version/Makefile b/‎Source/Data/bin/C_Version/Makefile‎ ‎Source/Data/bin_legacy/C_Version/Makefile‎Source/Data/bin/C_Version/Makefile renamed to Source/Data/bin_legacy/C_Version/Makefile
diff --git a/‎Source/Data/bin/C_Version/count.bash‎ ‎…rce/Data/bin_legacy/C_Version/count.bash‎Source/Data/bin/C_Version/count.bash renamed to Source/Data/bin_legacy/C_Version/count.bash b/‎Source/Data/bin/C_Version/count.bash‎ ‎…rce/Data/bin_legacy/C_Version/count.bash‎Source/Data/bin/C_Version/count.bash renamed to Source/Data/bin_legacy/C_Version/count.bash
diff --git a/‎…ce/Data/bin/C_Version/count.occurrence.c‎ ‎…/bin_legacy/C_Version/count.occurrence.c‎Source/Data/bin/C_Version/count.occurrence.c renamed to Source/Data/bin_legacy/C_Version/count.occurrence.c b/‎…ce/Data/bin/C_Version/count.occurrence.c‎ ‎…/bin_legacy/C_Version/count.occurrence.c‎Source/Data/bin/C_Version/count.occurrence.c renamed to Source/Data/bin_legacy/C_Version/count.occurrence.c
@@ -161,20 +161,252 @@ Emojis/symbols are allowed but **MUST NOT be the default candidate**. Add to `Sy
 - Frequencies must be non-negative integers (0 acceptable, negatives NOT allowed)
 - All files must be C locale sorted for binary search compatibility
 
-## Python Tools in bin/
+## Python Package Structure
+
+The `curation/` package contains library modules organized into submodules:
+
+| Submodule | Purpose |
+|-----------|---------|
+| `curation.builders` | Data building and processing tools (frequency_builder, phrase_deriver) |
+| `curation.compilers` | Data compilation tools (main_compiler, plain_bpmf_compiler) |
+| `curation.validators` | Validation and analysis tools (score_validator) |
+| `curation.utils` | General utilities (text_filter) |
+
+Scripts with side effects are located in `scripts/`:
 
 | Script | Purpose |
 |--------|---------|
-| `cook.py` | Main build script: combines all source files into `data.txt` |
-| `cook-plain-bpmf.py` | Builds `data-plain-bpmf.txt` for traditional Bopomofo mode |
-| `buildFreq.py` | Generates `PhraseFreq.txt` from `phrase.occ` and `exclusion.txt` |
-| `derive_associated_phrases.py` | Generates associated phrase suggestions from `data.txt` |
-| `bpmfmap.py` | Helper for automatic Bopomofo mapping |
-| `count.occurrence.py` | Counts phrase occurrences in text corpus |
-| `self-score-test.py` | Validates and scores dictionary data quality |
-| `nonCJK_filter.py` | Filters out non-CJK characters |
-| `audit_encoding.swift` | Checks file encoding issues |
+| `count_occurrences.py` | Counts phrase occurrences in text corpus |
+| `analyze_data.py` | Analyzes dictionary data and generates reports |
+| `map_bpmf.py` | Helper for automatic Bopomofo mapping |
+
+### Python Development Guidelines
+
+**CRITICAL RULES for AI coding assistants:**
+
+#### 1. Module Organization Rules
+
+**Library modules** (in `curation/` package) **MUST NOT** have side effects at module level:
+- **PROHIBITED**: Opening files, reading/writing data, printing output at module level
+- **PROHIBITED**: Executing code immediately when module is imported
+- **REQUIRED**: All initialization must be in functions
+- **REQUIRED**: Module must be importable without executing code
+
+**Example of BAD module (violates rules):**
+```python
+# BAD: Has side effects at import time
+import configparser
+config = configparser.ConfigParser()
+config.read('config.ini')  # WRONG: Reads file at import!
+corpus = open('corpus.txt').read()  # WRONG: Opens file at import!
+```
+
+**Example of GOOD module:**
+```python
+# GOOD: No side effects, importable as library
+import configparser
+
+def load_config(config_path='config.ini'):
+    """Load configuration from file."""
+    config = configparser.ConfigParser()
+    config.read(config_path)
+    return config
+
+def main():
+    """Main entry point for CLI usage."""
+    config = load_config()
+    # ... rest of logic
+
+if __name__ == '__main__':
+    main()
+```
+
+#### 2. Import Guidelines
+
+- **REQUIRED**: All imports at top of file
+- **NEVER** use inline imports (unless explicitly necessary for specific technical reasons)
+- Use relative imports within package (e.g., `from .compiler_utils import HEADER`)
+- Use absolute imports for external packages (e.g., `import argparse`)
+
+**Example of BAD imports:**
+```python
+# BAD: Inline import
+def process_data():
+    import pandas as pd  # WRONG: NEVER do this!
+    return pd.DataFrame(data)
+```
+
+**Example of GOOD imports:**
+```python
+# GOOD: All imports at top
+import argparse
+import sys
+from typing import List, Dict
+from .compiler_utils import HEADER
+
+def process_data(input_data):
+    result = []
+    for item in input_data:
+        result.append(item.upper())
+    return result
+```
+
+#### 3. Side Effect Management
+
+**Scripts with side effects belong in `scripts/` directory, NOT in `curation/` package.**
+
+Scripts that do any of the following must be in `scripts/`:
+- Read configuration files at module level
+- Open and process data files at module level
+- Print output or generate reports at module level
+- Execute analysis immediately when imported
+
+**When to use `scripts/` vs `curation/`:**
+
+| Location | Purpose | Characteristics |
+|----------|---------|-----------------|
+| `scripts/` | Pure CLI tools | Has side effects; not importable as library; immediate execution |
+| `curation/` | Library modules | No side effects; importable; reusable functions |
+
+#### 4. Script vs Library Separation
+
+**Library modules** in `curation/` should:
+- Provide reusable functions and classes
+- Have a `main()` function for CLI usage
+- Be declared in `pyproject.toml` `[project.scripts]`
+- Follow pattern: `mcbpmf-tool-name = "curation.module:main"`
+
+**Scripts** in `scripts/` should:
+- Be standalone executables
+- Have all logic in `if __name__ == '__main__':` block
+- Be called directly: `python3 scripts/script_name.py`
+- NOT be imported by other modules
+
+#### 5. PEP-8 Naming Conventions
+
+- Module names: `lowercase_with_underscores.py`
+- Function names: `lowercase_with_underscores()`
+- Class names: `CapitalizedWords`
+- Constants: `UPPERCASE_WITH_UNDERSCORES`
+
+**Examples:**
+- GOOD: `frequency_builder.py`, `main_compiler.py`, `text_filter.py`
+- BAD: `buildFreq.py`, `nonCJK_filter.py`, `cook-plain-bpmf.py`
+
+#### 6. Package Installation
+
+Install as editable package for development:
+```bash
+pip install -e .              # Install package
+pip install -e ".[dev]"       # Install with dev dependencies
+pip install -e ".[notebook]"  # Install with notebook dependencies
+```
+
+After installation, use console scripts:
+```bash
+mcbpmf-build-freq              # Instead of: python3 -m curation.builders.frequency_builder
+mcbpmf-compile                 # Instead of: python3 -m curation.compilers.main_compiler
+mcbpmf-validate-scores         # Instead of: python3 -m curation.validators.score_validator
+```
+
+## Project Path Configuration
+
+All scripts and modules use centralized path constants from the `curation` package:
+
+```python
+from curation import PROJECT_ROOT, CONFIG_FILE
+
+# PROJECT_ROOT = Source/Data/ directory (where pyproject.toml lives)
+# CONFIG_FILE = Source/Data/textpool.rc
+
+# Example usage in scripts
+config = configparser.ConfigParser()
+config.read(CONFIG_FILE)
+corpus_path = Path(config.get('data', 'corpus_path')).expanduser()
+```
+
+**Do NOT** compute paths relatively in individual scripts:
+- BAD: `Path(__file__).parent.parent`
+- BAD: `os.path.abspath(sys.argv[0]).split('/')`
+- GOOD: `from curation import PROJECT_ROOT`
+
+This ensures:
+- Single source of truth for project structure
+- Easy refactoring if directory structure changes
+- Consistent behavior across all tools
+
+## Historical Context: Tool Evolution (2012-2025)
+
+### Migration from bin/ to curation/ (October 2024)
+
+Prior to October 2024, all Python tools were located in the `bin/` directory (now renamed to `bin_legacy/`). This directory accumulated tools over 13+ years with contributions from multiple developers.
+
+#### Tool Creation Timeline
+
+- **2012-08-06**: `cook.py` created by Mengjuei Hsieh, replacing Ruby implementation
+- **2012-09-16**: `buildFreq.py` created, replacing bash version
+- **2013-01-02**: `self-score-test.py` added for quality validation
+- **2013-01-21**: C version moved to `C_Version/` subdirectory ("phasing out")
+- **2024-03-15**: `derive_associated_phrases.py` added by Lukhnos Liu (v2 system)
+- **2024-08-25**: `audit_encoding.swift` added by zonble
+- **2025-03-08**: `cook.py` modernized with Black formatting and argparse
+
+#### Why Migration Was Needed
+
+The bin/ structure had accumulated issues:
+1. Flat organization (~30 files, no logical grouping)
+2. Mixed concerns (library code, CLI scripts, config files, legacy tools)
+3. Inconsistent naming conventions
+4. Some modules had side effects at import time
+5. Not installable as proper Python package
+6. Each script calculated paths differently
+
+#### What Was Migrated vs Preserved
+
+**Migrated to `curation/` package** (actively used):
+- All compilation and build tools
+- Frequency calculation
+- Data validation
+- Text processing utilities
+
+**Moved to `scripts/`** (CLI-only, with side effects):
+- Corpus occurrence counting
+- Data analysis reports
+- BPMF mapping helpers
+
+**Preserved in `bin_legacy/`** (historical reference):
+- `audit_encoding.swift` - Still usable standalone tool (2024)
+- `C_Version/` - Fast C implementation, phased out in 2013
+- `Sample_Prep/` - Corpus preparation methodology
+- `disabled/` - Legacy Perl/Ruby/Bash implementations
+
+#### Path Configuration Changed
+
+- **Before**: Each script calculated paths relatively
+- **After**: Import from `curation` package: `from curation import PROJECT_ROOT, CONFIG_FILE`
+
+#### Using Legacy Tools
+
+The `audit_encoding.swift` tool is still functional:
+```bash
+cd bin_legacy
+swift audit_encoding.swift  # Validates BPMFBase.txt encoding categories
+```
+
+C version (for performance comparison):
+```bash
+cd bin_legacy/C_Version
+export TEXTPOOL=/path/to/corpus
+./count.bash 測試詞彙
+```
+
+For complete migration history and tool details, see `bin_legacy/DEPRECATED.md`.
+
+#### Key Contributors
 
+- **Mengjuei Hsieh**: Original Python implementation (2012-2013)
+- **Lukhnos Liu**: Modernization and associated phrases v2 (2024-2025)
+- **zonble**: Encoding audit tool (2024)
 
 ## Data Generation Pipeline
 
 
@@ -4,13 +4,13 @@ all: data.txt data-plain-bpmf.txt associated-phrases-v2.txt
 
 install: all
 
-data-plain-bpmf.txt: bin/cook-plain-bpmf.py BPMFBase.txt BPMFPunctuations.txt
-	bin/cook-plain-bpmf.py BPMFBase.txt BPMFPunctuations.txt data-plain-bpmf.txt
+data-plain-bpmf.txt: curation/compilers/plain_bpmf_compiler.py BPMFBase.txt BPMFPunctuations.txt
+	python3 -m curation.compilers.plain_bpmf_compiler BPMFBase.txt BPMFPunctuations.txt data-plain-bpmf.txt
 
-data.txt: bin/cook.py BPMFBase.txt BPMFMappings.txt BPMFPunctuations.txt \
+data.txt: curation/compilers/main_compiler.py BPMFBase.txt BPMFMappings.txt BPMFPunctuations.txt \
           PhraseFreq.txt phrase.occ Symbols.txt Macros.txt\
           heterophony1.list heterophony2.list heterophony3.list
-	bin/cook.py \
+	python3 -m curation.compilers.main_compiler \
 		--heterophony1 heterophony1.list \
 		--heterophony2 heterophony2.list \
 		--heterophony3 heterophony3.list \
@@ -22,11 +22,11 @@ data.txt: bin/cook.py BPMFBase.txt BPMFMappings.txt BPMFPunctuations.txt \
 		--macros Macros.txt \
 		--output data.txt
 
-associated-phrases-v2.txt: data.txt bin/derive_associated_phrases.py associated-punctuation.txt
-		bin/derive_associated_phrases.py $< $@ associated-punctuation.txt
+associated-phrases-v2.txt: data.txt curation/builders/phrase_deriver.py associated-punctuation.txt
+		python3 -m curation.builders.phrase_deriver $< $@ associated-punctuation.txt
 
-PhraseFreq.txt: bin/buildFreq.py phrase.occ exclusion.txt
-	bin/buildFreq.py
+PhraseFreq.txt: curation/builders/frequency_builder.py phrase.occ exclusion.txt
+	python3 -m curation.builders.frequency_builder
 
 clean:
 	rm -f data.txt data-plain-bpmf.txt phrase.list
@@ -68,7 +68,7 @@ tidy:
 	@sed -i '' -e 's/1/˙/g;s/2/ˊ/g;s/3/ˇ/g;s/4/ˋ/g' BPMFMappings.txt
 
 _phrase.occ: phrase.list
-	@bin/count.occurrence.py phrase.list > tmp && mv tmp phrase.occ
+	@python3 scripts/count_occurrences.py phrase.list > tmp && mv tmp phrase.occ
 
 phrase.list: BPMFBase.txt BPMFMappings.txt
 	awk 'length($$1)<4{print $$1}' BPMFBase.txt > tmp
@@ -91,6 +91,6 @@ _mydebugbuild:
           -derivedDataPath build \
           build
 _mycodecheck:
-	@for myfile in bin/*.py; do \
+	@for myfile in curation/*/*.py scripts/*.py; do \
             pycodestyle --ignore=E501,E722,E701,E221,W605 $$myfile; \
         done
@@ -1,22 +1,54 @@
-├── BPMFBase.txt		Bopomofo Table for chars
-├── BPMFMappings.txt		Bopomofo Table for 2-char words to 6-char words
-|                                  Originally simplified from tsi.src of libtabe
-|                                  (BSD Licensed) with my own and zonble's
-|                                  modification.
-├── BPMFPunctuations.txt	Punctuation table
-├── heterophony1.list		table for char heterophony, first order
-├── heterophony2.list		table for char heterophony, second order
-├── heterophony3.list		table for char heterophony, third order
-├── phrase.occ			table for appearance of a word in a text pool
-├── Makefile			for conveniences for developers
-└── bin
-    ├── cook.py			The tool to build the data.txt for McBopomofo
-    ├── buildFreq.bash		The awk script for counting occurance
-    ├── bpmfmap.py		mapping bpmf automatically and primitively
-    ├── cook-plain-bpmf.py	build data file for tranditional bpmf ime
-    ├── count.bash		a wrapper for a C-based counting facility
-    ├── count.occurrence.c	a C-based counting facility for command line
-    └── count.occurrence.py	a python-based counter for the whole list
+Dictionary Data Structure
+=========================
+
+├── BPMFBase.txt          Single character Bopomofo mappings
+├── BPMFMappings.txt      Multi-character phrases (2-6 chars)
+│                         Originally simplified from tsi.src of libtabe
+│                         (BSD Licensed) with modifications
+├── BPMFPunctuations.txt  Punctuation marks
+├── heterophony1.list     Heterophony reading priorities (1st order)
+├── heterophony2.list     Heterophony reading priorities (2nd order)
+├── heterophony3.list     Heterophony reading priorities (3rd order)
+├── phrase.occ            Phrase frequency/occurrence data
+├── exclusion.txt         Phrase frequency exclusions
+├── Symbols.txt           Special symbols (era names, etc.)
+├── Macros.txt            Text macros (date/time)
+├── Makefile              Build automation
+├── pyproject.toml        Python package configuration
+├── textpool.rc           Corpus counting configuration
+│
+├── curation/             Python package for data processing
+│   │                     (exports PROJECT_ROOT, CONFIG_FILE for path resolution)
+│   ├── builders/         frequency_builder, phrase_deriver
+│   ├── compilers/        main_compiler, plain_bpmf_compiler, compiler_utils
+│   ├── utils/            text_filter
+│   ├── validators/       score_validator
+│   ├── notebooks/        Jupyter analysis notebooks
+│   └── requirements.txt  Package dependencies
+│
+├── scripts/              Standalone CLI tools (with side effects)
+│   │                     (imports paths from curation package)
+│   ├── count_occurrences.py  Count phrase occurrences in corpus
+│   ├── analyze_data.py       Analyze dictionary data quality
+│   └── map_bpmf.py           Primitive BPMF mapping helper
+│
+├── tests/                Unit tests for curation package
+│
+└── bin_legacy/           DEPRECATED (Oct 2024): Historical tools (2012-2025)
+    ├── DEPRECATED.md     Complete migration guide & tool history
+    ├── audit_encoding.swift  Still usable encoding validator
+    ├── C_Version/        Fast C implementation (phased out 2013)
+    ├── Sample_Prep/      Historical corpus preparation methods
+    └── disabled/         Legacy Perl/Ruby/Bash implementations
+
+----- Build System -----
+Use Makefile targets (automatically uses new curation/ structure):
+  make all      # Build all data files
+  make sort     # Sort all data files
+  make check    # Validate data integrity
+  make clean    # Remove generated files
+
+For detailed build pipeline, see AGENTS.md and ../algorithm.md
 
 ----- Editorial Rule -----
 * when in doubt, use google/yahoo search to confirm the rarity of the phrases