TTS Corpus Corrector for Hungarian

A highly sophisticated, multi-stage pipeline for cleaning and correcting transcribed Hungarian text corpora, specifically designed for preparing high-quality datasets for Text-to-Speech (TTS) model training.

This project uses a hybrid approach, combining traditional linguistic tools (Hunspell), modern NLP models (HuSpaCy), and Large Language Models (Google Gemini) with a robust, interactive user interface (Streamlit) for human-in-the-loop validation.

Key Features

• Hybrid Error Detection: Uses Hunspell for basic typos and HuSpaCy for complex contextual errors like incorrect word splits/merges.
• Two-LLM Validation: Employs a two-stage LLM process (e.g., a fast model for initial suggestions, a powerful model for secondary validation) to increase the reliability of automated corrections.
• Structured LLM Output: Enforces a strict JSON output schema using Pydantic and Gemini's structured output feature, ensuring reliable and predictable results.
• Intelligent Prompt Engineering: Utilizes the CRISPE (Capacity, Request, Input, Steps, Policies, Example) framework for prompting, enabling the LLM to handle complex cases like contextual merges and reconstruction of heavily distorted words.
• Automated Adjudication: An intelligent comparison script automatically approves high-confidence corrections and flags ambiguous ones for human review.
• Interactive Review GUI: A powerful Streamlit application provides a user-friendly interface for manually reviewing and correcting flagged errors, with features for bulk and individual editing.
• Robust Workflow Management: The entire process is managed through a status system within a SQLite database, making the pipeline resilient and restartable at any stage.

---

The Workflow: How It Works

The correction process is a four-stage pipeline, with each stage handled by a dedicated script.

➡️ Step 1: 1_identify_errors.py - Error Identification

This script is the foundation of the pipeline. It reads the source .csv files and identifies potential errors.

• Function: Scans text to find potential errors.
• Method:
  1. It uses Hunspell to find basic spelling mistakes.
  2. It leverages the HuSpaCy transformer model to perform deep linguistic analysis, identifying more complex issues like incorrectly split words (szeret nék) or incorrectly merged words (pirosautó).
  3. It populates the database, creating a UniqueError for each distinct error string and an ErrorOccurrence for every place it appears.
• Output: The database is filled with errors, all in new status.

➡️ Step 2: 2_run_llm_correction...py - LLM Correction

This is the core of the automated correction. It sends the identified errors to LLMs for correction suggestions.

• Function: Gets correction suggestions from two different LLMs in a sequential process.
• Method:
  1. Categorization: It first categorizes all new errors. Errors occurring frequently (above a set threshold) are marked pending_human_review. Infrequent errors are marked pending_llm.
  2. First LLM Run (e.g., gemini-2.0-flash): Processes all errors with pending_llm status. The results are saved to the database, and the status is updated to pending_second_llm.
  3. Second LLM Run (e.g., gemini-2.5-flash): Processes all errors with pending_second_llm status. The results are saved, and the status is updated to pending_comparison.
• Special Features:
  • It can be run in --single-run mode to use only one LLM.
  • The professional CRISPE prompt instructs the LLM to handle complex cases and even find additional errors in the context that the first script might have missed.

➡️ Step 3: 3_compare_and_decide.py - Automated Adjudication

This script acts as an automated quality assurance agent.

• Function: Compares the suggestions from the two LLMs and makes an automated decision.
• Method:
  1. It queries all errors in pending_comparison status.
  2. It performs "sanity checks" on both suggestions using a flexible, relative Levenshtein distance rule to ensure the correction is reasonable.
  3. Decision Logic:
     • If the corrected text from both LLMs is identical, the correction is automatically approved, and the status is set to agreement_correction.
     • If the suggestions differ in any way, or if they fail the sanity checks, the error is flagged for manual review by setting the status to pending_human_review.

➡️ Step 4: 4_human_review_app.py - Manual Review GUI

This is the human-in-the-loop part of the pipeline, powered by Streamlit.

• Function: Provides an interactive web interface for a human to review, correct, or ignore the flagged errors.
• Method:
  1. It queries all errors in pending_human_review status.
  2. For each unique error, it displays a "workspace" showing the original error, the LLM suggestions, and a list of all its occurrences with their context.
  3. The user can perform bulk actions (e.g., "correct all selected with this text") or edit each occurrence individually.
  4. The user can also choose to ignore a specific error type, which adds it to the ignored_words table to prevent it from being flagged in the future.
  5. Once the user is finished with a unique error, they click a "Finish & Next" button to finalize the changes and move to the next error.

---

Setup and Installation

1. Prerequisites:

   • Python 3.10+
   • Hunspell library. On Debian/Ubuntu, install with:

     sudo apt-get update
     sudo apt-get install libhunspell-dev

   • Hungarian Hunspell dictionaries. On Debian/Ubuntu:

     sudo apt-get install hunspell-hu

2. Clone the Repository:

   git clone <your-repo-url>
   cd <your-repo-directory>

3. Install Dependencies: It is highly recommended to use a virtual environment.

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   pip install -r requirements.txt

4. Configuration:

   • Create a config.py file in the root directory.
   • Add your Google API key and define your database file and data directory:

     # config.py
     GOOGLE_API_KEY = "YOUR_API_KEY_HERE"
     DATABASE_FILE = "corrections.db"
     DATA_DIRECTORY = "path/to/your/csv/files"
     TEXT_COLUMN_INDEX = 1 # The column index of the transcription in your CSV

5. Setup Database: Run the setup script to create the initial database schema.

   python database_setupv5.py

Usage: Running the Pipeline

Run the scripts sequentially from your terminal in the project's root directory.

1. Identify Errors:

   python scripts/1_identify_errors.py

2. Run First LLM (e.g., Gemini 2.0 Flash):

   python scripts/2_run_llm_correction_structured_testvclaude.py --model gemini-2.0-flash

3. Run Second LLM (e.g., Gemini 1.5 Pro):

   python scripts/2_run_llm_correction_structured_testvclaude.py --model gemini-1.5-pro-latest

   Note: To use only one model, add the --single-run flag to the command.

4. Compare and Decide:

   python scripts/3_compare_and_decide.py

5. Start the Manual Review App:

   streamlit run scripts/4_human_review_app.py

   Then open the provided URL (e.g., http://localhost:8501) in your browser.

---

Database Management & Queries

You can inspect the state of the database at any time using the SQLite CLI.

Connect to the database:

sqlite3 corrections.db
.headers on
.mode column

Useful Queries

Phase 1: After Error Identification

-- Count total unique error types found
SELECT COUNT(*) FROM unique_errors;

-- Count total occurrences of all errors
SELECT COUNT(*) FROM error_occurrences;

-- List the 20 most common errors
SELECT error_word, occurrence_count FROM unique_errors ORDER BY occurrence_count DESC LIMIT 20;

Phase 2: After LLM Runs

-- Check the status distribution (most important query!)
-- See how many errors are waiting for the second LLM or for comparison.
SELECT status, COUNT(*) as count FROM unique_errors GROUP BY status ORDER BY count DESC;

-- Review suggestions from the first LLM
SELECT error_word, llm_correction_g2_0, llm_correction_type_g2_0 FROM unique_errors WHERE status = 'pending_second_llm' LIMIT 10;

-- Review suggestions from both LLMs
SELECT error_word, llm_correction_g2_0, llm_correction_g2_5 FROM unique_errors WHERE status = 'pending_comparison' LIMIT 10;

-- Find errors found by the LLM in the context (if any)
SELECT error_word, llm_correction_g2_5, notes FROM unique_errors WHERE status = 'llm_generated';

Phase 3: After Comparison

-- How many corrections were auto-approved?
SELECT COUNT(*) FROM unique_errors WHERE status = 'agreement_correction';

-- How many items are now waiting for human review?
SELECT COUNT(*) FROM unique_errors WHERE status = 'pending_human_review';

-- Show 10 items to be reviewed by a human
SELECT id, error_word, llm_correction_g2_0, llm_correction_g2_5, notes FROM unique_errors WHERE status = 'pending_human_review' LIMIT 10;

Future Development & Fine-Tuning

This project has a solid foundation, but there are many avenues for future improvement:

• Final "Apply Corrections" Script: A 5_apply_corrections.py script is needed to take the final_correction from the database and write the changes back to the original CSV files.
• Lemmatization for Ignore List: Instead of ignoring exact words, the system could ignore word stems (lemmas) to automatically handle all inflected forms (e.g., ignoring "Budapest" would also ignore "Budapesten", "Budapestre", etc.).
• GUI Database Interaction: The Streamlit app's database writing logic could be made more robust to handle real-time updates for each individual occurrence, not just at the UniqueError level.
• Model Performance Tracking: Log and analyze the agreement/disagreement rates between the two LLMs to evaluate their performance and cost-effectiveness over time.
• Smarter Error Identification: The 1_identify_errors.py script could be enhanced with n-gram analysis or other statistical methods to find unlikely word combinations, even if the individual words are spelled correctly.

---

❤️ Support the Project

If you find this project useful and want to support its development, you can buy me a coffee! Any contribution is greatly appreciated.

---

License

This project is licensed under the MIT License. See the LICENSE file for details.