📊 CSV Data Cleaning Copilot (GenAI Project)

🚀 Live Frontend URL

https://data-cleaner-frontend.vercel.app/

🚀 Backend API URL (HuggingFace Space – Mandatory)

https://subhankar7-datacleanerapi.hf.space

---

🧩 Overview — Problem Statement

Government departments and citizen-service datasets often contain:

• Missing values
• Invalid data formats
• Outliers
• Typographical errors
• Duplicate rows
• Irregular category values

Cleaning such data manually is slow and error-prone.

✅ Solution

This project is an AI-powered CSV Data Cleaning Copilot that:

• Detects data issues automatically
• Suggests intelligent, LLM-powered corrections
• Allows users to accept or reject fixes
• Cleans and exports corrected CSV files
• Shows visual charts and column profiling

---

⚙️ Features

• 📂 Upload CSV (20–200 rows)
• 🔍 Automated issue detection
• 🤖 LLM-based correction suggestions
• 📊 Visual issue distribution charts
• 📄 Column profiling summary
• ✔ Apply selected fixes
• ⬇️ Download cleaned CSV
• 🔌 Fully deployed backend (HF Spaces) + frontend (Vercel/Netlify/GitHub Pages)

---

🏗 Architecture Diagram

User → React Frontend → FastAPI Backend → LLM (HuggingFace zephyr-7b) ↑ Rule-based Analyzer

yaml Copy code

---

📁 Folder Structure

backend/ ├── app.py ├── rules.py ├── llm_chain.py ├── requirements.txt ├── Dockerfile

frontend/ ├── src/components/ ├── src/App.js ├── package.json ├── build/

yaml Copy code

---

🛠️ Installation (Local Development)

1️⃣ Backend Setup (FastAPI)

cd backend pip install -r requirements.txt uvicorn app:app --reload

shell Copy code

2️⃣ Frontend Setup (React)

cd frontend npm install npm start

yaml Copy code

---

🧪 Sample Test CSV

first_name,last_name,email,phone,date john,doe,john@gmial.com,999999,2020/55/12 sam,,sam@example,12345,2020-22-10 mike,ross,mikeross@,987654321999,32-10-2021

yaml Copy code

The system will detect:

• Missing last_name
• Invalid email (“gmial” → “gmail”)
• Invalid phone (too short/long)
• Invalid date format
• Outliers
• Duplicates

---

🤖 GenAI Workflow

1. Hallucination Reduction

• Prompt forces LLM to output only the corrected value
• No verbose responses
• If LLM fails → fallback deterministic correction

2. Token Optimization

• Short 20-token limit
• Compact prompts
• Single shared inference client

3. Accuracy Improvements

• Fuzzy category correction via fuzzywuzzy
• Email parsing via email-validator
• Date parsing via dateutil
• Numeric outlier detection via IQR

4. Unique Logic

• Hybrid LLM + rule-engine
• Confidence scores (High/Medium/Low)
• UI cell-highlighting for issues
• AI auto-suggestions panel
• Accept single-fix or batch fixes

5. Future Enhancements

• Domain-specific RAG validation
• PDF summary exports
• Bulk multi-file processing
• Finetuned LLM for gov-data cleaning

---

📹 Demo Video

(Google Drive link here)

https://drive.google.com/file/d/1sBi4hMnvDWhYn4whPblLDl8DgVOGPXsm/view?usp=drivesdk

🧑‍💻 Author

Subhankar Swain
Full-Stack GenAI Developer