Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 35 additions & 78 deletions uc-0a/README.md
Original file line number Diff line number Diff line change
@@ -1,104 +1,61 @@
# UC-0A — Complaint Classifier

**Framework:** R.I.C.E · CRAFT
**Done with your facilitator**
**Core failure modes:** Taxonomy drift · Severity blindness · Missing justification · Hallucinated sub-categories · False confidence on ambiguity

---

## Scenario
## Your Input File
```
../data/city-test-files/test_[your-city].csv
```
15 rows per city. `category` and `priority_flag` columns are stripped — you must classify them.

The City Operations team receives hundreds of complaints per week.
An AI classifier reads each complaint and outputs a category, priority,
reason, and flag. The output feeds the Director's dashboard every Monday.
## Your Output File
```
uc-0a/results_[your-city].csv
```

A naive prompt produces a classifier that invents category names, misses
urgent complaints involving children and injuries, and gives confident
answers on genuinely ambiguous inputs.
## Run Command
```bash
python classifier.py \
--input ../data/city-test-files/test_pune.csv \
--output results_pune.csv
```

---

## The Five Failure Modes
## Classification Schema — Your Enforcement Must Reference These Exactly

| # | Failure Mode | What it looks like |
| Field | Allowed values | Rule |
|---|---|---|
| 1 | **Taxonomy drift** | Same complaint type gets different category names across rows |
| 2 | **Severity blindness** | "Child fell near school" → Priority: Standard |
| 3 | **Missing justification** | No reason field in output |
| 4 | **Hallucinated sub-categories** | AI outputs "Pedestrian Safety Incident" — not in schema |
| 5 | **False confidence on ambiguity** | Ambiguous complaint classified confidently without NEEDS_REVIEW |

---

## Classification Schema

| Field | Allowed values |
|---|---|
| `category` | Pothole · Flooding · Streetlight · Waste · Noise · Road Damage · Heritage Damage · Heat Hazard · Drain Blockage · Other |
| `priority` | Urgent · Standard · Low |
| `reason` | One sentence citing specific words from the description |
| `flag` | NEEDS_REVIEW or blank |
| `category` | Pothole · Flooding · Streetlight · Waste · Noise · Road Damage · Heritage Damage · Heat Hazard · Drain Blockage · Other | Exact strings only — no variations |
| `priority` | Urgent · Standard · Low | Urgent if severity keywords present |
| `reason` | One sentence | Must cite specific words from description |
| `flag` | NEEDS_REVIEW or blank | Set when category is genuinely ambiguous |

**Severity keywords — must trigger Urgent:**
`injury` · `child` · `school` · `hospital` · `ambulance` · `fire` · `hazard` · `fell` · `collapse`

---

## Enforcement Rules Your agents.md Must Include

1. Category must be exactly one value from the allowed list. No variations.
2. Priority must be Urgent if description contains any severity keyword.
3. Every output row must include a reason field citing specific words from the description.
4. If category cannot be determined confidently — output `category: Other` and `flag: NEEDS_REVIEW`.
5. Never invent category names outside the allowed list.
**Severity keywords that must trigger Urgent:**
`injury`, `child`, `school`, `hospital`, `ambulance`, `fire`, `hazard`, `fell`, `collapse`

---

## Skills to Define in skills.md

### `classify_complaint`
- Input: one complaint row (dict with description, location fields)
- Output: dict with category, priority, reason, flag
- Error handling: vague/short descriptions → Other + NEEDS_REVIEW

### `batch_classify`
- Input: path to test CSV file
- Output: path to results CSV file
- Error handling: malformed rows logged and skipped, processing continues
- `classify_complaint` — one complaint row in → category + priority + reason + flag out
- `batch_classify` — reads input CSV, applies classify_complaint per row, writes output CSV

---

## Input File

```
data/city-test-files/test_[your-city].csv
```

Choose the file matching your city: pune / hyderabad / kolkata / ahmedabad

## Output File

```
uc-0a/results_[your-city].csv
```

## Run Command

```bash
cd uc-0a
python3 classifier.py --input ../data/city-test-files/test_pune.csv \
--output results_pune.csv
```
## What Will Fail From the Naive Prompt
Run `"Classify this citizen complaint by category and priority."` first.
Then look for:
1. Category names that vary across rows for the same type of complaint
2. Injury/child/school complaints classified as Standard instead of Urgent
3. No reason field in the output
4. Category names that are not in the allowed list above
5. Confident classification on genuinely ambiguous complaints

---

## CRAFT Commit Formula

## Commit Formula
```
UC-0A Fix [failure mode]: [why it failed] → [what you changed]
```

Examples:
```
UC-0A Fix severity blindness: no keywords in enforcement → added injury/child/school/hospital triggers
UC-0A Fix taxonomy drift: no fixed enum → restricted category to allowed list only
```
28 changes: 8 additions & 20 deletions uc-0a/agents.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,15 @@
# agents.md — UC-0A Complaint Classifier
# INSTRUCTIONS:
# 1. Open your AI tool
# 2. Paste the full contents of uc-0a/README.md
# 3. Use this prompt:
# "Read this UC README. Using the R.I.C.E framework, generate an
# agents.md YAML with four fields: role, intent, context, enforcement.
# Enforcement must include every rule listed under
# 'Enforcement Rules Your agents.md Must Include'.
# Output only valid YAML."
# 4. Paste the output below

role: >
[FILL IN]
You are an expert citizen complaint classifier. Your operational boundary is strictly limited to analyzing incoming complaint descriptions to categorize them, determine urgency, and provide explicit reasoning based solely on the text provided.

intent: >
[FILL IN]
Produce a structured, verifiable classification containing an exact category match, a priority level (Urgent, Standard, or Low), a one-sentence reason citing original words from the complaint, and an optional flag.

context: >
[FILL IN]
You are allowed to use ONLY the provided complaint description text. You must not use external knowledge to infer or extrapolate details. You are explicitly excluded from creating new or varying categories, hallucinating sub-categories, or showing false confidence on ambiguous issues.

enforcement:
- "[FILL IN: category enum rule]"
- "[FILL IN: severity keyword rule — list the keywords]"
- "[FILL IN: reason field rule]"
- "[FILL IN: ambiguity refusal rule]"
- "[FILL IN: no invented categories rule]"
- "Category must be exactly one of: Pothole, Flooding, Streetlight, Waste, Noise, Road Damage, Heritage Damage, Heat Hazard, Drain Blockage, Other."
- "Priority must be Urgent if the description contains any of the following severity keywords: injury, child, school, hospital, ambulance, fire, hazard, fell, collapse. Otherwise, assign priority as Standard or Low."
- "Every output row must include a reason field that is exactly one sentence and explicitly cites specific words from the description."
- "If the category cannot be determined from the description alone, or is genuinely ambiguous, output category: Other, flag: NEEDS_REVIEW and Priority: Low."
- "You cannot use OpenAI or external API."
135 changes: 117 additions & 18 deletions uc-0a/classifier.py
Original file line number Diff line number Diff line change
@@ -1,31 +1,130 @@
"""
UC-0A — Complaint Classifier
classifier.py — Starter file

Build this using your AI coding tool:
1. Share agents.md, skills.md, and uc-0a/README.md
2. Ask the AI to implement this file
3. Run: python3 classifier.py --input ../data/city-test-files/test_pune.csv \
--output results_pune.csv
"""
import argparse
import csv
import os

def classify_complaint(row: dict) -> dict:
"""
Classify a single complaint row.
Returns dict with: complaint_id, category, priority, reason, flag
Classify a single complaint row using strict python string matching
based on the RICE prompt rules from agents.md. No external APIs allowed.
"""
raise NotImplementedError("Build this using your AI tool + agents.md")
description = row.get("description", "").lower()

# 1. Determine priority via severity keywords
severity_keywords = ["injury", "child", "school", "hospital", "ambulance", "fire", "hazard", "fell", "collapse"]
matched_severity = next((kw for kw in severity_keywords if kw in description), None)

# 2. Determine category via strict string matching
# Categories: Pothole, Flooding, Streetlight, Waste, Noise, Road Damage, Heritage Damage, Heat Hazard, Drain Blockage, Other.
keyword_map = {
"pothole": "Pothole",
"flood": "Flooding",
"streetlight": "Streetlight",
"lights out": "Streetlight",
"dark": "Streetlight",
"garbage": "Waste",
"waste": "Waste",
"dead animal": "Waste",
"music": "Noise",
"noise": "Noise",
"road surface": "Road Damage",
"tiles broken": "Road Damage",
"crack": "Road Damage",
"heritage": "Heritage Damage",
"heat": "Heat Hazard",
"drain": "Drain Blockage"
}

category = "Other"
matched_cat_word = ""

for kw, cat in keyword_map.items():
if kw in description:
category = cat
matched_cat_word = kw
break

# 3. Apply constraints
priority = "Standard"
reason = ""
flag = ""

# Enforcement: "If the category cannot be determined from the description alone, or is genuinely ambiguous,
# output category: Other, flag: NEEDS_REVIEW and Priority: Low."
if category == "Other":
flag = "NEEDS_REVIEW"
priority = "Low"
reason = "Category could not be determined unambiguously from the description text."
else:
# Enforcement: "Priority must be Urgent if the description contains any of the following severity keywords..."
if matched_severity:
priority = "Urgent"
reason = f"Classified as Urgent because description mentions '{matched_cat_word}' and severity keyword '{matched_severity}'."
else:
priority = "Standard"
reason = f"Classified primarily because the description explicitly cites '{matched_cat_word}'."

return {
"category": category,
"priority": priority,
"reason": reason,
"flag": flag
}


def batch_classify(input_path: str, output_path: str):
"""Read input CSV, classify each row, write results CSV."""
raise NotImplementedError("Build this using your AI tool + agents.md")
"""
Reads an input CSV of complaints, applies classify_complaint to each row,
and writes the results to an output CSV.
"""
results = []
if not os.path.exists(input_path):
print(f"Error: Input file {input_path} not found.")
return

with open(input_path, mode='r', encoding='utf-8-sig') as infile:
reader = csv.DictReader(infile)
if not reader.fieldnames:
print("Error: Input CSV has no headers.")
return

fieldnames = list(reader.fieldnames)

# Add output columns from skills.md
for col in ["category", "priority", "reason", "flag"]:
if col not in fieldnames:
fieldnames.append(col)

for row in reader:
try:
classification_result = classify_complaint(row)
merged_row = {**row, **classification_result}
results.append(merged_row)
except Exception as e:
print(f"Failed to process row ID {row.get('complaint_id', 'UNKNOWN')}: {e}")
# Create an error record
error_row = {**row}
error_row["category"] = "Other"
error_row["priority"] = "Low"
error_row["reason"] = f"Processing Error: {e}"
error_row["flag"] = "NEEDS_REVIEW"
results.append(error_row)

if not results:
print("No rows processed.")
return

writer_fieldnames = list(results[0].keys())
with open(output_path, mode='w', encoding='utf-8', newline='') as outfile:
writer = csv.DictWriter(outfile, fieldnames=writer_fieldnames)
writer.writeheader()
writer.writerows(results)

print(f"Successfully processed {len(results)} rows.")


if __name__ == "__main__":
parser = argparse.ArgumentParser(description="UC-0A Complaint Classifier")
parser.add_argument("--input", required=True)
parser.add_argument("--output", required=True)
parser.add_argument("--input", required=True, help="Path to test_[city].csv")
parser.add_argument("--output", required=True, help="Path to write results CSV")
args = parser.parse_args()
batch_classify(args.input, args.output)
print(f"Done. Results written to {args.output}")
16 changes: 16 additions & 0 deletions uc-0a/results_ahmedabad.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
complaint_id,date_raised,city,ward,location,description,reported_by,days_open,category,priority,reason,flag
AM-202401,2024-04-24,Ahmedabad,Ward 13 – Sabarmati,Kankaria Lake promenade,Tarmac surface melting at 44°C. Footwear sticking. Park users unsafe.,Email,1,Other,Low,Category could not be determined unambiguously from the description text.,NEEDS_REVIEW
AM-202402,2024-04-23,Ahmedabad,Ward 12 – Maninagar,"Naroda Industrial Area, main gate",Metal bus shelter reaching dangerous temperatures. Commuters refusing to use.,Citizen Portal,4,Other,Low,Category could not be determined unambiguously from the description text.,NEEDS_REVIEW
AM-202405,2024-05-08,Ahmedabad,Ward 11 – Vastrapur,"Sabarmati Riverfront, North section",Dead trees with split branches. Fall risk to walkers. 3 trees affected.,WhatsApp Helpline,19,Other,Low,Category could not be determined unambiguously from the description text.,NEEDS_REVIEW
AM-202406,2024-04-21,Ahmedabad,Ward 15 – Chandkheda,"Chandkheda, Sector 22 park",Irrigation system broken. Grass dying in heatwave conditions.,Phone Helpline,18,Heat Hazard,Standard,Classified primarily because the description explicitly cites 'heat'.,
AM-202407,2024-04-25,Ahmedabad,Ward 15 – Chandkheda,"Thaltej, residential park",Broken bench and upturned paving. Child injured last week.,Email,20,Other,Low,Category could not be determined unambiguously from the description text.,NEEDS_REVIEW
AM-202410,2024-05-02,Ahmedabad,Ward 12 – Maninagar,SG Highway near Prahladnagar,Pothole on main highway causing morning rush lane closure.,Ward Office Walk-in,4,Pothole,Standard,Classified primarily because the description explicitly cites 'pothole'.,
AM-202414,2024-04-20,Ahmedabad,Ward 14 – Odhav,"Jodhpur Village, off SG Highway",Residential colony unlit after 9pm. Wiring theft reported.,Citizen Portal,10,Other,Low,Category could not be determined unambiguously from the description text.,NEEDS_REVIEW
AM-202417,2024-05-08,Ahmedabad,Ward 13 – Sabarmati,"Manek Chowk, Old City",Night market waste not cleared before morning. Heritage area affected.,Councillor Referral,21,Waste,Standard,Classified primarily because the description explicitly cites 'waste'.,
AM-202421,2024-05-03,Ahmedabad,Ward 14 – Odhav,"CG Road, commercial zone",Club music audible at residential buildings at 2am.,Citizen Portal,18,Noise,Standard,Classified primarily because the description explicitly cites 'music'.,
AM-202424,2024-05-04,Ahmedabad,Ward 11 – Vastrapur,"Shahibaug, near zoo",Zoo approach road surface bubbling at 45°C. Visitor complaints.,Councillor Referral,19,Road Damage,Standard,Classified primarily because the description explicitly cites 'road surface'.,
AM-202429,2024-04-24,Ahmedabad,Ward 11 – Vastrapur,Ellis Bridge walking track,River walk surface temperature unbearable. Installed temperature reads 52°C.,Councillor Referral,3,Other,Low,Category could not be determined unambiguously from the description text.,NEEDS_REVIEW
AM-202431,2024-04-26,Ahmedabad,Ward 14 – Odhav,"Paldi, Ratanpol area",Old city road subsidence near ancient step well. Heritage concern.,Citizen Portal,18,Heritage Damage,Standard,Classified primarily because the description explicitly cites 'heritage'.,
AM-202435,2024-05-07,Ahmedabad,Ward 11 – Vastrapur,"New CG Road, Chandkheda",Black metal road dividers storing heat. Motorists reporting burns on contact.,WhatsApp Helpline,16,Heat Hazard,Standard,Classified primarily because the description explicitly cites 'heat'.,
AM-202444,2024-05-15,Ahmedabad,Ward 12 – Maninagar,"Jivraj Park, commercial area",Restaurant waste bins overflowing on Sunday night. Health risk.,Councillor Referral,20,Waste,Standard,Classified primarily because the description explicitly cites 'waste'.,
AM-202445,2024-04-19,Ahmedabad,Ward 13 – Sabarmati,Ranip Bus Rapid Transit stop,BRT shelter roof glass broken. Users exposed to full sun.,Councillor Referral,9,Other,Low,Category could not be determined unambiguously from the description text.,NEEDS_REVIEW
Loading