CoRUS

Language model users often embed personal and social context in their questions. The asker's role -- implicit in how the question is framed -- creates specific needs for an appropriate response. However, most evaluations, while capturing the model's capability to respond, often ignore who is asking.

CoRUS (Community-driven Roles for User-centric Question Simulation) is a framework that embeds role-based context implicitly into a given question for user-centric evaluation of conversational AI. The framework:

1. Extracts information-seeking questions from Reddit posts (r/opiatesrecovery)
2. Defines user roles (Patient, Caregiver, Healthcare Practitioner) via a taxonomy with four facets: role, goal, behaviour, experience (the taxonomy is extracted from Reddit posts)
3. Simulates role-specific query reformulations using a language model
4. Queries models to be evaluated with the simulated queries
5. Evaluates responses on knowledge and support using pretrained BERT classifiers, readability using Flesch score, and abstention using keyword and prefix based classifier

This repository contains the full pipeline to reproduce our results. CoRUS can be applied to any community-sourced text corpus to enable role-based evaluation in new domains.

• 📄 Paper: Who's Asking? Simulating Role-Based Questions for Conversational AI Evaluation (ACL Findings 2026)
• 🤗 Dataset: navreeetkaur/CoRUS on HuggingFace

---

🗂️ Repository Structure

CoRUS/
├── src/
│   ├── utils.py               # Shared utilities
│   ├── config.yaml            # Model and pipeline configuration
│   ├── taxonomy.yaml          # Persona taxonomy (roles + facets)
│   ├── prompts.yaml           # Prompts
│   ├── process_questions.py   # Step 1: extract and quality-check questions from Reddit
│   ├── simulate_queries.py    # Step 2: simulate role-based queries
│   ├── batch_process.py       # Step 2 (large scale): Batch API
│   ├── llm_judge.py           # Step 3: evaluate simulated query quality
│   ├── get_llm_responses.py   # Step 4: collect LLM responses
│   ├── analyze_responses.py   # Step 5: score and analyze responses
│   └── plot_results.ipynb     # Step 6: plots and figures
├── data/
│   ├── reddit/                # Raw Reddit posts
│   ├── extracted_questions.csv
│   ├── role-facet-summaries   # facet-based summaries 
│   ├── simulated_queries/     # Step 2 output (per simulation model)
│   ├── responses/             # Step 4 output (LLM responses)
│   └── 10dims/                # Step 5 output (scored CSVs)
├── .env.example               # API key template
├── requirements.txt
└── LICENSE

---

⚙️ Setup

1. Install dependencies

Requires Python 3.10+.

git clone https://github.com/navreeetkaur/CoRUS.git
cd CoRUS
pip install -r requirements.txt

The knowledge and social support classifiers (Step 5) require pretrained BERT weights from minjechoi/10dimensions. Clone that repo alongside this one and follow its weights/README.md to download the weights.

2. Configure API keys

Copy .env.example to .env and fill in your API keys:

cp .env.example .env

ANTHROPIC_API_KEY=...
OPENAI_API_KEY=...
GOOGLE_API_KEY=...
NEBIUS_API_KEY=...

---

📦 Provided Data

There is one sim_query_{model}_10dims.csv per evaluation model (e.g. gpt, gemini, llama8B, llama70B, openbiollm). Each row is a single (query, role, model-response) triple with its scores.

Column	Description
post_id	ID of the source Reddit post the question was extracted from
extracted_query	Role-agnostic question extracted from the post (Step 1)
simulated_query	Role-specific reformulation of extracted_query (Step 2)
simulated_role	Role used for the reformulation (Patient, Caregiver Healthcare_Practitioner)
model_response	The evaluation model's response to simulated_query (Step 4)
model	Display name of the evaluation model that produced the response
model_social_support	Social support score for the model_response from the BERT classifier (0-1)
model_knowledge	Knowledge score for the model_response from the BERT classifier (0-1)
model_flesch_score	Flesch reading ease score for the model_response
model_flesch_score_normalized	model_flesch_score rescaled to 0-1

---

🔄 Pipeline

The persona taxonomy (src/taxonomy.yaml) was built using a domain-agnostic LLM-based summarization and clustering pipeline for inducing role/facet taxonomies from text. It's code is in a separate repo, please follow its instructions for building a taxonomy for your own domain.

Step 1: Extract and process questions from Reddit posts

Extract questions, check for role-revealing language, and reformulate as role-agnostic queries:

python src/process_questions.py \
  --input data/reddit/r_opiatesrecovery_posts.jsonl.csv \
  --output data/extracted_questions.csv \
  --extract_model gpt-3.5-turbo \
  --quality_model gpt-4o-mini-2024-07-18

Note: The models above are what we used in the paper. gpt-3.5-turbo is now deprecated by OpenAI; substitute a current model (e.g., gpt-4o-mini) if needed. The pre-extracted output is already provided in the data/extracted_questions.csv, so this step can be skipped entirely.

Output columns: index, post_id, title, body, is_question, identify_question, extracted_query, reveals_role.

Step 2: Simulate role-based queries

Run query simulation for all roles using the taxonomy:

python src/simulate_queries.py \
  --input_csv data/extracted_questions.csv \
  --output_dir data/simulated_queries \
  --facet_summaries_dir data/role-facet-summaries

--facet_summaries_dir is only required when is_fewshot: true in config.yaml. When is_fewshot: false (the default), it can be omitted.

The input CSV must have columns post_id and extracted_query. The output is saved in output_dir with the original columns plus one <Role>_query column per role defined in taxonomy.yaml (e.g., Patient_query, Caregiver_query, Healthcare_Practitioner_query, Community_Participant_query).

For large-scale runs, use the Anthropic Batch API (requires is_fewshot: true and facet summaries from Step 0). Batches can take a while, so the flow is two steps — submit, then fetch later.

Submit (writes batch_ids.json to --output_dir and exits):

python src/batch_process.py \
  --input_csv data/extracted_questions.csv \
  --facet_summaries_dir data/role-facet-summaries \
  --output_dir data/simulated_queries

Fetch (re-run with the same args plus --batch_ids_file once batches finish):

python src/batch_process.py \
  --input_csv data/extracted_questions.csv \
  --facet_summaries_dir data/role-facet-summaries \
  --output_dir data/simulated_queries \
  --batch_ids_file data/simulated_queries/batch_ids.json

Fetch is one-shot: if any batch is still in progress it prints status and exits without writing the CSV — safe to re-run. If you already have raw .jsonl files (e.g., from the Anthropic console), use --jsonl_dir <path> instead.

To run ablations across facet subsets and models:

python src/simulate_queries.py \
  --input_csv data/extracted_questions.csv \
  --output_dir data/simulated_queries/ablations \
  --run_ablations

Step 3: Evaluate quality of simulated queries using LLM-judge

Use the per-model directory written by Step 2 (e.g., data/simulated_queries/claude/ for the default simulation, or data/simulated_queries/ablations/claude/ if you ran --run_ablations).

python src/llm_judge.py \
  --input_dir data/simulated_queries/claude \
  --output_dir data/simulated_queries/llm_judge_evals

Default judge model is gpt-4o-mini-2024-07-18. To change, use --model.

Step 4: Collect LLM responses

Pass the per-model directory from Step 2 (e.g., data/simulated_queries/claude); the script picks up the single CSV inside it and queries every evaluation model.

python src/get_llm_responses.py \
  --input_dir data/simulated_queries/claude \
  --output_dir data/responses

Output: data/responses/full_llm_responses.csv, with one {key}_response column per evaluation model (e.g., gpt_response, llama8B_response). Use --models <key1> <key2> to query a subset.

Step 5: Score and analyze responses

Skip this step if you're using the provided data/10dims/ CSVs — go straight to Step 6.

Scores each response on knowledge and social_support (BERT classifiers from minjechoi/10dimensions), plus Flesch readability, then reports means with bootstrapped CIs across personas.

Setup: clone the 10dimensions repo as a sibling of this repo, then download BERT weights for knowledge and social_support into 10dimensions/weights/BERT/{dim}/pytorch_model.bin (see that repo's weights/README.md).

python src/analyze_responses.py \
  --responses_path data/10dims \
  --raw_responses data/responses/full_llm_responses.csv \
  --bert_weights_dir ../10dimensions/weights/BERT

Results are cached at data/10dims/sim_query_{model}_10dims.csv; subsequent runs skip BERT and only need --responses_path.

Step 6: Plot results

src/plot_results.ipynb reads the scored CSVs from Step 5 and produces the figures from the paper:

• box-plots for per-role distributions of knowledge, support, readability for each model.
• heatmaps of mean scores.
• scatter plot of percentage change in knowledge, support, readability w.r.t role-agnostic query with bootstrapped CIs.
• abstention proportions per role/model.

Input paths default to ../data/10dims (i.e., assume the notebook is launched from src/); override SCORED_RESPONSES_DIR at the top of the notebook if your layout differs.

---

🛠️ Configuration

Edit src/config.yaml to change:

• simulation_models — LLMs used to simulate queries (Step 2)
• all_facets — persona facets to include
• is_fewshot — use few-shot examples from cluster summaries
• thinking_mode — enable extended thinking for the simulation model
• evaluation_models — LLMs queried in Step 4 and analyzed in Step 5. Each entry has full_name, provider (openai / gemini / nebius), and display_name. Add/remove keys here to change the pipeline.

---

✏️ Citation

If you find our codebase and dataset beneficial, please consider citing our work:

@article{kaur2025whosaskingsimulatingrolebased,
      title={Who's Asking? Simulating Role-Based Questions for Conversational AI Evaluation}, 
      author={Kaur, Navreet and Ayad, Hoda and Jung, Hayoung and Mittal, Shravika and De Choudhury, Munmun and Mitra, Tanushree},
      year={2025},
      eprint={2510.16829},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.16829}, 
}

🤝 Contact

If you have any questions, please email kanavr[at]uw.edu. Thanks!