Automating Generation of Long-Form Queries

Official code for the SIGIR 2026 paper:

Automating Generation of Long-Form Queries
Shivani Upadhyay, Daniel Campos, Nandan Thakur, Ronak Pradeep, Nick Craswell, Jimmy Lin
SIGIR '26 · Melbourne, VIC, Australia · ACM DL

---

Overview

This repository provides a pipeline for automatically generating human-style long-form queries (narratives) from raw short search queries.

---

Setup

1. Clone the repository

git clone https://github.com/castorini/narr-gen.git
cd narr-gen

2. Install dependencies

pip install -r requirements.txt
python -m spacy download en_core_web_sm

3. Configure environment variables

cp .env.example .env

Edit .env:

OPEN_AI_API_KEY=your_openai_key
GCLOUD_PROJECT=your_gcp_project_id
GCLOUD_REGION=us-central1

# Optional — only needed with --use_azure
AZURE_OPENAI_API_KEY=
AZURE_OPENAI_API_VERSION=
AZURE_OPENAI_API_BASE=

---

Usage

Cluster and Generate Narratives

Place raw query topic files (.txt, tab-separated qid\tquery) in data/raw_topics/.

bash scripts/narr_create.sh [--min_cluster_size 4] [--max_clusters 50]

Produces data/narratives/clusters_raw.json and data/narratives/narratives_generated.json, each cluster containing four narrative variants:

Field	Description
gpt_narrative	GPT-4.1 original
gemini_narrative	Gemini 2.5 Flash original
gemini_rw_gpt_narrative	Gemini rewrite of GPT's output
gpt_rw_gemini_narrative	GPT rewrite of Gemini's output

Narrative Selection

For TREC 2025 RAG narratives generation manual selection process was adopted.

Human-selected data: A human-selected narratives file is included in this repository at data/narratives/trec25_rag_narratives_selected.json and can be used directly for style transformation and evaluation without running previous steps.

Style Transformation (Optional)

Rewrites selected narratives into the TREC 2006 ciQA third-person analyst style (1–2 sentences).

bash scripts/style_transform.sh [--input FILE] [--field FIELD]

Default input: data/narratives/trec25_rag_narratives_selected.json
Default output: data/style_transformed/narratives_ciqa_style.json

Evaluation

Computes linguistic and semantic complexity features and runs Mann-Whitney U tests with Cohen's d, comparing generated narratives against the human ciQA baseline (stored in data/narratives/human_narratives.json).

bash scripts/eval.sh [--generated FILE] [--narrative_field FIELD] \
                     [--style_transformed FILE] [--rag24_topics FILE]

Default output: results/eval_summary.csv

Using an External Narrative File

If you have narratives in a simple [{"id": "...", "narrative": "..."}] format, convert them for use with the pipeline:

python scripts/convert_narratives.py \
  --input  my_narratives.json \
  --output data/narratives/narratives_selected.json

---

Data Format

Input: Raw Topic Files (data/raw_topics/*.txt)

Tab-separated, one query per line:

q1001	what does a grassroots organization mean
q1002	how can nation building be promoted in a community
q1003	meaning of community in science

Output: Generated Narratives (data/narratives/narratives_generated.json)

[
  {
    "topic_id": "0",
    "original": ["what does a grassroots organization mean", "..."],
    "gpt_narrative": "I want a thorough understanding of ...",
    "gemini_narrative": "I'm looking for a comprehensive understanding of ...",
    "gemini_rw_gpt_narrative": "I want a thorough understanding of ...",
    "gpt_rw_gemini_narrative": "I'm interested in understanding ..."
  }
]

---

Citation

@inproceedings{upadhyay2026longform,
  author    = {Upadhyay, Shivani and Campos, Daniel and Thakur, Nandan and
               Pradeep, Ronak and Craswell, Nick and Lin, Jimmy},
  title     = {Automating Generation of Long-Form Queries},
  booktitle = {Proceedings of the 49th International ACM SIGIR Conference on
               Research and Development in Information Retrieval},
  series    = {SIGIR '26},
  year      = {2026},
  location  = {Melbourne, VIC, Australia},
  doi       = {10.1145/3805712.3809917},
  publisher = {ACM}
}

---

License

This project is licensed under the Creative Commons Attribution 4.0 International License.