You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Ensure there's no trailing newline if passages_str is empty, though unlikely.
18
+
passages_str=passages_str.strip()
19
+
if"gpt-4.1"inself.model_name:
20
+
prompt=f"""
21
+
Given a query and a list of passages, your task is to re-rank these passages based on their relevance to the query. {TASK2PROMPT[self.task]}
22
+
23
+
Please perform the following steps:
24
+
1. **Understand the Query**: First, carefully read and understand the user's query to identify the core information need.
25
+
2. **Analyze Each Passage**: For each passage, critically evaluate its content and determine how well it addresses the query. Consider factors like:
26
+
- Directness of the answer
27
+
- Completeness of the information
28
+
- Presence of supporting evidence or details
29
+
- Absence of irrelevant or distracting information
30
+
3. **Compare and Contrast**: Compare the passages against each other. Identify which passages are more relevant and why. Note any subtle differences in relevance.
31
+
4. **Reasoning for Ranking**: Explicitly state your reasoning for the rank you assign to each passage. Explain why a passage is ranked higher or lower than others. This step-by-step thought process is crucial.
32
+
5. **Assign Ranks**: Based on your analysis and reasoning, assign a unique rank to each passage, starting from 1 for the most relevant.
33
+
34
+
**Output Format:**
35
+
Your final output must be a list of ranks, corresponding to the original order of the passages. For example, if there are 3 passages, and you decide the second passage is most relevant, the first is second most relevant, and the third is least relevant, your output should be:
36
+
[2] > [1] > [3]
37
+
38
+
No other text or explanation should be present in the final output, only the list of ranks.
39
+
40
+
**Query:**
41
+
{query}
42
+
43
+
**Passages:**
44
+
{passages_str}
45
+
46
+
**Your Step-by-Step Reasoning (before the final output list):**
47
+
[Provide your detailed thought process here, analyzing each passage and justifying your ranking decisions. This section will not be part of the final output but helps in arriving at the correct ranking.]
48
+
49
+
**Ranks (only this list will be parsed):**
50
+
"""
51
+
else:
52
+
prompt=f"""
53
+
Given a query and a list of passages, your task is to re-rank these passages based on their relevance to the query. {TASK2PROMPT[self.task]}
54
+
55
+
Please perform the following steps:
56
+
1. **Understand the Query**: First, carefully read and understand the user's query to identify the core information need.
57
+
2. **Analyze Each Passage**: For each passage, critically evaluate its content and determine how well it addresses the query. Consider factors like:
58
+
- Directness of the answer
59
+
- Completeness of the information
60
+
- Presence of supporting evidence or details
61
+
- Absence of irrelevant or distracting information
62
+
3. **Compare and Contrast**: Compare the passages against each other. Identify which passages are more relevant and why. Note any subtle differences in relevance.
63
+
4. **Assign Ranks**: Based on your analysis and reasoning, assign a unique rank to each passage, starting from 1 for the most relevant.
64
+
65
+
**Output Format:**
66
+
Your final output must be a list of ranks, corresponding to the original order of the passages. For example, if there are 3 passages, and you decide the second passage is most relevant, the first is second most relevant, and the third is least relevant, your output should be:
67
+
[2] > [1] > [3]
68
+
69
+
No other text or explanation should be present in the final output, only the list of ranks.
70
+
71
+
**Query:**
72
+
{query}
73
+
74
+
**Passages:**
75
+
{passages_str}
76
+
"""
77
+
returnprompt
78
+
79
+
# Example usage (optional, for testing)
80
+
if__name__=='__main__':
81
+
logging.basicConfig(level=logging.DEBUG) # Changed to INFO for less verbose default
82
+
# Ensure OPENAI_API_KEY is set as an environment variable for testing
83
+
# export OPENAI_API_KEY='your_openai_api_key'
84
+
85
+
# Example documents (doc_id, doc_text)
86
+
docs_to_rerank= [
87
+
("doc1", "The quick brown fox jumps over the lazy dog."),
88
+
("doc2", "A lazy dog sits under the tree."),
89
+
("doc3", "Foxes are omnivorous mammals belonging to several genera of the family Canidae."),
90
+
("doc4", "The study of dogs is known as cynology."),
91
+
("doc5", "Quick feet help the fox escape predators."),
92
+
("doc6", "Brown bears are found in North America."),
93
+
("doc7", "Lazy rivers flow slowly."),
94
+
("doc8", "Jumping requires strong leg muscles."),
95
+
("doc9", "The quicksand trapped the unlucky traveler."),
96
+
("doc10", "Dog breeds vary greatly in size and temperament."),
97
+
]
98
+
query_str="information about quick brown foxes and lazy dogs"
99
+
100
+
try:
101
+
# Retrieve API key from environment variable
102
+
# The TogetherListwiseReranker base class should handle the api_key and api_base arguments
103
+
# if they are passed to its __init__ method.
104
+
# Since OpenAIReranker now uses parent's __init__, we can pass these.
105
+
openai_api_key=os.getenv("OPENAI_API_KEY")
106
+
ifnotopenai_api_key:
107
+
logger=logging.getLogger(__name__)
108
+
logger.warning("OPENAI_API_KEY environment variable not set. API calls might fail.")
109
+
# Or raise ValueError("OPENAI_API_KEY must be set for testing")
110
+
111
+
# Instantiate OpenAIReranker
112
+
# model_name defaults to "gpt-4-turbo-preview" in parent if not specified.
113
+
# The parent __init__ takes api_key and api_base (which is called base_url there)
114
+
ranker=OpenAIReranker(
115
+
model_name="o3-mini",
116
+
window_size=4,
117
+
stride=2,
118
+
api_key=openai_api_key,
119
+
base_url=None,
120
+
task="biology"# Provide a task value as parent expects it
121
+
)
122
+
123
+
logger=logging.getLogger(__name__)
124
+
logger.info(f"Testing OpenAIReranker with model: {ranker.model_name}")
parser.add_argument('--llm', type=str, default=None, help="Model name for Claude, OpenAI, or Sentence Transformer rerankers (e.g., 'claude-3-opus-20240229', 'gpt-4-turbo', 'mixedbread-ai/mxbai-rerank-xsmall-v1')")
238
+
parser.add_argument('--together_model', type=str, default=None, help="Model name for TogetherListwiseReranker (e.g., 'mistralai/Mixtral-8x7B-Instruct-v0.1')")
0 commit comments