Skip to content

Commit 85cfe4d

Browse files
better629better629
authored
Merge pull request #2 from FoundationAgents/evaluation
merge evaluation into main
2 parents 3893874 + 71c8613 commit 85cfe4d

File tree

20 files changed

+1393
-1
lines changed

20 files changed

+1393
-1
lines changed

‎.gitignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,7 +165,7 @@ cython_debug/
165165
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
166166
# and can be added to the global gitignore or merged into this file. For a more nuclear
167167
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
168-
#.idea/
168+
.idea/
169169

170170
# PyPI configuration file
171171
.pypirc

‎evaluation/LongBench/LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2023 THU-KEG & Zhipu AI
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎evaluation/LongBench/README.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
We use [LongBench v1](https://github.com/THUDM/LongBench/tree/main/LongBench) to evaluate the trained model. And we delete the un-related files in the repo to avoiding confusing.
2+
3+
Since the model has thinking output, we have made appropriate updates based on the basic retention of the original information of `dataset2maxlen.json` and `dataset2prompt.json`.
4+
5+
### Install
6+
```shell
7+
conda create -n longbench python=3.10 # with independent environment
8+
conda activate longbench
9+
10+
cd CARE/evaluation/LongBench
11+
pip3 install -r requirements.txt
12+
```
13+
14+
Remind, if you want to test with official model, you should replace the config files with [official config](https://github.com/THUDM/LongBench/tree/main/LongBench/config)
15+
16+
### Evaluate
17+
The datasets inference result
18+
```shell
19+
CUDA_VISIBLE_DEVICES=0,1,2,3 python pred.py --model Qwen2.5-7B-CARE # replace with your one or multi cards id
20+
```
21+
22+
The metric calculation from the `pred` folder.
23+
```shell
24+
python3 eval.py --model Qwen2.5-7B-CARE
25+
```

‎evaluation/LongBench/config/dataset2maxlen.json‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"narrativeqa": 4096,
3+
"qasper": 4096,
4+
"multifieldqa_en": 4096,
5+
"multifieldqa_zh": 4096,
6+
"hotpotqa": 4096,
7+
"2wikimqa": 4096,
8+
"musique": 4096,
9+
"dureader": 4096,
10+
"gov_report": 4096,
11+
"qmsum": 4096,
12+
"multi_news": 4096,
13+
"vcsum": 4096,
14+
"trec": 4096,
15+
"triviaqa": 4096,
16+
"samsum": 4096,
17+
"lsht": 4096,
18+
"passage_count": 4096,
19+
"passage_retrieval_en": 4096,
20+
"passage_retrieval_zh": 4096,
21+
"lcc": 4096,
22+
"repobench-p": 4096,
23+
"2hop_qa": 4096
24+
}

‎evaluation/LongBench/config/dataset2prompt.json‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"narrativeqa": "You are given a story, which can be either a novel or a movie script, and a question. Answer the question. \n\nStory: {context}\n\nNow, answer the question based on the story. \n\nQuestion: {input}\n\n",
3+
"qasper": "You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write \"unanswerable\". If the question is a yes/no question, answer \"yes\", \"no\", or \"unanswerable\". Do not provide any explanation.\n\nArticle: {context}\n\n Answer the question based on the above article as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write \"unanswerable\". If the question is a yes/no question, answer \"yes\", \"no\", or \"unanswerable\". Do not provide any explanation.\n\nQuestion: {input}\n\nAnswer:",
4+
"multifieldqa_en": "Read the following text and answer.\n\n{context}\n\nNow, answer the following question based on the above text.\n\nQuestion: {input}\n",
5+
"multifieldqa_zh": "阅读以下文字并用中文简短回答：\n\n{context}\n\n现在请基于上面的文章回答下面的问题，只告诉我答案，不要输出任何其他字词。\n\n问题：{input}\n回答：",
6+
"hotpotqa": "Answer the question based on the given passages. \n\nThe following are given passages.\n{context}\n\nAnswer the question based on the given passages. \n\nQuestion: {input}\n",
7+
"2wikimqa": "Answer the question based on the given passages. \n\nThe following are given passages.\n{context}\n\nAnswer the question based on the given passages. \n\nQuestion: {input}\n",
8+
"musique": "Answer the question based on the given passages. \n\nThe following are given passages.\n{context}\n\nAnswer the question based on the given passages. \n\nQuestion: {input}\n",
9+
"dureader": "请基于给定的文章回答下述问题。\n\n文章：{context}\n\n请基于上述文章回答下面的问题。\n\n问题：{input}\n回答：",
10+
"gov_report": "You are given a report by a government agency. Write a one-page summary of the report.\n\nReport:\n{context}\n\nNow, write a one-page summary of the report.\n\nSummary:",
11+
"qmsum": "You are given a meeting transcript and a query containing a question or instruction. Answer the query in one or more sentences.\n\nTranscript:\n{context}\n\nNow, answer the query based on the above meeting transcript in one or more sentences.\n\nQuery: {input}\nAnswer:",
12+
"multi_news": "You are given several news passages. Write a one-page summary of all news. \n\nNews:\n{context}\n\nNow, write a one-page summary of all the news.\n\nSummary:",
13+
"vcsum": "下面有一段会议记录，请你阅读后，写一段总结，总结会议的内容。\n会议记录：\n{context}\n\n会议总结：",
14+
"trec": "Please determine the type of the question below. Here are some examples of questions.\n\n{context}\n{input}",
15+
"triviaqa": "Answer the question based on the given passage. Only give me the answer and do not output any other words. The following are some examples.\n\n{context}\n\n{input}",
16+
"samsum": "Summarize the dialogue into a few short sentences. The following are some examples.\n\n{context}\n\n{input}",
17+
"lsht": "请判断给定新闻的类别，下面是一些例子。\n\n{context}\n{input}",
18+
"passage_count": "There are some paragraphs below sourced from Wikipedia. Some of them may be duplicates. Please carefully read these paragraphs and determine how many unique paragraphs there are after removing duplicates. In other words, how many non-repeating paragraphs are there in total?\n\n{context}\n\nPlease enter the final count of unique paragraphs after removing duplicates. The output format should only contain the number, such as 1, 2, 3, and so on.\n\nThe final answer is: ",
19+
"passage_retrieval_en": "Here are 30 paragraphs from Wikipedia, along with an abstract. Please determine which paragraph the abstract is from.\n\n{context}\n\nThe following is an abstract.\n\n{input}\n\nPlease enter the number of the paragraph that the abstract is from. The answer format must be like \"Paragraph 1\", \"Paragraph 2\", etc.\n\nThe answer is: ",
20+
"passage_retrieval_zh": "以下是若干段落文字，以及其中一个段落的摘要。请确定给定的摘要出自哪一段。\n\n{context}\n\n下面是一个摘要\n\n{input}\n\n请输入摘要所属段落的编号。答案格式必须是\"段落1\"，\"段落2\"等格式\n\n答案是：",
21+
"lcc": "Please complete the code given below. \n{context}Next line of code:\n",
22+
"repobench-p": "Please complete the code given below. \n{context}{input}Next line of code:\n",
23+
"2hop_qa": "Answer the question based on the given passages. \n\nThe following are given passages.\n{context}\n\nAnswer the question based on the given passages. \n\nQuestion: {input}\n"
24+
}

‎evaluation/LongBench/config/model2maxlen.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"llama2-7b-chat-4k": 3500,
3+
"longchat-v1.5-7b-32k": 31500,
4+
"xgen-7b-8k": 7500,
5+
"internlm-7b-8k": 7500,
6+
"chatglm2-6b": 31500,
7+
"chatglm2-6b-32k": 31500,
8+
"chatglm3-6b-32k": 31500,
9+
"vicuna-v1.5-7b-16k": 15500,
10+
"Qwen2.5-7B-Instruct": 32768,
11+
"Qwen2.5-7B-CARE": 32768,
12+
"Meta-Llama-3.1-8B-Instruct": 32768,
13+
"Llama-3.1-8B-CARE": 32768
14+
}

‎evaluation/LongBench/config/model2path.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"llama2-7b-chat-4k": "meta-llama/Llama-2-7b-chat-hf",
3+
"longchat-v1.5-7b-32k": "lmsys/longchat-7b-v1.5-32k",
4+
"xgen-7b-8k": "Salesforce/xgen-7b-8k-inst",
5+
"internlm-7b-8k": "internlm/internlm-chat-7b-8k",
6+
"chatglm2-6b": "THUDM/chatglm2-6b",
7+
"chatglm2-6b-32k": "THUDM/chatglm2-6b-32k",
8+
"vicuna-v1.5-7b-16k": "lmsys/vicuna-7b-v1.5-16k",
9+
"Qwen2.5-7B-CARE": "Qwen/Qwen2.5-7B-Instruct",
10+
"Llama-3.1-8B-CARE": ""
11+
}

0 commit comments

Comments
 (0)