RAGChecker Metric Input Requirements and Missing Data Handling

[RyosukeSakaguchi]

Hello,

I'm working with RAGChecker to evaluate my RAG system, and I'm trying to clarify the input data requirements for each metric and how the calculations behave when certain data is missing.

I'm referring to the following metrics:

• precision
• recall
• f1
• claim_recall
• context_precision
• context_utilization
• noise_sensitivity_in_relevant
• noise_sensitivity_in_irrelevant
• hallucination
• self_knowledge
• faithfulness

Based on the code, these metrics rely on intermediate results like answer2response, response2answer, retrieved2answer, and retrieved2response.

My questions are:

1. What are the specific, mandatory input data fields required to calculate each of the metrics listed above? For example, does calculating precision require useUserInput, useResponse, and useReference (as defined in some of the RAGChecker configurations)? A table summarizing the dependencies would be ideal.

2. What happens when one or more of the required input data fields are missing for a specific metric? Does the metric:

   • Return a value of 0?
   • Throw an error or exception?
   • Attempt to calculate the metric based on the available data, potentially leading to an inaccurate result?
   • Behave in some other way?

3. Specifically, which of the metrics can or cannot calculate a valid score if the RAG pipeline does not use Reference answers?

Understanding how RAGChecker handles missing data is crucial for interpreting the evaluation results accurately.

Thank you for your assistance!